Academy · Course · Based on Deep Learning by Goodfellow, Bengio and Courville
Interactive lessons that follow the book section by section. Every equation is clickable, every idea has a demo you can break, and every section ends with a checkpoint. Learn the whole path from the start, or jump to the one topic you need.
Guided path
Go through the modules in book order. A progress bar follows you down the path, each question earns XP, and every module ends with a mastery score that tells you what to revisit.
Free browsing
Open any section or subsection straight away. Your answers still count towards your XP and mastery, so you can mix the two ways of learning.
Your progress is stored in this browser only and never leaves your device. Clearing the site data erases it.
The opening chapter of the book, played step by step: why learning beat hand-written rules, why the way data is described matters, how deep models build concepts from simpler ones, how depth is measured, who the book is for, and the three waves of neural network research.
12 of 12 sections available as interactive modules.
Read the four kinds of arrays that deep learning is written in, pick out any element, row, column or subset, flip a matrix across its diagonal, and add arrays of different shapes by broadcasting.
See how a matrix acts on a vector, how two matrices combine, which algebra rules the product obeys and which it breaks, and how a pile of equations collapses into the single line Ax = b.
Meet the matrix that does nothing, the matrix that undoes another one, and the short argument that solves Ax = b with it. Then see why the book calls the inverse a theoretical tool and not a daily one.
See a matrix as a set of directions you can travel along, and use that picture to decide when Ax = b has no solution, one, or infinitely many, and exactly when a matrix can have an inverse.
Measure the size of a vector and a matrix with the Lp norms, know which properties make a function a norm, and choose between the squared L2, L1 and max norms.
Learn to recognize diagonal, symmetric and orthogonal matrices, and unit and orthogonal vectors, and see why each shape makes arithmetic cheap or geometry simple.
Hunt for the directions a matrix only stretches, write the matrix as eigenvectors times eigenvalues, and read singularity, extremes and definiteness off the eigenvalues.
Pull a matrix apart into turn, stretch, turn. Every real matrix, square or not, factors this way, and the pieces connect back to eigenvectors.
Solve Ax = y when A has no inverse: see why such systems have no answer or many, define the pseudoinverse as a limit, compute it with the SVD, and recognize what it returns in each case.
Add up a diagonal and gain a tool: write the Frobenius norm without sums, move factors around inside a trace, and rewrite a quadratic form as a trace.
See the determinant as the factor by which a matrix scales area and volume, connect it to the product of the eigenvalues, and recognize what a determinant of 0 or 1 does to space.
Squash points onto a line and rebuild them: derive PCA from linear algebra alone, then find the directions that lose the least.
14 of 14 sections available as interactive modules.
Why a learning system cannot avoid uncertainty: the three sources it comes from, why a simple uncertain rule can beat a perfect one, and what a probability number means.
What a random variable is, why it needs a probability distribution before it says anything about chance, how to read the notation for a variable and its values, and how discrete and continuous variables differ.
How to write down chance: a probability mass function for discrete variables, a joint distribution for several variables, a density for continuous ones, and the uniform distribution in both settings.
How to get the distribution of one variable out of a joint distribution: add up the other variable in the discrete case, integrate it out in the continuous case, and see what the result loses.
How to update the probability of one thing once you know another happened: the definition as a renormalized slice of the joint table, the one restriction it needs, why the direction of the condition matters, and why seeing is not doing.
Any joint distribution over many variables can be rebuilt from one small conditional distribution per variable. You will see why this is true, why the order does not matter, and why it is the recipe behind sampling and behind modern sequence models.
When does knowing one variable tell you nothing about another? You will learn independence as a factorization test, see why finite data always looks slightly dependent, and meet conditional independence, the assumption that makes large probabilistic models affordable.
The three standard summaries of a distribution: the average value of a function (expectation), how widely it spreads (variance), and how two variables move together (covariance). You will also see why zero covariance does not mean independence.
The small family of distributions that deep learning uses again and again: Bernoulli, multinoulli, Gaussian, exponential, Laplace, Dirac and mixtures. For each you will see what its parameters do, why it is useful, and what it cannot express.
Meet the sigmoid, which turns any score into a probability, and the softplus, which turns any score into a positive scale. Watch the sigmoid saturate, then check the identities that tie the two functions together.
Turn a conditional probability around. You will compute the probability of a cause after you see an observation, from how likely the observation is under each cause and from the prior, and you will see why the base rate can decide the answer.
Three technical ideas that make continuous probability safe to use: sets of measure zero, properties that hold almost everywhere, and the Jacobian factor that corrects a density when a variable is transformed.
Measure surprise, uncertainty and the gap between two distributions: self-information, Shannon entropy, differential entropy, the KL divergence and its direction, cross-entropy, and the convention 0 log 0 = 0.
Describe a distribution over many variables as a product of small factors, draw the factorization as a directed or undirected graph, and read independences off the graph. Edit the graphs yourself and watch the formulas follow.
5 of 5 sections available as interactive modules.
Why a computer cannot hold every real number, how exp() blows up or vanishes, and the one-line shift that makes softmax and log softmax safe to compute.
How a matrix can magnify tiny errors in its input, how the condition number measures that magnification, and why the sensitivity belongs to the matrix and not to the algorithm.
Minimize a function by following its slope: the gradient and its directions, step sizes, the Jacobian and Hessian, saddle points, Newton’s method, and the guarantees that Lipschitz continuity and convexity give.
How to minimize a function when the answer must stay inside a set: projecting back, changing variables, and the generalized Lagrangian with its multipliers, the minimax form and the KKT conditions.
How to minimize a least squares objective by gradient descent and by Newton’s method, and how to add the constraint that the solution has length at most 1 with a Lagrangian and gradient ascent on its multiplier.
11 of 11 sections available as interactive modules.
What it means for a program to learn: the task, the performance measure and the experience, and then the first complete learning algorithm, linear regression, with its normal equations and bias term.
Why low training error is not the goal: generalization, the capacity of a model, the U-shaped error curve, the no free lunch theorem and how regularization expresses preferences.
Why some settings must be chosen outside the training set, how a validation set and k-fold cross-validation estimate generalization, and why a reused test set goes stale.
Treat an estimate as a random quantity, split its error into bias and variance, and see why a slightly biased estimator can beat an unbiased one.
Derive estimators from one principle, see why it reduces to minimizing a KL divergence, a cross-entropy or a mean squared error, and learn when it can be trusted.
Treat a parameter as uncertain, update the belief with Bayes rule, predict with the whole posterior, and see why a Gaussian prior on the weights becomes weight decay when you keep only the most probable parameters.
Four classic ways to learn a rule from labeled examples: logistic regression, support vector machines with the kernel trick, nearest neighbors and decision trees. Run each one, break it, and see what it assumes.
How to find a useful description of data without any labels: PCA rotates the data until its coordinates are uncorrelated, and k-means replaces each point by the index of its cluster.
Learn why one small random minibatch can steer training, how its error shrinks with the batch size, why a constant learning rate leaves a noise floor, and why one update does not get more expensive as the data grows.
See that many learning algorithms are one recipe with four swappable parts, a dataset, a cost, a model and an optimizer, and build the combinations yourself.
See why simple learners fail on high-dimensional data: configurations multiply, local smoothness runs out, and the manifold idea gives deep learning a way forward.
6 of 6 sections available as interactive modules.
Learn why one straight line cannot compute XOR, and build a two-unit ReLU network that computes it, one matrix at a time.
Choose a cost function and an output unit that match the data, so that the gradient stays large where the model is wrong and training keeps moving.
Choose the bend of a hidden unit by checking its slope and its saturation, because those two properties decide whether gradient descent keeps learning through the unit.
Read a network as a chain of layers, count its parameters, and see what the universal approximation theorem does and does not promise. You will also see how depth multiplies the number of pieces a rectifier network can make, and what skip and sparse connections change.
Compute a gradient by applying the chain rule backwards through a computational graph. You will see why reusing shared work saves so much computation, how the same idea trains a small network, and where the method stops being the cheapest choice.
Trace how the chain rule, gradient descent and back-propagation reached deep learning, why the early linear models stalled, and why cross-entropy and rectified units changed training. Each idea has a small demo you can test.
Add a price for size to a loss, predict how that price pulls the best weights towards zero, and see why the L2 and L1 prices leave different marks on the solution.
See a norm penalty as a limit on the size of the weights, follow the penalty strength as a Lagrange multiplier, and compare a hard limit that is enforced by reprojection with a soft price.
See why some problems have no single answer, how a ridge term makes the inverse exist, and why the pseudoinverse is the limit of ridge regression as the penalty shrinks to zero.
Make more training examples from the ones you have, keep the label when you change the input, add noise to inputs as a kind of augmentation, and compare methods only when they share the same augmentation.
See how noise on the inputs of a linear model acts like a weight penalty, why noise on the weights leaves a linear model unchanged, and how label smoothing keeps a classifier from chasing probabilities of exactly one.
Learn how one model can use many unlabeled examples together with a few labeled ones, through a learned representation and a shared parameter, and when that choice helps.
Learn how several related tasks can share a part of one network, why pooling their examples constrains the shared part, and when that sharing fails.
Learn how training time acts as a hyperparameter, how early stopping chooses it, and why stopping early behaves like L2 regularization.
Learn how two models can be tied by a penalty or by sharing one set of weights, why a convolution reuses its kernel at every position, and what shifting the input does to the output.
Learn how sparse representations differ from sparse parameters, how an L1 penalty on activations sets units exactly to zero, and how OMP-k limits the number of nonzero entries.
Average several models, measure how much the average gains from the correlation of their errors, and build a bagged ensemble by resampling the training set with replacement.
Train an exponentially large ensemble of sub-networks by removing units at random, then average them cheaply with the weight scaling rule, and see why the randomness regularizes.
Find inputs that a trained network gets wrong, see why small changes add up in linear models, and use adversarial training to make a network locally constant near its data.
Use the manifold hypothesis to make a classifier invariant to movement along the data manifold: tangent distance, the tangent prop penalty, the ReLU problem, and a local stand-in for the manifold tangent classifier.
8 of 7 sections available as interactive modules.
Explain why training minimizes a proxy for the goal, why minimizing the training error can overfit, why a surrogate loss and early stopping are used, and why a minibatch gives an honest but noisy gradient.
Explain the main obstacles to optimizing a neural network: ill-conditioning, local minima and symmetries, saddle points, cliffs, inexact gradients, and the gap between local moves and global structure, with the book's limits on what any algorithm can guarantee.
Explain stochastic gradient descent, why its learning rate must decay for convergence, what momentum and Nesterov momentum do to the path, and what the terminal velocity and the physical picture of momentum mean.
One subsection of the book, taught step by step: the velocity that remembers past gradients, the speed limit it creates, and the physical picture behind it.
Explain why the initial parameters must break symmetry, how the scale of random weights controls signal and gradients, why the biases and variance parameters are set separately, and how initialization acts as a prior.
Give every parameter its own learning rate, and see how AdaGrad, RMSProp and Adam change that rate from the history of their gradients.
Use curvature to choose better steps: Newton’s method, its regularized form, conjugate gradients and BFGS, and the cost each one pays to avoid the full Hessian.
Change the model, the optimizer, the starting point or the output to make deep networks easier to train: batch normalization, coordinate descent, Polyak averaging, pretraining, better model design and continuation.
Compute a convolution by sliding a kernel along an input, in one and two dimensions, see what the kernel flip does, and write the same operation as a sparse matrix.
Explain why convolution uses sparse connections, shared parameters and translation equivariance, and count the savings for a real image.
Explain what a pooling function summarizes, why max pooling tolerates small shifts, how a stride shrinks a layer, and how pooling gives a fixed number of summaries for inputs of any length.
Explain why convolution and pooling act as infinitely strong priors, build a local and shared layer from a dense one, and see when such a prior underfits.
Read the convolution that neural networks really use: many kernels at once, strides, zero padding, locally connected and tiled layers, the gradients that training needs, and bias terms. Each variant has a demo that you can change.
Turn one label per image into one label per pixel. See why the output map can shrink, how a recurrent network refines its first guess, and how neighboring pixels form regions.
See the shapes of data that convolution reads, from audio to color video. Learn why a shift in the input shifts the output, how to get a fixed-size output from a variable-size input, and when convolution does not fit.
Make convolution cheaper without changing its answer. Compare the cost of a full kernel and a separable kernel, build a kernel from two vectors, and see how the Fourier transform gives the same result by a different route.
Explain why a convolutional network can learn useful kernels without supervised training, cluster patches into a bank of kernels, see why random filters followed by pooling already respond to moved patterns, and count the cost of each way of training.
Trace the link from simple and complex cells in the visual cortex to convolutional layers, build a Gabor filter yourself, recover a hidden filter from noisy responses, and see where the cartoon stops matching the brain.
Trace the milestones that made convolutional networks central to early deep learning, test why they worked first, count parameters against a dense layer, and see how stacked layers grow what one unit can see.
Turn a recurrent definition into a chain of identical steps, count how many copies of the same rule a finite chain contains, and explain why one set of parameters can serve every time step.
Run the forward pass of a recurrent network, read its loss over a sequence, compute gradients by back-propagation through time, and see how teacher forcing, graphical models and context change what the network can learn.
Explain why some tasks need the future as well as the past, read the two recurrences of a bidirectional network, measure how much each input counts for an output, and see how the idea extends to grids.
Map an input sequence to an output sequence of a different length with an encoder, a context and a decoder, see why a fixed-size context can run out of room, and see how attention lets the decoder read the input in parts.
Split one step of a recurrent network into its three blocks, add hidden layers to each block, and count how the length of the route between time steps changes.
Read a sequence as a tree instead of a chain, count how the depth of a balanced tree grows, and follow one forward pass with weights shared across the tree.
Multiply by one weight, then by a matrix, then by many random layers, and see why repeated products vanish or explode.
Fix the recurrent weights, learn only a linear readout, and measure how long the state remembers, how fast it forgets, and how a non-normal matrix can stretch a single step.
Build memory at several speeds with leaky averages, skip connections and slow groups, and read off how far each part of the model can reach back in time.
Read an LSTM cell and a GRU as a set of valves that decide what to keep, what to write and what to show, and test each valve on a memory task.
Tame steep cliffs in the loss surface by capping the gradient, compare two ways of clipping, and read a regularizer that keeps the gradient from shrinking as it moves back through time.
Compare knowledge held in weights with knowledge held in memory cells, read memory with soft and hard addressing, and see why content and location addressing differ.
Choose the number that matches the goal: see why accuracy can mislead on rare events, trade precision against recall with a threshold, and refuse uncertain cases to buy accuracy with coverage.
Start with the model family that fits the shape of your data, a piecewise linear unit, a decaying learning rate and early stopping, and copy a known model whenever a similar task already has one.
Check the training error first, then the test error and the gap between them. Gather more data when the gap stays large after regularization, and grow the data on a doubling scale.
Choose the settings that training does not learn: how capacity, the learning rate and the search method decide the result, and how grid, random and model-based search differ.
Find out whether a poor result is the method or a bug: read the worst mistakes, compare training and test error, fit one example, and check gradients numerically.
Follow a real project from metric to fix: trade coverage against accuracy, score a whole address, read the errors, and widen the crop.
Explain why deep learning needs large and parallel computation, and what each family of tricks in the book buys: cache-friendly and branch-free code, asynchronous updates, smaller models for inference, cascades and gaters, and low-precision numbers.
Explain why computer vision needs a common pixel scale, what global and local contrast normalization do, and how dataset augmentation and test-time voting reuse the same invariances. Every demo is a toy built for this page.
Explain speech recognition as a most-probable-sequence search, how audio is cut into frames, why the acoustic and language parts are weighed against each other, and how deep networks replaced parts of the GMM-HMM pipeline.
Explain n-gram language models and their counting estimates, why smoothing and distributed word representations are needed, how the output layer over a large vocabulary is made cheaper, and how attention lets a translation model read a whole sentence.
Explain how recommender systems predict user-item associations with embeddings and a bilinear rule, why exploration is needed when the system only sees the items it shows, and how knowledge bases store facts as triplets that are ranked by corruption.
Describe data as a few hidden factors plus noise, read the covariance WWᵀ + ψ off a factor model, and see how probabilistic PCA turns into PCA as the noise shrinks.
Recover hidden sources from mixtures: why independence needs more than decorrelation, why the sources must be non-Gaussian, and how the choice of prior and the constraint on the mixing matrix shape the method.
Learn features that change slowly over time: the slowness principle, the linear SFA objective with its three constraints, why the features must be decorrelated, and how a quadratic basis lets SFA find slowness hidden in a nonlinear signal.
Model data as a sparse combination of a dictionary of features: the Laplace and Student-t priors, inference as a lasso problem with exact zeros, learning by alternating encoding and decoding, and why sparse codes do not make good samples.
Read PCA as fitting a flat pancake of high probability to data: the encoder and decoder of a linear autoencoder, the minimum reconstruction error as the sum of the discarded eigenvalues, and the equivalence with maximising the variance of the code.
9 of 9 sections available as interactive modules.
Explain what an autoencoder is, why a code narrower than the input forces it to keep the most important structure, how a linear undercomplete autoencoder recovers the principal direction of the data, and why too much capacity lets it copy without learning anything.
Explain how a penalty on the code, a corrupted input, or the slope of the code can keep an autoencoder useful even when its code is as wide as its input, and check each idea with an interactive demo.
Explain why an autoencoder can gain from a deep encoder and decoder, what one hidden layer can and cannot enforce, and how a stack of shallow autoencoders is trained, with interactive demos for each idea.
Explain how the decoder of an autoencoder can be a probability distribution, how the output unit follows the noise model, and how encoders and decoders can be stochastic, with demos for each idea.
Explain how a denoising autoencoder learns to undo corruption, how its reconstruction becomes a vector field that estimates a score, and what a fixed noise level does to the distribution it learns, with interactive demos for each idea.
Explain why data concentrate near a manifold, what a tangent measures, how a reconstruction that is flat around its data learns the shape of the manifold, why neighbour-graph methods need many examples, and why translations make manifolds complicated, with an interactive demo for each idea.
Explain the contractive penalty on the encoder Jacobian, what it means for a linear operator to be contractive, how the leading singular vectors read as tangent directions, why saturating sigmoid units shrink the Jacobian, why tied weights block a scale trick, how the penalty relates to denoising, and why deep contractive stacks stay contractive, with an interactive demo for each idea.
Explain how predictive sparse decomposition trains a fast encoder to predict the codes that a slow sparse search would find, what each term of the cost in Equation 14.19 does, and why training alternates between searching for codes and updating the parameters.
Explain how autoencoders reduce dimension and support retrieval: why a smaller code helps, how a binary code turns search into a table lookup, and how saturated sigmoid units produce bits that survive noise, with honest toy demos for each idea.
Explain how a deep network can be pretrained one layer at a time without labels, why the start point it gives can matter, and why the method faded after 2006 except in language.
Explain when a representation learned for one setting helps another, how the shared part depends on whether inputs or outputs carry the same meaning, and how one-shot and zero-shot learning work through a shared representation.
Explain why a representation that separates the causes of the data helps semi-supervised learning, when unlabelled data reveals the classes through p(x), and why a fixed cost such as mean squared error can miss causes that matter.
Explain why a code of many separately set features names far more concepts than one symbol per concept, why that helps a learner generalize, and how many regions a set of linear features can carve.
Count the straight pieces of a folded function, see why a shallow network needs exponentially more units to match it, and state what this result does and does not promise.
Sort the clues that help a learner find underlying causes, from labels and smoothness to sparsity and independence, and test each clue on a small toy.
7 of 7 sections available as interactive modules.
Count how many numbers a lookup table needs for a distribution over many variables, see why the four main tasks need the whole input, and see how keeping only direct links between variables makes the same question cheap.
Read a graph as a list of direct interactions. Build directed and undirected models from it, count their parameters, compute the normalizing constant, test independence by separation, convert between the two graph languages, and draw factor graphs.
Draw samples from a directed graph by following its arrows, see where that shortcut needs a harder answer, and run Gibbs sampling on a loop of variables that has no first variable.
Count what a graph saves in numbers, work and sampling steps, read each missing edge as an assumption, and see why keeping structure separate from algorithms lets one method serve many models.
See how hidden variables create dependencies between visible ones, count what it costs to connect everything directly, measure what a search for structure costs, and read a hidden variable as a feature.
See what inference asks for, why the identity (16.9) lets maximum likelihood work, why exact inference counts solutions and becomes intractable, and how a variational guess leaves a gap equal to a KL divergence.
See how deep learning keeps the tools of graphical models but changes the design choices, then follow one small restricted Boltzmann machine through its energy, its conditionals, its block Gibbs sampling and its derivative.
Estimate a sum or an integral by averaging samples drawn from a distribution, and read the size of the error from the spread of those samples.
Rewrite a sum as an average under a different distribution, weight each sample to correct for the change, and choose that distribution so the estimate stays steady, including when the normalising constant is unknown.
Sample a distribution you cannot draw from directly by running a chain of random moves, and check when the chain has settled into the distribution you want.
Build a Markov chain by updating one variable at a time from its conditional distribution, see why correlated variables slow it down, and find which groups of variables can move together.
See why a Markov chain can get stuck in one region of a distribution, and try three ways the book describes to help it move between regions: flatten the peaks with tempering, or make the representation deeper.
Explain why the gradient of an undirected model splits into a positive and a negative phase, derive the gradient of the log partition function, and estimate its negative phase by sampling.
Compare the three ways of approximating the negative phase in Algorithms 18.1 to 18.3, see why contrastive divergence misses spurious modes, and explain why persistent chains and fast weights help.
Explain why ratios of probabilities avoid the partition function, write the pseudolikelihood and its generalized form, and say where the approximation helps and where it fails.
Explain how score matching trains a model without the partition function, why the data score can be dropped, which models it suits, and how ratio matching handles binary data through neighbouring states.
Explain why a model fitted to a finite sample needs smoothing, derive the denoising target for Gaussian noise, and see how a denoising score matching model learns a density that is wider than the data.
See how noise-contrastive estimation turns the normaliser into a free constant, trains a classifier between data and noise, and recovers the constant as minus the log partition function.
Compare two models without computing either partition function, then estimate a ratio of partition functions with importance sampling, annealed importance sampling and bridge sampling, and see where each one breaks.
Explain why inference is hard in deep models, and show that the evidence lower bound turns inference into an optimization over a guess q. You will be able to rearrange the bound and judge a guess by its gap.
Run the expectation maximization loop on a toy with two hidden clusters, explain why the E-step closes the gap that the M-step opens, and say what each step is allowed to change.
Explain the difference between a most likely hidden value and a full posterior, why a point guess gives a loose bound, and how sparse coding with a Laplace prior turns inference into a shrinkage problem you can solve by hand.
Restrict the approximate posterior to a factorial family, optimize its parameters with fixed point updates, and see how the same bound yields a Gaussian when latent variables are continuous.
Train a network to do inference directly, using samples from the model in the wake-sleep algorithm, and see where such a network can go wrong when the data differs from the samples it was trained on.
Explain how a Boltzmann machine assigns a probability to every binary pattern through an energy function, why hidden units add power, why learning uses the difference between two sets of statistics, and why the partition function is the obstacle.
Explain why a restricted Boltzmann machine keeps its two layers apart, why that makes the hidden units independent given the visible units and the reverse, and how block Gibbs sampling draws from a model whose total probability cannot be computed exactly.
Explain how a deep belief network stacks RBMs with directed links below a top undirected pair, how it is sampled and trained one layer at a time, and how its weights become a feed-forward network for classification.
Explain how a deep Boltzmann machine stacks undirected layers, why its posterior needs mean-field fixed points, how variational stochastic maximum likelihood and layer-wise pretraining train it, and how joint training with centering and multi-prediction avoids pretraining.
Explain how Gaussian-Bernoulli RBMs model real-valued data through a conditional mean, why covariance models such as mcRBM, mPoT and the spike and slab RBM capture pixel relationships, and what each costs in sampling and in positive definiteness.
Explain how convolution makes Boltzmann machines practical for images, why max pooling is costly in an energy model, how probabilistic max pooling keeps only n + 1 states per pooling region, and where the approach limits overlap, boundaries and input size.
Explain how a Boltzmann machine becomes a conditional model for structured outputs and for sequences, how the conditional RBM of Taylor et al. makes its visible bias depend on the past, and how the RNN-RBM lets a recurrent network emit every RBM parameter and is trained through contrastive divergence and back-propagation through time.
Explain how Boltzmann machines can be trained with a discriminative criterion, how most models use second-order energy terms while higher-order terms add masks, class interactions and gating, and why designing a new Boltzmann machine needs an energy function that keeps every conditional tractable.
Explain how gradients pass through a random draw: the reparametrization trick for continuous samples, REINFORCE for discrete ones, and baselines that cut the variance of the estimate without biasing it.
Explain how directed generative networks produce samples, how a generator turns noise into data, how variational autoencoders, GANs and moment matching train such generators, and how auto-regressive models such as NADE factor a joint distribution one variable at a time.
Run the Markov chain of a denoising autoencoder, sample a conditional distribution by clamping some units, and see why walk-back training starts its walks at training examples.
Build a generative chain from a latent state and a visible state, check when its spread settles with a linear toy, train it on the clamped reconstruction log-probability, and see how a discriminant variant reuses the same chain.
Build a generative model by reversing a diffusion one small step at a time, see why one step is easier to learn than the full way back, and sample by rejecting candidates until a summary of them matches its target.
Find out why a generative score can mislead: units, binarization, a background variance that games the likelihood, the direction of the KL divergence, and copies that pass a coverage check.
The lessons follow Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville (MIT Press, 2016), which is free to read at deeplearningbook.org. All explanations, demos and questions on these pages were written by Mathalyse Research.