↵ select ↓ ↑ navigate esc close

Paper Feed: The Little Book of Generative AI Foundations: An Intuitive Mathematical Primer

Paper Feed ·

Listen: https://serve.buildtall.systems/147f9609591f09ddb9f31a5dc4f1aa2d845020ec65c570b6da8f884e35ad5e07.mp3

Source: https://arxiv.org/pdf/2605.29713


This episode covers The Little Book of Generative AI Foundations: An Intuitive Mathematical Primer, a preprint by Tianhua Chen of the University of Huddersfield. It is a book-length tutorial that sets out the mathematics behind the main families of generative AI models.

First, the citation record: according to OpenAlex, the work is not retracted and has been cited 0 times as of 4 November 2026. The book, a working preprint in its second version, argues that much of generative AI rests on a small set of mathematical principles, including latent variables, likelihoods, variational bounds, invertible transformations, noising processes, score fields, adversarial comparison and energy landscapes. Its aim is a compact but careful route through these ideas, built step by step from first principles rather than a catalogue of architectures or recent refinements. The author says it is aimed at foundations rather than the research frontier, and that it contains more derivation than a high-level overview strictly needs. The kind of work is a tutorial primer, not an experiment: the text reports no datasets, trials or benchmark results, so its support is derivational, and the strength it shows is the careful unpacking of each model family, from principal component analysis through probabilistic PCA, the variational autoencoder, diffusion, score-based models, normalising flows, autoregressive models, GANs, Wasserstein GANs and energy-based models, with appendices that work through the algebra. The limits are the author's own statements: the scope is selective, the references are kept focused rather than forming a survey bibliography, and readers already working in the field may find much of it familiar. The preface says the work was not developed under a funded research grant, commissioned institutional output, or formal course documentation, and that any remaining errors are the author's own. The preface calls it a working preprint that may be revised, and says a revised version may be submitted for formal book publication. On where it is published, the document describes itself as an arXiv preprint that followed an earlier SSRN preprint, and its second version is dated 8 September 2026. The OpenAlex record lists an arXiv preprint published on 28 May 2026. The two dates differ, and each source states its own.

The preface begins by placing the book in a fast-moving field. Generative artificial intelligence has developed rapidly, with new models, systems and applications appearing at remarkable speed. The author argues that beneath this surface, many central ideas rest on a smaller set of mathematical principles: latent variables, likelihoods, variational bounds, invertible transformations, stochastic noising processes, score fields, adversarial comparison, and energy landscapes. Those principles are the book's subject.

The book sets out to give a compact but rigorous route through them. It does not try to catalogue every modern architecture, implementation detail or state-of-the-art refinement. The stated aim is to make the foundations of the major model families understandable, and to make them more accessible "without removing their mathematical substance." The intended readers are mathematically curious researchers, practitioners and students who want to understand generative modelling from first principles.

The preface explains that the word "little" in the title is about scope rather than depth. The book is selective about what it covers, but within that scope the treatment is meant to be careful, with core ideas built step by step so that the mathematical structure is visible rather than hidden behind black-box recipes. The author adds that the book is aimed at foundations rather than the research frontier, so readers already working there may find much of it familiar. The references are kept deliberately focused, supporting the main historical and technical connections rather than serving as a survey bibliography.

The central purpose, in the preface's account, is to connect the main families of generative models through one mathematical narrative. The ordering is chosen so that the mathematics unfolds "as naturally and compactly as possible," rather than to follow a strict historical chronology. Tools such as linear algebra, probability, calculus, Gaussian algebra and density transformations are introduced when a modelling idea needs them, and are woven into the narrative instead of being gathered into a long preliminary review. The preface also admits that the book contains more derivation than a high-level overview strictly needs:

The book contains more derivation than is strictly necessary for a high-level overview.

It calls this intentional, because the objectives of many ideas become clearer once they are unpacked.

The preface then outlines the route. The book begins with PCA and autoencoders, which introduce linear transformations, projection, reconstruction and latent structure. Probabilistic PCA follows as the first latent-variable generative model in the book. The next stage covers variational generative modelling, introducing the evidence lower bound, the expectation-maximisation algorithm and variational inference, and leading to the variational autoencoder. The diffusion material comes next: denoising diffusion probabilistic models are first presented in discrete time as sequential latent-variable models, then comes the calculus needed for continuous time, and then the score-based view, in which generation is guided by learned score fields and reverse-time stochastic dynamics. The chapter on exact density models covers normalising flows and autoregressive factorisations, which keep likelihoods tractable by design, either through invertible transformations or through the chain rule of probability. The final chapter looks beyond tractable likelihoods, introducing GANs, Wasserstein GANs and energy-based models, where learning proceeds through comparison, geometric discrepancy or scalar energy landscapes rather than explicit likelihood evaluation.

On its origin, the preface describes the book as an independently authored scholarly project. It states that the work was not developed as part of a funded research grant, commissioned institutional output, formal module development, or official course documentation for any specific module or programme. Earlier short-form versions of some of these ideas appeared as an SSRN preprint and later in revised form as an arXiv preprint, and the present book expands that line of thought into a fuller primer. This is the second preprint version. Compared with the first, it adds visual explanations throughout, along with minor clarifications. The preface calls it a working preprint that may receive further revisions, corrections, clarifications or additional material, and says a revised version may be submitted for formal book publication.

The reuse statement says the manuscript is made available for scholarly reading, citation, research reference and study. Readers are welcome to cite it and to use it as a supporting resource for research, learning and teaching, with appropriate attribution. Substantial reproduction, adaptation, redistribution, or conversion into derivative teaching, training or course resources requires the author's permission, unless applicable copyright exceptions or a separate licence attached to a future version allow otherwise.

The acknowledgements say the book grew out of the author's continuing effort to explain the mathematical foundations of modern generative AI from first principles. The author credits the wider academic community whose papers, books, lectures and discussions shaped the context of the work, and states that any remaining errors, omissions or unclear explanations are the author's own.

The chapter on linear algebra foundations begins by explaining why principal component analysis, usually called PCA, is a good entry point for the whole book. The authors give two reasons. First, PCA is a widely known way of reducing the number of dimensions in data, and its ideas lead on to later models. They describe its progression "from algebraic projection, to reconstruction, to probabilistic latent-variable modelling" as foreshadowing techniques in Variational Autoencoders and diffusion models. Second, PCA illustrates core linear algebra: projections, changes of basis, eigenvalues and eigenvectors. Generative models work in high-dimensional spaces, where data are represented and transformed with matrices, so covariance matrices, matrix decompositions and changes of basis keep appearing as tools for understanding the structure of data.

The chapter moves in steps. It starts with matrices as transformations, then with two ways of understanding matrix multiplication, then with eigenvectors. From there it sets up PCA as a search for directions of maximal variance, shows how PCA can be read as a change of basis, and ends by recasting PCA as reconstruction, which leads to autoencoders.

A matrix, in this account, is not just a table of numbers. Matrices are best understood, in the chapter's words, "not merely as tables of numbers, but as representations of transformations acting on vectors." A linear transformation is a rule that moves vectors while preserving addition and scaling. Any vector in a plane can be written as a combination of two standard basis directions, one pointing along each axis. The identity matrix leaves every vector unchanged. A 90 degree rotation sends the first basis direction to the second and the second to the negative of the first, and because every vector is built from the basis, the same rotation applies to the whole plane. The authors draw the general lesson:

A linear transformation is completely determined by where it sends the basis vectors.

A shear, by contrast, keeps one basis direction fixed and tilts the other, and the columns of the matrix need not be perpendicular or of unit length.

The chapter then gives two views of multiplying a matrix by a vector. In the column view, the output is a weighted combination of the matrix's columns, with the entries of the input vector as the weights. In the row view, each entry of the output is a dot product between a row of the matrix and the input vector. The authors explain why this dot product has a geometric meaning. It measures how much one vector lies along another, which is a projection. The dot product equals the length of one vector times the projection of the other onto it, and when the direction has unit length the dot product is exactly that projection. Multiplying two matrices corresponds to composing two transformations: applying the product to a vector is the same as applying the second matrix first and then the first.

Eigenvectors come next. For a square matrix, an eigenvector is a nonzero vector that stays on its own line when the matrix acts on it, and the eigenvalue is the factor by which it is scaled. The chapter's summary is that

Eigenvectors can therefore be viewed as directions of pure scaling under the transformation defined by A.

If a matrix has enough independent eigenvectors, they can be stacked as columns to form a new coordinate system. In that system the matrix only stretches each coordinate on its own, and the eigenvalues sit on the diagonal. The authors describe the process as three steps: change to the eigenvector coordinates, scale each coordinate independently, then convert back to the original coordinates.

Symmetric matrices are a special and important case. Their eigenvalues are always real, and their eigenvectors can be chosen to be orthogonal, meaning perpendicular and of unit length. Because the change of coordinates is then a rotation-like move that keeps lengths and angles, the chapter states that "symmetric matrices can always be diagonalised using an orthonormal change of basis." This matters for PCA because the covariance matrix of data is always symmetric.

The PCA discussion rests on two principles. The first is that directions with high variance tend to separate data points more clearly and so carry more useful structure. The second is sequential orthogonal maximisation: the first component captures the most variance possible, and each later one captures as much of the remaining variance as it can while staying perpendicular to those before it. To set the problem up, the data are first centred so that their average is zero. Projecting each data point onto a unit-length direction gives a scalar value, and the variance of those values measures how spread out the data are along that direction. The authors show that this variance can be written as a quadratic form built from the covariance matrix, which averages the outer products of the centred data points. PCA then becomes the problem of finding the unit direction that maximises this quantity.

The solution uses the eigen-decomposition of the covariance matrix. Expressing the candidate direction in the eigenvector basis leaves its length unchanged, and the objective becomes a weighted sum of the eigenvalues, where the weights are the squared coordinates of the direction and add up to one. A weighted average of numbers cannot exceed the largest of them, so the variance is bounded by the largest eigenvalue. The bound is reached only when all the weight sits on the eigenvector with that eigenvalue. The first principal component is therefore the leading eigenvector of the covariance matrix, and the maximum variance equals its eigenvalue. The k-th principal component is the eigenvector with the k-th largest eigenvalue, orthogonal to the ones before it. The resulting coordinates are uncorrelated.

For dimensionality reduction, the chapter keeps the top k principal components and stacks them as columns. Encoding a data point means projecting it onto these directions to get a shorter list of coordinates. Decoding means mapping that list back into the original space, which gives an approximation. If k equals the full dimension, the reconstruction is exact. Otherwise some information is lost. To measure how much is kept, the authors use the explained variance ratio: the sum of the top k eigenvalues divided by the sum of all of them. They describe it as "a principled, data-driven criterion for selecting the number of components," widely used in practice.

The chapter then looks at PCA as a change of basis. It introduces the trace of a matrix, which is the sum of its diagonal entries. For a covariance matrix, the diagonal entries are the variances of the individual coordinates, so the trace is the total variance of the data. A change of basis can redistribute variance across coordinates, but it cannot change the total. The reason is the cyclic property of the trace, which makes the trace of a similarity transformation equal to the trace of the original matrix. Applying PCA transforms the covariance matrix in exactly this way, so the off-diagonal entries that represent correlations between coordinates vanish, and the eigenvalues appear on the diagonal as the variances along the principal directions. The authors conclude that

PCA does not create or destroy variance.

It re-expresses the same total variance in coordinates that match the natural directions of the data.

The singular value decomposition, or SVD, extends these ideas to any real matrix, whether or not it is square or symmetric. Eigen-decomposition only looks for directions that a matrix maps onto themselves, and only works cleanly for certain square matrices. The SVD instead asks which orthogonal input directions are mapped to which orthogonal output directions, and how much each is stretched. Each input direction is sent to an output direction scaled by a nonnegative singular value. When the matrix is symmetric, the input and output directions coincide, and the singular values are the absolute values of the eigenvalues. The chapter also presents a rank-one expansion, in which the matrix is a sum of simple terms, each of which projects onto one input direction, scales, and reorients along one output direction. In the authors' words, "any matrix can be understood as a superposition of these elementary outer-product transformations." The chapter's broader point is that

complex linear transformations often become much easier to understand once the right coordinate system or decomposition is chosen.

The final major move reframes PCA as a reconstruction problem. Instead of asking which directions hold the most variance, the chapter asks which k-dimensional representation loses the least information when data are compressed and then rebuilt. The authors write the question as:

If we compress the data into a k-dimensional representation and then reconstruct it, which k-dimensional representation loses the least information?

The reconstruction of each point is its orthogonal projection onto the chosen subspace, and the loss is the squared distance between the point and its reconstruction, summed over all points. The authors state that this reconstruction formulation gives the same optimal subspace as the variance formulation, and that it is closely related to the classical best low-rank approximation of a matrix.

This view makes PCA look like an encoder followed by a decoder, both linear and set by the eigenvectors. Autoencoders generalise this by learning the encoder and decoder from data, by minimising reconstruction error. In PCA the two maps are fixed. In an autoencoder they are learnable functions. The chapter then asks when a linear autoencoder recovers PCA. Two things matter. Both encoder and decoder must be linear, so that all reconstructions lie in one flat subspace. And the autoencoder does not have to learn an orthonormal basis, because the key object is the subspace itself. The authors put it this way:

The key geometric object for dimensionality reduction is not a particular basis, but the subspace spanned by that basis.

If an orthonormal basis is wanted later, it can be obtained by orthogonalising the learned decoder, for instance with a QR decomposition, which changes the basis but not the subspace.

When the encoder and decoder are nonlinear, typically neural networks, the reconstructions no longer have to lie in a flat subspace. They can trace a curved low-dimensional surface called a manifold. The authors present this as the reason autoencoders became central to modern representation learning, and PCA as the simplest member of a larger family of reconstruction models.

The chapter ends by noting the limits of deterministic autoencoders. Each input maps to one latent code and one reconstruction, with no probabilistic account of how latent variables generate data. That means the model cannot assign probabilities to latent codes, cannot generate new samples by drawing latent variables from a chosen distribution, and cannot be trained by likelihood. Closing the gap requires probabilistic latent-variable models, starting with probabilistic PCA and then the Variational Autoencoder, where reconstruction is kept but placed inside a probabilistic framework.

The chapter's references point to the sources behind these ideas. Strang's textbook is offered as a broad reference for linear algebra and the matrix viewpoint. Jolliffe and Cadima give a modern overview of PCA and its interpretation. The classical origins of PCA go back to Pearson and to Hotelling. Baldi and Hornik discuss the connection between linear autoencoders and PCA, and Hinton and Salakhutdinov developed deep autoencoders.

The chapter on probabilistic PCA opens by placing its subject between the book's earlier chapter and its later ones. The authors note that principal component analysis and autoencoders "remained fundamentally deterministic": once a latent representation is chosen, the reconstruction is fixed. Probabilistic PCA, or PPCA, is introduced as the simplest model in the book that makes the move to a probabilistic view explicit. The authors list what it brings that recurs through the rest of the book:

a prior over latent variables, a conditional distribution of observations given latent causes, a marginal likelihood for learning, and a posterior distribution for inference.

A latent variable is a hidden quantity that is not observed directly but is assumed to lie behind the data and explain part of its structure.

The authors stress that PPCA is linear and Gaussian, and that this makes its central quantities derivable in closed form. They call this "rare tractability" and say it makes PPCA an ideal bridge model. Because every step stays transparent, the model lets the reader study likelihood-based learning, posterior inference, Jensen's inequality, the evidence lower bound, and the expectation maximisation algorithm before meeting them in more expressive models.

The generative process has two steps. First, a latent vector is drawn from a standard Gaussian prior. Second, an observation is produced by applying a linear transformation to that vector, adding the mean of the data, and then injecting Gaussian noise. The latent vector is lower dimensional than the observation. A matrix maps latent coordinates into the observation space, and a single noise level controls how spread out the noise is in every direction. The paper's key point is that the latent representation no longer fixes one observation. In the authors' words, "the latent representation no longer determines a single point exactly. Instead, it determines the centre of a distribution of plausible observations."

Given a latent vector, the observation is Gaussian, centred on the linear image of that vector plus the mean. The authors explain that as the latent value changes, the centres trace a line through observation space when the latent space is one-dimensional. The latent variable sets the position along that low-dimensional subspace, while the noise lets observations scatter away from it.

The paper then connects PPCA back to classical PCA through the zero-noise limit. As the noise shrinks to zero, each latent coordinate maps to a single deterministic point, and the randomness disappears. The authors conclude:

In this sense, PCA may be understood as the zero-noise limit of PPCA.

PCA is therefore a special limiting case of PPCA, and PPCA adds

uncertainty around the reconstruction.

To fit the model, the authors maximise the log-likelihood of the dataset. They average it over the training points, which can be written as an expectation under the empirical distribution, that is, equal weight on each observed point. The logarithm is used because, as the authors say, it is "both numerically safer and algebraically simpler." It turns a product of many small probabilities into a sum, which avoids numerical underflow and is easier to analyse.

The quantity that matters is the marginal likelihood of an observation, obtained by integrating the latent variable out. In most latent-variable models that integral cannot be solved by hand. The authors explain why PPCA escapes this. Three facts about linear Gaussian models do the work: a linear transformation of a Gaussian is Gaussian; a sum of independent Gaussians is Gaussian; and marginalising a Gaussian latent variable leaves a Gaussian distribution over the observation. The authors summarise:

PPCA is special precisely because its linear-Gaussian structure keeps this marginalisation analytically tractable.

The chapter works out the mean and covariance directly. The mean of the observation is the data mean, because the latent and noise terms both have zero mean. For the covariance, the latent part contributes the matrix formed by the mapping times its transpose, and the noise contributes the noise level times the identity. The two add together. The authors conclude that

the latent variable has been integrated out exactly, and the resulting marginal over observations remains Gaussian.

The two parts of the covariance have different meanings. The latent part is low rank. The authors write that "It is low rank," so the variation it creates is confined to at most a k-dimensional subspace of the d-dimensional observation space, where k is the latent dimension. The noise part is isotropic, meaning it adds the same variance in every observation direction. PPCA therefore models variability both along the latent subspace and around it.

The authors then turn to learning, and offer three ways to think about it. The first is direct analytic maximum likelihood. One writes the log-likelihood, sets its derivatives with respect to the parameters to zero, and solves. For PPCA this succeeds, and the solution is tied to the eigenstructure of the sample covariance matrix. The authors also flag the limits of this route. It depends on the model being simple enough to solve exactly. Once the model becomes nonlinear or non-Gaussian, the closed-form route typically breaks down. Even when a solution exists, it may need expensive operations such as matrix inversion or eigen-decomposition in high dimensions.

The second route is gradient-based optimisation. The parameters are updated repeatedly by gradient ascent on the log-likelihood. This does not require solving the problem exactly, and the authors note that gradients can be used "whenever the objective or its gradient can be computed". They describe this as foreshadowing how most modern generative models are trained. For PPCA, however, they judge gradient ascent not the most illuminating route, because it does not reveal the latent-variable structure.

The third route is the expectation maximisation algorithm, which the authors credit to Dempster, Laird, and Rubin. Rather than solving in one step or treating the latent variables as nuisance inside a black-box objective, EM alternates between estimating the latent variables under current parameters and updating the parameters using those estimates. The authors say that in PPCA EM is not strictly necessary, but it reveals a general principle for latent-variable models that survives when exact solutions do not.

To see why EM is general, the authors step back to the case where the marginal likelihood is no longer tractable. The logarithm sits outside an integral over the latent variable, which makes direct optimisation hard. The remedy is Jensen's inequality. A concave function, such as the logarithm, satisfies a known inequality that lets the logarithm move inside an expectation, trading exactness for a lower bound that is easier to work with. The bound is called the evidence lower bound, or ELBO.

The derivation introduces an auxiliary distribution over the latent variable. Any valid distribution can be chosen, and it lets the marginal likelihood be written as an expectation of a ratio. Applying Jensen's inequality then gives a bound on the log-likelihood. The authors stress that this distribution is freely chosen and that the step is essential, not a trick of notation. The bound becomes exact when the auxiliary distribution equals the true posterior over the latent variable. The authors conclude that

The ELBO is not merely an arbitrary surrogate objective.

Its tightness depends on how well the distribution approximates the posterior.

The chapter is careful to say what PPCA does not need. The authors write that

PPCA is not special because it avoids the latent-variable problem, but because it solves it exactly.

Its posterior and marginal are both tractable, so PPCA does not strictly require the ELBO. The bound matters for the general case, where exact Gaussian calculations break down.

EM then follows from the bound. Each iteration has two steps. In the E-step, with the current parameters fixed, the auxiliary distribution is set to the posterior over the latent variable, which makes the bound tight. In the M-step, the distribution is held fixed and the parameters are updated to maximise the expected complete-data log-likelihood. The authors note that the entropy term does not depend on the parameters, so it drops out of the M-step.

The authors explain why EM never decreases the likelihood. After the E-step the bound equals the log-likelihood at the current parameters. The M-step does not decrease the bound. And because the bound is a lower bound, the log-likelihood at the new parameters is at least the bound. Together these show that "one full EM iteration cannot decrease the observed-data log-likelihood." The authors add a caveat: standard EM assumes the posterior can be computed exactly. When it cannot, one must use an approximation, and that step leads from EM to variational inference.

For PPCA the posterior is Gaussian, and the E-step yields the expected value of the latent variable and its second moment. The M-step then substitutes these into the expected complete-data log-likelihood and maximises. This gives closed-form updates. The authors conclude that the exact PPCA solution can be read as the fixed point of an EM procedure. In their words, "EM does not give a different model; it gives a different view of learning the same model."

EM therefore frames learning in latent-variable models as alternation between inference, estimating hidden variables given the data, and learning, updating the model parameters from those inferred variables. The chapter also introduces the aggregate posterior, which is the posterior averaged over the whole dataset. It describes how the dataset as a whole is represented in latent space.

The authors identify the limitation that motivates the next chapter. PPCA introduces a prior, but it does not regularise the aggregate posterior to match that prior. The paper says PPCA "does not yet provide the kind of flexible and regularised inference framework" that modern latent-variable models need. The variational autoencoder is presented as a nonlinear and variational extension of the PPCA story. It keeps latent variables generating observations, but learns the mapping and the inference together. The authors are careful to say that moving from PPCA to the variational autoencoder is not a move from a non-generative model to a generative one, since

PPCA is already generative.

The change is from a linear-Gaussian model to a more flexible nonlinear one.

The chapter closes by summarising its arc. PPCA is the first probabilistic latent-variable model in the book. Its linearity and Gaussianity make prior, likelihood, marginal, and posterior all computable, and its covariance splits into a low-rank latent part and isotropic noise. Of the three learning routes, EM best reveals the structure of latent-variable learning, and that structure leads to Jensen's inequality and the ELBO. The authors credit the original PPCA formulation to Tipping and Bishop, and point readers wanting a more pedagogical treatment of Gaussian latent-variable models and EM to the books by Bishop and by Murphy.

The chapter on the Variational Autoencoder, or VAE, joins two ideas the book has already built up. The overview describes the VAE as bringing together "the two strands developed in the previous chapters": autoencoders, which compress data into a smaller code and rebuild the input from it, and probabilistic PCA, which treats that compressed description as a hidden quantity in a generative model. A hidden quantity of this kind is called a latent variable. The authors explain that in a deterministic autoencoder each input maps to one latent code, while in probabilistic PCA the latent variables are probabilistic but the model stays linear and Gaussian. The VAE, they write, "combines these ideas in a more flexible nonlinear setting." It keeps the probabilistic parts of probabilistic PCA, namely a prior, a likelihood, a posterior and a marginal likelihood, but "replaces exact Gaussian inference with a learned approximate posterior." The prior is the belief about latent variables before seeing data, and the posterior is the belief after seeing a data point.

The chapter first makes the encoder probabilistic. A standard autoencoder encoder is a function that returns exactly one latent vector for each input. In a VAE, the encoder instead defines a distribution over latent variables, which serves as a learned approximation to the true posterior. In practice a neural network reads the input and outputs the mean and variance of a Gaussian, a bell-shaped distribution, in latent space, and a latent code is then sampled from it. The authors sum up the change in one sentence:

the latent representation is no longer a single code, but a distribution from which codes may be sampled.

The true posterior is usually out of reach because the networks are nonlinear, so the approximation is needed.

The decoder changes in the same way. Where a deterministic decoder returns one reconstruction, the VAE decoder defines a conditional distribution over observations given a latent variable, which is the generative part of the model. For continuous data this is often a Gaussian, and for binary data a Bernoulli distribution. The authors say that

the decoder does not output a single deterministic reconstruction, but a distribution over possible observations given the latent variable.

In practice the mean or the most likely value is often used as the reconstruction, but the full distribution is what the model is built on.

The standard VAE uses a simple prior, a standard Gaussian with mean zero and identity covariance. The prior does two jobs. It is the starting point for generation: after training, a latent variable is drawn from the prior and the decoder turns it into a data point. It is also the reference the inferred latent distributions are pulled towards during training. The authors point out that probabilistic PCA had a prior but did not explicitly push its inferred latent representations towards it, whereas in the VAE "that alignment will become part of the learning objective itself." The chapter stresses the cost of this added flexibility. The VAE's encoder and decoder are nonlinear neural networks, which "greatly increase modelling power while also destroying the analytic tractability we previously relied on." So the marginal likelihood and the posterior can no longer be written down in closed form, and the chapter asks:

If the true marginal likelihood and posterior are intractable, what objective should we optimise instead?

The answer is the Evidence Lower Bound, or ELBO. Training by maximum likelihood would mean maximising the logarithm of the marginal likelihood, which is an integral over all latent variables. For a nonlinear decoder that integral has no closed form. The ELBO replaces the true posterior with the encoder's approximation and gives a quantity that is always at most the log-likelihood. The chapter writes the objective for a single data point in three equivalent forms, each showing a different side of the problem.

The first is the joint-density form, which expresses the bound as an expectation, under the approximate posterior, of the log of the joint distribution of data and latent variables minus the log of the approximate posterior itself. The second form uses Bayes' rule to show the bound as the exact log-likelihood minus a mismatch term. That mismatch is a Kullback–Leibler divergence, or KL divergence, between the approximate posterior and the true posterior. The chapter explains that KL divergence is never negative and is zero only when the two distributions are the same. It is not a true distance, the authors note, "because it is asymmetric, but it is extremely useful in probabilistic modelling." Because the mismatch cannot be negative, the ELBO can never exceed the true log-likelihood, which is why it is a lower bound. Maximising it means raising the bound on the model's evidence while shrinking the gap between the approximate and true posteriors.

The third form is the one most often used in practice. It splits the objective into a reconstruction term, the expected log-probability the decoder assigns to the data given sampled latent codes, and a regularisation term, the KL divergence between the encoder's distribution and the prior. The reconstruction term rewards accurate data fitting. The regularisation term, in the authors' words, "prevents the encoder from placing each data point arbitrarily in latent space," so that the latent space keeps a coherent global structure and sampling from the prior produces meaningful data. The chapter presents this balance as the reason the VAE is "more than an autoencoder with noise." Its objective is "to perform approximate probabilistic inference inside a latent-variable generative model."

The last part of the chapter covers how the objective is optimised. The reconstruction term is an expectation over samples drawn from the encoder, and the sampling step depends on the encoder's parameters. Treated as a black box, the sampling step blocks ordinary gradient-based learning from reaching the encoder. The fix is the reparameterisation trick. For a Gaussian with diagonal covariance, the chapter draws noise from a standard normal distribution and forms the latent sample as the encoder's mean plus the encoder's standard deviation multiplied by that noise. The randomness now sits in noise that does not depend on the encoder, so the sample is a differentiable function of the encoder's outputs. The authors caution that "the trick does not remove stochasticity; it relocates it to the auxiliary noise," which is independent of the parameters. With that in place, encoder and decoder are trained jointly by backpropagating through one computation graph, which the authors call one of the central achievements of the VAE. Training is then an instance of variational inference: the true posterior is never computed, and a tractable approximation is learned.

The chapter also describes the beta-VAE, which adds a weight to the KL term. With a weight above one, the latent space is regularised more strongly, which often gives more structured or disentangled representations, usually at a cost in reconstruction quality. The authors use this to show a central tension in latent-variable models: the latent space must be "regular enough to support generation, but informative enough to support useful representation and reconstruction."

In summary, the chapter presents the VAE as the step from probabilistic PCA to modern deep generative models. It replaces exact inference with a learned approximation, rewrites the ELBO as reconstruction minus regularisation, and uses the reparameterisation trick to train the whole model with gradients. The original VAE was introduced by Kingma and Welling, and Rezende, Mohamed and Wierstra developed closely related stochastic-gradient variational methods.

The text places diffusion models among the most important modern families of generative models, the kind of systems that create new data such as images. It says they achieve "strong performance in a wide range of applications, particularly in image synthesis." The core idea is that,

instead of generating a sample in a single step, learn to transform noise gradually into structured data through a sequence of denoising operations.

The authors focus on the discrete-time denoising diffusion probabilistic model, usually called DDPM, which they describe as providing "one of the clearest probabilistic formulations of this idea." They explain the intuition with physics. A drop of ink released into water slowly spreads until it is indistinguishable from the water around it. Diffusion models reverse that picture:

instead of watching order dissolve into noise, we learn how to start from noise and gradually reconstruct structure.

The authors approach diffusion through the probabilistic, latent-variable view they built with probabilistic PCA and the variational autoencoder. A latent variable is a hidden quantity that is not directly observed but is assumed to generate what is observed. The key difference, in their account, is that "instead of using a single latent variable or a single stochastic encoding step, diffusion introduces a whole sequence of latent states connected through time." That gives the model two linked phases: a fixed forward process that corrupts data into noise, and a learned reverse process that rebuilds structure from noise. The authors say this temporal design adds flexibility, but it also changes how inference and learning have to be formulated.

The argument runs in a clear order. First comes the forward process, which takes a clean data sample and corrupts it with Gaussian noise over a fixed number of steps, producing a trajectory of increasingly noisy states. Next, the reverse process is introduced as the learned generative model. It starts from a simple Gaussian distribution at the final step and learns a chain of denoising transitions. Then the authors recast diffusion in the evidence lower bound, or ELBO, a tractable quantity that sits below the log-likelihood of the data and can be pushed up in its place. They then split that bound into local denoising terms, one per time step. After that, they derive the exact reverse posterior, which is the true one-step denoising distribution for a known clean sample, and rewrite it in terms of noise. The result is the practical DDPM training objective, which turns learning into noise prediction.

The forward process is the simpler half. It is built from a noise schedule, a set of small numbers that decide how much fresh noise enters at each step. At every step the state keeps a portion of the previous signal and adds a small Gaussian perturbation. The authors write that the forward process is "fixed and known, rather than learned from data." Each state depends only on the one just before it, which is the Markov assumption that keeps the chain analytically manageable.

The text then gives a closed form that matters in practice. Rather than simulating every step, one can jump straight from the clean sample to the noisy state at any time step. In the authors' words,

This turns out to be one of the most useful features of the forward diffusion process, because it lets us jump directly to any time step t without having to simulate the entire chain.

The derivation shows that the many noise steps add up to a single Gaussian. A noisy state therefore keeps a shrinking fraction of the original signal, and the rest is replaced by noise. The text says that after enough steps the original structure is washed out and the sample is driven close to isotropic Gaussian noise, meaning noise with the same spread in every direction. An illustration over a thousand steps, using a linear schedule, shows the signal fading as the noise grows.

The reverse process is the part that has to be learned. Starting from a random Gaussian sample, it generates a slightly cleaner state at each step, with the aim that the final state looks like a genuine data sample. The authors note that "the reverse transitions are not known in closed form and must therefore be learned from data." They use a neural network with parameters to represent these transitions, and they again assume the reverse chain is Markovian.

The obvious shortcut is to reverse the forward rule, but the authors explain why that does not work directly. Bayes' rule says the reverse step depends on the distribution of the states at the two time steps, and those distributions are not simple fixed Gaussians. They are induced by pushing the unknown data distribution through the noising chain. As the authors put it, "although the forward step itself is simple, the true reverse conditional depends implicitly on the underlying data distribution and is therefore not available as a simple rule that we can directly evaluate or sample from."

A second difficulty concerns likelihood. The model should assign high probability to real data, which means computing the marginal likelihood of the clean sample. That requires integrating over the whole high-dimensional trajectory of latent states, which is generally intractable. So the central question is posed plainly:

how can we optimise the model when the exact marginal likelihood is not available in closed form?

The answer, as in the variational autoencoder, is the ELBO.

The authors map the diffusion setting onto the general ELBO. The observed variable is the clean sample, and the latent variable is the whole noisy trajectory. What differs from the variational autoencoder is the auxiliary distribution. The variational autoencoder needs a learned encoder to approximate the posterior. In diffusion, the known forward process already supplies it. The text puts it this way:

So the auxiliary distribution required by the ELBO is not an extra learned construction added on top of the model; it is already built into the model as the forward diffusion process itself.

It follows that "diffusion does not need to learn a separate encoder network for the latent variables in the way the VAE does." The authors then state the bound and take an outer expectation over the training data,

This is the ELBO specialised to the diffusion setting.

Diffusion models, in their framing, remain latent-variable models, but the latent is now a time-indexed sequence rather than a single code.

The next step expands that bound so that the learning problem splits across time steps. The forward chain appears in the denominator of the bound, but the model is trying to learn a reverse process. So the learned denoising transition should be compared with the true reverse posterior, not with the forward corruption rule. The authors distinguish the two carefully. The learned reverse transition is the model being trained. The true reverse posterior is the exact one-step denoising distribution implied by the forward process, once the clean sample is fixed. Conditioning on the observed clean sample is what makes the posterior analytically tractable. During training the clean sample is known, so the target is available even though the unconditional reverse distribution is not.

To expose that posterior, the authors apply Bayes' rule while keeping every term conditioned on the clean sample. The logarithm of the product then splits into a sum, and most of the terms that involve only the forward process telescope and cancel. What remains is a degenerate constant and no learnable parameters, so it can be dropped. The authors then sort the result into three roles:

This decomposition separates three roles: prior matching at the noisy endpoint, local reverse-transition matching at intermediate steps, and reconstruction at the data end.

The first term asks whether the final noisy state is close to the Gaussian distribution the reverse process starts from. The last term measures how well the model reconstructs the clean sample from its first noisy state. The middle terms, one per step, ask the model to match the true reverse posterior at each time.

The authors then turn to the form of that posterior. The forward process is Gaussian, so the posterior is Gaussian too. In the text's words,

Because the forward process is Gaussian, the true reverse posterior is also Gaussian.

Its mean is a weighted combination of the clean sample and the current noisy state. The authors interpret it as "the most plausible slightly cleaner state by combining knowledge of both the clean origin and the current noisy observation." Its variance is fixed by the noise schedule, so the reverse step's uncertainty is already determined by the forward design. The model is therefore not asked to match an arbitrary denoising distribution, but a specific Gaussian target.

There is a catch. The authors note that

Although this posterior is exact, it still depends explicitly on the clean data sample x0.

That is acceptable during training, when the clean sample is known, but "it is not usable directly at generation time, when x0 is precisely the unknown object we wish to create."

The fix is reparameterisation. The forward process gives a direct relation between the clean sample, the noisy state and a standard Gaussian noise variable. Solving that relation for the clean sample and substituting it into the posterior mean removes the explicit dependence on the clean data. The posterior mean can then be written in terms of the current noisy state and the noise that produced it. The authors draw the consequence:

So instead of trying to recover x0 directly, the model can aim to predict the underlying noise.

This gives the standard DDPM parameterisation. A neural network takes the noisy state and the time step and predicts the noise. The learned reverse transition is then a Gaussian whose mean is built from that prediction, with the same fixed variance as the true posterior. If the prediction is accurate, the learned mean matches the exact one. Because both distributions share the same variance, their KL divergence reduces to a squared difference between the two means. Substituting the two mean expressions shows that the difference is proportional to the gap between the true noise and the predicted noise, up to a time-dependent scaling factor. The authors summarise:

the local variational terms reduce, up to a time-dependent coefficient, to a regression problem in the added Gaussian noise.

The commonly used simplified objective, introduced by Ho and colleagues, drops that time-dependent weighting and trains on plain squared error in the noise. That is the version most practitioners use.

Training, in the text's account, is conceptually simple. For each clean sample, one picks a random time step, draws a Gaussian noise sample, and constructs the noisy state using the closed-form relation. The network is then trained to predict the noise from the noisy state and the time step. Because the noisy state can be generated directly, training never needs to simulate the full forward chain.

Generation runs the learned reverse process. It starts from a sample of pure Gaussian noise and, for each time step from the last to the first, draws a slightly cleaner state. At each step the model uses its predicted noise to compute the reverse mean and remove a small amount of corruption. The final state is the generated sample.

The authors close their account by summarising the design. The forward process is fixed, the reverse process is learned, the ELBO reduces to local denoising targets, and the practical objective becomes noise prediction. They credit Sohl-Dickstein and colleagues with the original framing, which drew on nonequilibrium thermodynamics and established a fixed corruption process, a learned reverse process, and a sequence of intermediate latent states linking data to noise. They credit Ho, Jain and Abbeel with the modern formulation, which trained the sequential model through a variational objective that simplifies to noise prediction. Nichol and Dhariwal are credited with refinements that improved training and sampling. The authors also say that the score-based and continuous-time views developed later in the book offer a complementary interpretation of the same denoising idea. The focus here is the discrete latent-variable construction.

The chapter opens with a question about step size:

what happens when the step size becomes smaller and smaller, so that the evolution is better viewed as continuous in time?

It looks back at the previous chapter's discrete-time diffusion models, where generation is a sequence of finite noising and denoising steps, and notes that many methods in machine learning follow the same pattern of repeated small updates. In the concrete form it gives, the next state equals the current state plus a structured change scaled by a small time step, plus an optional random push. In discrete time such rules are applied step by step. In continuous time the same idea becomes infinitesimal motion, and that is where the language of calculus becomes essential.

The chapter moves in three stages: discrete updates, then continuous motion, then density evolution. The last stage matters because generative modelling cares less about where one particle goes than about how a whole cloud of particles reshapes its probability density. The stated aim is to build the calculus foundation for the next chapter, which connects these ideas to score-based generative modelling and fully continuous diffusion.

The opening stretch introduces the basic vocabulary. A derivative measures the rate of change of a quantity, often read as velocity when the quantity is position. The text describes an infinitesimal change as rate times duration. It then uses a first-order Taylor expansion, which approximates a smooth function near a point using its slope there. When the displacement is very small, the discarded higher-order terms become much smaller than the linear term, so an update rule can be read as a derivative. A differential equation, in this picture, is a rule of motion that assigns a direction of travel to every point in space.

The first motion pattern is deterministic contraction. In one dimension, velocity is proportional to position with the opposite sign, so the state is pulled toward the origin, and the pull is stronger the farther away the state is. Solving the equation gives an exponential decay toward zero, and a rate parameter alpha controls how fast that decay happens. In two dimensions the chapter uses a diagonal matrix, so each coordinate contracts independently at its own rate. The solution is written with the matrix exponential, which is defined by the same power series as the scalar exponential, with the matrix standing in for the number. The point is that different directions may shrink at different speeds while the motion still heads toward the origin.

The second pattern is Brownian motion, also called Wiener motion. The chapter lists three defining properties: the process starts at zero, increments over disjoint time intervals are independent, and an increment over any time step is normally distributed with mean zero and variance equal to the step length. Splitting a time interval into many small pieces and adding the increments shows that the position at time t has mean zero and variance t, so its spread grows linearly with time. Because the variance equals the step, the typical size of a step scales with the square root of the step length, not the step length itself, and the text says this square-root scaling will reappear in diffusion models. Brownian motion, the chapter stresses, "does not pull the state toward any preferred location; instead, it describes random wandering whose uncertainty increases as time passes." The text also calls it the canonical model of continuous-time random fluctuation, and it adds that the increments are Gaussian with variance growing linearly in time.

The third pattern combines the first two in the Ornstein–Uhlenbeck process. A drift term pulls the state back toward the origin, and a noise term injects Brownian perturbations. The chapter gives its explicit solution: the initial value decays exponentially, and all past random perturbations are accumulated but weighted by a decay factor, so older noise fades while recent noise matters more. This is why the process is described as mean-reverting. In the chapter's words, it "remains random while retaining a stable restoring tendency." A figure places the three motion patterns side by side to show how the Ornstein–Uhlenbeck process mixes the two basic effects.

The chapter then shifts from individual particles to whole probability densities. The simplest principle is that

probability is conserved under deterministic transport.

If a particle moves from a starting point to a new location under a fixed flow, the probability in a small interval travels with it. The density at the new location therefore equals the initial density divided by the factor by which the flow stretched that interval. In one dimension that stretching factor is the derivative of the flow map. Stretching lowers the density because the same mass spreads over more space, and compression raises it. In several dimensions the same role is played by the determinant of the Jacobian matrix, which measures how a small volume is scaled.

From this finite-time result the chapter derives an infinitesimal law, the one-dimensional Liouville equation. It fixes a point in space and tracks probability flux, which is density times velocity, flowing in through one boundary of a small interval and out through the other. Net outflow changes the mass inside the interval, and in the limit the rate of change of density equals the negative spatial derivative of the flux. Expanding that derivative with the product rule separates two effects. A transport term carries the existing density profile along with the velocity, while a deformation term depends on how the velocity changes with position and makes nearby particles either spread apart or bunch together.

Following a single particle instead of a fixed point, the transport term cancels. The chapter finds that the logarithm of the density changes along a trajectory at a rate equal to the negative spatial slope of the velocity. From the particle's own viewpoint, the density changes only through local expansion or compression. The text puts it directly:

What remains is only the local expansion or compression effect.

The multidimensional version replaces the slope with divergence. The Jacobian matrix describes how a small displacement changes under the flow, and its determinant gives the local volume scaling. For a small step, the determinant equals one plus the trace of the Jacobian times the step, and the trace is the divergence of the velocity field. The chapter defines divergence as the quantity that "measures the net outflow of a vector field from a tiny region of space." Conservation of probability then gives the multidimensional Liouville equation: the rate of change of density equals the negative divergence of density times velocity. Along a trajectory, the log density again changes at the negative divergence of the velocity, so the picture matches the one-dimensional case. Where the divergence is positive, volume expands and density falls, and where it is negative, volume shrinks and density rises.

Next the chapter adds randomness. Under pure Brownian motion, the density at a point after a short time is a weighted average of the earlier density nearby, with weights given by a narrow Gaussian kernel. Probability diffuses in from neighbouring locations and diffuses away from the point. The chapter describes this as convolution with a narrow Gaussian, which smooths and spreads the density. Writing the density as a Taylor series in the Gaussian increment and taking the expectation, the first-order term vanishes because the increment has mean zero. The second-order term survives because the variance is the time step. Taking the limit as the step goes to zero gives a diffusion equation in which the rate of change of density equals half the diffusion coefficient times the second spatial derivative.

Thus pure Brownian motion gives the pure diffusion equation.

Adding drift to the diffusion gives the one-dimensional Fokker–Planck equation, which combines the transport term from the Liouville equation with the diffusion term. In several dimensions the diffusion term uses the Laplacian, the sum of second spatial derivatives. The chapter compares three cases with the same starting Gaussian density. Drift alone moves the mean toward the origin while shrinking the variance. Diffusion alone keeps the mean fixed while increasing the variance. Together, as in the Ornstein–Uhlenbeck process, the mean reverts toward zero while the variance approaches a stationary value equal to the diffusion coefficient squared divided by twice the restoring rate. The text's short summary of the comparison is:

Drift transports probability; diffusion spreads it.

The chapter presents the Liouville and Fokker–Planck equations together as the main bridge from elementary continuous-time motion to the density dynamics behind fully continuous diffusion models, and it points ahead to the next step, where density evolution is linked to the gradient of the log density.

Score-based modelling starts from a different question than the one the previous chapter asked about denoising diffusion. That chapter treated diffusion as a discrete latent-variable construction, in which data are gradually corrupted into noise and a learned reverse process gradually rebuilds structure. This chapter adds a second reading. Instead of asking only how to write down a reverse Markov chain, it asks a more local and geometric question: if a sample sits at a given point, which direction should it move so that it becomes more likely under the distribution the model is meant to generate from? The answer is the score, the gradient of the log-density, which points in the direction of steepest local increase of that log-density.

The chapter presents this as a change of viewpoint, and it stresses the change.

A probability density describes where mass is concentrated, but the score describes how to move through the space defined by that density.

The density becomes a vector field, a set of local arrows that can guide noisy samples back toward regions of high probability. In the chapter's framing, score-based modelling replaces the problem of explicitly evaluating a density with the problem of learning the local geometry of that density. The chapter is clear that this complements the DDPM viewpoint rather than replacing it. Its concept map runs in order: local density geometry, score learning, multi-scale denoising, continuous-time reverse diffusion, and a final return to DDPMs.

The first stage treats the score as a drift field. In one dimension the score is the derivative of the log-density, which is the slope of the density divided by the density itself. The chapter notes that this division makes the score a relative slope, a more meaningful field of motion than an absolute slope. For a one-dimensional Gaussian with mean mu and variance sigma squared, the score works out to the negative of the distance from the mean, divided by the variance. A point to the right of the mean gets a negative score and a point to the left gets a positive one, so both point back toward the centre, and the further out a point sits, the stronger the pull. The chapter says that "the score behaves like a restoring field that directs particles toward the high-density region." In several dimensions the score becomes the negative inverse covariance matrix times the offset from the mean. The chapter draws out the consequence:

Directions with smaller variance produce stronger restoring forces, while directions with larger variance produce weaker ones.

A figure on a two-component mixture shows the same idea for a multimodal density, where the score arrows bend toward the locally more influential mode. Its arrows are normalised so that direction, not magnitude, is emphasised.

Following the score as a deterministic flow moves particles uphill. The chapter then names the weakness of this:

A purely deterministic score flow tends to move particles uphill toward modes.

Such a flow can collapse samples into concentrated regions. That is useful for optimisation but not enough for sampling, so noise has to be added, which produces Langevin dynamics. Langevin dynamics combines a drift given by the score, which pulls particles toward higher density, with Brownian noise, which keeps them exploring. In discrete time, each step moves the sample by a small multiple of the score and then adds a Gaussian perturbation whose scale is the square root of twice the step size. The noise stops the sample from simply collapsing into one mode. Because the process is drift plus noise, its density follows a Fokker–Planck equation. The chapter reads that equation as two competing effects: the score term concentrates probability in high-density regions, and the diffusion term spreads it outward. The chapter then compares this with the Ornstein–Uhlenbeck process from earlier. That process has a linear pull toward the origin and a Gaussian stationary law. Langevin dynamics generalises it by replacing the fixed linear restoring force with the score of any chosen target. For a standard Gaussian target the two coincide, up to a noise scaling.

The chapter then checks that the target density is stationary under Langevin dynamics, meaning that once the density reaches the target it stays there. Substituting the target into the Fokker–Planck equation, the score term turns into the Laplacian of the target, which exactly cancels the diffusion term, so the rate of change is zero. The chapter calls this one of the central conceptual results behind score-based generative modelling: using the score as the drift, with the right noise, preserves the target. It is careful about the limits. The argument establishes stationarity, not a full proof that every starting density converges to the target. The chapter says that a full convergence theory "requires additional assumptions and is beyond the present tutorial."

The next question is how to learn the score when the target density is not available in closed form and only samples are given. The chapter models the score with a parameterised function, typically a neural network, and measures the error with the Fisher divergence, the average squared difference between the learned score and the true data score. The difficulty is that this objective seems to need the true score, which depends on the unknown density. Score matching gets around this with integration by parts. In one dimension, the chapter rewrites the cross term, the expected product of the model score and the true score, by moving the derivative from the unknown density onto the model. This costs only a boundary term, which vanishes under the assumption that the data density decays quickly. The resulting objective is the expected value of half the squared model score plus the derivative of the model score. In several dimensions the derivative becomes a divergence, following from the divergence theorem, and the final objective is half the squared norm of the model score plus its divergence. The chapter credits this idea to Hyvärinen, who showed that models could be estimated by matching gradients of log-densities without needing the normalising constant.

Directly learning the score of real data is hard, because real data tend to concentrate near low-dimensional manifolds in a high-dimensional space. The density can then be sharply concentrated and irregular, and the learned field can behave erratically where there is little data. The remedy is Gaussian corruption: add noise to each clean sample, so that the noisy distribution is the original data smoothed by a Gaussian. Probability spreads into nearby regions and sharp structure softens. The chapter then shows that the score of this smoothed density equals one over the noise variance, times the expected difference between the clean point and the noisy point, given the noisy point. In its words, "the score points toward the conditional average clean point that could have produced it." This target still cannot be computed, because it needs the posterior and the marginal density. But pairs of clean and noisy points can be sampled easily, by drawing a clean point and adding noise. The chapter uses a basic property of least-squares regression: the best squared-error predictor is the conditional mean of the target given the input. Using the noisy point as input and the scaled difference as target, the optimal predictor is exactly the smoothed score. The chapter states that "Individual training pairs provide tractable conditional targets," and that averaging them recovers the score of the smoothed density. A figure illustrates this with thin arrows for individual targets averaging into one thick arrow. The chapter concludes that

Denoising score matching therefore replaces an intractable score estimation problem with a supervised regression problem defined entirely through a known corruption process.

It also notes a limit: for a fixed noise level, the model learns the score of the corrupted density, not the data density directly, so the choice of noise scale matters.

A single noise level is not enough. With large noise the score gives stable global guidance, but fine details are blurred away. With small noise the score captures sharp local detail, but it is hard to estimate far from the data. The chapter puts it this way:

Thus one noise level cannot simultaneously provide robust global guidance and precise local refinement.

A figure shows the same two-component density smoothed at three scales, where large noise reveals coarse geometry and small noise exposes separate modes. The solution is to learn a family of score fields over a decreasing sequence of noise levels, using a model that takes the noise level as an input. Sampling can then use annealed Langevin dynamics. It starts from broad noise and runs Langevin updates at the largest scale, then moves to smaller scales. The process moves from random noise to a coarse data region, then to local structure, then to fine detail. The chapter notes that annealed Langevin dynamics is a generative procedure but not yet a diffusion model in the DDPM sense, because a diffusion model arranges the intermediate distributions as the marginals of one forward noising process.

When the noise levels become many and close together, the chapter replaces the level index with a continuous time variable. The noisy distributions become a continuous family running from the data distribution at the start to a simple noise distribution at the end, and the score becomes time-dependent. The chapter is explicit that learning scores at many levels does not by itself define a diffusion model. One needs a stochastic dynamics whose evolving distribution is that family. The forward process is a stochastic differential equation with a drift term and a noise term scaled by a coefficient that depends on time. Its job is not to generate new data. It defines a controlled path from data to noise. Generation runs that path backward. The chapter shows that the reverse-time dynamics has a drift equal to the forward drift minus the squared noise coefficient times the score, plus reverse-time noise, simulated from the terminal time back to zero. Since the true score is unknown, a learned time-conditioned model takes its place. Generation starts from a simple noise sample and evolves backward using the learned score. The chapter concludes that "a diffusion model is not simply a score estimator, and score matching alone is not yet a complete generative model." Instead, "the forward diffusion process defines a structured path from data to noise, while score learning provides the local information needed to reverse that path."

The chapter then links this back to the DDPM noise-prediction objective. In the DDPM forward process, a noisy sample can be drawn directly from a clean sample, with the clean signal weighted by a cumulative factor. For this Gaussian, the conditional score with the clean sample treated as known is the negative of the added noise, scaled by a known factor. So a network that predicts the noise is also predicting a conditional score, up to that scaling. The chapter is careful that this conditional score is not the marginal score the reverse process needs. The marginal score comes from averaging over all clean samples that could have produced the noisy observation, just as in denoising score matching. Noise prediction is therefore a practical regression route to the kind of information score-based models need. The chapter's unified view places the two perspectives side by side. The DDPM view emphasises the discrete latent-variable structure and the reverse Markov chain, while the score view emphasises the geometry of the intermediate densities and the continuous-time dynamics. The chapter says that

The main conceptual message is not that all formulations are identical in their details.

What they share is the idea to

learn how noisy samples should be locally corrected, and use those learned corrections to transform noise into data.

The chapter closes by tracing its sources. It credits Hyvärinen with score matching, Vincent with connecting score matching to denoising, Song and Ermon with learning scores across multiple Gaussian noise levels and using annealed Langevin dynamics, especially when data concentrate near lower-dimensional structures, and Song and colleagues with the stochastic differential equation formulation and reverse-time sampling. Ho and colleagues are credited with the DDPM formulation. Together, the chapter says, these works provide the main conceptual foundation for understanding diffusion models as "both probabilistic denoising models and score-guided generative dynamics."

The chapter starts by setting its subject against the approaches covered earlier. Latent-variable models, diffusion models and score-based models all reach a probability indirectly, through a hidden variable, a learned reversal of a noising process, or a learned score field. This chapter takes a different route. In its words,

Instead of relying on an intractable latent marginal, or learning to reverse a stochastic corruption process, we impose mathematical structure that keeps likelihood evaluation tractable from the start.

The aim is a model in which the probability of an observed data point can be computed directly. The chapter names two such routes, normalising flows and autoregressive factorisations, and says both matter because their core ideas keep recurring in newer systems: change of variables, tractable Jacobians, conditional factorisation and ordered generation.

The first route is normalising flows. A flow starts from a simple base distribution, such as a standard Gaussian, and passes it through a chain of differentiable, invertible transformations. The key tool is the change-of-variables formula, which says exactly how probability density changes under an invertible map. The chapter gives the intuition in two parts. The density of the output must account for where its probability came from, since each output has a single source point. It must also account for stretching:

A transformation that spreads points further apart should reduce the density, while one that compresses points should increase it.

The Jacobian, a matrix of local slopes, measures this stretching at each point, and its determinant gives the local change in volume.

The chapter makes this concrete with a one-dimensional example. A variable uniformly spread over the interval from zero to one is doubled. The doubled variable now spreads over the interval from zero to two, and the density falls to one half. In the words of the chapter,

the same total probability now fills an interval twice as wide, so its density is halved.

In two dimensions, a linear map that doubles one direction and triples the other multiplies local area by six, so the density is divided by six. The general formula says the same thing for any differentiable invertible map: the density of the output equals the density of the input point times the reciprocal of the absolute Jacobian determinant.

The chapter also explains why the transformations must keep the dimension. The Jacobian has to be square so that its determinant measures volume change. A map into a lower-dimensional space collapses information, so different inputs can land on the same output. A map into a higher-dimensional space produces points that lie on a thinner surface, which has no ordinary full-dimensional density unless extra structure is added. The chapter concludes that "standard normalising flows preserve dimension."

A linear map cannot bend space, so the chapter introduces planar flows as a first nonlinear layer. As the concept map puts it,

Planar flows introduce nonlinearity through a residual rank-one perturbation.

The layer takes the input, projects it onto a learned direction to get a single number, passes that number through a smooth squashing function such as tanh, multiplies the result by a learned vector, and adds the outcome back to the input. The chapter calls this a residual structure, in the spirit of residual connections in deep learning. The addition of the input keeps the full space available. Without it, the output would lie along the single direction of the learned vector.

The Jacobian of this layer has a useful form. It is the identity matrix plus a rank-one matrix, meaning a matrix whose outputs all point along one vector. The chapter states,

The Jacobian is a perturbation of the identity by a rank-one matrix.

This matters because the determinant of a full high-dimensional matrix is expensive, usually costing an amount that grows with the cube of the dimension. The matrix determinant lemma reduces the calculation to a single scalar. The chapter notes that the planar flow therefore does not deform all dimensions at once. It applies a controlled one-directional change whose volume effect can be computed in closed form.

Invertibility is the harder part. The planar layer has no simple closed-form inverse. The chapter instead gives a sufficient condition that keeps the determinant from reaching zero, which would mean local volume collapse. For a squashing function whose slope lies between zero and one, such as tanh, it is enough that a scalar combination of the learned weight and direction stays above minus one. This can be enforced by reparameterising the direction. The chapter sums up the trade-off in these terms: planar flows keep the determinant analytically tractable and make invertibility controllable through a scalar condition, but they do not offer an easy inverse.

The chapter then turns to affine coupling layers, introduced in the NICE and RealNVP work by Dinh and colleagues, as a practical alternative. The input is split into two parts. The first part passes through unchanged. The second part is scaled and shifted elementwise, with the scale and shift computed from the first part by functions that are usually neural networks. The first part therefore acts as context that controls how the second part is transformed. The layer deliberately gives up full mixing inside one step in exchange for exact inversion and a cheap determinant.

The Jacobian of a coupling layer has a block structure. The block for the unchanged part is simply the identity, and the block linking the two parts is the one that may be complicated, since it involves the derivatives of the neural networks. The chapter notes that this complicated block does not affect the determinant. The remaining block is diagonal, so the log-determinant reduces to the sum of the scale outputs. In the chapter's words,

The triangular block structure is the key computational simplification: the off-diagonal block may be complicated, but it does not enter the determinant.

The determinant is therefore computed in linear time in the dimension, even though the networks inside the layer are highly nonlinear.

Inversion is just as direct. The unchanged part is copied across, and the transformed part is recovered by subtracting the shift and dividing by the scale. The chapter concludes that

No root finding or iterative inversion is needed.

The inverse costs about the same as the forward pass, which matters for large models that must evaluate likelihoods and draw samples many times. The chapter says coupling works because the dependency is one-way: one subset controls the other, but not the reverse, which gives both the triangular Jacobian and the closed-form inverse.

A single layer can only deform space modestly, so the expressive power of flows comes from composition. The chapter explains that

many simple, invertible transformations are stacked so that their small deformations accumulate into a rich global mapping.

Because each layer is invertible, the whole stack is invertible, and the model supports two operations in opposite directions. For likelihood evaluation, a data point is mapped backward through the inverses to the base space, and the base density at that point is adjusted by the sum of the log-determinants along the way. Each layer adds one volume correction to the final log-likelihood. For sampling, a point is drawn from the base distribution and pushed forward through the layers.

The chapter stresses that composition alone is not enough. If the same split were used in every layer, some variables would act only as context and never be transformed. To prevent this, practical flows interleave the nonlinear layers with mixing operations, such as permutations of the coordinates or general invertible linear maps. The chapter says,

The purpose of these mixing operations is to change which coordinates interact in later layers.

The usual pattern is to mix, apply a tractable nonlinear deformation, and mix again. The chapter names RealNVP and Glow, the latter by Kingma and Dhariwal using invertible one-by-one convolutions, as architectures built on this pattern and trained by exact maximum likelihood.

The second route, autoregressive factorisation, reaches exact likelihoods in a different way. Instead of transforming a base variable into data, the model writes the joint probability of a data point as a product of conditional probabilities, using the chain rule of probability. A data point is treated as a sequence of components, each modelled given the components before it. The chapter explains that this shifts where structure is imposed. In flows, the structure sits in the transformation. In autoregressive models, it sits in the dependencies among the components. The chapter notes,

This identity is exact. It makes no approximation and imposes no modelling assumption by itself.

The modelling choice comes from the ordering and from how each conditional is parameterised.

The product is a valid probability density as long as each conditional is itself normalised, meaning its values add up to one over all outcomes. The chapter shows that integrating out the last component, then the one before it, and so on, leaves one at each step, so the whole product integrates to one. Once that holds, the model gives exact likelihoods by construction, with no latent variables, variational lower bounds or Jacobian determinants. The price, the chapter says, is that

they impose an explicit ordering, and sampling becomes sequential.

In practice, a single shared network produces the parameters of all the conditionals, not a separate model for each factor. The chapter stresses that

The same parameters are reused across the factorisation; what changes from one conditional to the next is the conditioning context.

For continuous values, a common choice is a Gaussian conditional, whose mean and variance are functions of the earlier components. For discrete values, the network outputs the probabilities of a categorical distribution. The chapter walks through a three-variable case: the first component has no predecessors, so its parameters are unconditional, the second is conditioned on the first, and the third on the first two. The same network does all three jobs, with a different amount of context each time.

The model is trained and used in different ways, and the chapter treats this asymmetry as central. During training, every component of the observed data point is known, so each conditional can be evaluated against the real earlier values, and many of these terms can be computed at once. During sampling, the components must be generated in order, because each new one depends on the ones already drawn. Each sample therefore takes as many sequential steps as the data has components, which is the source of the slowdown the chapter describes.

To make this work in a neural network, the parameters for each component must depend only on the components before it. The chapter explains that this is enforced by masking, which blocks forbidden connections so that information flows only from earlier variables to later ones. Written as a dependency pattern, the allowed connections form a lower-triangular structure. The chapter observes that this avoids cycles: a component can depend on earlier ones, but those earlier ones never depend on it. As a result, the joint log-likelihood splits into a sum of conditional log-likelihoods. The chapter says,

In this sense, triangular dependency in autoregressive models plays a role analogous to triangular Jacobians in coupling-based normalising flows.

The chapter links this principle to well-known systems. MADE, by Germain and colleagues, applies masking to feedforward networks. PixelRNN and PixelCNN, by van den Oord and colleagues, model images one pixel at a time. WaveNet, from the same group, uses causal convolutions to generate raw audio. These examples show the autoregressive route driving sequence, audio and image generation.

The chapter closes by bringing the two routes together. Normalising flows get exact likelihoods from invertible maps with tractable Jacobians, and autoregressive models get them from ordered conditional factorisations. The chapter concludes that

Across both model families, the central message is that exact likelihoods are made possible by structural constraints.

Both show that high-dimensional probability models can stay computable when their transformations or dependencies are carefully organised.

The chapter opens by looking back over the book. Probabilistic PCA and variational autoencoders introduced latent-variable modelling and approximate inference. Diffusion models turned generation into sequential noising and denoising, and score-based models reread that denoising through learned score fields and stochastic dynamics. Normalising flows and autoregressive models kept exact likelihoods by building transformations or factorisations with a known structure. The chapter then asks a different question: what happens when a tractable likelihood is no longer the central object? In many important generative models, the data distribution is not an explicitly normalised density that can be evaluated at any point. Learning instead proceeds through comparison, through geometry, or through unnormalised scalar landscapes. Three routes are named: generative adversarial networks, Wasserstein GANs, and energy-based models. The chapter's central theme is stated in its concept map as "how learning can proceed when probabilities are implicit, unnormalised, or replaced by comparisons."

The first route is the generative adversarial network, or GAN, which the chapter attributes to Goodfellow and colleagues. It starts from a simpler question, which it poses directly:

do we need to compute probabilities at all in order to generate realistic data?

If the goal is samples that look as though they came from the data, the chapter argues that learning by comparison may be enough. Rather than asking how likely a sample is, the model asks whether it is indistinguishable from real data. Learning proceeds by matching samples, not by evaluating densities.

A GAN has two networks. The first is a generator, a parameterised function that takes a random latent variable, drawn from a simple distribution such as a standard Gaussian or a uniform distribution, and maps it into data space. Because the generator defines a distribution only by pushing noise through a function, the chapter says that distribution is implicit: samples can be drawn from it, but its density usually cannot be evaluated at a given point. The chapter sets this against earlier models. The latent variable is not inferred from data as in a VAE, it does not come from a forward noising process as in diffusion, and the generator need not be invertible as in flows, so no Jacobian determinant or explicit normalisation is needed. The generator "behaves like a deterministic simulator: it specifies how to generate data, but not how likely a given data point is." At generation time, sampling is direct: draw the latent variable and compute the generator's output. The chapter's verdict is that GANs provide "direct sampling, but not direct likelihood evaluation."

The second network is the discriminator, which supplies the training signal. It maps an input to a number between zero and one, read as its confidence that the input came from the real data rather than from the generator. It is trained as a binary classifier, scoring real samples near one and generated samples near zero. The chapter stresses that

The discriminator is not an end in itself.

It defines no density and is not the final object used after training. Its job is to provide a moving, adaptive measure of the gap between real and generated samples. The two networks play a minimax game. The discriminator tries to maximise a score that rewards separating real from generated samples, and the generator tries to minimise it by producing samples the discriminator rates as real. In practice the two steps alternate, and the generator gradually reshapes its output until the comparison stops being useful.

To see what this game does to whole distributions, the chapter first fixes the generator and asks what the best possible discriminator would be. With the generator fixed, the discriminator is solving binary classification between the real distribution and the generator's distribution. Working input by input, the best score is the real density divided by the sum of the real and generator densities. The chapter draws the consequence:

the optimal discriminator is a local density-ratio object, even though neither density is explicitly available to the model.

A figure makes this concrete. The optimal output rises toward one where the data density dominates, falls toward zero where the generator density dominates, and sits at one half where the two are equal.

Substituting that best discriminator back into the value function gives what the chapter treats as the core theoretical result. The value becomes a constant, minus log 4, plus two times the Jensen–Shannon divergence between the data distribution and the generator's distribution. The constant does not depend on the generator, so in this idealised setting, training amounts to minimising Jensen–Shannon divergence. To explain that divergence, the chapter first recalls the Kullback–Leibler divergence from the VAE discussion. KL measures how costly it is to use one distribution to approximate another. It is asymmetric, and it becomes infinite when the approximating distribution assigns zero density where the other has mass. Jensen–Shannon fixes both problems by comparing each distribution with their shared mixture and averaging the two divergences. It is symmetric and bounded between zero and log 2. The chapter's reading is that the generator is not maximising likelihood. It adjusts its implicit distribution to reduce the divergence, which is distribution matching through comparison. The global optimum is reached when the generator's distribution equals the data distribution. At that point the best discriminator is one half everywhere, because real and generated samples cannot be told apart.

Practical training departs from the original generator objective in a known way. Early on, the discriminator can confidently reject generated samples, so the generator's term is almost flat there and its gradient becomes very small even when its samples are poor. The non-saturating loss instead asks the generator to raise the discriminator's confidence that its samples are real, which gives a much stronger learning signal. The chapter notes that the two objectives share the same desired equilibrium, but the non-saturating loss is not an algebraic simplification of the first. It is preferred in practice even though the original objective has the cleaner link to Jensen–Shannon divergence. The adversarial setup can encourage sharp, convincing outputs rather than blurry averages. The chapter also names the weakness, mode collapse. Because the generator only has to fool the discriminator, it may model a convincing subset of the data and ignore other modes. Likelihood-based models are penalised for missing modes, but GANs optimise distinguishability rather than coverage. When two distributions barely overlap, a classification-based gap may say mainly that they are distinguishable, not how one should move toward the other. That limitation motivates the next route.

The Wasserstein route replaces overlap with geometry. Probability distributions are treated as mass that can be moved through space. Small displacements mean the distributions are close, and long transports mean they are far apart. The 1-Wasserstein distance, also called the Earth Mover's distance, is the minimal cost of transporting mass, with cost proportional to the distance moved. The chapter gives the practical form, the Kantorovich–Rubinstein duality. The distance equals the supremum, over all real-valued functions that are one-Lipschitz, of the expected value of the function under one distribution minus its expected value under the other. Lipschitz means the function cannot change faster than unit rate with respect to distance. The chapter explains that without this constraint the comparison would be meaningless, because the function could assign arbitrary values to the two sets of samples.

A worked example makes the point. Put one distribution entirely at zero and another entirely at ten. Any admissible function can differ between those two points by at most ten, and the function minus x reaches that bound, so the distance is ten, matching the intuition that all the mass moves ten units. The chapter notes that an overlap-based divergence can detect disjoint supports but cannot say whether the two distributions are one unit or ten units apart. A second figure shows the same contrast for translated Gaussians of equal shape. As overlap vanishes, Jensen–Shannon divergence approaches its ceiling of log 2 and stops responding to further separation. The Wasserstein distance equals the shift and keeps tracking it.

Wasserstein GANs use this idea inside the adversarial loop. The generator is trained to reduce a Wasserstein estimate, while a neural network that approximates the optimal function, now called a critic, is trained to widen the gap in expected scores. The terminology changes. The critic outputs a real-valued score, with no sigmoid, no probability interpretation, and no requirement of normalisation. Only differences in critic values matter. Its purpose is not a real-or-fake decision but to assign scores whose average gap reflects the geometric separation between the distributions. The generator can still receive useful gradients when the two distributions do not overlap, because the distance changes continuously as the generator moves. The chapter warns that the word score here means something different from the score in diffusion models, which is the gradient of a log-density. The critic's output is a scalar used for comparison, constrained through its Lipschitz behaviour rather than through probabilistic normalisation.

The remaining practical challenge is enforcing the Lipschitz constraint, which neural networks do not satisfy automatically. The original WGAN clipped each weight to a fixed interval. The chapter calls this crude, because it can reduce the critic's capacity and destabilise optimisation. The gradient penalty, introduced in WGAN-GP, instead constrains how the critic's output responds to its input. It adds a weighted penalty that pushes the norm of the critic's gradient toward one, evaluated at points sampled along straight lines between real and generated samples. The chapter argues that these interpolation points lie precisely where the critic must learn how to compare the two distributions. The weighting coefficient controls how strongly the constraint is enforced. The result keeps the transport meaning of the distance while giving more reliable training. The chapter adds that generative learning need not rely on explicit likelihoods, since it can be guided by a learned real-valued function whose geometry pushes generated samples toward the data.

The third route, energy-based models, goes a step further. The chapter describes them as beginning with a single scalar function over the data space. Each configuration receives an energy value. Lower energy indicates compatibility with the structure of the data, and higher energy indicates implausibility. The energy is deliberately unconstrained. It need not be positive, bounded or normalised, and by itself it is not a probability. The chapter prefers to read it as preference, cost or compatibility. Learning means shaping a landscape in which observed data occupy valleys. Only relative energy differences matter. Adding a constant to every energy shifts all points equally and leaves their ordering unchanged. The modelling question becomes

where should the model place low-energy regions in the data space?

and the answer is learned from data.

To connect energies to distributions, the chapter uses the Boltzmann form. The density is the exponential of negative energy, divided by a normalising constant called the partition function, which is the integral of that exponential over all configurations. The negative exponential turns low energy into large weight, and the partition function normalises the result globally. The difficulty is that this constant is usually intractable in high dimensions. For images, audio, or text representations, the integral cannot be evaluated exactly. The chapter states the trade-off plainly:

This is the central trade-off of energy-based modelling.

Energy functions can be very flexible because they avoid the structural constraints that exact density models need, but that flexibility costs normalisation. The model usually defines only an unnormalised density. Normalising flows and autoregressive models are designed so that density can be evaluated exactly. Energy-based models place the emphasis on the landscape itself, where energy differences and energy gradients remain useful even when the normalising constant is unknown.

Learning must be contrastive. Lowering the energy of observed samples is not enough, because lowering energy everywhere would not separate data from implausible configurations. The chapter takes maximum likelihood as the principled objective. Maximising the average log-probability of the data lowers the energy of data samples, while the partition function term stops energy from being lowered everywhere, since that term grows when too much of the space gets low energy. Differentiating the objective gives two terms. A data term lowers the energy of observed samples, and a model term raises the energy of samples drawn from the model's current distribution. The chapter calls these the positive and negative phases. The negative phase pushes back against configurations that the model itself currently favours too much.

The chapter identifies the practical problem that follows. The data expectation can be approximated from the training set, but the model expectation requires drawing samples from the current model, which is itself a hard problem. That creates a circular dependence. Improving the energy needs samples from the model, but good samples need an energy landscape that has already been well shaped. The chapter names this as one of the main reasons energy-based models are powerful yet difficult to train.

Because an energy model provides no direct sampler, generation is framed as movement across the landscape. The chapter starts with deterministic descent: begin from a configuration and repeatedly step against the energy gradient, scaled by a step size. That alone can get stuck in a local minimum and never explore other plausible regions. Adding noise gives a Langevin-style update, in which the gradient step is combined with a random Gaussian nudge. The deterministic term moves toward lower energy, and the noise prevents collapse and lets the dynamics explore. Under suitable conditions and a continuous-time interpretation, this dynamics has the energy-based distribution as its stationary distribution. The chapter notes that this is the Langevin idea from the score-based discussion, with the drift written as a negative energy gradient rather than as a score.

The chapter then makes the link explicit. For a density proportional to the exponential of negative energy, the score, which is the gradient of the log density, equals the negative energy gradient. The partition function does not depend on the data point, so its gradient vanishes. That identity bridges the two families: the local direction toward higher probability is the same as the direction toward lower energy. The difference lies in representation. An energy model learns a scalar and derives the vector field by differentiating it, which automatically gives a conservative field that comes from one underlying function. A score model learns the vector field directly. That is more flexible, but the field need not be the gradient of any scalar unless extra structure is imposed. For generation, the chapter concludes, both approaches learn local directions that guide samples toward regions of higher data density.

Score matching offers another way around the partition function. Rather than matching normalised probabilities, it matches local score fields. The chapter recalls the derivation from the score-based discussion and states its energy form: half the squared length of the energy gradient, minus the Laplacian of the energy, which is the sum of its second derivatives. Because the score is the negative energy gradient, learning the score means learning the shape of the energy near the data. The chapter puts the goal this way:

It asks for the local direction in which probability should increase, or equivalently the direction in which energy should decrease.

It credits Hyvärinen with introducing score matching for unnormalised models, and Vincent with connecting it to denoising.

The final step connects energy and diffusion. Diffusion models learn a time-dependent score, approximating the gradient of the log density of each noisy version of the data. Each noisy distribution therefore has an implicit energy landscape, defined as minus its log density plus an arbitrary constant that has no effect on gradients. A diffusion model thus learns the negative gradients of a whole family of energy landscapes, indexed by noise level. The chapter describes this family as a multi-scale hierarchy. At high noise, the distribution is smooth and close to a simple one, so its landscape is broad and regular. At low noise it resembles the data and carries finer structure. Training therefore learns the sequence from coarse landscapes to the fine, data-level landscape. High noise levels give global guidance, and low noise levels give local refinement.

The chapter also explains why diffusion avoids the traditional difficulties. Classical energy-based maximum likelihood needs samples from the current model for its negative phase. Diffusion instead builds noisy training pairs through a known forward corruption process and learns denoising directions across the noise levels. The chapter says the result remains deeply connected to energy gradients, but training is far more stable. Energy-based models give the broader landscape view, and score-based diffusion is one especially effective modern way to learn the energy-gradient directions along a multi-scale path from noise to data.

The chapter summarises its argument. GANs train a generator through adversarial comparison rather than likelihood maximisation. The optimal discriminator links the original objective to Jensen–Shannon divergence, and practical training uses the non-saturating loss for stronger gradients. Wasserstein GANs replace overlap-based classification with a geometric distance, estimated through a real-valued Lipschitz critic whose score differences reflect how far mass must move. Energy-based models assign each configuration a scalar energy. Because they define an unnormalised density, learning and sampling run on energy differences and energy gradients rather than on normalised likelihoods. The identity linking energy gradients and scores shows that energy-based and score-based models share the same local geometry, and diffusion extends that geometry across a hierarchy of noisy distributions.

The chapter credits its main ideas to specific work. Goodfellow and colleagues introduced GANs as a minimax game and linked the optimal discriminator to Jensen–Shannon divergence. Arjovsky and colleagues introduced Wasserstein GANs, and Gulrajani and colleagues developed the gradient penalty. LeCun and colleagues offer a broad tutorial on learning as the shaping of energy landscapes. Classical energy-based examples include Hopfield networks and Boltzmann machines. Hyvärinen introduced score matching for estimating unnormalised models without evaluating the partition function, and Vincent connected it to denoising. Song and Ermon, and Song and colleagues, are recalled as linking energy-gradient ideas to multi-scale score learning and diffusion.

The book's closing reflection describes its own route as narrow on purpose. It calls the path "a deliberately selective path through the mathematical foundations of generative modelling," rather than a survey of every model in the field. It then retraces that route from the simplest starting point: linear representation, matrices as transformations, principal component analysis as a variance-preserving projection, and autoencoders as reconstruction-based compression. Probabilistic PCA turned that deterministic picture into a latent-variable generative model, with the prior, likelihood, marginal likelihood and posterior as its central objects.

The variational autoencoder carried this idea into the nonlinear setting, where, the book notes, exact marginal likelihoods and posteriors are no longer available. That gap led to the evidence lower bound, variational inference, approximate posteriors and the reparameterisation trick. Diffusion models then moved the story from a single latent variable to a whole trajectory of noisy latent states, which the book describes as turning generation into "the learned reverse of a gradual noising process."

The continuous-time and score-based chapters, the book says, offered a second view of the same problem. Instead of describing generation only as a discrete chain, they study motion, density evolution, Brownian noise, the Liouville equation, the Fokker–Planck equation, Langevin dynamics, score matching and reverse-time diffusion. The book concludes that "generative modelling can also be understood as learning directions of motion through probability space."

The later chapters examine two complementary routes. Normalising flows and autoregressive models keep exact likelihoods by design, either through invertible transformations with tractable Jacobians or through ordered conditional factorisation. Generative adversarial networks, Wasserstein GANs and energy-based models move beyond tractable likelihoods, relying instead on comparison, geometric discrepancy or scalar energy landscapes.

The book argues that these chapters present generative AI "not as a collection of unrelated architectures, but as a set of recurring mathematical ideas." It lists those ideas as representation, transformation, marginalisation, approximation, noising and denoising, density evolution, score fields, likelihood construction, distributional comparison and energy shaping, and it says these ideas keep reappearing in newer frameworks.

As examples of later work, the book points to flow matching and rectified flows, which it reads as developments of the continuous-time transport view. They "learn vector fields that move probability mass from a simple distribution toward the data distribution." It also points to consistency models, which it presents as another response to the sampling-efficiency challenge in diffusion, "aiming to retain the quality of iterative denoising while reducing the number of sampling steps."

The stated purpose of the book is to build "a compact mathematical base" from which such later developments can be approached more confidently, so that many modern extensions become easier to place within a common conceptual map.

The appendix offers a second route to the probabilistic PCA results. The main chapter derives the marginal distribution of the data with a direct linear-Gaussian argument, which the appendix calls "the quickest and most transparent for the main flow of the chapter." This appendix instead uses completing the square, which it describes as "the algebraic tool that performs this rewriting." Its advantage in this setting is that one derivation yields both the posterior distribution over the hidden variable z and, after that variable is integrated out, the marginal distribution of the data x.

The idea rests on a simple fact. A Gaussian density is an exponential of a quadratic form, so when the log of a density is a quadratic function of a variable, it can be rewritten in the standard Gaussian shape. The mean and covariance can then be read off directly. The appendix states a matrix version of the trick for a symmetric positive definite matrix A and a vector b. Minus one half times x transposed A x, plus b transposed x, equals minus one half times the quantity x minus A inverse b, transposed, times A, times that same quantity, plus one half of b transposed A inverse b. The vector A inverse b is the centre of the quadratic, and A is the precision matrix, whose inverse gives the covariance. The authors check the identity by expanding the right-hand side, and the extra constant term cancels out, leaving the original expression.

With that tool in hand, the appendix starts from the PPCA model. The hidden variable z has a standard normal distribution, and the data x, given z, is normal with mean W z plus mu and covariance alpha squared times the identity. Taking the log of the joint distribution gives a function that, with x held fixed, is quadratic in z. Expanding the squared norm makes this explicit. The appendix then defines two quantities. The first is M, equal to W transposed W plus alpha squared times the identity. The second is m, equal to M inverse times W transposed times the quantity x minus mu. Because a fixed matrix multiplying the quadratic in z can be grouped this way, the z-dependent part of the log-joint can be written as a Gaussian kernel centred at m.

The appendix explains why this identifies a Gaussian posterior. Any term that does not involve z is absorbed into the normalising constant, so only the z-dependent quadratic matters. Because M is positive definite, that quadratic is concave in z, which means it defines a proper Gaussian distribution. The posterior is therefore normal with mean m, that is, M inverse times W transposed times the quantity x minus mu, and covariance alpha squared times M inverse. In the appendix's framing, the mean m is the latent estimate for a given data point.

To get the marginal distribution of the data, the appendix integrates z out of the joint distribution. Since the z-dependent part is already a Gaussian centred at m, integrating over z removes it and contributes only a constant. What remains is a quadratic function of x minus mu, so the marginal must also be Gaussian. The appendix states that "carrying out the remaining algebra gives" a normal distribution for x with mean mu and covariance W W transposed plus alpha squared times the identity. It does not write out that remaining algebra line by line, so a reader who wants every step will need to fill it in.

The appendix is candid about its role. It says that "this derivation is not needed for the main conceptual flow of PPCA, but it is useful because the same quadratic-completion trick will reappear later whenever Gaussian latent-variable models are manipulated explicitly." Its value lies in showing the algebra behind results the main text uses, and in preparing the reader for the same move in later models.

Earlier in the paper, the authors state that the true reverse posterior, the probability distribution over a slightly less noisy version of the data given the current noisy version and the original clean data, is Gaussian, and they give its mean and variance in closed form. This derivation shows where those expressions come from. The authors say it rests on the same algebraic idea used earlier for probabilistic PCA:

once all relevant terms are Gaussian, we can collect the quadratic terms in the variable of interest and complete the square.

A Gaussian is the familiar bell-shaped distribution, and completing the square is a standard rewriting of a quadratic expression so that its centre and spread can be read off directly.

The derivation starts from Bayes' rule, which flips a conditional probability around: the reverse probability is the forward probability multiplied by the prior, divided by the evidence. The authors note that the denominator does not depend on the variable being solved for, so it "acts only as a normalising constant" and can be dropped when only the shape of the distribution matters. They then use the fact that the forward process is Markovian, meaning each noisy step depends only on the step immediately before it. That lets the probability of the current step given the previous step and the clean data be replaced by the probability given the previous step alone. What remains are two Gaussian factors: one describing the move from the previous noisy state to the current one, and one describing the previous noisy state given the clean data.

Multiplying two Gaussians amounts to adding their exponents. The authors expand the two squared distances, keep only the terms involving the unknown earlier state, and gather them into the standard quadratic form of a Gaussian. Completing the square then gives a Gaussian whose covariance is the inverse of a combined coefficient, which the paper calls A, and whose mean is A inverse multiplied by a linear term.

The paper then simplifies A. Using the relations that one minus the step's alpha equals its beta, and that the cumulative alpha is the product of the step alphas, the numerator collapses to one minus the cumulative alpha at the current step. The resulting variance is the beta at the current step, scaled by the ratio of one minus the previous cumulative alpha to one minus the current cumulative alpha. The mean works out to a weighted blend of two things: the clean data point, and the noisy point at the current step, with weights set by the alpha and beta terms. Putting it together, the paper concludes that the earlier step is Gaussian, with that mean and that variance spread equally across every direction, as the identity matrix denotes.

The paper says what this result is for. It is "the exact local reverse target that the learned denoising transition ... is asked to match inside the ELBO." In other words, the model being trained learns to predict the reverse step, and the evidence lower bound, the training objective the paper uses, measures how closely it does so against this exact target.

The main takeaway is that the book treats generative AI as a set of recurring mathematical ideas rather than unrelated architectures, and that generation can be understood as learning directions of motion through probability space. The author is open about the limits of that view. The text is selective by design, it is an early preprint open to further revision, and some results are left as pointers rather than full treatments. The score-based chapter, for example, notes that a full convergence theory needs additional assumptions beyond the scope of the tutorial, and the closing remarks point to later developments such as flow matching, rectified flows and consistency models as directions the book prepares the reader for rather than covers in depth. The book's own stated aim is to provide a compact mathematical base from which such later work can be approached more confidently.

  1. The Little Book of Generative AI Foundations: An Intuitive Mathematical Primer