Probability

There is a beautiful connection between probability theory and partial differential equations (PDEs), given by the famous Feynman–Kac theorem. This result allows us to translate between finite-dimensional stochastic problems and infinite-dimensional deterministic problems. It appears throughout finance, physics, control theory, and machine learning (generative modeling and reinforcement learning). What makes this connection especially useful is that it works in both directions. The solution to a high-dimensional PDE (which can be very expensive to compute numerically) can be evaluated at a single point simply by simulating a random process. Conversely, a difficult question about a stochastic process can be transformed into a deterministic PDE and tackled using standard techniques. ...

How do we know if a machine learning model will perform well on unseen data? What happens if you continue to add samples to the dataset? Is it better to have a more complex model or a simpler one? These questions have been around for many years and are central to the field of statistical learning theory. Generalization theory provides mathematical guarantees and bounds on the generalization capability of families of functions. I have prepared a few notes on the basics of generalization theory. The only prerequisite is probability theory, and it is intended to be self-contained. It includes the most important results, such as Dudley’s Theorem and McDiarmid’s Theorem. ...