I derive the expected value of a random variable that is left-truncated and lognormally distributed.

1

I work through a simple Python implementation of geometric Brownian motion and check it against the theoretical model.

2

In probability theory, Bienaymé's identity is a formula for the variance of random variables which are themselves sums of random variables. I provide a little intuition for the identity and then prove it.

3

I derive some basic properties of the lognormal distribution.

4

A useful view of a covariance matrix is that it is a natural generalization of variance to higher dimensions. I explore this idea.

5

I discuss moving or rolling averages, which are algorithms to compute means over different subsets of sequential data.

6

I discuss and prove the Gauss–Markov theorem, which states that under certain conditions, the least squares estimator is the minimum-variance linear unbiased estimator of the model parameters.

7

How do we know when a parameter estimate from a random sample is significant? I discuss the use of standard errors and confidence intervals to answer this question.

8

I provide a numerical demonstration that the mutual information of two random variables, the observations and latent variables in a Gaussian mixture model, is symmetric.

9

The mutual information (MI) of two random variables quantifies how much information (in bits or nats) is obtained about one random variable by observing the other. I discuss MI and show it is symmetric.

10

I derive the entropy for the univariate and multivariate Gaussian distributions.

11

Why are a distribution's moments called "moments"? How does the equation for a moment capture the shape of a distribution? Why do we typically only study four moments? I explore these and other questions in detail.

12

Under certain regularity conditions, maximum likelihood estimators are "asymptotically efficient", meaning that they achieve the Cramér–Rao lower bound in the limit. I discuss this result.

13

The Cramér–Rao lower bound allows us to derive uniformly minimum–variance unbiased estimators by finding unbiased estimators that achieve this bound. I derive the main result.

14

I document several properties of the Fisher information or the variance of the derivative of the log likelihood.

15

I walk the reader through a proof the Rao–Blackwell Theorem.

16

I discuss a straightforward proof of the law of total expectation with three standard assumptions.

17

I show how several properties of the distribution of a random variable—the expectation, conditional expectation, and median—can be viewed as solutions to optimization problems.

18

Probability distributions that are members of the exponential family have mathematically convenient properties for Bayesian inference. I provide the general form, work through several examples, and discuss several important properties.

19

Many probabilistic models assume random noise is Gaussian distributed. I explain at least part of the motivation for this, which is grounded in the Central Limit Theorem.

20

The KL divergence, also known as "relative entropy", is a commonly used metric for density estimation. I re-derive the relationships between probabilities, entropy, and relative entropy for quantifying similarity between distributions.

21

Bessel's correction is the division of the sample variance by $N - 1$ rather than $N$. I walk the reader through a quick proof that this correction results in an unbiased estimator of the population variance.

22