Gaussian Distribution

1. One-dimensional Gaussian Distribution

p (x) = N (μ, σ^{2}) = \frac{1}{σ 2 π} exp (- \frac{( x - μ ) ^{2}}{2 σ ^{2}}), x \in R, μ \in R, σ > 0

A random vector is said to be k-variate normally distributed if every linear combination of its k components has a univariate normal distribution.

p (x) = N (x; μ, Σ) = \frac{1}{( 2 π ) ^{d /2} ∣ Σ ∣ ^{1/2}} exp (- \frac{1}{2} (x - μ)^{T} Σ^{- 1} (x - μ)) x \in R^{d}, μ \in R^{d}, Σ \in R^{d \times d}

特殊情况 $Σ = σ^{2} I$ 时，

p (x) = N (x; μ, σ^{2} I) = \frac{1}{( 2 π ) ^{d /2} σ ^{d}} exp (- \frac{1}{2 σ ^{2}} ∥ x - μ ∥^{2})

p_{1} (x) = N (x; μ_{1}, Σ_{1}), p_{2} (x) = N (x; μ_{2}, Σ_{2}), x \in R^{d}

D_{K L} (p_{1} ∣∣ p_{2}) = \frac{1}{2} (lo g \frac{∣ Σ _{2} ∣}{∣ Σ _{1} ∣} + Tr (Σ_{2}^{- 1} Σ_{1}) + (μ_{2} - μ_{1})^{T} Σ_{2}^{- 1} (μ_{2} - μ_{1}) - d)

Assume $Σ \in R^{d \times d}$ is an symmetric positive definite matrix, then

N (x; μ, Σ) = \frac{1}{( 2 π ) ^{d} ∣ Σ ∣} exp (- \frac{1}{2} (x - μ)^{T} Σ^{- 1} (x - μ))

If $Σ = σ^{2} I$ , then

\nabla_{x} ln N (x; μ, σ^{2}) = \nabla_{x} - \frac{1}{2 σ ^{2}} (x - μ)^{T} (x - μ) = - \frac{1}{σ ^{2}} (x - μ)