高斯分布

One-dimensional Gaussian Distribution

$$p(x)=\mathcal{N}(\mu ,{\sigma }^2)=\frac{1}{\sigma \sqrt{2\pi }}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right),x\in \mathbb{R},\mu \in \mathbb{R},\sigma >0$$

Multi-dimensional Gaussian Distribution

A random vector is said to be k-variate normally distributed if every linear combination of its k components has a univariate normal distribution.

多维高斯分布的概率密度函数(在能写出概率密度的情况下)

$$\begin{array}{c}p(\mathbf{x})=\mathcal{N}(\mathbf{x};\mu ,\mathbf{\Sigma })=\frac{1}{(2\pi {)}^{d/2}|\mathbf{\Sigma }{|}^{1/2}}\exp \left(-\frac{1}{2}(\mathbf{x}-\mu {)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mu )\right)\\ \mathbf{x}\in {\mathbb{R}}^d,\mu \in {\mathbb{R}}^d,\mathbf{\Sigma }\in {\mathbb{R}}^{d\times d}\end{array}$$

特殊情况 $\Sigma ={\sigma }^{2}I$时，

$$p(\mathbf{x})=\mathcal{N}(\mathbf{x};\mu ,{\sigma }^{2}I)=\frac{1}{(2\pi {)}^{d/2}{\sigma }^d}\exp \left(-\frac{1}{2{\sigma }^2}\Vert \mathbf{x}-\mu {\Vert }^2\right)$$

KL Divergence between two Gaussian Distributions

$$p_1(\mathbf{x})=\mathcal{N}(\mathbf{x};{\mu }_1,{\mathbf{\Sigma }}_1),p_2(\mathbf{x})=\mathcal{N}(\mathbf{x};{\mu }_2,{\mathbf{\Sigma }}_2),\mathbf{x}\in {\mathbb{R}}^d$$ $$D_{KL}(p_1||p_2)=\frac{1}{2}\left(\log \frac{|{\Sigma }_2|}{|{\Sigma }_1|}+\text{Tr}({\Sigma }_2^{-1}{\Sigma }_1)+({\mu }_2-{\mu }_1{)}^T{\Sigma }_2^{-1}({\mu }_2-{\mu }_1)-d\right)$$

Derivative of Gaussian Distribution

Assume $\mathbf{\Sigma }\in {\mathbb{R}}^{d\times d}$ is an symmetric positive definite matrix, then

$$\mathcal{N}(\mathbf{x};\mu ,\mathbf{\Sigma })=\frac{1}{\sqrt{(2\pi {)}^d|\mathbf{\Sigma }|}}\exp \left(-\frac{1}{2}(\mathbf{x}-\mu {)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mu )\right)$$

If $\mathbf{\Sigma }={\sigma }^2\mathbf{I}$, then

$$\begin{array}{rl}{\nabla }_x\ln \mathcal{N}(x;\mu ,{\sigma }^2)& ={\nabla }_x-\frac{1}{2{\sigma }^2}(\mathbf{x}-\mu {)}^T(\mathbf{x}-\mu )\\ & =-\frac{1}{{\sigma }^2}(\mathbf{x}-\mu )\end{array}$$