posterior mean

Tweedie’s formula

---

Let $p(\mathbf{y}|\eta )$ belong to the exponential family distribution

$p(\mathbf{y}|\eta )=p_0(\mathbf{y})\exp ({\eta }^{\top }T(\mathbf{y})-\phi (\eta ))$

where $\eta$ is the canonical vector of the family, $T(\mathbf{y})$ is some function of $\mathbf{y}$, and $\phi (\eta )$ is the cumulant generation function which normalizes the density, and $p_0(\mathbf{y})$ is the density up to the scale factor when $\eta =0$. Then, the posterior mean $\hat{\eta }:=\mathbb{E}[\eta |\mathbf{y}]$ should satisfy

$({\nabla }_{\mathbf{y}}T(\mathbf{y}){)}^{\top }\hat{\eta }={\nabla }_{\mathbf{y}}\log p(\mathbf{y})-{\nabla }_{\mathbf{y}}\log p_0(\mathbf{y})$

Posterior mean of VP-SDE (continuous DDPM)

---

For the case of VP-SDE or DDPM sampling, $p({\mathbf{x}}_0|{\mathbf{x}}_t)$ has the unique posterior mean at

$${\hat{\mathbf{x}}}_0:=\mathbb{E}[{\mathbf{x}}_0|{\mathbf{x}}_t]=\frac{1}{\sqrt{\bar{\alpha }(t)}}({\mathbf{x}}_t+(1-\bar{\alpha }(t)){\nabla }_{{\mathbf{x}}_t}\log p_t({\mathbf{x}}_t))$$

Proof. For the case of VP-SDE or DDPM forward sampling, we have

$$p({\mathbf{x}}_t|{\mathbf{x}}_0)=\mathcal{N}({\mathbf{x}}_t;\sqrt{\bar{\alpha }(t)}{\mathbf{x}}_0,(1-\bar{\alpha }(t))I)$$ $$p({\mathbf{x}}_t|{\mathbf{x}}_0)=\frac{1}{(2\pi (1-\bar{\alpha }(t)){)}^{d/2}}\exp \left(-\frac{\Vert {\mathbf{x}}_t-\sqrt{\bar{\alpha }(t)}{\mathbf{x}}_0{\Vert }^2}{2(1-\bar{\alpha }(t))}\right),$$

which is a Gaussian distribution. The corresponding canonical decomposition is then given by

$$p({\mathbf{x}}_t|{\mathbf{x}}_0)=p_0({\mathbf{x}}_t)\exp \left({\mathbf{x}}_0^{\top }T({\mathbf{x}}_t)-\phi ({\mathbf{x}}_0)\right),$$

where

$$\begin{array}{rl}{p}_0({\mathbf{x}}_t)& :=\frac{1}{(2\pi (1-\bar{\alpha }(t)){)}^{d/2}}\exp \left(-\frac{\Vert {\mathbf{x}}_t{\Vert }^2}{2(1-\bar{\alpha }(t))}\right)\\ T({\mathbf{x}}_t)& :=\frac{\sqrt{\bar{\alpha }(t)}}{1-\bar{\alpha }(t)}{\mathbf{x}}_t\\ \phi ({\mathbf{x}}_0)& :=\frac{\bar{\alpha }(t)\Vert {\mathbf{x}}_0{\Vert }^2}{2(1-\bar{\alpha }(t))}\end{array}$$

Therefore, we have

$$\frac{\sqrt{\bar{\alpha }(t)}}{1-\bar{\alpha }(t)}{\hat{\mathbf{x}}}_0={\nabla }_{{\mathbf{x}}_t}\log p_t({\mathbf{x}}_t)+\frac{1}{1-\bar{\alpha }(t)}{\mathbf{x}}_t$$

which leads to

$${\hat{\mathbf{x}}}_0=\frac{1}{\sqrt{\bar{\alpha }(t)}}({\mathbf{x}}_t+(1-\bar{\alpha }(t)){\nabla }_{{\mathbf{x}}_t}\log p_t({\mathbf{x}}_t))$$

This concludes the proof.

Posterior mean of General SDE

A general SDE of diffusion is given by

$$d\mathbf{x}=\mathbf{f}(\mathbf{x},t)dt+G(t)d{\mathbf{w}}_t$$

where $\mathbf{f}:{\mathbb{R}}^d\times [0,T]\to {\mathbb{R}}^d$ and $G:[0,T]\to {\mathbb{R}}^{d\times d}$ are given functions.

$TODO$