Kronecker-factored Approximate

1. 约定

默认所有向量都是列向量

求 $F^{- 1} v$

Fisher 信息阵 $F = E [\frac{\partial L}{\partial θ} (\frac{\partial L}{\partial θ})^{T}]$

把神经网络的每个权重矩阵逐列向量化，然后按一个确定的顺序排列这些向量，得到一个长向量 $θ$
对于 $y = W x + b$ , 令 $g = \frac{\partial L}{\partial y}$

\frac{\partial L}{\partial W} = \frac{\partial L}{\partial y} \cdot x^{T}

\frac{\partial L}{\partial b} = \frac{\partial L}{\partial y} \cdot I

把 $x$ 和 $I$ 在后文中用 $a$ 表示

F = E [\frac{\partial L}{\partial θ} \cdot \frac{\partial L}{\partial θ}^{T}] = E [\frac{\partial L}{\partial θ _{1}} \cdot \frac{\partial L}{\partial θ _{1}}^{T}] E [\frac{\partial L}{\partial θ _{1}} \cdot \frac{\partial L}{\partial θ _{2}}^{T}] \dots E [\frac{\partial L}{\partial θ _{2}} \cdot \frac{\partial L}{\partial θ _{1}}^{T}] E [\frac{\partial L}{\partial θ _{2}} \cdot \frac{\partial L}{\partial θ _{2}}^{T}] \dots ⋮ E [\frac{\partial L}{\partial θ _{n}} \cdot \frac{\partial L}{\partial θ _{1}}^{T}] E [\frac{\partial L}{\partial θ _{n}} \cdot \frac{\partial L}{\partial θ _{2}}^{T}] \dots E [\frac{\partial L}{\partial θ _{1}} \cdot \frac{\partial L}{\partial θ _{n}}^{T}] E [\frac{\partial L}{\partial θ _{2}} \cdot \frac{\partial L}{\partial θ _{n}}^{T}] ⋮ E [\frac{\partial L}{\partial θ _{n}} \cdot \frac{\partial L}{\partial θ _{n}}^{T}]

$θ_{i}$ 表示按前述顺序排列的第 i 个权重矩阵的向量化表示。
取 $F$ 的对角线上的矩阵作为 $F$ 的近似。

E [\frac{\partial L}{\partial θ _{i}} \cdot \frac{\partial L}{\partial θ _{i}}^{T}] = E [v ec (g_{i} \cdot a_{i}^{T}) v ec (g_{i} \cdot a_{i}^{T})^{T}] = E [(a_{i} \otimes g_{i}) (a_{i}^{T} \otimes g_{i}^{T})] = E [(a_{i} a_{i}^{T}) \otimes (g_{i} g_{i}^{T})] \approx E [a_{i} a_{i}^{T}] \otimes E [g_{i} g_{i}^{T}] =: A_{i} \otimes G_{i} (未证明)

F \approx d ia g (A_{1} \otimes G_{1}, A_{2} \otimes G_{2}, \dots, A_{n} \otimes G_{n}) = A_{1} \otimes G_{1}, 0, 0, ⋮, 0, 0, A_{2} \otimes G_{2}, 0, ⋮, 0, 0, 0, A_{3} \otimes G_{3}, ⋮, 0, \dots, \dots, \dots, ⋱, \dots, 000 ⋮ A_{n} \otimes G_{n} (未证明)

把 $A_{i} \otimes G_{i}$ 记作 $F_{i}$
$A_{i}$ 和 $G_{i}$ 都是对称矩阵，对称矩阵的逆矩阵为对称矩阵。

F_{i}^{- 1} \cdot v_{i} \approx (A_{i}^{- 1} \otimes G_{i}^{- 1}) v ec (V_{i}) = v ec (G_{i}^{- 1} V_{i} (A_{i}^{- 1})^{T}) = v ec (G_{i}^{- 1} V_{i} A_{i}^{- 1}) = v ec (E [g_{i} \cdot g_{i}^{T}]^{- 1} V_{i} E [a_{i} \cdot a_{i}^{T}]^{- 1})

A \otimes B = a_{11} B a_{21} B ⋮ a_{m 1} B a_{12} B a_{22} B ⋮ a_{m 2} B \dots \dots ⋱ \dots a_{1 n} B a_{2 n} B ⋮ a_{mn} B

v ec (u v^{T}) = v \otimes u

(A \otimes B)^{T} = (A^{T} \otimes B^{T})

(A \otimes B) (A^{T} \otimes B^{T}) = A A^{T} \otimes B B^{T}

(A \otimes B)^{- 1} = A^{- 1} \otimes B^{- 1}

(A \otimes B) v ec (X) = v ec (BX A^{T})

(A + b c^{T})^{- 1} = A^{- 1} + \frac{A ^{- 1} b c ^{T} A ^{- 1}}{1 - c ^{T} A ^{- 1} b}

(A + b c^{T})^{- 1} d = A^{- 1} d + \frac{A ^{- 1} b c ^{T} A ^{- 1} d}{1 - c ^{T} A ^{- 1} b}

(F - \frac{\partial L}{\partial θ} \frac{\partial L}{\partial θ}^{T})^{- 1} d = F^{- 1} d + \frac{F ^{- 1} \frac{\partial L}{\partial θ} \frac{\partial L}{\partial θ} ^{T} F ^{- 1} d}{1 - \frac{\partial L}{\partial θ} ^{T} F ^{- 1} \frac{\partial L}{\partial θ}} = F^{- 1} d + (\frac{\frac{\partial L}{\partial θ} ^{T} F ^{- 1} d}{1 - \frac{\partial L}{\partial θ} ^{T} F ^{- 1} \frac{\partial L}{\partial θ}}) F^{- 1} \frac{\partial L}{\partial θ} = F^{- 1} d + (\frac{\frac{\partial L}{\partial θ} ^{T} ( F ^{- 1} d )}{1 - \frac{\partial L}{\partial θ} ^{T} ( F ^{- 1} \frac{\partial L}{\partial θ} )}) (F^{- 1} \frac{\partial L}{\partial θ})