Adaptive Stepsize

\(E_n'-E_n = \eta\cdot d_n\)

For gradient-descent-like methods: \(d=-1\cdot\nabla L\)
For (quasi)-Newton-like methods: \(d=-1\cdot (H_L)^{-1}\nabla L \quad\Rightarrow\quad d^\dagger H d = -1\cdot d^\dagger\nabla L\)

Based on Taylor-Series

1. Order

\(L'=L + 2\cdot \Re\left\{\sum_n (E_n'-E_n)^\dagger \nabla_nL\right\}\)

\(\eta = \frac{L'-L}{2\cdot \Re\{ d^\dagger\cdot \nabla L \}}\)

In Copra: \(L'=-L \quad\rightarrow\quad\) \(\eta = \frac{-L}{\Re\{ d^\dagger\cdot \nabla L \}}\)

2. Order

\(L'=L + 2\cdot \Re\left\{\sum_n (E_n'-E_n)^\dagger \nabla_nL\right\} + \Re\left\{\sum_{np} (E_n'-E_n)^\dagger H_{np}(E_p'-E_p)\right\}\)

\(\eta = 2\cdot\left(1 \mp\sqrt{1-\frac{L'-L}{2\Re\{ d^\dagger\cdot \nabla L \}}}\right)\)

(with signed sqrt to keep numerical stability, is the case for all sqrt here)

Based on Pade-Approximation

\(R_{nm}(x)=\frac{P(x)}{Q(x)} = \frac{\sum\limits_{i=0}^{n} a_i x^i}{1+\sum\limits_{i=1}^{m} b_i x^i}\)

If Taylor-series is: \(\quad T(x)=f_0 + f_1 x +f_2x^2 + \mathcal{O}(x^3)\)

\(R_{01}(x)=\frac{a_0}{1+b_1x} = \frac{f_0}{1-\frac{f_1}{f_0}x}\)

\(R_{11}(x)=\frac{a_0+a_1x}{1+b_1x} = \frac{f_0 + \left(f_1-\frac{f_2}{f_1}f_0\right)x}{1-\frac{f_2}{f_1}x}\)

\(R_{02}(x)=\frac{a_0}{1+b_1x+b_2x^2} = \frac{f_0}{1-\frac{f_1}{f_0}x + \left(\left(\frac{f_1}{f_0}\right)^2-\frac{f_2}{f_0}\right)x^2}\)

\(\eta_{01}=\frac{L}{L'}\cdot\frac{L'-L}{2\cdot\Re\{ d^\dagger\cdot \nabla L \}}\)

\(\eta_{11}=\frac{2(L'-L)}{4\cdot\Re\{ d^\dagger\cdot \nabla L \} - (L'-L)}\)

\(\eta_{02}=\frac{L}{4\cdot\Re\{ d^\dagger\cdot \nabla L \}+L}\cdot\left(1\pm\sqrt{1 - 4\cdot\left(1 + \frac{L}{4\cdot\Re\{ d^\dagger\cdot \nabla L \}}\right)\cdot\frac{L'-L}{L'}}\right)\)