Gradient Descent, Four Ways It Goes Wrong梯度下降：四种出错的方式

Gradient descent takes a step downhill, measures the slope again, and repeats until the steps get tiny. I wrote it from scratch, about twenty lines with no libraries, and plotted every step on four one-variable functions so each failure is easy to see.

Where you start. On the parabola x² + x + 1 with a step size of 0.1, starting at 3 took 31 steps to reach the bottom at −0.5, and starting at −1 took 22. On a parabola every step shrinks the remaining distance by the same fraction, so the step count grows with how far away you start.

A function with no bottom. On x⁵ − 2x⁴ + 2, starting at −0.4 with a small step size of 0.01, the cost keeps falling and the steps keep growing, from −1.2 to −2.8 in the last three steps alone, and only the 30-step limit stops them from overflowing. Nothing is wrong with the step size. The function goes down forever in that direction, and the slope grows as it goes, so there is no minimum to find.

The step size. On x⁴ − 4x² + 5, starting at 3, a step size of 0.001 was still at 1.54 after 100 steps and never reached the minimum at √2 ≈ 1.41. A step size of 0.01 got there in 65. At 0.2, starting from 2, the steps overshoot so badly that the path jumps between the two valleys and never settles.

The wrong valley. On x⁴ − 4x² + 0.5x + 5, which has a shallow valley and a deeper one, the same step size sends a start at −0.5 into the deep valley and a start at 0.5 into the shallow one, as on the cover. Gradient descent only ever sees the slope where it stands, which is why practical training uses random restarts, momentum and adaptive methods like Adam.

梯度下降的做法是：朝下坡方向走一步，重新测一次斜率，然后不断重复，直到每一步都变得非常小。我从零写了这个算法，大约二十行代码，没有用任何库，并在四个单变量函数上把每一步都画了出来，这样每种失败都一目了然。

起点在哪里。 在抛物线 x² + x + 1 上，步长取 0.1，从 3 出发要走 31 步才能到达 −0.5 处的最低点，从 −1 出发则要 22 步。在抛物线上，每一步都会把剩下的距离缩小同样的比例，所以起点离得越远，需要的步数就越多。

没有最低点的函数。 在 x⁵ − 2x⁴ + 2 上，从 −0.4 出发，步长只取很小的 0.01，代价（函数值）却一直在下降，步子也越迈越大，光是最后三步就从 −1.2 走到了 −2.8，只是因为有 30 步的上限，才没有让数值溢出。步长本身没有问题。这个函数在那个方向上会一直往下走，而且越往下斜率越大，所以根本没有最小值可找。

步长。 在 x⁴ − 4x² + 5 上，从 3 出发，步长为 0.001 时走了 100 步还停在 1.54，始终没到达 √2 ≈ 1.41 处的最小值。步长为 0.01 时，65 步就到了。步长为 0.2、从 2 出发时，每一步都冲过头太多，路径在两个山谷之间来回跳，永远停不下来。

走错山谷。 x⁴ − 4x² + 0.5x + 5 有一个浅山谷和一个更深的山谷。用同样的步长，从 −0.5 出发会落进深山谷，从 0.5 出发则会落进浅山谷，就像封面图里那样。梯度下降只能看到自己脚下的斜率，所以实际训练中会用随机重启、动量，以及 Adam 这样的自适应方法。