Quantify progress under smoothness and curvature

Try

For f(x)=x², gradient descent uses x_next=x−α·2x=(1−2α)x. Starting at x=4 with α=1/4 halves the iterate each time: 4,2,1,1/2. The objective falls by a factor of four per step. With α=1, the same method alternates between 4 and−4 without progress. With a larger step its magnitude grows. A convex objective does not protect an arbitrarily chosen numerical step from failure.

Predict

Predict which scale controls a safe step for a general smooth objective. A large curvature can make the gradient direction change quickly, so the tangent approximation remains accurate only over shorter distances. A Lipschitz constant L for the gradient bounds this change. The usual step 1/L is therefore a statement about curvature and coordinates, not a universal learning rate. Rescaling variables changes L and can materially change algorithm behavior.

Experiment

Compare exact recurrence factors for the same quadratic. Inspect both iterate and objective behavior rather than only whether one step happened to reduce the value.

Step α	Iterate factor 1−2α	Behavior from nonzero x
1/4	1/2	Converges without sign changes
3/4	−1/2	Converges with alternating signs
1	−1	No decay
5/4	−3/2	Diverges in magnitude

The complete convergence interval for this quadratic is 0<α<1. The conservative 1/L rule uses L=2 and gives α=1/2, solving this particular quadratic in one step. A sufficient general bound need not describe every step that works for a specific function.

Understand

If ∇f is L-Lipschitz along the segment from x to x+d, integrate its change to obtain f(x+d)≤f(x)+∇f(x)·d+(L/2)||d||². Indeed the error beyond the tangent equals the integral from zero to one of [∇f(x+td)−∇f(x)]·d, bounded above by the integral of Lt||d||². Substituting d=−∇f(x)/L gives decrease at least ||∇f(x)||²/(2L).

If f is bounded below and this common smoothness bound holds throughout the iterates and steps, summing decreases bounds the sum of squared gradient norms, so those norms tend to zero. This alone does not prove that iterates converge, or that a limiting stationary point is globally optimal. Convexity upgrades stationary points to global minimizers. With μ-strong convexity and an existing minimizer, minimizing the strong tangent lower bound gives ||∇f||²≥2μ(f−f⋆). Combining this with descent yields a function-gap contraction by at most 1−μ/L for the 1/L step.

Apply

For f(x,y)=(x²+100y²)/2, the Hessian eigenvalues are 1 and 100. The safe step 1/100 makes the y coordinate vanish immediately but multiplies x by 99/100. The ratio L/μ=100 predicts slow worst-direction progress. This is conditioning, not a failure of the gradient formula. A coordinate transformation or a curvature-aware method can change the geometry seen by the algorithm, but must preserve the interpretation of the original decision variables.

As a stopping exercise, suppose the gradient norm is small. Under strong convexity, it bounds objective suboptimality using the gradient-gap inequality. Without such a condition it may indicate a flat nonoptimal region or a saddle. State which theorem converts your numerical residual into a useful optimization guarantee.

Continue

Keep the objective lower bound, smoothness region, step rule and claimed convergence type explicit. A finite decreasing trace is not a convergence proof. Continue to Line search and convergence for a procedure that selects a step using observed decrease while retaining a theoretical termination condition. It reduces dependence on knowing a usable global L, but does not remove the need for smoothness assumptions in global analysis.