Use curvature without confusing local and global guarantees

Try

For f(x,y)=(x−2)²+5(y+1)², the gradient is (2(x−2),10(y+1)) and the Hessian is diag(2,10). Solving Hd=−g gives d=(2−x,−1−y). A full Newton step reaches the minimizer from any starting point. The quadratic model is exact here. For a nonquadratic function, the same step minimizes only a local quadratic approximation and can behave very differently far from the solution.

Predict

Predict whether Newton always seeks a minimum. On f(x)=−x² it moves every nonzero point to zero, the maximum. Newton solves a stationarity equation; a positive-definite Hessian is what turns its direction into a descent direction. Also predict whether forming an explicit matrix inverse is necessary. It is not: solve the linear system. This distinction matters numerically, especially when the Hessian is poorly conditioned or sparse.

Experiment

Compare Newton and gradient steps from (0,0) in the opening quadratic. Use gradient step α=1/10.

Method	Direction or displacement	Next point
Gradient step	(2/5,−1)	(2/5,−1)
Full Newton	(2,−1)	(2,−1)

The gradient step respects the largest curvature but progresses slowly in the flatter direction. Newton rescales both directions using curvature. This example explains its appeal without claiming that a Hessian solve is computationally free. Count derivative evaluation, matrix factorization and line-search work when comparing methods.

Understand

Newton’s direction satisfies H(x)d=−∇f(x). If H is positive definite and the gradient nonzero, ∇fᵀd=−∇fᵀH⁻¹∇f<0, proving descent. Near a minimizer with positive-definite Hessian, continuity keeps nearby Hessians invertible. If the Hessian is locally Lipschitz, Taylor’s theorem for the gradient gives a remainder of order ||e||², where e is the error from the solution. Multiplying by a uniformly bounded inverse Hessian yields ||e_next||≤C||e||² for sufficiently close starting points. This is the local quadratic-convergence mechanism; it does not give a global guarantee from arbitrary initialization.

Farther away, an indefinite or singular Hessian can produce ascent, an undefined solve or an excessively large step. Common safeguards modify H by adding a suitable positive multiple of the identity, use a positive-definite approximation, or fall back to the negative gradient. A line search then controls distance along a descent direction. State these modifications explicitly: their convergence analysis concerns the safeguarded algorithm, not an unmodified full Newton iteration.

Apply

Quasi-Newton methods learn curvature from changes between iterates. Let s=x_next−x and y=∇f(x_next)−∇f(x). The secant equation Bs=y approximates the gradient change using a Hessian-like matrix. Positive definiteness requires sᵀy>0 for nonzero s, since sᵀBs is positive. The BFGS update B_new=B−BssᵀB/(sᵀBs)+yyᵀ/(yᵀs) preserves positive definiteness when B is positive definite and sᵀy>0. Its quadratic form consists of a nonnegative projected part plus a positive rank-one correction; equality in both terms would force the tested nonzero vector to vanish.

Do not claim that the secant equation alone determines B: it gives only one vector relation. Nor does BFGS automatically enjoy quadratic Newton convergence. Its stronger local rates require additional assumptions and are outside the elementary Newton proof presented here. The update’s curvature condition must be checked or maintained by a suitable line-search or damping strategy.

Continue

Distinguish solving for a direction, proving descent and accepting a step. Continue to Equality constraints, where directions must respect a feasible tangent space and stationarity involves multipliers. Curvature remains useful, but the relevant second-order object will include constraint curvature rather than only the objective Hessian.