Try
Start the three-group binomial example at β=(0,0). All probabilities are 1/2, so the fitted success counts are (2,2,2), while the observations are (1,2,3). Which coefficient should move: the intercept, the slope, or both?
Let X have rows (1,−1), (1,0), (1,1). The score U is the vector of first derivatives of the total log likelihood. It measures local change in each coefficient direction. Here the observed-minus-expected vector is (−1,0,1), so
U = Xᵀ(−1,0,1) = (0,2).
The information I is the negative Hessian, the matrix of second derivatives with its sign reversed. All three variance weights are 4(1/2)(1/2)=1, giving I=XᵀX=diag(3,2). Solve Iδ=U before reading the next section.
Predict
The proposed Newton increment is δ=(0,1). It raises the slope and keeps the intercept zero. The exact optimum has slope log 3≈1.098612, so one step does not finish the problem. Predict which of the three fitted probabilities changes most and why the middle one stays 1/2.
| x | Initial p | Proposed p after β=(0,1) | Observed count | Proposed expected count |
|---|
| −1 | 1/2 | 1/(1+exp 1) | 1 | 4/(1+exp 1) |
| 0 | 1/2 | 1/2 | 2 | 2 |
| 1 | 1/2 | exp 1/(1+exp 1) | 3 | 4 exp 1/(1+exp 1) |
At the exact optimum, weights are (3/4,1,3/4). Recompute the information: I=diag(5/2,3/2). Its inverse is diag(2/5,2/3). This inverse is a local curvature matrix; a sampling interpretation needs additional assumptions and an asymptotic theorem.
Experiment
Return to the GLM workbench, initialize the worked binomial data and take one step. Compare the trace with U=(0,2), I=diag(3,2) and the accepted coefficient vector (0,1).
Now initialize the worked Poisson data. At zero coefficients its means are the exposures (2,1,2), so residual counts are (0,1,6), U=(7,6) and I=diag(5,4). The full Newton proposal is δ=(7/5,3/2), but that is not the accepted update in this example. The line search rejects the full length and accepts half, giving β=(0.7,0.75). Compare likelihood before and after the accepted step. A rejected proposal must not be reported as the current fit.
At the exact Poisson optimum β=(log 2,log 2), means are (2,2,8), and
I = ((12,6),(6,10)), with I⁻¹ = (1/84)((10,−6),(−6,12)).
Multiply the two matrices to check the inverse. Iterating numerical values is guided execution; the following calculation explains why the method has this form.
Understand
Vector derivatives and positive curvature
Write η=o+Xβ, with fixed offset vector o. Use y for observed counts: successes for binomial, events for Poisson. Let μ be expected counts and W the diagonal matrix of conditional count variances at the current β. The scalar derivatives from the first lesson give
U(β)=Xᵀ(y−μ), and I(β)=XᵀWX.
For any coefficient direction d,
dᵀI d = Σᵢ wᵢ(aᵢᵀd)².
At finite parameters the weights are positive when totals or exposures are positive. Thus I is positive definite exactly when X has full column rank. The log likelihood is then strictly concave: if a finite stationary point exists, it is the unique global maximizer. This does not prove that such a point exists; a strictly concave function can approach an unattained supremum.
For these canonical models, the observed negative Hessian does not depend separately on y, so its conditional expectation equals itself. Newton and Fisher scoring therefore coincide here. That equality need not hold for arbitrary links or models.
IRLS is a computational identity
Iteratively reweighted least squares, or IRLS, expresses the same update as a weighted linear solve. Define the working response
z = η + W⁻¹(y−μ).
At the current iterate, minimize Σᵢwᵢ(zᵢ−oᵢ−aᵢᵀb)² over the next coefficient vector b. Its normal equations are
XᵀWX b = XᵀW(z−o) = XᵀWX β + Xᵀ(y−μ).
Hence b=β+δ with Iδ=U. The offset must be subtracted before solving; it is not an unknown coefficient. The working response and weights change with the iterate, and the observations have not become Gaussian. Use a stable factorization to solve the weighted system rather than explicitly forming an inverse. The QR lesson explains this computational distinction.
Why a line search is needed
If U≠0 and I is positive definite, the derivative along δ=I⁻¹U is UᵀI⁻¹U>0. Differentiability therefore guarantees improvement for sufficiently small positive step lengths. It does not guarantee improvement for length 1. A backtracking line search tries shorter lengths, accepts only its specified likelihood improvement, and reports failure if no supported trial is accepted.
An accepted ascent step, a small score, a finite-existence certificate and a valid uncertainty approximation answer different questions. They must remain separate in both your proof and the trace. Near separation, large predictors or poorly scaled columns can make arithmetic unreliable even before a formal mathematical boundary is reached.
Apply
Change the binomial totals to 6 and the successes to (1,3,5), retaining x=−1,0,1. Compute the first full Newton proposal from β=0, then verify the exact stationary point (0,log 5). State why the proposal differs from the answer.
Check the changed vector calculation
At zero the residual counts are (−2,0,2), giving U=(0,4). The weights are all 3/2, so I=diag(9/2,3) and δ=(0,4/3). At slope log 5 the probabilities are (1/6,1/2,5/6), making every residual count zero. Newton used curvature at the starting point, which is not constant along the path. Positive weights and rank give uniqueness of this stationary point. The next lesson supplies the missing general existence argument.
Continue
Proceed to bounded fitting and finite existence. Keep your proposed step, accepted length and numerical stopping status distinct when comparing the trace with the proof.
