Curvature Without the Hessian: Power Iteration, Trace Estimation, and Quantization

The Hessian describes how a loss gradient changes when parameters move. Forming every Hessian entry can be prohibitively expensive, but many useful questions require only the Hessian’s action on selected vectors.

Three operations organize this note: double backward computes a Hessian-vector product; power iteration extracts a dominant eigenpair from repeated products; and Hutchinson’s estimator recovers the trace in expectation. Their outputs answer different questions, especially when the Hessian has negative eigenvalues.

What curvature means before computing it

Let a scalar loss depend on a parameter vector:

Assume the loss is twice continuously differentiable near the point being examined. Define a one-dimensional path through a perturbation:

The chain rule gives

and

Integrating the derivative twice yields

If the Hessian changes little along the displacement, replace it inside the integral by its value at the starting point. Since the integral of the weight is one-half,

The linear term measures slope. The quadratic term measures curvature. A trained checkpoint need not have a negligible full gradient, and a small nonzero linear term can dominate sufficiently small perturbations because it scales with displacement rather than squared displacement.

For a constructed parameter count of one million, a dense Hessian has one trillion entries. At four bytes per entry, that is four trillion bytes before accounting for other memory. The calculation illustrates quadratic storage growth; it is not a measurement of a particular model.

Inability to store the matrix does not make every second-order method impossible. Iterative linear solvers can use Hessian-vector products to approximate damped Newton steps. Matrix storage and linear-system solution are related but distinct obstacles.

Directional curvature and the meaning of “top”

For a unit vector, directional curvature is the Rayleigh quotient:

Because the Hessian is symmetric under the stated differentiability assumption, expand the vector in an orthonormal eigenbasis:

Then

The quotient is a weighted average of eigenvalues. It lies between the smallest and largest algebraic eigenvalues, with the extremes attained at the corresponding eigenvectors.

For a positive semidefinite Hessian, every directional quadratic contribution is nonnegative. At an indefinite point, some directions decrease the loss to second order while others increase it.

“Top eigenvalue” is therefore ambiguous unless the ordering is specified:

• The largest algebraic eigenvalue identifies the most positive curvature.
• The smallest algebraic eigenvalue identifies the most negative curvature.
• The eigenvalue of largest magnitude identifies the strongest absolute amplification by the matrix.

Ordinary power iteration targets the last of these. It targets the most positive curvature automatically only when the spectrum and dominant magnitude make those quantities coincide.

Deriving a Hessian-vector product with double backward

Let a probe vector be fixed with respect to the parameters. Form the scalar

Differentiate its coordinate expression:

This is the vector-Jacobian product of the gradient, or

Symmetry gives

The computation consists of a gradient followed by a gradient of a scalar constructed from that gradient. It never needs to assemble the entries of the Hessian.

Holding the probe fixed is essential. If the probe itself depends on the parameters, the product rule gives an extra term:

That is not the desired fixed-vector product. Detaching the probe from automatic differentiation prevents the unwanted term.

The first differentiation must retain differentiable operations in its result. A numerical gradient vector with no derivative graph is insufficient for a second differentiation. Retaining an ordinary forward graph and constructing a graph of the derivative are different requirements.

Computational memory still includes the model’s derivative graph and saved activations. “No dense Hessian” does not mean “no substantial memory cost.” What is avoided is storing a parameter-by-parameter matrix.

A double-backward example with an explicit answer

Use the quadratic

Its gradient and Hessian are

Choose

Direct multiplication gives

The double-backward route reaches the same result without matrix construction:

Differentiating this scalar gives the expected product.

A minimal PyTorch implementation is:

import torch

w = torch.tensor(
    [1.0, -1.0],
    dtype=torch.float64,
    requires_grad=True,
)
v = torch.tensor([2.0, -1.0], dtype=torch.float64)

loss = 1.5 * w[0] ** 2 + w[0] * w[1] + 1.5 * w[1] ** 2

(gradient,) = torch.autograd.grad(loss, w, create_graph=True)
directional_derivative = (gradient * v.detach()).sum()
(hv,) = torch.autograd.grad(directional_derivative, w)

# Algebraically expected: tensor([5., -1.], dtype=torch.float64)

For a model with multiple parameter tensors, the scalar directional derivative sums the elementwise products over all corresponding gradient and probe tensors. A completely unused parameter or a locally constant first derivative needs explicit zero handling; not every such case leaves a differentiable tensor for a second call.

Custom operations must support higher derivatives. At nondifferentiable activation boundaries, the classical Hessian may not exist, and the returned automatic-differentiation result must be interpreted according to the implemented derivative rules.

Checking that the product represents one fixed objective

A finite-difference comparison provides an independent check:

To see the connection, expand each gradient around the central point. Their first-order changes have opposite signs, so subtraction doubles the Hessian-vector term. For the constructed quadratic, the equality holds for every nonzero step in exact arithmetic because the gradient is linear.

In floating-point computation, an extremely small step can amplify cancellation error. Testing several moderate step sizes is more informative than trusting one arbitrarily tiny value.

Symmetry supplies another check:

If this fails substantially on a smooth deterministic example, inspect the graph, parameter ordering, probe handling, and objective evaluation.

Repeated products used by an eigensolver must refer to the same matrix. Changing the minibatch, dropout mask, parameter values, or normalization state between calls changes the operator. Standard power-iteration reasoning no longer applies directly.

Freezing randomness or using evaluation mode must be an explicit objective choice. Evaluation-mode curvature and training-mode curvature need not describe the same function. A reproducible report records the loss reduction, data subset, model mode, regularizer, and parameter subset.

Why power iteration works

Start with a unit vector and repeatedly compute

If a product is zero, normalization is impossible. The current vector lies in the nullspace, so a restart may be needed.

Ignoring the intermediate normalizations, the direction after repeated products is

Expand the starting vector in eigenvectors, ordering eigenvalues by magnitude:

Then

Factor out the largest-magnitude eigenvalue:

If its magnitude is strictly greater than the others and the initial coefficient is nonzero, the remaining relative contributions vanish geometrically.

The convergence speed is controlled by

A ratio close to one means slow separation. Equal dominant magnitudes can prevent convergence to a unique direction. A negative dominant eigenvalue can cause alternating signs, which do not prevent the Rayleigh quotient from estimating that negative eigenvalue.

After each iteration, estimate the eigenvalue using

This requires only the same matrix-vector operation.

Power iteration carried through numerically

For the earlier constructed Hessian,

the eigenpairs are

and

Start from the first coordinate axis. The first product and normalization give

The Rayleigh quotient is

The next normalized direction is

Its quotient is

The relative unwanted eigenvector coefficient halves with every multiplication because the eigenvalue ratio is one-half.

A useful stopping diagnostic is the residual:

For the second iterate in this example,

A stable-looking quotient can therefore still accompany a noticeable residual.

Expanding the residual in the eigenbasis gives

At least one eigenvalue must lie within the residual norm of the quotient. But a small residual does not establish that the eigenvalue is the largest: an exact non-dominant eigenvector also has zero residual.

The indefinite-Hessian correction

Consider

A generic power-iteration start converges in direction toward the first axis because negative five has the largest magnitude. The Rayleigh quotient approaches negative five, even though the largest algebraic eigenvalue is two.

Reporting that result as the “largest positive curvature” would be incorrect.

If a valid spectral lower bound is available, shifting the matrix can make its spectrum nonnegative:

Products remain cheap:

The eigenvectors are unchanged and every eigenvalue shifts by the same amount. With a shift of six in this example, the eigenvalues become one and eight. Power iteration then identifies eight, from which subtracting the shift recovers two.

The shift must be justified. Choosing an arbitrary shift does not guarantee the intended ordering for an unknown indefinite spectrum. Symmetric Krylov methods can estimate multiple spectral extremes using products, but their outputs likewise require convergence diagnostics.

Hutchinson’s estimator from coordinate expectations

The trace is the sum of diagonal entries and also the sum of eigenvalues:

Let a random probe satisfy

Independent Rademacher coordinates, each equally likely to be positive or negative one, satisfy this condition. Their squared coordinates are one, while distinct-coordinate products have zero expectation.

Expand the quadratic form:

Taking expectations gives

The off-diagonal contributions disappear in expectation. This is the entire unbiasedness argument.

With independent probes, the estimator is

Each sample requires one Hessian-vector product and one dot product. Averaging preserves unbiasedness and divides the variance by the number of probes.

Normalization needs care. A random unit direction has

in a parameter space of dimension \(P\). Its quadratic form estimates average eigenvalue, not trace. Multiplying by the dimension restores a trace estimate.

Deriving the Rademacher estimator’s variance

For a symmetric matrix and Rademacher probe,

The diagonal contribution is deterministic because every squared coordinate equals one.

When the centered expression is squared, cross terms between distinct unordered index pairs have zero expectation. At least one independent sign appears to an odd power. The remaining squared pair terms equal one, giving

Therefore,

This explains a useful special case: for a diagonal matrix in the chosen coordinates, every Rademacher probe gives the exact trace.

For the constructed two-coordinate Hessian,

The four possible probes produce

Their mean is six, exactly the trace, and their variance is

An average over independent probes has standard deviation

This is the exact sampling variability for the constructed matrix.

Rotating this matrix into its eigenbasis gives a diagonal matrix with eigenvalues four and two. Its trace is still six, but Rademacher probes in those coordinates have zero estimator variance. The trace is coordinate-invariant under orthogonal changes; this probe distribution’s variance need not be.

Trace and extreme eigenvalues summarize different risks

For a positive semidefinite Hessian and perturbation budget,

the largest quadratic contribution is

The eigenbasis expansion proves the upper bound, and a perturbation along the leading eigenvector attains it.

The trace instead connects to an average over isotropic perturbations. Let a random perturbation have mean and covariance

Write the perturbation as its mean plus a zero-mean remainder. Expanding the quadratic and taking expectations removes the cross terms:

The expected Taylor change is therefore

For zero-mean isotropic perturbations,

this simplifies to

Top eigenvalue and trace are consequently answers to different perturbation models. Their usefulness is not resolved by declaring one universally superior.

For example,

have equal trace. Their average quadratic response to equal isotropic noise is the same, while the worst directional response is twice as large for the first.

For an indefinite Hessian, trace is signed. The matrix

has zero trace and substantial curvature in both directions. A near-zero trace is not a certificate of flatness.

Applying the perturbation model to quantization

Quantization replaces weights with nearby representable values. Its error is a parameter perturbation, so the Taylor expression provides a local sensitivity model.

As a deliberately simplified model, suppose each coordinate’s rounding error is independent and uniform over an interval of width \(h\) centered at zero. Its variance is

Thus,

and the predicted average quadratic loss change is

For the constructed Hessian with trace six and interval width one-fifth,

Because that loss is exactly quadratic, the Taylor calculation is exact under the stipulated random-error model. Actual deterministic rounding does not automatically generate independent uniform errors. Clipping, unequal scales, correlated weights, and biased errors change the relevant covariance and mean.

For a parameter block, the corresponding Hessian block and perturbation covariance determine its local contribution. Cross-block Hessian terms vanish in expectation only when the assumed cross-covariances and means make them vanish. Independently ranking blocks can miss interactions.

Block size also matters. Trace sums curvature over coordinates; average curvature divides by block dimension. Equal per-coordinate noise and equal total perturbation energy imply different comparisons. A sensitivity report should state the perturbation budget before selecting a normalization.

Calibration statistics do not resolve these issues automatically. Matching activation means and variances constrains moments, not a complete distribution. A symmetric two-point variable and a standard Gaussian both have mean zero and variance one, despite different distributions. Such moment matching cannot by itself establish that calibration inputs reproduce clinically relevant cases.

Keeping curvature measurements comparable

Loss normalization directly scales curvature. If a loss is multiplied by a constant, its gradient, Hessian eigenvalues, and trace all receive that constant. Summed losses and mean losses cannot be compared as if their curvature values shared a scale.

Adding a squared-norm penalty contributes

to the Hessian. Every eigenvalue increases by the penalty coefficient, and the trace increases by that coefficient times the parameter count. Whether the reported Hessian includes regularization must therefore be explicit.

Parameterization matters too. Under a linear change of variables,

the transformed Hessian is

This follows by applying the chain rule twice. A uniform parameter rescaling can therefore change every curvature value even when the represented predictions are unchanged after the corresponding reparameterization.

Finally, the method should match the question. An HVP reveals response to one chosen direction. Power iteration searches for a dominant spectral direction. Trace estimation averages directional information. None of them directly measures clinical reliance on an input structure; they characterize a declared loss with respect to declared parameters.

Revision checklist

Why it matters for my work

Curvature can help me study the sensitivity of an ultrasound model to weight perturbation or compression. I need to keep that question separate from sensitivity to clinical evidence in an image, while checking whether compression changes the evidence-use behavior I actually care about.

What I have not resolved

• Which perturbation model best approximates the quantization scheme I would deploy?
• Compare predicted local loss changes with actual quantization changes and patient-separated clinical faithfulness audits.