Use the delta method to estimate the variance of the maximum likelihood estimator of a function of the parameter(s).

An MLE for $\theta$ is rarely what you want to report. You want a survival probability, a quantile, an LER, or a premium. The delta method converts the variance of $\hat{\theta }$ into the variance of $g(\hat{\theta })$ in one line of calculus.

MLE invariance. The functional invariance property says the MLE travels through any function. If $\hat{\theta }$ maximizes the likelihood for $\theta$, then $g(\hat{\theta })$ is the MLE for $\tau =g(\theta )$. You do not re-maximize. You substitute. This is half of the delta-method package. It gives you the point estimate. The other half (variance) needs calculus.

The univariate delta method. Asymptotically the MLE satisfies $\sqrt{n}(\hat{\theta }-\theta )\to N(0,1/I(\theta ))$, where $I(\theta )$ is the Fisher information per observation. For a smooth function $g$, a first-order Taylor expansion around $\theta$ gives $g(\hat{\theta })\approx g(\theta )+g^{\prime }(\theta )(\hat{\theta }-\theta )$. Variance flows through linearly.

$$\text{Var}\left(g(\hat{\theta })\right)\approx {\left[g^{\prime }(\theta )\right]}^2\text{Var}(\hat{\theta })$$

In practice you plug $\hat{\theta }$ into both $g^{\prime }(\theta )$ and $\text{Var}(\hat{\theta })$. The estimated information $I(\hat{\theta })$ gives $\widehat{\text{Var}}(\hat{\theta })=1/[nI(\hat{\theta })]$, or you read it off the inverse observed information directly.

Common mistakes

• Dropping the cross-term $2g_1{g}_2{\sigma }_{12}$ in the multivariate formula. Costs you the correlation correction. With Example 2 numbers, the SE is wrong by a factor of 2.5.
• Forgetting to square $g^{\prime }(\theta )$. The formula is $[g^{\prime }(\theta ){]}^2\text{Var}(\hat{\theta })$, not $g^{\prime }(\theta )\text{Var}(\hat{\theta })$. The latter can be negative.
• Leaving $\theta$ symbolic in the final variance. You must plug $\hat{\theta }$ into both $g^{\prime }$ and the information.

Bottom line

• MLE invariance: if $\hat{\theta }$ is the MLE of $\theta$, then $g(\hat{\theta })$ is the MLE of $g(\theta )$, with no re-optimization.
• Univariate delta: $\text{Var}(g(\hat{\theta }))\approx [g^{\prime }(\theta ){]}^2\text{Var}(\hat{\theta })$, with both $g^{\prime }$ and the information evaluated at $\hat{\theta }$.
• Multivariate delta: $\text{Var}(g(\hat{\mathbf{\theta }}))\approx \nabla g^{\top }\mathbf{\Sigma }\nabla g$, keeping off-diagonals, where $\mathbf{\Sigma }$ is the inverse Fisher information (inverse Hessian of the negative log-likelihood).
• SE and CI: standard error is the square root of the delta-method variance; the 95% Wald CI is $g(\hat{\theta })\pm 1.96\text{SE}$.

Exam shortcut

Write $\hat{\tau }=g(\hat{\theta })$ first, then $\widehat{\text{Var}}(\hat{\tau })=\nabla g^{\top }\mathbf{\Sigma }\nabla g$. Two lines if scalar, three if vector. Do not derive the information from scratch when the problem hands you $\mathbf{\Sigma }$. For one-parameter MLEs, memorize $\widehat{\text{Var}}(\hat{\theta })={\hat{\theta }}^2/n$ for Exponential and $\hat{p}(1-\hat{p})/n$ for Bernoulli. These cover most univariate delta problems. When two parameters are negatively correlated (Gamma, Pareto, Lognormal), expect the cross-term to reduce the SE of a product or sum of the MLEs.

The full lesson (about 3,090 words, 21 min read) adds 3 worked examples, all 9 common mistakes, a self-check, free in the app.

Learning objectives

• 4c

Browse all free Exam ASTAM lessons or jump into free Exam ASTAM practice questions.