THE GARDEN

Catastrophic Cancellation

Updated 15 Sep 2026

• •

  What it is

  • •

    severe loss of significant digits that occurs when subtracting two nearly-equal numbers ($x \approx y \Rightarrow x - y$), since their leading digits cancel out and only rounding-error digits remain

• •

  Mitigation strategies (each with a worked example)

  • •

    algebraic rationalization — $\sqrt{x+1}-\sqrt{x}$ loses 100% of its precision computed directly at large $x$; rewriting via the conjugate as $\dfrac{1}{\sqrt{x+1}+\sqrt{x}}$ recovers full precision

  • •

    alternate quadratic formula — the standard formula $x = \dfrac{-b+\sqrt{b^2-4ac}}{2a}$ catastrophically fails for $x^2+10^6x+1=0$ (returns $0.0$ instead of the correct $-1\times10^{-6}$); the algebraically equivalent form $x = \dfrac{-2c}{b+\sqrt{b^2-4ac}}$ avoids the cancellation entirely

  • •

    trigonometric identity substitution — $1-\cos(x)$ for small $x$ cancels to zero directly; the double-angle identity $1-\cos(x) = 2\sin^2(x/2)$ computes correctly

  • •

    Taylor series substitution — $\dfrac{1-e^{-x}}{x}$ for very small $x$ loses precision directly; expanding $e^{-x}$ via its Taylor series and simplifying gives a numerically stable formula

  • •

    two-pass variance formula — the textbook shortcut $s^2 = \frac{1}{n-1}\left(\sum x_i^2 - \frac{(\sum x_i)^2}{n}\right)$ can catastrophically fail for data with large values close together (e.g. $[1000001, 1000002, 1000003]$), potentially even producing a mathematically impossible negative variance; computing deviations from the mean directly ($\sum(x_i-\bar{x})^2$) avoids this

• •

  Related

  • •

    Error Propagation

  • •

    Machine Epsilon

◌ Explore connections in Graph view