Catastrophic Cancellation

Updated 01 Oct 2026

• •

  What it is

  • •

    severe loss of significant digits that occurs when subtracting two nearly-equal numbers ($x \approx y \Rightarrow x - y$), since their leading digits cancel out and only rounding-error digits remain

• •

  Exapmle

  • •

    for $x^2+10^6x+1=0$, the small root from $(-10^6+\sqrt{10^{12}-4})/2$ subtracts nearly equal numbers.

  • •

    in a typical float64 calculation, it gives approximately $-1.00000761449337\times10^{-6}$, rather than necessarily returning zero.

  • •

    the rationalized expression $-2/(10^6+\sqrt{10^{12}-4})$ gives approximately $-1.000000000001\times10^{-6}$.

• •

  Mitigation strategies (each with a worked example)

  • •

    Algebraic reformulation can substantially reduce cancellation, but does not universally guarantee full precision or eliminate every source of error.

    • •

      Rationalization: $\sqrt{x+1}-\sqrt{x}=1/(\sqrt{x+1}+\sqrt{x})$. The second form can reduce cancellation; attainable accuracy still depends on the inputs and arithmetic.

    • •

      For the quadratic example above, the rationalized expression improves the small-root calculation. The direct float64 calculation loses accuracy but does not return zero in the checked example.

    • •

      trigonometric identity substitution — $1-\cos(x)$ for small $x$ cancels to zero directly; the double-angle identity $1-\cos(x) = 2\sin^2(x/2)$ computes correctly

    • •

      Taylor series substitution — $\dfrac{1-e^{-x}}{x}$ for very small $x$ loses precision directly; expanding $e^{-x}$ via its Taylor series and simplifying gives a numerically stable formula

    • •

      two-pass variance formula — the textbook shortcut $s^2 = \frac{1}{n-1}\left(\sum x_i^2 - \frac{(\sum x_i)^2}{n}\right)$ can catastrophically fail for data with large values close together (e.g. $[1000001, 1000002, 1000003]$), potentially even producing a mathematically impossible negative variance; computing deviations from the mean directly ($\sum(x_i-\bar{x})^2$) avoids this

• •

  Related

  • •

    Error Propagation

  • •

    Machine Epsilon

◌ Explore connections in Graph view