THE GARDEN

Numerical error, conditioning, and stability

Updated 11 Oct 2026

• •

  Error measures

  • •

    Absolute error -> $|x-\hat{x}|$.

  • •

    Relative error -> $\frac{|x-\hat{x}|}{|x|}$ when $x\ne0$.

  • •

    true error needs the exact answer; approximate relative error compares successive iterates and is only a stopping estimate.

  • •

    near zero, use an absolute tolerance or a combined test such as $|x_k-x_{k-1}|\leq\mathrm{atol}+\mathrm{rtol}|x_k|$.

• •

  Error sources

  • •

    Chopping -> discards digits.

  • •

    Rounding -> selects a nearby representable value according to the rounding rule.

  • •

    Truncation error -> replaces an exact operation or infinite process with a finite approximation.

  • •

    Roundoff -> arises from finite precision.

  • •

    Error propagation -> input uncertainty or earlier rounding affects later results.

  • •

    Catastrophic cancellation -> loses significant digits when subtracting nearly equal quantities.

• •

  Conditioning and stability

  • •

    Conditioning -> describes sensitivity of the mathematical problem to input changes.

  • •

    Stability -> describes whether the algorithm adds excessive error while solving it.

  • •

    Important distinction -> a problem can be ill-conditioned even when solved with a stable algorithm.

  • •

    Large condition number -> warns that small input or rounding errors may cause large output changes.

  • •

    Patriot missile failure -> small binary representation error became operationally important after repeated computation.

• •

  Convergence and stopping

  • •

    Order of convergence -> describes how quickly error shrinks asymptotically.

  • •

    Small-step caution -> a small change between iterates does not prove correctness.

    • •

      a method may stagnate or converge to an unsuitable root.

  • •

    Stopping checks -> residual, step change, error bound where available, iteration limit.

  • •

    RMSE in regression -> $\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}$.

    • •

      summarizes residual scale in outcome units.

◌ Explore connections in Graph view