Round-off Error
The difference between the exact mathematical value of a function and the value computed using finite-precision floating-point arithmetic.
Governing formula
Machine Epsilon ε_mach ≈ 2⁻⁵³ ≈ 1.11 × 10⁻¹⁶ (IEEE 754 Double Precision)
SI unit
Floating-Point Precision Error
In depth
Computers store real numbers using finite bit lengths (e.g. 64-bit IEEE 754 double precision). Repeated arithmetic operations can accumulate round-off errors, causing catastrophic cancellation in subtraction of nearly equal numbers.
Examples in the real world
Minimizing catastrophic cancellation when computing quadratic formula roots for near-zero discriminant values.