Quantitative careers / Independent preparationOriginal books · Free worked examples

Research practice / 3 minute read

Explain a floating-point mismatch in a quant research project

Use an original cancellation example to separate stored inputs, arithmetic order, display rounding and model error.

Quant Finance Playbook editorial · How the material is developed

A financial newspaper showing a historical stock chart
Illustrative photograph by Markus Spiske on Unsplash.
On this page

Before investigating numerical precision, confirm what the inputs mean. The return-unit exercise separates basis points, percent and decimal returns; its resource comparison helps distinguish a unit-definition gap from a broader project need.

A numerical mismatch is not automatically evidence that a research model is wrong. First identify the arithmetic being performed and the precision of the stored values. Formatting two results to look the same does not establish that the calculations agree.

The Python floating-point tutorial explains why many decimal fractions are approximated in binary storage and why summation methods can differ. The exercise below uses exactly representable integer inputs so that you can isolate arithmetic order rather than blame decimal input conversion.

Three inputs, two calculation paths

Let the mathematical inputs be 2 raised to the power 54, one, and minus 2 raised to the power 54. Their exact sum is one. On a usual IEEE754 binary64 Python float implementation, compare these two explicitly parenthesized calculations:

large = float(2 ** 54)
left_first = (large + 1.0) - large
cancel_first = (large - large) + 1.0

The first result is 0.0; the second is 1.0. At that magnitude, adding one to the large floating-point value cannot preserve the unit increment in the rounded intermediate result. Subtracting the large value afterwards cannot recover it. In the second path, the equal large values cancel before the unit is added.

These expressions are equal in exact real arithmetic. The finite-precision intermediate operations explain their different computed values. This is a deliberately extreme diagnostic, not a claim about the error in any real trading dataset.

Compare a diagnostic reference

import math
from fractions import Fraction

values = [large, 1.0, -large]
accurate_float_total = math.fsum(values)
exact_stored_total = sum(Fraction.from_float(x) for x in values)

For these inputs, the results are 1.0 and the exact fraction 1. A rational calculation over stored floats answers a precise question: what is the exact sum of the numbers already represented? It does not recover information lost before those numbers were stored. Likewise, a more accurate sum does not correct wrong units, missing rows or future information in a feature.

Avoid using the built-in sum as a universal example of naive left-to-right addition. Its floating-point summation behavior has changed across Python versions. Record the interpreter version and the actual expression when explaining a result. The math reference describes fsum and its implementation qualifications.

Write a useful discrepancy note

RecordExample for this exercise
Expected mathematical quantitySum of the three stated integers, equal to one
Stored inputsThree exactly representable binary64 values
Divergent operationAddition of one to the large positive value before cancellation
Reference comparisonExact rational sum over the stored inputs
Scope of conclusionArithmetic-order sensitivity demonstrated for this input set

A defensible note says: “The mismatch is reproduced by the intermediate rounding in the first expression. Reordering this example or using the accurate summation reference retains the unit. I have not established the appropriate algorithm or error tolerance for the full project.”

Choose the next investigation

For a real project, define the required quantity and units, inspect the input scale, compare a suitable reference on a bounded case, and check whether the difference matters to the stated decision. Choose tolerances from that task rather than rounding until the comparison passes. A relative tolerance alone needs special care when the expected quantity is near zero.

Use simulation model checks when the arithmetic agrees but the generator may represent the wrong event. Use numerical-practice resource selection to decide whether this free exercise, language documentation or the existing Research Project Lab addresses your next gap.