arithmetic · activeDigit-level arithmetic
Multi-digit arithmetic, counting, and symbol manipulation fail in ways tied to tokenization and memorization.
Also called: numerical reasoning, token-level math
Tags: data-analysis, general, math, reasoning
A capable model performs multi-digit addition and multiplication, counts characters and items, and manipulates digits reliably regardless of how the numbers happen to be tokenized or how often they appeared in training.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
Digit-level arithmetic and numeric manipulation in text: multi-digit addition and multiplication, carries, place value, counting characters or items in a text sequence. In scope when the paper studies whether the model computes correctly. NOT in scope: visual object counting in images (that is a perception result), or math word problems where the difficulty is the reasoning chain rather than the calculation -- those belong with reasoning unless the paper isolates the arithmetic step.
Claims
- mechanismsingle paperArithmetic accuracy in frontier models depends strongly on how digits are chunked into tokens; a different digit grouping changes error rates.
- mechanismreplicatedcontestedWriting out reasoning steps improves how a problem is decomposed but does not fix the arithmetic inside a step — the model can decompose correctly and still get the calculation wrong, at the same confident register as when it is right. For exact arithmetic the fix is to move the computation out of the text, not to add more steps.
- mechanismsingle paperFew-shot arithmetic accuracy correlates with how often the specific numbers appear in pretraining data, evidence of memorization.
- mechanismsingle paperIntroduced grade-school math word problems on which 2021 models made frequent arithmetic slips, motivating verifier training.
- mechanismsingle paperMulti-digit multiplication accuracy falls to near zero as problem size grows, and models rely on pattern matching rather than a learned algorithm.
- mechanismsingle paperOffloading the computation to generated code and an interpreter removes most arithmetic errors on math word problems.
- mechanismsingle paperPosition embeddings that encode digit place let small transformers add 100-digit numbers, showing the limit is representational not fundamental.
- mechanismmechanism reasoningOffloading the calculation to generated code removes the arithmetic as a source of error but does not remove error — what remains is the translation from problem to program, which is now the only thing the model is doing. Verification has to move there with it.
Techniques
- Encode digit position in the modelarchitectureAdd positional embeddings that tell the model which place each digit occupies.
- Offload arithmetic to codetoolingHave the model write a short program for the numeric part and run it, instead of computing in text.
- Show intermediate stepspromptingPrompt the model to write out intermediate reasoning steps before the final answer.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.