# FluxEM: Algebraic Embeddings for Neural Arithmetic

Hunter Bown Shannon Labs
December 2024

## Abstract

Neural networks often struggle with arithmetic because they treat numbers as arbitrary tokens. FluxEM is an embedding scheme where arithmetic operations map to geometric operations: addition becomes vector addition, and multiplication becomes addition in log space. The result is systematic generalization from algebraic structure without learned parameters, with accuracy evaluated within floating-point tolerance. The approach draws on Lewin's Generalized Interval Systems (1987), a framework from music theory that formalizes transformation-based structure.

## 1. The Problem

Large language models can describe calculus but still fumble arithmetic like `1847 x 392`. Prior work (NALU, xVal, Abacus) focuses on training models to learn arithmetic more reliably.

The core issue is representation: when numbers are embedded as arbitrary token vectors, arithmetic relationships must be inferred from data. This inference often fails to generalize to new ranges.

## 2. The Insight

What if numbers were encoded so arithmetic operations are geometric operations?

For addition, embed numbers along a fixed direction. Then vector addition in embedding space equals arithmetic addition:

embed(a) + embed(b) = embed(a + b)

For multiplication, embed in log space. Addition in log space corresponds to multiplication in linear space:

log_embed(a) + log_embed(b) = log_embed(a x b)

These are algebraic identities over the reals for the magnitude component; under IEEE-754, they hold within floating-point precision bounds.

## 3. Formal Definition

### 3.1 Linear Embedding

For addition and subtraction:

e_lin(n) = (n / scale) * v

where v in R^d is a fixed unit vector. The homomorphism property e_lin(a) + e_lin(b) = e_lin(a + b) follows directly.

### 3.2 Logarithmic Embedding

For multiplication and division:

e_log(n) = (log|n| / scale) * v_mag + sign(n) * v_sign

where v_mag and v_sign are orthogonal unit vectors. Magnitude and sign are tracked separately, and the sign component is recombined in the operator definition.

Zero is handled explicitly outside the log embedding via a masking branch.

## 4. Results

Operation | OOD Accuracy | Test Range
--- | --- | ---
Addition | Within tolerance | [-100000, 100000]
Subtraction | Within tolerance | [-100000, 100000]
Multiplication | Within tolerance | [10, 1000] x [10, 1000]
Division | Within tolerance | [100, 10000] / [10, 100]
Powers | Within tolerance | Various
Roots | Within tolerance | Various

"Within tolerance" means all samples achieved < 1% relative error (or < 0.5 absolute error when |expected| <= 1). These bounds are intended for neural approximation contexts rather than calculator-grade financial or scientific workloads. Errors arise from IEEE-754 floating-point precision.

## 5. Connection to Music Theory

This approach has historical roots in music theory. Euler's Tonnetz (1739) embedded pitches geometrically, and Lewin's Generalized Musical Intervals and Transformations (1987) formalized a transformation-based framework.

A Generalized Interval System (GIS) consists of:

- S: a set of objects (pitches, or in this case, numbers)
- IVLS: an interval group (Z_12 for pitch classes, R for numbers)
- int: a function mapping pairs to intervals

FluxEM applies this structure to numbers: arithmetic is encoded directly as interval relations in embedding space.

## 6. Prior Work

Approach | Method | Limitation
--- | --- | ---
NALU (Trask 2018) | Log/exp with learned gates | Gates can fail to generalize
xVal (Golkar 2023) | Learned scaling direction | Direction must be learned
Abacus (McLeish 2024) | Positional digit encoding | Character-level, not continuous
FluxEM | Fixed algebraic structure | Parameter-free

## 7. Limitations

- Precision bounded by IEEE-754 floating point
- Zero requires explicit masking (log(0) undefined)
- Sign tracked separately from magnitude in log space
- Not a general reasoning system: arithmetic only

## 8. Code

Requires Python 3.10+.

Note: PyPI currently serves `fluxem==0.1.0`. For the latest source (0.2.0), install from GitHub.

    git clone https://github.com/Hmbown/FluxEM.git
    cd FluxEM
    python3 -m pip install -e .

    python3 -m venv .venv
    source .venv/bin/activate
    python3 -m pip install fluxem

    from fluxem import create_unified_model

    model = create_unified_model()
    model.compute("1847*392")    # -> 724024.0
    model.compute("123456+789")  # -> 124245.0
    model.compute("2**16")       # -> 65536.0

    # Windows (PowerShell)
    python -m venv .venv
    .\.venv\Scripts\Activate.ps1
    python -m pip install fluxem

Source: github.com/Hmbown/FluxEM

## 9. Lean4 Notes

These snippets illustrate how the linear and logarithmic magnitude properties can be stated in a proof assistant. They are illustrative and omit proof details.

    -- Linear embedding property (sketch)
    -- e_lin (a + b) = e_lin a + e_lin b

    -- Log embedding magnitude property (sketch)
    -- proj_mag (e_log (a * b)) = proj_mag (e_log a) + proj_mag (e_log b)

## References

- Euler, L. (1739). Tentamen novae theoriae musicae.
- Lewin, D. (1987). Generalized Musical Intervals and Transformations. Yale University Press.
- Trask, A. et al. (2018). Neural Arithmetic Logic Units. NeurIPS.
- Golkar, S. et al. (2023). xVal: A Continuous Number Encoding. arXiv.
- McLeish, S. et al. (2024). Transformers Can Do Arithmetic with the Right Embeddings. arXiv.