Neural Matter Networks — Activation-Free Neural Computation
Description: The ⵟ-product: A kernel-based activation-free neural network architecture that unifies alignment and proximity for geometrically-aware computation.

## The Problem with Traditional Neural Networks

Modern neural networks rely on a paradigm that separates geometry from non-linearity: linear transformations (dot products) followed by element-wise activation functions (ReLU, sigmoid).

Consider ReLU: it maps the entire spectrum of negative pre-activations—representing varying degrees of dissimilarity—to a uniform zero, discarding nuanced geometric relationships.

📐

#### Dot Product Limitation

Captures alignment but ignores spatial proximity. Vectors can be aligned yet arbitrarily far apart.

🎯

#### Activation Information Loss

ReLU collapses half-spaces to zero. Sigmoid saturates extremes. Geometric structure is lost.

🔄

#### Architectural Complexity

Requires normalization layers, attention mechanisms, and regularization to stabilize training.

#### Topological Distortion by Activations

Original
After ReLU

⚠️ ReLU creates sharp folds - discontinuous gradient at zero

## The ⵟ-Product: Unifying Alignment & Proximity

Inspired by inverse-square laws in physics, the ⵟ-product creates a unified operator that captures both directional alignment and spatial proximity.

$$\text{ⵟ}(\mathbf{w}, \mathbf{x}) = \frac{\langle \mathbf{w}, \mathbf{x} \rangle^2}{\|\mathbf{w} - \mathbf{x}\|^2 + \epsilon}$$
⟨w, x⟩²
Numerator
Squared dot product captures alignment between vectors
÷
‖w − x‖² + ε
Denominator
Squared distance captures proximity with epsilon (ε) for stability
🎓

#### Mercer Kernel

Symmetric and positive semidefinite, connecting to established kernel theory.

Theorem 2.1
∞

#### Universal Approximation

NMNs are dense in C(𝒳) under uniform norm on compact domains.

Theorem 2.4
📉

#### Self-Regularization

Responses remain bounded and gradients decay at infinity without explicit normalization.

Proposition 3.1
∇

#### Stable Gradients

Gradients vanish for distant inputs, providing natural localization during training.

Proposition 3.2

## Interactive Visualizations

Explore how the ⵟ-product differs from traditional similarity measures

### Similarity Measures & Gradient Fields

Drag the anchor point (⭐) to see how each measure responds. The top row shows similarity values, the bottom row shows gradient fields. Notice how the ⵟ-product creates a localized potential well with vortex-like gradients.

Anchor W: (3.0, 4.0) ε: 1.0
Similarity Measures (drag ⭐ to move anchor)
Dot Product
w · x
Euclidean Distance²
‖w − x‖²
ⵟ-Product
(w·x)² / (‖w−x‖² + ε)
Cosine Similarity
w·x / (‖w‖‖x‖)
Gradient Fields ∇ (also draggable)
Dot Product
Euclidean
ⵟ-Product
Cosine

### XOR Problem: Single Neuron Solution

The classic XOR problem cannot be solved by a single linear neuron. The ⵟ-product's intrinsic non-linearity enables a single unit to solve it.

#### Linear Neuron

Cannot separate XOR

#### ⵟ-Product Neuron

Solves XOR with w = [1, -1]

Input x | XOR Output | w·x | ⵟ(w,x)
--- | --- | --- | ---
(0,0) | 0 | 0 | 0
(0,1) | 1 | -1 | >0
(1,0) | 1 | 1 | >0
(1,1) | 0 | 0 | 0

### Decision Boundary Comparison

Linear neurons create hyperplane boundaries. ⵟ-product neurons create vortex-like territorial fields around learned prototypes.

#### Linear Model Boundaries

Unbounded half-space partitions

#### ⵟ-Product Boundaries

Localized vortex territories

### Loss Landscape Comparison

The optimization landscape for the XOR problem. Dot product has a spurious minimum; ⵟ-product creates exploitable valleys.

#### Dot Product + Sigmoid

#### ⵟ-Product

## Neural Matter Network Architectures

NMN layers serve as drop-in replacements for Linear + Activation, providing intrinsic non-linearity through geometry.

#### NMN Layer

Dense
$$h(\mathbf{x}) = s \cdot \sum_{i=1}^n \frac{(\mathbf{w}_i^\top\mathbf{x} + b_i)^2}{\|\mathbf{w}_i - \mathbf{x}\|^2 + \epsilon}$$

Replaces Linear + ReLU with a single geometric operation.

#### ⵟ-Conv

Convolution
$$(\text{ⵟ-Conv}(K, I))_{i,j} = \frac{\langle K, I_{i,j} \rangle^2}{\|K - I_{i,j}\|^2 + \epsilon}$$

Geometrically-aware feature extraction for spatial data.

#### ⵟ-Attention

Transformer
$$\text{ⵟ-Attn}(Q,K,V) = \text{softmax}(s \cdot Q \text{ⵟ} K^T) V$$

Query-key similarity through geometric alignment and proximity.

### Implemented Architectures

Architecture | Base Model | Design | Key Change
--- | --- | --- | ---
AetherResNet | ResNet | ⵟ-Conv → Linear Conv per block | No activation functions
AetherGPT | GPT-2 | MHA + NMN → Linear | No normalization layers

## Experimental Results

### Vision Benchmarks

Architecture | CIFAR-10 | CIFAR-100 | STL-10 | Tiny-ImageNet
--- | --- | --- | --- | ---
ResNet-18 | 94.23% | 72.15% | 78.42% | 56.89%
Aether-ResNet-18 | 92.37% | 74.83% | 80.91% | 59.34%
ViT-Small | 91.78% | 69.91% | 75.13% | 52.76%
Aether-ViT-Small | 92.45% | 70.58% | 78.89% | 51.42%

### Language Modeling (Fineweb 2.5B tokens)

#### GPT-2 Baseline

FP32 Loss 2.43
BF16 Loss 3.03

#### Aether-GPT2

FP32 Loss 2.29
BF16 Loss 2.69
11.2% improvement in BF16

### Training Dynamics

Validation and training loss over 35k steps. Aether-GPT2 (blue) consistently outperforms the linear baseline (green).

### Learned Prototypes: Linear vs ⵟ-Product

Linear models produce diffuse, blurry prototypes. ⵟ-product neurons learn sharp, geometrically coherent digit representations.

## Mathematical Foundations

Explore the rigorous theoretical guarantees behind Neural Matter Networks. Each theorem builds upon the previous, creating a complete mathematical framework.

Theorem 1

### The ⵟ-Product is a Mercer Kernel

Theorem (Mercer's Condition): The kernel $k_{\text{ⵟ}}(\mathbf{x}, \mathbf{w}) = \frac{(\mathbf{x} \cdot \mathbf{w})^2}{\|\mathbf{x} - \mathbf{w}\|^2 + \varepsilon}$ is symmetric and positive semi-definite, hence a valid Mercer kernel on $\mathbb{R}^d$.

#### What Does This Mean?

A Mercer kernel is a special type of similarity function that has two critical properties:

- Symmetry: $k(\mathbf{x}, \mathbf{w}) = k(\mathbf{w}, \mathbf{x})$ — the similarity between x and w is the same as between w and x.
- Positive Semi-Definiteness (PSD): For any set of points, the kernel matrix has all non-negative eigenvalues.

#### Why Is This Important?

Being a Mercer kernel means the ⵟ-product implicitly computes an inner product in a high-dimensional feature space without ever explicitly computing that space. This is the famous "kernel trick" from machine learning theory.

💡
Key Insight: The ⵟ-product is a product of two PSD kernels: the squared dot product $(\mathbf{x} \cdot \mathbf{w})^2$ (a polynomial kernel) and the inverse multiquadric $\frac{1}{\|\mathbf{x} - \mathbf{w}\|^2 + \varepsilon}$ (an RBF-like kernel). By the Schur product theorem, their product is also PSD.

#### What Does This Offer?

- Theoretical Foundation: Connects NMNs to 50+ years of kernel method research (SVMs, Gaussian Processes, etc.)
- Reproducing Kernel Hilbert Space (RKHS): Guarantees existence of a rich function space for learning
- Optimization Guarantees: Many kernel-based optimization results apply directly

Theorem 2

### Universal Approximation Theorem

Theorem: Let $\mathcal{X} \subset \mathbb{R}^d$ be compact. The class of single-hidden-layer ⵟ-product networks $f(\mathbf{x}) = \sum_{i=1}^n \alpha_i \cdot g(\mathbf{x}; \mathbf{w}_i, b_i) + c$ is dense in $C(\mathcal{X})$ under the uniform norm. That is, NMNs can approximate any continuous function to arbitrary precision.

#### What Does This Mean?

The Universal Approximation Theorem (UAT) is the fundamental existence theorem for neural networks. It guarantees that given enough neurons, a network can approximate any continuous function as closely as desired.

This theorem proves that NMNs lose no expressive power by removing activation functions. The ⵟ-product's inherent non-linearity is sufficient.

#### The Proof Strategy

The proof is elegant and leverages the kernel structure:

1. Recover IMQ kernel: By differentiating $g(\mathbf{x}; \mathbf{w}, b)$ twice with respect to bias $b$, we recover the inverse multiquadric (IMQ) kernel: $\partial_b^2 g = \frac{2}{\|\mathbf{x} - \mathbf{w}\|^2 + \varepsilon}$
2. Fourier analysis: The IMQ kernel has a strictly positive Fourier transform (Bessel function)
3. Uniqueness: Any measure orthogonal to all IMQ translates must be zero
4. Density: By Hahn-Banach/Riesz duality, the span is dense in $C(\mathcal{X})$

🔬
Key Insight: The bias term $b$ is crucial — it allows the network to "shift" response fields and span the entire function space through differentiation. This is why we use $(\mathbf{w}^\top\mathbf{x} + b)^2$ rather than just $(\mathbf{w}^\top\mathbf{x})^2$.

#### What Does This Offer?

- No Power Loss: NMNs are as expressive as ReLU/Sigmoid networks
- Geometric Localization: Unlike ReLU (unbounded growth), the ⵟ-product achieves density through localized geometric units
- Practical Corollary: Single hidden layer is sufficient in theory — though deeper networks may learn more efficiently

Proposition 3

### Self-Regulation & Bounded Outputs

Proposition: For any fixed weight vector $\mathbf{w}$, the ⵟ-product output remains bounded and converges as $\|\mathbf{x}\| \to \infty$: $$\lim_{\|\mathbf{x}\| \to \infty} \text{ⵟ}(\mathbf{w}, \mathbf{x}) = \|\mathbf{w}\|^2 \cos^2\theta$$ where $\theta$ is the angle between $\mathbf{w}$ and the direction of $\mathbf{x}$.

#### What Does This Mean?

Unlike ReLU which can grow unboundedly with input magnitude, or dot products which scale linearly, the ⵟ-product naturally self-regulates. As inputs get very large, the output converges to a finite value that depends only on the direction, not magnitude.

#### Corollary: Dimensional Self-Normalization

Corollary: At initialization with random i.i.d. weights, both the numerator $(\mathbf{w}^\top\mathbf{x})^2$ and denominator $\|\mathbf{w} - \mathbf{x}\|^2$ scale as $\mathcal{O}(d)$ with dimension $d$, so their ratio remains $\mathcal{O}(1)$.

This means NMNs don't need careful initialization schemes like Xavier or He — they're dimensionally stable by design!

#### Corollary: Mitigating Internal Covariate Shift

Corollary: As input magnitudes grow large, the mean and variance of neuron activations across a batch become independent of input magnitudes, depending only on angular distribution.

This explains why NMNs can often operate without batch normalization — the "covariate shift" problem that plagues deep networks is naturally mitigated.

⚡
Practical Impact: No gradient explosion from large inputs. No need for gradient clipping in most cases. Simpler, more stable training dynamics.

#### What Does This Offer?

- No Exploding Activations: Outliers don't cause numerical instabilities
- Reduced Normalization Needs: Less reliance on BatchNorm, LayerNorm
- Memory Efficiency: 15-25% reduction from eliminating normalization layers
- Robust Training: Stable even with varying input distributions

Proposition 4

### Stable Learning & Gradient Localization

Proposition: The gradient of the ⵟ-product with respect to input vanishes for distant inputs: $$\lim_{\|\mathbf{x}\| \to \infty} \|\nabla_{\mathbf{x}} \text{ⵟ}(\mathbf{w}, \mathbf{x})\| = 0$$ Specifically, $\|\nabla_{\mathbf{x}} \text{ⵟ}\| \sim \mathcal{O}(1/k)$ as $\|\mathbf{x}\| = k \to \infty$.

#### What Does This Mean?

Each ⵟ-product neuron creates a localized learning region. Points far from the weight vector contribute vanishingly small gradients. This is fundamentally different from:

- Linear neurons: Gradients are constant regardless of distance (no localization)
- ReLU neurons: Gradients are either constant (positive side) or zero (negative side)

#### Regularity Properties

##### Analyticity (Lemma)

The ⵟ-product is $C^\infty$ — infinitely differentiable. Perfect for physics-informed neural networks (PINNs) where you need higher-order derivatives.

##### Lipschitz Continuity (Proposition)

The kernel is globally Lipschitz continuous: $|K(\mathbf{w}, \mathbf{x}) - K(\mathbf{w}, \mathbf{y})| \leq L \|\mathbf{x} - \mathbf{y}\|$. This bounds how fast outputs can change.

🎯
Geometric Intuition: Each neuron is like a "gravity well" — it strongly attracts nearby points but has negligible influence on distant ones. During training, each neuron learns to "own" a local region of the input space.

#### What Does This Offer?

- Outlier Robustness: Distant outliers don't cause large, destabilizing gradient updates
- Local Learning: Each neuron specializes in its region without interfering with others
- Smoother Optimization: Lipschitz bounds provide theoretical guarantees on loss landscape
- PINN-Friendly: $C^\infty$ smoothness enables any-order derivative computation

Theorems 5-6

### Information-Geometric Foundations

Theorem (Minimal Similarity): For probability distributions $\mathbf{p}, \mathbf{q} \in \Delta^{n-1}$: $$\text{ⵟ}(\mathbf{p}, \mathbf{q}) = 0 \iff \text{supp}(\mathbf{p}) \cap \text{supp}(\mathbf{q}) = \emptyset$$ Furthermore, disjoint support implies $\text{KL}(\mathbf{p} \| \mathbf{q}) = \infty$.
Theorem (Maximal Similarity): $\text{ⵟ}(\mathbf{p}, \mathbf{q}) \to \infty \iff \mathbf{p} = \mathbf{q}$, and this corresponds to $\text{KL}(\mathbf{p} \| \mathbf{q}) = 0$.

#### What Does This Mean?

When the ⵟ-product is applied to probability distributions (like softmax outputs), it exhibits deep connections to information theory. The extremes of the ⵟ-product correspond precisely to extremes in KL divergence!

ⵟ = 0
⟺
Disjoint Support
Distributions never overlap
KL divergence = ∞
ⵟ → ∞
⟺
Identical Distributions
p = q everywhere
KL divergence = 0

#### Duality of Orthogonality Concepts

The ⵟ-product unifies three distinct notions of "orthogonality":

Type | Condition | Interpretation
--- | --- | ---
Euclidean | $\mathbf{p} \cdot \mathbf{q} = 0$ | Vectors perpendicular in space
Combinatorial | $\text{supp}(\mathbf{p}) \cap \text{supp}(\mathbf{q}) = \emptyset$ | No shared non-zero entries
Information-Theoretic | $\text{KL}(\mathbf{p} \| \mathbf{q}) = \infty$ | Infinite surprise/divergence

All three are equivalent when $\text{ⵟ}(\mathbf{p}, \mathbf{q}) = 0$!

🌉
Bridge Between Worlds: The ⵟ-product acts as a bridge connecting Euclidean geometry (dot products, distances) with probabilistic reasoning (KL divergence, entropy). This explains why NMNs work well with cross-entropy loss — there's deep mathematical harmony.

#### What Does This Offer?

- Natural Compatibility: Works harmoniously with entropy-based loss functions
- Interpretable Outputs: Similarity scores have information-theoretic meaning
- Theoretical Unity: Unifies geometric and probabilistic perspectives in one framework
- Distribution Modeling: Natural fit for generative models and density estimation

Theorem 7

### Topological Organization: Neural Fiber Bundles

Theorem: Let $\mathcal{M} \subset \mathbb{R}^d$ be a smooth compact data manifold and $\{w_c\}_{c=1}^C$ be class prototypes. The classification rule $\hat{c}(x) = \arg\max_c \text{ⵟ}(w_c, x)$ partitions $\mathcal{M}$ into decision regions.
Separation Property: If prototypes are orthogonal ($\langle w_i, w_j \rangle = 0$ for $i \neq j$), then:

1. Prototypes are maximally dissimilar: $\text{ⵟ}(w_i, w_j) = 0$
2. Decision boundaries are spatially separated from prototype cores

#### What Does This Mean?

Traditional linear classifiers create hyperplane decision boundaries — flat cuts through space that extend infinitely. The ⵟ-product creates something fundamentally different: vortex-like territorial fields around each prototype.

Class A
Class B

Linear: Unbounded half-spaces

A
B
C

ⵟ-Product: Localized vortex territories

#### Why Orthogonal Prototypes?

When class prototypes are orthogonal, something magical happens:

- Each prototype's "core" (where response is maximal) is completely separate from other prototypes
- At point $x = w_i$: the response to class $i$ is $\|w_i\|^4/\varepsilon$ (very large), while response to class $j$ is exactly 0
- Decision boundaries are forced into "neutral zones" between prototypes

🌀
Vortex Intuition: Imagine each prototype as a whirlpool. Points are "sucked in" toward their nearest prototype with strength proportional to alignment and inverse-square of distance. The decision boundary is where the "pull" from two prototypes exactly balances.

#### Fiber Bundle Structure

For those familiar with differential geometry: the classification can be viewed through the lens of fiber bundles. The data manifold $\mathcal{M}$ is the base space, each point $x$ has a "fiber" of response values $[\text{ⵟ}(w_1, x), ..., \text{ⵟ}(w_C, x)]$, and classification is projection to the dominant fiber component.

#### What Does This Offer?

- Geometric Interpretability: Decision regions have intuitive spatial meaning
- Natural Clustering: Each class "owns" a territory, not just a half-space
- Robust Boundaries: Orthogonal prototypes guarantee maximum class separation
- Manifold-Aware: Works naturally with curved data manifolds, not just flat spaces

### The Complete Picture

These six theorem groups form a coherent mathematical framework:

1 Mercer: Valid kernel ⟹ rich function space
↓
2 UAT: Can approximate any function
↓
3 Self-Regulation: Bounded, stable outputs
↓
4 Stable Gradients: Trainable via localized learning
↓
5 Info Theory: Works with entropy losses
↓
6 Topology: Geometric decision boundaries

Together, these guarantee that NMNs are theoretically sound, practically trainable, and geometrically interpretable — without a single activation function.

## Quick Start

### Installation

    pip install nmn

    # Framework-specific
    pip install "nmn[torch]"    # PyTorch
    pip install "nmn[keras]"    # Keras/TensorFlow
    pip install "nmn[nnx]"      # Flax NNX (JAX)
    pip install "nmn[all]"      # Everything

    import torch
    from nmn.torch.nmn import YatNMN

    # Replace nn.Linear + activation with single layer
    layer = YatNMN(
        in_features=128,
        out_features=64,
        epsilon=1e-5
    )

    x = torch.randn(32, 128)
    y = layer(x)  # (32, 64) — inherently non-linear!

    import keras
    from nmn.keras.nmn import YatNMN

    # Drop-in replacement for Dense + activation
    layer = YatNMN(
        features=64,
        epsilon=1e-5
    )

    x = keras.ops.zeros((32, 128))
    y = layer(x)  # (32, 64)

    import jax.numpy as jnp
    from flax import nnx
    from nmn.nnx.nmn import YatNMN

    layer = YatNMN(
        in_features=128,
        out_features=64,
        rngs=nnx.Rngs(0)
    )

    x = jnp.zeros((32, 128))
    y = layer(x)  # (32, 64)

### Available Layers

YatNMN Dense
YatConv1D Conv
YatConv2D Conv
YatConv3D Conv
YatConvTranspose Conv
MultiHeadAttention Attention
YatLSTMCell RNN
YatGRUCell RNN

## Citation

    @article{bouhsine2025nomoredelulu,
      author = {Taha Bouhsine},
      title = {No More DeLuLu: A Kernel-Based Activation-Free Neural Networks},
      year = {2025},
      url = {https://github.com/azettaai/nmn}
    }

Theorem 1

## The ⵟ-Product is a Mercer Kernel

🧒

#### Explain Like I'm 5

Imagine you have a special measuring stick that tells you how "similar" two toys are. A good measuring stick should follow two rules:

- 🔄 Fair both ways: If toy A is "very similar" to toy B, then toy B should also be "very similar" to toy A.
- ✅ Makes sense together: If you measure lots of toys, the numbers should all "agree" with each other (no contradictions).

The ⵟ-product is like a magic measuring stick that follows both rules perfectly! This means we can trust it to compare things fairly, and mathematicians have already figured out lots of cool tricks we can use with measuring sticks like this.

Theorem (Mercer's Condition): The kernel $k_{\text{ⵟ}}(\mathbf{x}, \mathbf{w}) = \frac{(\mathbf{x} \cdot \mathbf{w})^2}{\|\mathbf{x} - \mathbf{w}\|^2 + \varepsilon}$ is symmetric and positive semi-definite, hence a valid Mercer kernel on $\mathbb{R}^d$.

#### 🎯 The Problem This Solves

When we invented the ⵟ-product as a new way to measure similarity, we needed to answer a critical question: Is this mathematically legitimate?

Without being a valid kernel, the ⵟ-product would be just another arbitrary formula. By proving it's a Mercer kernel, we unlock 50+ years of kernel methods research — SVMs, Gaussian Processes, kernel PCA, and more — all of which now apply to NMNs.

#### 📐 The Mathematics In Depth

A Mercer kernel is a function $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ satisfying two properties:

1. Symmetry: $k(\mathbf{x}, \mathbf{w}) = k(\mathbf{w}, \mathbf{x})$ for all $\mathbf{x}, \mathbf{w}$
2. Positive Semi-Definiteness: For any $n$ points $\{x_1, ..., x_n\}$ and any coefficients $\{c_1, ..., c_n\}$: $$\sum_{i=1}^{n} \sum_{j=1}^{n} c_i c_j k(x_i, x_j) \geq 0$$

The proof proceeds in three steps:

Step 1: Decompose into known kernels

We write $\text{ⵟ}(\mathbf{w}, \mathbf{x}) = k_1(\mathbf{w}, \mathbf{x}) \cdot k_2(\mathbf{w}, \mathbf{x})$ where:

- $k_1(\mathbf{w}, \mathbf{x}) = (\mathbf{w}^\top \mathbf{x})^2$ — the squared dot product (degree-2 polynomial kernel)
- $k_2(\mathbf{w}, \mathbf{x}) = \frac{1}{\|\mathbf{w} - \mathbf{x}\|^2 + \varepsilon}$ — the inverse multiquadric (IMQ) kernel

Step 2: Verify each component is PSD

- Polynomial kernel: $(\mathbf{w}^\top \mathbf{x})^2 = \langle \phi(\mathbf{w}), \phi(\mathbf{x}) \rangle$ where $\phi$ maps to the space of outer products. This is PSD by construction.
- IMQ kernel: Has a known positive Fourier transform (modified Bessel function $K_0$), which by Bochner's theorem implies PSD.

Step 3: Apply Schur Product Theorem

The Schur product theorem states: if $K_1$ and $K_2$ are PSD kernel matrices, their element-wise (Hadamard) product $K_1 \circ K_2$ is also PSD.

Since both $k_1$ and $k_2$ are PSD, their product $\text{ⵟ} = k_1 \cdot k_2$ is PSD. ∎

#### 💥 The Consequences

🌐

##### Reproducing Kernel Hilbert Space (RKHS)

Every Mercer kernel defines an RKHS — a rich function space where learning has nice properties. The ⵟ-product implicitly projects data into this infinite-dimensional space.

🔮

##### The Kernel Trick

We can compute inner products in the high-dimensional feature space without ever computing the features explicitly. This is computationally efficient and theoretically powerful.

📊

##### Representer Theorem Applies

Optimal solutions to regularized learning problems lie in the span of kernel evaluations at training points. This gives theoretical guarantees on generalization.

🔗

##### Connection to SVMs & GPs

All kernel-based algorithms (Support Vector Machines, Gaussian Processes, kernel PCA) can now use the ⵟ-product as their kernel function.

#### 🎓 What This Really Means

This theorem is the foundation stone of NMN theory. It answers the question: "Why should we believe this strange formula has any mathematical meaning?"

By proving Mercer's condition, we establish that the ⵟ-product isn't just a heuristic — it's a principled similarity measure with deep connections to functional analysis, optimization theory, and statistical learning.

💡
Key Insight: The ⵟ-product combines the alignment sensitivity of polynomial kernels with the locality of RBF kernels. This hybrid nature is what makes it so effective for neural computation.

#### 📜 Historical Context

Mercer's theorem dates back to 1909, when James Mercer proved that certain integral operators could be decomposed using orthonormal functions. This became the foundation of kernel methods in machine learning, popularized by SVMs in the 1990s.

By connecting NMNs to this rich history, we inherit decades of theoretical insights and practical algorithms — while introducing something genuinely new: activation-free neural networks.

Theorem 2

## Universal Approximation Theorem

🧒

#### Explain Like I'm 5

Imagine you have a magic box that can draw any picture you want. You just need to tell it "draw a cat" or "draw a house" and it will draw it perfectly!

The Universal Approximation Theorem says our ⵟ-product networks are like that magic box — they can learn to do anything (well, any continuous function) if we give them enough "drawing tools" (neurons).

Even though we removed the activation functions (like ReLU), we didn't lose any power! The ⵟ-product is already "magical" enough on its own.

Theorem: Let $\mathcal{X} \subset \mathbb{R}^d$ be compact. The class of single-hidden-layer ⵟ-product networks $f(\mathbf{x}) = \sum_{i=1}^n \alpha_i \cdot g(\mathbf{x}; \mathbf{w}_i, b_i) + c$ is dense in $C(\mathcal{X})$ under the uniform norm. That is, NMNs can approximate any continuous function to arbitrary precision.

#### 🎯 The Problem This Solves

When we removed activation functions, critics asked: "Can you still learn complex functions?"

Traditional neural networks rely on activation functions (ReLU, sigmoid) to create non-linearity. Without them, a network would just be a series of linear transformations — which can only learn linear functions.

This theorem proves that the ⵟ-product's inherent geometric non-linearity is sufficient. We don't need separate activation functions because the ⵟ-product itself is non-linear.

#### 📐 The Mathematics In Depth

The proof is elegant and leverages the kernel structure established by Theorem 1:

Step 1: Recover IMQ Kernel

Consider the ⵟ-product with bias: $g(\mathbf{x}; \mathbf{w}, b) = \frac{(\mathbf{w}^\top\mathbf{x} + b)^2}{\|\mathbf{w} - \mathbf{x}\|^2 + \varepsilon}$

By differentiating twice with respect to $b$: $$\partial_b^2 g(\mathbf{x}; \mathbf{w}, b) = \frac{2}{\|\mathbf{x} - \mathbf{w}\|^2 + \varepsilon}$$

This is the inverse multiquadric (IMQ) kernel — a well-studied kernel in approximation theory.

Step 2: Fourier Analysis

The IMQ kernel has a strictly positive Fourier transform (related to the modified Bessel function $K_0$). This is a key property for density results.

Step 3: Uniqueness via Orthogonality

If a measure $\mu$ is orthogonal to all IMQ translates (i.e., $\int k(\mathbf{x}, \mathbf{w}) d\mu(\mathbf{x}) = 0$ for all $\mathbf{w}$), then by the positivity of the Fourier transform, $\mu$ must be the zero measure.

Step 4: Density via Hahn-Banach/Riesz Duality

By the Hahn-Banach theorem and Riesz representation theorem, if the span of $\{g(\cdot; \mathbf{w}, b)\}$ is not dense, there exists a non-zero continuous linear functional that vanishes on the span. This functional corresponds to a measure, which by Step 3 must be zero — a contradiction.

Therefore, the span is dense in $C(\mathcal{X})$. ∎

🔬
Key Insight: The bias term $b$ is crucial! It allows the network to "shift" response fields and span the entire function space through differentiation. This is why we use $(\mathbf{w}^\top\mathbf{x} + b)^2$ rather than just $(\mathbf{w}^\top\mathbf{x})^2$.

#### 💥 The Consequences

∞

##### No Expressive Power Loss

NMNs are as expressive as ReLU/Sigmoid networks. Single hidden layer is sufficient in theory (though deeper networks may learn more efficiently).

🎯

##### Geometric Localization

Unlike ReLU (unbounded growth), the ⵟ-product achieves density through localized geometric units — creating "vortex-like" territorial fields.

🔧

##### Simpler Architecture

No need for complex activation functions. The geometric operator itself provides the non-linearity, leading to simpler, more interpretable networks.

📊

##### Kernel Method Connection

The proof connects NMNs to kernel methods, opening the door to using kernel-based optimization techniques and theoretical guarantees.

#### 🎓 What This Really Means

This is the fundamental existence theorem for NMNs. It answers the question: "Can we really learn complex functions without activation functions?"

The answer is a resounding yes. The ⵟ-product's geometric structure provides sufficient non-linearity to approximate any continuous function.

This theorem bridges the gap between theoretical possibility and practical feasibility, showing that activation-free networks are not just a curiosity — they're a viable alternative with the same expressive power.

#### 📜 Historical Context

Universal approximation theorems date back to the 1980s, with seminal work by Cybenko (1989) and Hornik et al. (1989) showing that single-hidden-layer networks with sigmoidal activations are universal approximators.

Our theorem extends this tradition, showing that geometric non-linearity (via the ⵟ-product) can replace functional non-linearity (via activations) without losing approximation power.

Theorem 3

## Self-Regulation & Bounded Outputs

🧒

#### Explain Like I'm 5

Imagine you have a volume knob that can go from 0 to 100. If you turn it all the way up, it might break your speakers! But what if the knob had a safety limit that prevents it from going too loud?

The ⵟ-product is like that safe volume knob. Even if you give it really big numbers, it automatically limits itself and never explodes. It's self-regulating!

This means we don't need extra "safety equipment" (like normalization layers) — the ⵟ-product is already safe by design.

Proposition: For any fixed weight vector $\mathbf{w}$, the ⵟ-product output remains bounded and converges as $\|\mathbf{x}\| \to \infty$: $$\lim_{\|\mathbf{x}\| \to \infty} \text{ⵟ}(\mathbf{w}, \mathbf{x}) = \|\mathbf{w}\|^2 \cos^2\theta$$ where $\theta$ is the angle between $\mathbf{w}$ and the direction of $\mathbf{x}$.

#### 🎯 The Problem This Solves

Traditional neural networks suffer from unbounded growth:

- ReLU: Output grows linearly with input magnitude — can explode with outliers
- Dot products: Scale linearly with dimension — requires careful initialization
- Internal Covariate Shift: Activation statistics change during training, requiring normalization

The ⵟ-product solves all three problems naturally, without explicit normalization.

#### 📐 The Mathematics In Depth

As $\|\mathbf{x}\| \to \infty$, we can write $\mathbf{x} = k \mathbf{u}$ where $k = \|\mathbf{x}\|$ and $\mathbf{u}$ is a unit vector. Then:

$$\text{ⵟ}(\mathbf{w}, \mathbf{x}) = \frac{(\mathbf{w}^\top k\mathbf{u})^2}{\|k\mathbf{u} - \mathbf{w}\|^2 + \varepsilon} = \frac{k^2 (\mathbf{w}^\top\mathbf{u})^2}{k^2 - 2k(\mathbf{w}^\top\mathbf{u}) + \|\mathbf{w}\|^2 + \varepsilon}$$

Dividing numerator and denominator by $k^2$ and taking the limit:

$$\lim_{k \to \infty} \text{ⵟ}(\mathbf{w}, k\mathbf{u}) = \frac{(\mathbf{w}^\top\mathbf{u})^2}{1} = \|\mathbf{w}\|^2 \cos^2\theta$$

where $\cos\theta = \frac{\mathbf{w}^\top\mathbf{u}}{\|\mathbf{w}\|}$ is the cosine of the angle between $\mathbf{w}$ and $\mathbf{u}$.

#### 💥 The Consequences

📉

##### No Exploding Activations

Outliers don't cause numerical instabilities. The output is bounded by $\|\mathbf{w}\|^2$, regardless of input magnitude.

🎯

##### Dimensional Self-Normalization

At initialization, both numerator and denominator scale as $\mathcal{O}(d)$, so their ratio remains $\mathcal{O}(1)$. No need for Xavier/He initialization!

🔄

##### Mitigates Internal Covariate Shift

As inputs grow large, activation statistics depend only on angular distribution, not magnitude. This naturally mitigates the covariate shift problem.

💾

##### Memory Efficiency

15-25% reduction in memory from eliminating normalization layers (BatchNorm, LayerNorm). Simpler, faster training.

#### 🎓 What This Really Means

This proposition shows that the ⵟ-product has built-in stability. Unlike ReLU or linear layers, it doesn't need external mechanisms to prevent numerical issues.

This is a geometric property — the inverse-square law in the denominator naturally creates a "safety valve" that prevents unbounded growth.

⚡
Practical Impact: No gradient explosion from large inputs. No need for gradient clipping in most cases. Simpler, more stable training dynamics.
Theorem 4

## Stable Learning & Gradient Localization

🧒

#### Explain Like I'm 5

Imagine you're learning to draw, and you have a teacher who gives you feedback. If you're drawing something very different from what the teacher wants, they might not give you much feedback (because it's too far off).

The ⵟ-product works the same way! When inputs are far away from the weight vector, the gradients (feedback) become very small. This means each neuron only "learns" from nearby examples — creating natural localization.

This is like having a teacher who focuses on helping you with things that are close to what you're trying to learn, rather than getting distracted by completely different things.

Proposition: The gradient of the ⵟ-product with respect to input vanishes for distant inputs: $$\lim_{\|\mathbf{x}\| \to \infty} \|\nabla_{\mathbf{x}} \text{ⵟ}(\mathbf{w}, \mathbf{x})\| = 0$$ Specifically, $\|\nabla_{\mathbf{x}} \text{ⵟ}\| \sim \mathcal{O}(1/k)$ as $\|\mathbf{x}\| = k \to \infty$.

#### 🎯 The Problem This Solves

Traditional neurons have global gradient influence:

- Linear neurons: Gradients are constant regardless of distance (no localization)
- ReLU neurons: Gradients are either constant (positive side) or zero (negative side) — binary, not smooth

This means every training example affects every weight, leading to:

- Slow convergence (too many conflicting signals)
- Difficulty learning local patterns
- Susceptibility to outliers

#### 📐 The Mathematics In Depth

Computing the gradient:

$$\nabla_{\mathbf{x}} \text{ⵟ}(\mathbf{w}, \mathbf{x}) = \frac{2(\mathbf{w}^\top\mathbf{x})\mathbf{w}}{\|\mathbf{w} - \mathbf{x}\|^2 + \varepsilon} - \frac{(\mathbf{w}^\top\mathbf{x})^2 \cdot 2(\mathbf{x} - \mathbf{w})}{(\|\mathbf{w} - \mathbf{x}\|^2 + \varepsilon)^2}$$

As $\|\mathbf{x}\| = k \to \infty$, the denominator grows as $k^2$, while the numerator grows as $k$. Therefore, $\|\nabla_{\mathbf{x}} \text{ⵟ}\| \sim \mathcal{O}(1/k) \to 0$.

#### 💥 The Consequences

🎯

##### Natural Localization

Each neuron creates a "territory" around its weight vector. Only nearby inputs contribute significant gradients, enabling local pattern learning.

∞

##### Infinitely Smooth (C∞)

The ⵟ-product is infinitely differentiable, perfect for physics-informed neural networks (PINNs) requiring higher-order derivatives.

📊

##### Lipschitz Continuity

The gradient is Lipschitz continuous, providing theoretical guarantees for optimization and stability during training.

🛡️

##### Robust to Outliers

Distant outliers contribute vanishingly small gradients, so they don't disrupt learning of local patterns.

#### 🎓 What This Really Means

This proposition shows that ⵟ-product neurons have spatial awareness. They naturally create "learning territories" — regions of input space where they actively learn, with smooth falloff outside.

This is fundamentally different from linear or ReLU neurons, which have global influence. The localization property enables:

- Faster convergence (less conflicting gradient signals)
- Better generalization (local patterns are more robust)
- Interpretability (each neuron "owns" a region of space)

Theorem 5

## Information-Geometric Foundations

🧒

#### Explain Like I'm 5

Imagine you have two ways to measure how "different" two things are:

- 📏 Ruler way: Measure the distance between them (like measuring with a ruler)
- 📊 Information way: Measure how surprised you'd be to see one when expecting the other

The ⵟ-product is special because it connects both ways! It's like having a magic bridge between measuring distances and measuring information.

This means we can use the ⵟ-product with information-theoretic losses (like KL divergence) and it still makes mathematical sense!

Theorem: The ⵟ-product exhibits a duality between Euclidean geometry and information geometry. Specifically, it can be related to KL divergence and entropy-based losses through its kernel structure.

#### 🎯 The Problem This Solves

Many machine learning tasks use information-theoretic losses:

- KL divergence for probabilistic models
- Cross-entropy for classification
- Mutual information for representation learning

Traditional neural networks use Euclidean geometry (dot products, distances), which doesn't naturally connect to information theory. This theorem bridges that gap.

#### 📐 The Mathematics In Depth

The connection comes from the kernel structure. Since the ⵟ-product is a Mercer kernel, it defines a Reproducing Kernel Hilbert Space (RKHS). In this space:

$$\text{ⵟ}(\mathbf{w}, \mathbf{x}) = \langle \phi(\mathbf{w}), \phi(\mathbf{x}) \rangle_{\mathcal{H}}$$

where $\phi$ maps to the RKHS $\mathcal{H}$.

Information geometry studies probability distributions using the Fisher information metric, which can be related to KL divergence. The kernel structure of the ⵟ-product allows us to interpret it in this framework.

#### 💥 The Consequences

🔗

##### Unified Framework

The ⵟ-product bridges Euclidean geometry (for optimization) and information geometry (for probabilistic modeling), creating a unified framework.

📊

##### Compatible with Information Losses

Can be used with KL divergence, cross-entropy, and other information-theoretic losses while maintaining geometric interpretability.

🎯

##### Dual Interpretation

The same operation can be interpreted as either geometric similarity (Euclidean) or information similarity (probabilistic), depending on context.

🔮

##### Rich Theoretical Connections

Connects to maximum entropy principles, variational inference, and other information-theoretic frameworks through the kernel structure.

#### 🎓 What This Really Means

This theorem shows that the ⵟ-product isn't just a geometric operator — it's a unifying bridge between two fundamental mathematical frameworks:

- Euclidean geometry: For optimization, distances, and spatial reasoning
- Information geometry: For probability, entropy, and statistical learning

This duality means NMNs can seamlessly work with both geometric and probabilistic objectives, making them versatile for a wide range of applications.

Theorem 6

## Topological Organization: Neural Fiber Bundles

🧒

#### Explain Like I'm 5

Imagine you're organizing your toys. You could put them in boxes with straight lines dividing them (like a grid), but that's boring and doesn't match how toys naturally group together.

Instead, imagine each toy has a magnetic field around it that attracts similar toys. These fields create swirling, vortex-like territories where similar toys naturally gather.

The ⵟ-product creates these "vortex territories"! Each neuron creates a swirling field around its weight vector, and inputs are classified based on which territory they fall into. It's like having smart, organic boundaries instead of boring straight lines.

Theorem: ⵟ-product classification creates a fiber bundle structure over the input space, where each neuron defines a "fiber" (territory) with vortex-like boundaries. The decision boundaries are smooth, localized, and topologically organized.

#### 🎯 The Problem This Solves

Traditional linear models create unbounded, half-space partitions:

- Decision boundaries are hyperplanes (straight lines/planes)
- Each class occupies an unbounded region
- No natural "territories" or localization

The ⵟ-product creates localized, vortex-like territories with smooth, organic boundaries that better match natural data distributions.

#### 📐 The Mathematics In Depth

In a classification setting with $K$ classes, we have $K$ weight vectors $\{\mathbf{w}_1, ..., \mathbf{w}_K\}$. The decision rule is:

$$\text{class}(\mathbf{x}) = \arg\max_k \text{ⵟ}(\mathbf{w}_k, \mathbf{x})$$

This creates a Voronoi-like partition, but with smooth, curved boundaries instead of straight lines. Each region is:

- Localized: Bounded around the weight vector (due to inverse-square decay)
- Smooth: Infinitely differentiable boundaries (due to C∞ property)
- Vortex-like: Creates swirling patterns due to the interaction of alignment and proximity

The fiber bundle structure comes from viewing each class as a "fiber" over the input space, with the ⵟ-product defining the connection.

#### 💥 The Consequences

🌀

##### Vortex Territories

Each neuron creates a localized "vortex" territory around its weight vector, with smooth, curved boundaries that naturally adapt to data distribution.

📐

##### Geometric Interpretability

The weight vectors act as "prototypes" or "centers" of their territories, making the learned representations geometrically interpretable.

🎯

##### Better Generalization

Localized territories prevent overfitting to distant outliers and create more robust decision boundaries that generalize better.

🔬

##### Topological Insights

The fiber bundle structure provides a rich mathematical framework for understanding how NMNs organize and partition the input space.

#### 🎓 What This Really Means

This theorem reveals the geometric organization of NMNs. Unlike linear models that create arbitrary half-spaces, ⵟ-product neurons create natural territories — regions of space that "belong" to each neuron.

This organization is:

- Localized: Each territory is bounded and centered around a prototype
- Smooth: Boundaries are infinitely differentiable, creating organic shapes
- Interpretable: Weight vectors directly represent the "center" of each territory
- Topologically Rich: The fiber bundle structure provides deep mathematical insights

This is why NMNs learn sharp, coherent prototypes (as seen in MNIST experiments) rather than diffuse, blurry representations — each neuron naturally organizes into a well-defined territory.

Drag the anchor point to interact