Language Experiments — Neural Matter Networks
Description: AetherGPT language modeling experiments, training dynamics, and performance on text generation tasks.

← Back to Theory Experiments

# Language Experiments

🧒

#### Explain Like I'm 5

You know how when someone starts a sentence, you can guess the next word? 🤔

- 📖 "Once upon a ____" → You'd guess "time"!
- 🤖 GPT is a computer that learned to do this by reading LOTS of books
- ✨ AetherGPT does the same thing, but using our ⵟ-product magic!

The cool part? AetherGPT makes fewer mistakes and uses less memory! 🎉

#### 🧠 AetherGPT Architecture

AetherGPT is a GPT-2 variant that replaces standard attention and MLP blocks with ⵟ-based components:

👀

##### ⵟ-Attention

Geometric query-key matching using alignment + proximity instead of pure dot product.

🔢

##### NMN Feed-Forward

Replaces MLP + GELU with NMN layers that have intrinsic non-linearity.

🚫

##### No LayerNorm

Self-regulation eliminates the need for normalization layers entirely.

📊

##### Same Parameters

124M parameters, matching GPT-2 small for fair comparison.

#### 📈 Training Dynamics

##### 📊 Training Loss Comparison

━━ GPT-2 (baseline) ━━ AetherGPT

#### 📊 Results Summary

Metric | GPT-2 | AetherGPT | Improvement
--- | --- | --- | ---
Validation Loss (BF16) | 3.03 | 2.69 | ↓ 11.2%
Validation Loss (FP32) | 3.05 | 2.78 | ↓ 8.9%
Peak Memory | Baseline | 15-25% less | ✓
Throughput | Baseline | ~4% slower | ~
Normalization Layers | Required | None | Simpler

#### 🔬 Key Observations

📈
BF16 Performance Boost: AetherGPT shows larger improvements in BF16 (11.2% vs 8.9% in FP32), suggesting that the geometric operations benefit from lower-precision floating point — possibly due to the bounded nature of the ⵟ-product.
💾
Memory Efficiency: The 15-25% memory savings come from not needing to store activation values for ReLU/GELU during backward pass. This can enable larger batch sizes or longer context lengths on the same hardware.

#### 🔧 Training Configuration

Hardware & Setup

- Dataset: OpenWebText (subset)
- Context Length: 1024 tokens
- Batch Size: 256 (effective)
- Learning Rate: 6e-4 with cosine decay
- Hardware: 8× A100 GPUs