Purely Rotational Foundation Models & Geometric Optimization

MatterWave AI researches and builds Geometric Rotation Transformers (GRT) — a foundation architecture that replaces dense linear weight matrices (nn.Linear) in attention (Q, K, V, O) and feed-forward (expand, contract) blocks with cascades of content-modulated 2×2 Givens rotations on SO(D) interleaved with butterfly permutations.

Build and Scale with Geometric Rotation Transformers

Because every transform in the GRT residual path is an orthogonal rotation cascade, hidden-state norms are preserved by construction rather than by post-hoc rescaling. GRT checkpoints execute deterministically with bit-for-bit greedy generation parity across PyTorch (CUDA, ROCm, Intel XPU), vLLM, and llama.cpp (CPU AVX2, Vulkan, and ROCm HIP).

Starting from our pretrained GRT-10M and GRT-20M NeurIPS submission checkpoints (trained with first-generation PairAdam) and our 24-hour Qwen 3 distilled GRT-30M base checkpoint, replacing the optimizer with HuygensSymplecticAdam and training across a multi-round curriculum delivers substantial improvements across the 10-task SLM-Bench evaluation suite (13,144 held-out multiple-choice examples across BoolQ, ARC-Easy, ARC-Challenge, OpenBookQA, PIQA, WinoGrande, HellaSwag, QNLI, and CommonsenseQA, plus 500 held-out GSM8K mathematical reasoning problems):

• GRT-10M-Curriculum (8.7M total / 2.2M backbone) — Lifts 9-task multiple-choice accuracy from 37.61% (PairAdam NeurIPS Baseline) to 41.71% (Single-Pass) to 45.74% (Multi-Round Curriculum, +8.13%), while GSM8K 4-way verification reaches 96.60% (perplexity reduced 5.77× from 145.85 to 25.28).
• GRT-20M-Curriculum (16.7M total / 10.3M backbone) — Lifts 9-task multiple-choice accuracy from 38.63% (PairAdam NeurIPS Baseline) to 45.03% (Single-Pass) to 48.17% (Multi-Round Curriculum, +9.54%), while GSM8K 4-way verification reaches 98.20% (perplexity reduced 7.86× from 60.12 to 7.65).
• GRT-30M-Curriculum (29.3M total / 9.74M backbone) — Lifts 9-task multiple-choice accuracy on the 24-hour Qwen 3 distilled base from 39.61% (PairAdam Baseline) to 43.86% (Single-Pass) to 48.07% (Multi-Round Curriculum, +8.46%), while GSM8K 4-way verification reaches 99.20% (perplexity reduced 3.68× from 27.99 to 7.60).

System-1 Cognitive Telemetry & Multimodal Specialists

Beyond base causal language modeling, the GRT-30M backbone powers specialized reasoning routers, retrieval encoders, and unified multimodal checkpoints:

• Open-S1-GRT-30M — 5-head System-1 cognitive telemetry router (action routing, escalation, loop guard, specification support, and verification gating) reaching 95.83% 5-family macro accuracy (92.00% MC_SHUFFLE_V2 transfer with HuygensSymplecticAdam; ECE = 0.0095 with SymplecticAdam).
• GRT Architecture Whitepaper (PDF) — Full mathematical formulation, kernel design, and multi-backend verification report.

Explore the MatterWave AI Model Suite

Browse our open-weights repositories by category: