James Routley

Break the 64 GB RAM Wall: Run 284B DeepSeek-V4-Flash (< 59 GiB RAM) & 30B MoEs (up to 111 tok/s) at 256K Context on Pure CPU — Zero GPUs Required.

⚡ Quickstart · 🧩 Supported Models · 🖥️ Supported Chips & Platforms · 🏆 Leaderboard · 📄 Preprint PDF (paper/) · 📐 Architecture · ❓ FAQ

284.33B

56.56 GiB

57.61 GiB

44.36 tok/s

60.4–111.6 tok/s

13.1 ms

Real-time Frontier & Agentic MoE inference on the CPU workstation you already own — 100% in RAM, zero NVMe swap:

the machine you have	RAM	recommended model	speed (0 GPUs)	what is going on
a developer laptop / mini-PC	16–32 GB	Nemotron-3.5-30B-A3B (`14.35 GiB`)	25.0–60.4 tok/s	46 Mamba-2 SSM layers + 1.64 MB L3 Stockfish-PVS drafter; 256K ctx in 6 GiB KV
a 64 GB desktop / server	64 GB	DeepSeek-V4-Flash 284B (`56.56 GiB`)	44.4 tok/s pp · 9.1–14.7 tok/s tg	SwiGLU-H2 shaves 20.31 GiB; entire 284B model + 256K ctx sits in RAM (57.61 GiB VmRSS)
a hybrid desktop (CPU + 16 GB GPU)	32–64 GB	Nemotron-30B / Qwen3-30B	80.80 tok/s	hot attention trunk on GPU (`-ngl 99`), cold routed MoE experts in CPU DDR5
an 8-channel workstation (Threadripper)	128 GB+	DeepSeek-284B / Nemotron-30B	111.62 tok/s (30B)	8-CCD AVX-512BW `_1x8` register tiling saturates 8-channel DDR5 memory bandwidth

TL;DR — The 3 Paradigm-Breaking Pillars of llama-modchip:

🐋 284.33B Frontier MoE in a 64 GB CPU Box: Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB on disk / 57.61–58.9 GiB VmRSS at full 256K context (44.36 tok/s prefill, 9.06–10.74 tok/s pure CPU decode, 11.5–14.7 tok/s with PVS) using closed-form Analytical SwiGLU-H2 Second-Order Hessian Compression + Fractal-64 (Q2_0, 3.75 bpw) MLA transcoding + 4-Level Register (ZMM AVX-512BW) + L1D (32 KB) + L2 (1 MB) + L3 (64 MB) Whole-Model Tiling + $L_2$-Quadrature MoE Pruning.

⚡ 30B MoE at GPU Speeds on Pure CPU: Runs NVIDIA-Nemotron-3.5-30B-A3B in 14.35 GiB at 25.0–60.4 tok/s on 16-core Zen 5 (111.6 tok/s on Threadripper, 80.8 tok/s on AMD Vulkan Hybrid, 44.97 tok/s on AWS Graviton3 ARM64) via a 1.64 MB Stockfish-PVS v2 + RouteCrack L2/L3 Drafter (96.4% acceptance) and _MM_HINT_NTA L3 cache isolation (HARPCache).

🤖 Zero-API Claude Code (./modchip claude): Eliminates the 45-second CPU agent prompt re-evaluation bottleneck via /dev/shm Slot-0 KV Warm Restore (18.9 ms on Nemotron-30B / 13.1–32.6 ms on DeepSeek-284B) with all 40 Claude Code tools active.

🐧 Linux & 🍎 macOS (1-Command Automated Setup & Chat)

git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat

⚡ Zero-Compile Direct Run (Pre-Compiled Linux `x86_64` Binary + Hugging Face `.gguf`)

⚠️ Important Compatibility Note (GGML_TYPE_Q2_0 = 40): Our Fractal-64 models use custom block-64 BF16-scale Gray-coded tensors (GGML_TYPE_Q2_0, type ID 40) and custom deepseek4 / Stockfish-PVS kernels. Opening them in unpatched llama.cpp, Ollama, or LM Studio will fail with unknown tensor type 40. Use either ./modchip above or our standalone single-file pre-compiled binary (llama-server-modchip-linux-x86_64, zero .so dependencies) bundled directly inside the Hugging Face repositories and GitHub Release v2.4.0:

# 1. For 30B Nemotron-3.5-A3B (14.35 GiB — runs in 16–32 GB RAM at 25–60.4 tok/s):
hf download Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64-CPU-GGUF \
  llama-server-modchip-linux-x86_64 NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf --local-dir .
chmod +x llama-server-modchip-linux-x86_64
./llama-server-modchip-linux-x86_64 -m NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf -c 262144 -t 16 --port 8080

# 2. For 284B DeepSeek-V4-Flash (56.56 GiB — fits in 64 GB RAM at 9.1–14.7 tok/s):
hf download Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF --local-dir ./dsv4
chmod +x ./dsv4/llama-server-modchip-linux-x86_64
./dsv4/llama-server-modchip-linux-x86_64 -m ./dsv4/DeepSeek-V4-Flash-0731-Fractal-00001-of-00003.gguf -c 32768 -t 16 -fa on --port 8080

🪟 Windows 10/11 (1-Click Hybrid GPU + CPU MoE Pack)

Open windows/ and double-click windows/AVVIA_MODCHIP.bat (or run ./modchip start in Git Bash). It automatically detects your CPU cores, free RAM, and GPU VRAM (nvidia-smi), downloads official llama.cpp CUDA 12.4 / Vulkan binaries, computes the optimal --n-cpu-moe split, and opens the Web Chat UI (http://127.0.0.1:8080).

`./modchip` Command	What It Does
`./modchip chat`	💬 Interactive terminal chat with live `tok/s`, TTFT, and `Stockfish-PVS` acceptance telemetry
`./modchip web`	🌐 Opens the Dark-Mode Web Chat & Live Telemetry HUD (`http://127.0.0.1:8787`)
`./modchip bench`	📊 Runs a 30-second 2-stage speed test and prints a shareable CPU Benchmark Scorecard
`./modchip claude`	🤖 Launches Claude Code CLI (`40 tools`) wired to the local 256K engine (`18.9–32.6 ms` warm restore)
`./modchip eval`	🧪 Runs the reproducible 100-Point Quality, Tool-Calling, Coding & 8.5K Needle Gate
`./modchip quantize`	🗜️ Transcodes `BF16`/`Q4_0`/`IQ2_XXS` GGUFs into `Fractal-64` (`3.75 bpw`) or `Fractal-MXFP4` (`4.25 bpw`)
`./modchip status` / `stop`	🔍 Inspects CPU readiness, `2MB HugePages`, and `/dev/shm` IPC, or stops background services

🏆 Can your CPU run a 284B Frontier MoE in 64 GB RAM? Test in 2 seconds (Zero Downloads):

curl -sL https://raw.githubusercontent.com/synchopate/llama-modchip/main/scripts/check_cpu_readiness.py | python3

👉 Paste your terminal scorecard in a 📊 1-Click GitHub Benchmark Issue or GitHub Discussions to be added to the Official Community CPU Leaderboard!

🧩 Supported Models & Model-Specific Optimizations

Each model family has a dedicated deep-dive optimization guide with exact mathematical proofs, operator-level latency breakdowns, and quality gate benchmarks:

Model Family	Total / Active Params	Disk (`GGUF`) & `256K` RAM (`VmRSS`)	Key Optimizations Implemented	Verified Speed (`16C Zen 5` / Hybrid)	Detailed Guide
🚀 `NVIDIA-Nemotron-3.5-30B-A3B`	`30.2B` / `3.0B`	`14.35 – 17.59 GiB`	• Block-64 `Fractal-64` (`3.75 bpw`) & `MXFP4` (`4.25 bpw`) (`100/100` Quality Gate)	`25.0 – 60.4 tok/s` (`0 GPU`)	📖 Nemotron Guide
🐋 `DeepSeek-V4-Flash-0731-Fractal`	`284.33B` / `13.17B`	`56.56 GiB` (`-20.3 GiB`)	• Analytical 2nd-Order `SwiGLU-H2` MoE + `Fractal-64` MLA Transcoding	`43.2–44.4 tok/s` (Prefill)	📖 DeepSeek-V4 Guide
⚡ `Qwen3-30B-A3B` & `Qwen3-14B`	`30.5B` / `3.3B -> 1.7B`	`13.70 – 17.28 GiB`	• Norm-Preserving Dynamic MoE Top-K (`8 -> 4`, `+38.5%` decode)	`34.80 tok/s` (`0 GPU` Top-4)	📖 Architecture §3.9
🏎️ `NVIDIA-Nemotron-H-8B-128K`	`8.0B` Dense	`4.60 – 8.00 GiB`	• 48-Layer Mamba-2 SSM Constant-Memory Backbone	`100+ tok/s` (`RTX 2080 Ti`)	📖 Nemotron Guide

🖥️ Supported Chips & Processor-Specific Optimizations

Full microarchitectural details and tuning flags for every supported processor and GPU hybrid are documented in docs/HARDWARE_OPTIMIZATIONS.md:

Processor / Platform	Verified Hardware	Hardware-Specific Optimizations Active	Platform Guide & Scripts
AMD Zen 5 (`Desktop / Mobile / Server`)	`Ryzen 9 9955HX`, `9950X`, `EPYC 9005 Turin`	• Single-cycle 512-bit AVX-512 (`VPDPBUSD`, `VPRORVQ`, `VBMI`, `BF16`)	📖 Hardware Guide §2.1
AMD Zen 4 Multi-CCD (`Threadripper / EPYC`)	`Threadripper PRO 7985WX` (`64C/128T`, `8 CCDs`, `256 MB` L3)	• Dynamic sysfs L3 CCD Topology Discovery (`scripts/llama_modchip.py`)	📖 Hardware Guide §2.2
AMD 3D V-Cache + Radeon Vulkan Hybrid	`Ryzen 9 7945HX3D` (`128 MB` L3) + `RX 9070 XT 16GB` (`RADV`)	• Tensor-Role Splitting (`-ngl 99 --n-cpu-moe N --load-mode none`)	📖 Hardware Guide §2.3
Intel / AMD + NVIDIA RTX (`Windows 10/11`)	`Core i9-9900K / 14900K` + `RTX 2080 Ti 11GB / 3090 / 4090`	• Single-Cycle AVX2 `vpsllvd + vpmaddwd` Sub-Scale Fusion	📖 Windows Guide
Apple Silicon (`M1 / M2 / M3 / M4`)	`Apple M4 Max` (`16C+40C`, `64 GB`), `M4 Pro` (`14C+20C`, `24 GB`)	• Zero-Copy Unified Memory Metal Shaders + ARM64 NEON/AMX (`238.9 GB/s` DRAM BW on M4 Max)	📖 Apple Silicon Guide
ARM64 Cloud & Edge (`AWS Graviton / Ampere`)	`AWS Graviton3 m7g.2xlarge` (`8 vCPU Neoverse-V1`, `32 MB` L3)	• ARMv8.4-A DotProd (`SDOT`) + 256-Bit SVE + 1-vCPU-per-core pinning (`-t 8`)	📖 Hardware Guide §1
AMD Zen 4 Compact Mini-PC (`Linux`)	`Ryzen 7 8745H / 8745HS` (`8C/16T`, `16 MB` L3 + `Radeon 780M`)	• AVX-512 VNNI + `16 MB` L3 window auto-tuning (`19.48–23.38 tok/s` on `96 GB` DDR5-5600)	📖 Fedora Guide

📊 Headline Benchmarks: Pure CPU & Hybrid Summary

1. `DeepSeek-V4-Flash 284B` (`13.17B` Active) on a `64 GB` Zen 5 CPU (`AMD Ryzen 9 9955HX`, `0 GPUs`)

Full operator-level profiling and mathematical proofs in docs/DEEPSEEK_V4_OPTIMIZATIONS.md:

`DeepSeek-V4-Flash-0731-Fractal` (`56.56 GiB` @ `256K` Context)	Prefill (`pp512`)	Pure Decode (`N=1`)	With `Stockfish-PVS`	Prob Cosine Sim	KL-Divergence ($\Delta$ PPL)	Top-1 / Top-3 Match
Standard `IQ2_XXS` (`76.87 GiB`, Unpatched)	🔴 OOM on 64GB	`5.91 tok/s` (`169.3 ms`)	N/A (Rollback Bug)	`1.00000`	`0.0000` (`0.00%`)	`100% / 100%`
1. `v2.4.0` Lossless (`Top-6` All Layers, Default)	`43.16 – 44.36 tok/s` (`+77.1%`)	`9.06 – 9.95 tok/s`	`11.5 – 14.7 tok/s`	`1.00000`	`0.0000` (`0.00%` Bit-Exact)	`100% / 100%`
2. `v2.4.0` Sweet-Spot (`Top-5` on `il >= 3`, $L_2$ Quadrature)	`43.16 – 44.36 tok/s`	`9.68 – 9.87 tok/s`	`12.2 – 15.1 tok/s`	`0.99714`	`0.0042 nats` (`+0.42%` PPL)	`95.8% / 100%`
3. `v2.4.0` `Top-5 L2` + Gate-First SwiGLU (`80/128` $L_2$)	`43.16 – 44.36 tok/s`	`10.15 tok/s` (`98.5 ms`)	`12.8 – 15.6 tok/s`	`0.99385`	`0.0531 nats`	`100% / 100%`
4. `v2.4.0` Turbo `10+` (`Top-4 L2` + Gate-First SwiGLU)	`43.16 – 44.36 tok/s`	`10.12 – 10.74 tok/s` (`93.2 ms`)	`13.0 – 16.0 tok/s`	`0.99126`	`0.0281 – 0.0694 nats`	`100% / 100%`

2. `NVIDIA-Nemotron-3.5-30B-A3B` (`3.0B` Active) at `256K` Context (`AMD Ryzen 9 9955HX`, `0 GPUs`)

Full ablation tables and 100/100 Quality Gate report in docs/NEMOTRON_OPTIMIZATIONS.md:

Quantization Profile	Disk Size	Idle `VmRSS` (`256K`)	Quality Gate (`100` pts)	Prefill (`pp512`)	Base Decode (`N=1`)	`Stockfish-PVS v2 (M=7)` Refactoring
`Baseline-Q4_0` (Official GGUF)	`17.59 GiB`	`21.41 GiB`	`100.0 / 100.0`	`241.0 – 306.8 tok/s`	`19.54 tok/s`	`40.91 avg / 45.43 peak tok/s`
`Fractal-MXFP4` (`4.25 bpw`)	`15.95 GiB`	`17.46 GiB`	`100.0 / 100.0`	`241.0 – 306.8 tok/s`	`26.08 tok/s` (`+33.5%`)	`48.39 avg / 60.40 peak tok/s` (`80.8` Vulkan)
`Fractal-64-BF16` (`3.75 bpw`)	`14.35 GiB`	`15.40 GiB` (`-6.01 GiB`)	`100.0 / 100.0`	`241.0 – 306.8 tok/s`	`25.02 tok/s` (`+28.0%`)	`48.39 avg / 60.40 peak tok/s` (`96.4%` acc)

🏆 Community CPU & Hybrid Leaderboard

Run ./modchip bench (or windows\BENCHMARK_PC.bat on Windows) and ➕ Submit Your Scorecard (1-Click GitHub Issue)!

Hardware Platform (`CPU / SoC / Hybrid`)	Cores / Cache	Memory	Prefill (`tok/s`)	Decode — Prose / C++20	Decode — `Stockfish M=7` Burst	Config & Contributor
AMD Threadripper PRO 7985WX (`Zen 4`)	`64C/128T` (`8 CCDs`, `256 MB` L3)	`256 GB` 8-Ch DDR5-6000 (`0 GPU`)	`386.00` (`-t 32`)	`73.84 – 80.14 tok/s` (`70.0%` acc)	`111.62 tok/s` warm / `80.14` cold (`-t 32` 4C/CCD)	`@aifartist` (Issue #1)
AMD Ryzen 9 7945HX3D + RX 9070 XT (`Hybrid`)	`16C/32T` (`128 MB` 3D V-Cache) + `16 GB` Vulkan	`94 GB` DDR5 + `16 GB` VRAM	`563.00` (256K ctx)	`75.10 tok/s` (`MXFP4` 15.2GB)	`80.80 tok/s` (Vulkan `SSM_SCAN` + `MXFP4` MoE)	`Community (RADV/Vulkan)` (`FAQ.md`)
AMD Ryzen 9 9955HX (`Zen 5`)	`16C/32T` (`2 CCDs`, `64 MB` L3)	`64 GB` DDR5 (`0 GPU`)	`241.0` (Peak `306.8`)	`25.02 – 34.5 tok/s` (`Fractal-64` / `MXFP4`)	`48.39 avg / 60.40 peak` (`len 6.90`)	`@synchopate` (`docs/NEMOTRON_OPTIMIZATIONS.md`)
Apple M4 Max (`16C CPU + 40C GPU`)	`16C` (`ARM64 NEON` + `40C Metal`)	`64 GB` LPDDR5X-8533 (`238.9 GB/s` BW)	`~451.0` (Audit)	`49.2 – 57.7 tok/s` (Prose / Code)	`70.9 – 120.0 tok/s` (Agent / Refactor)	`@jbeckman1974` (Issue #8)
Apple M4 Pro (`14-Core SoC`)	`14C CPU + 20C Metal`	`24 GB` Unified (`0 Swap`)	`256.0 – 549.0`	`47.0 – 48.0 tok/s` (128K)	`49.0+ tok/s`	`@jarorosFork` (`apple-silicon/`)
AWS Graviton3 `m7g.2xlarge` (`ARM64 Neoverse-V1`)	`8 vCPU` (`32 MB` L3, `SVE-256`)	`31 GB` DDR5 (`0 GPU`)	`101.67` (`-t 8`)	`41.05 tok/s` (C++20)	`44.97 tok/s` (Refactoring)	`@gerchik` (Issue #2)
AMD EPYC 7402P (`Zen 2 Rome` · `AVX2-only`)	`24C/48T` (`8 CCDs`, `128 MB` L3)	`128 GB` 8-Ch DDR4 (`0 GPU`)	`158.73` (`-t 24`)	`27.29 tok/s` (C++20, `567 ms` TTFT)	`31.22 tok/s` (Cold Run 1, `66.7%` acc)	`@AdamKlob` (Issue #6)
AMD Ryzen 9 9950X (`Zen 5 Desktop`)	`16C/32T` (`2 CCDs`, `64 MB` L3)	`64 GB` DDR5-6800 (`0 GPU`)	`135.64` (v2.1 `Q4_0`)	`20.78 tok/s` (C++20, `526 ms` TTFT)	`24.67 tok/s` (Cold Run 1, `66.5%` acc)	`@nadams` (Issue #5)
AMD Ryzen 7 8745H w/ Radeon 780M (`Zen 4`)	`8C/16T` (`16 MB` L3, `AVX-512`)	`96 GB` DDR5-5600 (`0 GPU`, Arch)	`128.81` (`-t 8`)	`19.48 tok/s` (C++20, `686 ms` TTFT)	`23.38 tok/s` (`69.8%` PVS acc)	`@arch-btw` (Issue #9)
Windows 11 Hybrid (`i9-9900K + RTX 2080 Ti 11GB`)	`8C/16T` (`16 MB` L3) + `11 GB` CUDA	`16 GB` DDR4 + `11 GB` GDDR6	Auto CUDA 12.4	`50–58 tok/s` (`14B`) / 30B Hybrid	`100+ tok/s` (`Nemotron-H-8B` 128K)	`windows/`
AMD Ryzen 7 8745HS (`GEEKOM A8`)	`8C/16T` (`16 MB` L3)	`64 GB` DDR5-5600	Verified Fedora 44	Verified (`ngram`)	Verified (`CPU` & `780M`)	`@jarorosFork` (`fedora/`)
🐋 `DeepSeek-V4-Flash 284B` on Ryzen 9 9955HX	`16C/32T` (`2 CCDs`, `64 MB` L3)	`64 GB` DDR5 (`58.9 GiB` RSS @ `256K`)	`43.16 – 44.36 tok/s` (`32.6 ms` Slot-0)	`9.06 – 10.74 tok/s` (`13.17B` active)	`11.50 – 14.69 tok/s` (`94.5%` PVS acc)	`@synchopate` (`docs/DEEPSEEK_V4_OPTIMIZATIONS.md`)

🛠️ Click to expand: Repository Structure & Manual CMake Build Instructions

🏢 Enterprise & Hardware Sponsorships

llama-modchip demonstrates that modern multi-core CPUs and compact Mini-PCs with 64–128 GB of DDR5 RAM can run 284B Frontier MoEs (DeepSeek-V4-Flash) and 30B Hybrid MoEs (Nemotron-3.5-30B-A3B) locally at full 256K context—eliminating $30,000+ multi-GPU hardware requirements for air-gapped enterprise code intelligence, security auditing, and autonomous agent fleets.

We actively partner with hardware manufacturers, cloud silicon teams, and AI research labs to build and validate architecture-specific kernel patches:

🖥️ CPU & Workstation OEMs (AMD Ryzen AI / Threadripper / EPYC, Minisforum, Framework, System76, Apple, Intel): Sponsor a physical test unit or workstation to receive dedicated L1D/L2/L3 cache-tiling & SIMD kernels and permanent verification on the Official README CPU Leaderboard.
☁️ Cloud & ARM64 Silicon Teams (AWS Graviton4, AmpereOne, Google Axion, Azure Cobalt): Sponsor cloud compute credits to validate SVE2 / SME / multi-socket NUMA kernels.
🤝 Enterprise Support & Custom Model Transcoding: Reach out via GitHub Issues or GitHub Sponsors to sponsor custom SwiGLU-H2 + Fractal-64 model transcoding and on-premise deployment.

Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU