Run the biggest language model you can on the smallest hardware you own.
TV boxes, old tablets, single-board computers and routers all have a CPU, some RAM and flash storage that sit idle most of the day. smolmaxxing finds the biggest model each of them can really run, then pushes it further with custom kernels, careful memory use and weights streamed from flash. Every number in this repo was measured on the device itself.
| 📺 Xiaomi TV Box S (3rd Gen) | 📱 iPad Air (2013) | |
|---|---|---|
| Chip | Amlogic S7D, 4× Cortex-A55 | Apple A7, 2 cores |
| RAM | 1.9 GB (about 1 GB usable) | 1 GB (about 700 MB usable) |
| OS | Android 14, no root | iOS 12.5.7, jailbroken |
| Everyday model | Qwen3.5-2B at 7.7 tok/s | Bonsai-1.7B (1-bit) at 24-26 tok/s |
| Biggest model | Ling-mini-2.0, 16B, at 5.1 tok/s | Ling-mini-2.0, 16B, at 2.6-4.5 tok/s |
| Full results | docs/tv-box.md | ipad-air/README.md |
Both devices run 8B and 16B mixture-of-experts models on about 1 GB of RAM by keeping only part of each model in memory and reading the rest from flash as it's needed.
More devices (phones, single-board computers, routers) are coming. The techniques aren't tied to one box: the 1-bit kernel needs only basic NEON, and the flash streaming works on any Linux or Android device with more model than RAM.
All of it is a patch on llama.cpp. The main ideas:
- Kernels built for the chip. A 1-bit lookup-table kernel, repacked 2-bit and ternary formats for the Cortex-A55, and versions of them for the A7, which has no dot-product instructions.
- Less RAM per model. Most of a small model's vocabulary is rarely used, so we drop those output rows. On the TV, a "headless" mode also stops the launcher and Google services while a model runs.
- Experts paged from flash. A mixture-of-experts model only uses a few experts per token. We keep the popular ones in RAM, read the others from flash on demand and prefetch the next layer's. This is how 8B and 16B models run on 1 GB.
- Streaming for dense models bigger than RAM, at the flash's full speed.
docs/how-it-works.md has every technique with its measured gain.
The quickest start is a ready-made model file from Hugging Face. They need the patched llama.cpp: stock llama.cpp can't load them.
- TV box or another ARMv8.2 Android device: docs/tv-box-setup.md
- Jailbroken iPad on iOS 12: ipad-air/README.md
- All model files and how to rebuild them: docs/models.md
| Page | What's in it |
|---|---|
| docs/tv-box.md | device #1: every model run on the TV box, speed and quality |
| ipad-air/README.md | device #2: the iPad Air results, setup and scripts |
| docs/how-it-works.md | the kernels, memory tricks, expert pager and streaming engine |
| docs/models.md | downloads for both devices, rebuild targets and checksums |
| docs/lessons.md | what we learned, including what didn't work |
| EXPERIMENT.md | the full lab notebook: 160 experiments, failures included |
| Path | What |
|---|---|
patches/ |
every change to llama.cpp for the TV box (the iPad's patch is in ipad-air/patches/) |
tools/ |
model rebuilding, GGUF rewriting, benchmarks and kernel checks |
tv-llm-ui/ |
the live token-stream chat page and its launcher |
ipad-air/ |
device #2: iOS 12 build, smolchat, deploy and benchmark scripts |
bench/ |
every chat benchmark (results.jsonl) and the raw logs |
data/ |
vocabulary lists and evaluation texts |
tiny-llm-web/ |
the first experiment: a 260K-parameter story model |
zigcc/ |
zig wrappers for cross-compiling static aarch64 binaries |
MIT. The llama.cpp patches are derivatives of llama.cpp (MIT). Model weights keep their own licenses, and the evaluation texts are Wikipedia excerpts (CC BY-SA 4.0).
Built on llama.cpp and llama2.c. Models from Qwen, unsloth (GGUFs and importance matrices), PrismML (Bonsai), inclusionAI (Ling-mini-2.0), tablinum (Moon-Beam), bartowski and mradermacher (GGUFs), Google (Gemma) and stories260K. The iPad is jailbroken with Amethyst. The banner uses Archivo (SIL OFL) and is rendered by tools/banner/render.sh. The 🤏 logo is the pinching-hand emoji from Google's Noto Emoji (Apache 2.0).