July 8, 2026
I came into Recurse Center with the goal of becoming a proficient user of open source LLMs, and some vague idea of hooking an open source model up with some kind of large-scale agent management framework like GasCity and seeing if I could get the system to make something useful.
I was inspired by Vicki Boykis' recent newsletter issue, Running local models is good now, in which she describes her experience successfully getting a local open weight model to serve as her primary agentic coding tool. I love the idea of getting the benefit of an AI coding assistant without bleeding money to Anthropic. My first week at RC, I spent about a day wrestling with the setup tutorial that Vicki links to but ran into one yak-shaving problem after another.
No shade to the tutorial – it's perfectly clear and the steps work perfectly. It's just that, when I followed the tutorial, I indeed got LM Studio up and running, I successfully downloaded the recommended Gemma model from HuggingFace, I hooked it up with Pi as my coding harness... And then watched my computer, with its poor sad 8GB of RAM, become completely unresponsive as soon as I started trying to use my new setup. Flipping back to Vicki's post, I saw that her computer has 64GB of RAM. Fancy!
Ok, fine, maybe local models are not in my immediate future. But luckily, Recurse Center has a partnership with Modal, where we can get some free credits to run AI projects in their cloud. My next idea was to host an inference server on Modal and have Pi, running on my machine, make calls out to that server. Modal has a tutorial and example script for spinning up an inference server, which I downloaded and ran.
Again, props to the folks at Modal – the tutorial script worked smoothly out of the box. I didn't change any of the defaults, just spun up the inference server and pointed Pi at it. And promptly burned $2.48 in Modal credits writing a Python function to generate the next number in a Fibonacci sequence.
Let's back up.
In his tutorial
(How to run a local coding agent with Gemma 4 and Pi),
Pat Loeber recommends using Gemma 4 26B A4B (Q4_K_M). He describes it as a
“mixture of experts” model, and suggests downloading the quantized GGUF
version. I have no idea what any of those words mean.
Vicki Boykis recommends gemma-4-12b-qat, noting that it is “more
recent and smaller and faster, without much sacrifice in accuracy”. Sounds good to
me, I like smaller, faster, and more accurate. Unfortunately, before I even tried to
download it, LM Studio informed me that this model is too large to run successfully on
my computer. Pat's model is much larger, an obvious non-starter.
The model that I did semi-successfully run locally was gemma-4-e2b. Is that a good model?
A bad model? If it had not completely frozen my computer, would it have done anything actually
useful? How am I to know?
| Model | Size |
|---|---|
| Gemma 4 26B A4B (Q4_K_M) | 15.64 GB |
| gemma-4-12b-qat | 7.15 GB |
| gemma-4-e2b | 5.94 GB |
Modal's inference tutorial script recommends the same model as Pat,
google/gemma-4-26B-A4B-it, but runs it on their third-most-expensive GPU,
an H200. Since I wasn't excited to set quite that much money on fire quite that fast, I
tried to sub out Vicki's recommended model.
Here's where it got exciting. I tried to modify the Modal script to use
gemma-4-12b-qat and a much cheaper GPU, only to discover that vLLM, the
model serving framework that the tutorial relies on, doesn't support the Gemma 4 12B
models
(open bug report here).
At this point, I had a series of very frustrating chats with Claude, in which I tried to select a different model, one that would effectively trade off low cost with adequate performance as a coding agent. But ultimately, I felt like I just really had no idea what was going on. What's the difference between 12B vs 26B vs E2B? What does QAT mean? Why would I pick a model from the Gemma 4 family vs Claude's suggestion of GPT-OSS-20B?
One of my goals during this batch at Recurse Center is to stop feeling like an imposter when it comes to large language models. Not so much how to use them, which honestly, I feel fine about, but to have a better mental model of what's going on under the hood, and how the different models vary. Maybe a good way to express this goal is, “I'd like to be able to read a model card on HuggingFace and understand what all the words mean, both individually and together.”
A fellow Recurser recommended Andriy Burkov's The Hundred-Page Languages Models Book as a good way to go all the way back to basics. I'm about halfway through Chapter 2 and finding it to be a fantastic read: clear, accessible, and the right mix of theory and implementation. I'll share more thoughts on what I've read so far, and what I've started to implement as a result, in a future post.