Ripple-65B Preview: U.S. Open-Weight SOTA on Agentic Coding

Ricursive Intelligence is a full-stack artificial intelligence lab building self-improving systems by co-optimizing hardware, software, and model.

To power our chip design platform, we trained a language model from scratch that sets a new U.S. open-weight SOTA on agentic coding. We optimized the model architecture for our training and serving platforms, driving substantial efficiency gains.

Today, we release Ripple-65B Preview, a general-purpose variant of the proprietary model that powers our AI chip design platform. The model is pretrained on 12 trillion tokens with support for up to 1M context length. Ripple-65B Preview delivers strong results on agentic coding, achieving U.S. open-weight state-of-the-art (80.52) on TerminalBench 2.1 and surpassing significantly larger US open-weight models such as Reflection Beam 501B (80.1), Meta Muse Spark Open 816B (76.4), Thinking Machines Inkling 975B (63.8), and NVIDIA Nemotron 3 Ultra 550B (56.4).

Ripple’s dense model architecture provides strong reasoning performance with low-latency token generation. The model’s small parameter envelope also enables it to run efficiently on local devices such as NVIDIA DGX Spark, which feature high compute FLOPs with low memory bandwidth relative to datacenter accelerators.

Ripple-65B Preview is available on Hugging Face, Together AI, GMI Cloud, and OpenRouter. Fireworks goes live later today.

LLM pretraining is legendarily expensive. In 2023, OpenAI CEO Sam Altman said that GPT-4 training costs exceeded $100 million. In the following year, Dario Amodei said that frontier AI models can cost up to $1 billion to train, projecting that costs would rise to $10-100B by 2026 to 2027 for pretraining alone.

In contrast, we developed and trained Ripple-65B Preview for only $32M, which is one-tenth the cost of the previous open-weight SOTA on agentic coding, as estimated to the best of our knowledge.¹ Note that $32M is our full R&D cost starting from scratch, including pretraining ladder development, data mixing, evals, debugging, synthetic data generation, posttraining recipe development, and the final hero pretraining and posttraining run.

Rather than investing in a brute-force RL climb, we carefully iterated on our SFT and RL recipes with a focus on compute efficiency. We opted not to buy any posttraining data, instead devising our own synthetic data pipelines, quality filtering, and coverage metrics. 

Our mixed RL recipe uses truncated importance sampling, score centering, and linear length penalty with high-quality long-horizon synthetic environments. The RL tasks are tuned for our model capabilities, and we continuously added harder tasks as the model improved.

Ripple-65B Preview’s architecture was designed to be trained and served efficiently on NVIDIA hardware. A key objective was to make design decisions which reduce pretraining and posttraining infrastructure complexity while still enabling strong reasoning and agentic performance. Therefore, we chose to train a modernized dense Transformer model rather than use Mixture-of-Experts (MoE) like other labs. This decision quickly paid off, and we were able to train a strong model from scratch while maintaining a solid MFU (40%) throughout pretraining and posttraining, even with simple sharding, no custom kernels, and no quantization.

Ripple-65B Preview uses a thin-and-deep model aspect ratio, with a narrow d_model (5120) and 80 total layers. To improve reasoning performance, we overprovision the number of attention heads while increasing the SWA-to-global layer ratio and reducing SWA window size to compensate for the increased cost. In addition, we added FFN activation clipping, gated attention, global NoPE, and value residuals.

We optimized Ripple’s pretraining recipe through a series of careful model laddering experiments. 

Our starting point was Olmo 3’s model ladder, from which we tested interventions at small scale with a focus on improving coding capabilities. Our v1.0 ladder implemented StepFun batch size and learning rate scaling laws, and v1.1 implemented our new ladder which smoothly scales architectural dimensions and model depth. Ladder v1.2 switched from the post-norm baseline to pre+post norm, which was neutral at small scale but showed improvements starting at 4B. Ladder v1.3 implemented sigmoid gated attention. Ladder v1.4 (2-stream mHC) showed good results at smaller scale but did not improve at 4B scale, and was therefore rejected. Ladder v1.5 switched to global NoPE and added a final LR cooldown to zero, and ladder v1.6 added value residuals. In the final version, we switched to our custom pretraining dataset with substantially upsampled code. 

Together, the compounded changes improved compute efficiency on code as measured by dolma_stack CE by up to 16x at 4B model scale. In other words, our final Ripple model ladder can be used to achieve similar code performance to the Olmo 3 ladder with 16x less compute. Note that the slope of the Olmo 3 ladder baseline is shallower than ladder v1.0+, which shows that the compute efficiency improvement will increase with additional scale.

We built Ripple-65B Preview to begin closing the loop between models and the hardware that fuels them. An internal variant of this model powers our AI chip design platform, and we look forward to seeing the capabilities that emerge as we scale toward the frontier.

We were fortunate to build on top of existing open-source contributions, including those from the Olmo team at the Allen Institute for AI (AI2), Prime Intellect, RadixArk, and NVIDIA. A strong open-source ecosystem is also self-improving, where contributors strengthen the ecosystem, which in turn draws more contributors. 

At Ricursive, we are selectively hiring exceptional researchers and engineers as we push the frontier of full-stack AI from model to silicon. If you’re interested, check out https://www.ricursive.com/careers.