A Year of Fast Apply — The Path to 10k Tokens per Second

A year ago today, we released our first Fast Apply model publicly. Since then, we’ve learned a lot about how to fine-tune small, specialized models for code-specific tasks.

Today, we’re open-sourcing what we've learned in training this series of models — dataset curation, training methods, and inference techniques that led to Relace Apply 3, our best model yet, capable of running at 10k+ tokens per second while maintaining state-of-the-art accuracy.

Error rate comparison on 500 randomly sampled production merge requests. For breakdown of error categories, see section on evaluating merges.

The Problem

When making edits to a codebase, it doesn't make sense to have an expensive LLM regenerate all the unchanged, preexisting code.

Editing a thousand line file (~10k tokens) by rewriting it from scratch with Claude 4.5 Sonnet takes over 100 seconds and costs at least $0.18. For coding agents, this is infeasible from a product perspective.

The solution is to have the frontier model output a diff that minimally expresses the changes to make (i.e. the hard tokens), and use a lightweight algorithm to efficiently apply the diff back into the preexisting code. Not only does this save on cost, but if the merging algorithm is fast, you also significantly speed up end-to-end generation time.

For a long time, LLMs were incapable of reliably producing diff formats that were mergeable by a fixed algorithm like string replace or UDiff.

Cursor pioneered a flexible workaround to this problem — let the frontier model produce "lazy" diffs and use a small, fast apply model as the merging algorithm. However, Cursor never made their model available for other companies to use outside of the IDE.

We decided to train our own apply model that anyone could use.

Why Use an LLM as the Merge Algorithm?

In practice, frontier models can produce a wide variety of pathological diffs. Any closed-form algorithm you write to perform the merge will be susceptible to edge cases. In agentic settings, where there is often no human oversight, these errors compound and produce incorrect code.

Example of a pathological initial_code, diff pair that is hard to address with a closed-form merging algorithm.

In the code above, the user starts with a function called handler. The intent of the diff on the right is to rename the handler function to messageHandler and add a parameter for the name column in the database query.

It would be difficult to write an algorithm that anticipates these two steps. A standard merging algorithm would likely just add a new function called messageHandler, duplicating the original function.

The advantage of using an LLM as the merge algorithm is that it can flexibly infer the intent of the diff. Pretraining on trillions of code tokens bakes in pattern recognition robust to the many edge cases you see in production.

Also, by splitting the task into two steps — diff generation and merge — you can use a much smaller, fast model to focus entirely on the merge. This separation of work based on difficulty is a pattern we exploit a lot at Relace to improve overall performance of coding agents.

To achieve good results without needing to pretrain a model, we fine tune off-the-shelf small models on a high quality dataset that matches the distribution in production.

Producing the Dataset

A training set for fast apply contains three components: initial_code, diff, and merged_code. The initial code and diff are passed in as model inputs, and the merged code is the output we train the model to produce.

For fine tuning, you need a small set of high quality examples. We found the size of the dataset to be much less important than the diversity and quality of merge data. Our first model was trained with only 30k data points, and we saw marginal gains beyond 100k data points.

Inputs

Early on, Kortix AI released an open-source dataset by post-processing data scraped from public GitHub repos. Given some initial code, they prompt a frontier model like Claude to (1) come up with a change to make and (2) produce the "lazy" diff for it.

This turns out to be the wrong approach, as it doesn't reflect the actual distribution of data in production environments. The rich variety of edge cases in diffs comes from context overload. The LLM must adhere to instructions in long system prompts while simultaneously inferring intent from noisy conversations with non-technical users.

To get high-quality, complex merges with plenty of edge cases into the training set, we partnered with prompt-to-app companies. We took snapshots of the real context for LLM coding tasks and reran them with additional instructions to produce the "lazy" edits. This allowed us to sample directly from the true distribution of initial_code and diff that would be seen in production.

Output

To generate the correct merged_code, we use distillation with rejection sampling. The idea is to feed the initial code and diff into a well-prompted frontier “teacher” model, guided by a set of rules for how merges should be handled.

This approach lets you produce a large amount of candidate data quickly, but it’s crucial to filter out the teacher’s mistakes. When done correctly, the trained student model can actually end up outperforming the teacher.

However, filtering correctly is hard. We developed a multi-stage process to build a high quality LLM-as-a-judge that could scale up to thousands of datapoints.

Evaluating Merges

We began by manually reviewing 500 randomly sampled examples to create a set of ironclad ground truths. For our first Apply model, these datapoints were fully synthetic, but we later repeated the process using real production data from earlier iterations of the hosted model.

To streamline the process while maintaining quality, we built our own internal evaluation tool: a Git-style diff viewer with annotation tools for categorizing merge outcomes.

A screenshot of Relace's internal merge evaluation tool.

Even with this tool, it took us over 40 hours to painstakingly ensure 100% correctness. Patterns started to emerge in the process, and we broke down merges into 6 categories:

1. Correct Merge: Model correctly implemented the intent of the diff.
2. Non-functional Error: Model implemented the intent of the diff except for inconsequential details such as variation in comments and formatting (e.g. function definitions in different orders).
3. Smoothing: Model fixed an error incorporated by the diff that would have resulted in uncompilable or broken code.
4. Functional/Merge Error: Model did not carry out the intent of the diff (e.g. omitting parts of code and inserting code in incorrect places).
5. Hallucination Error: Model added or changed code not specified by the diff (may be unsuccessful attempts at smoothing).
6. Truncation Errors: Model performed an incomplete merge (with parts of final code cut out) leading to uncompilable code.

Note: Categories 1-3 are considered correct merges, while 4-6 are considered incorrect. We keep the six-class system for more nuanced evaluations, but collapse to a binary classification when creating the LLM judge.

Aside on Smoothing

We often found that frontier LLMs produced diffs leading to slightly incorrect code. The most common example is when the diff uses a new library in the code without actually importing it.

This raised a question: should a fast apply model strictly follow the diff, or fix the error? In practice, we found customers building coding agents prefer that the model auto-corrects small mistakes to reduce friction for end users.

The hallucination category accounts for cases where the apply model overstepped and introduced more incorrect code to try and fix errors.

LLM-as-a-Judge

Once we had this set of 500 categorized merge examples we could trust, the next step was to align an LLM-as-a-judge to our human-annotated seed dataset. It would be completely infeasible to hand-evaluate enough code merges to create the full training dataset of 100k+ examples.

The LLM-as-a-judge technique exploits the generation/verification gap. i.e. It's easier for an LLM to evaluate whether an answer is correct than to actually generate the answer. Still, the LLM often makes mistakes, and it's important to align it properly with a human evaluation first.

As with any binary classification task, there are two error modes:

• False positive - a bad merge is incorrectly classified as good.
• False negative - a good merge is incorrectly classified as bad.

Since we have a large synthetic dataset, false positives are the more problematic error mode. It's much worse for bad merges to remain in the dataset than to reduce dataset yield by throwing away a few false negatives.

We used Claude 4 Sonnet as the judge and iteratively tuned the prompt until we hit a false positive rate of ~1%. For comparison, the initial naive judge with no tuning had a false positive rate of ~16%!

Scaling Up

With the aligned LLM judge, it's possible to filter the rest of the data at scale.

For Relace Apply 3, we started with 200k sets of initial_code, diff, and synthetically generated merged_code. We chose a representative distribution of examples across dozens of languages, but focused predominantly on TypeScript/Javascript, Markdown, Python, Ruby, and HTML.

To further cut down on mistakes, we added an extra post-processing step using a combination of static analysis tools — syntax verification with a code parser, deduplication, and regex-based filtering for common undesirable behaviors identified through customer feedback.

After all the filtering, we were left with a high-confidence training set of ~145k data points.

Training with LoRA

For small models, we've consistently found that data quality is the most important ingredient. With a clean, in-distribution dataset, the training just boils down to specializing a high quality base model for the merging task without catastrophic forgetting on general coding.

We trained our apply models using Supervised Fine-Tuning (SFT) on top of open-source coding models in the 3–8 billion parameter range. This gave us the right balance of expressiveness, inference speed, and cost efficiency.

We built our own training pipeline on top of the HuggingFace transformers library, but you can also just use out-of-the-box libraries like axolotl or unsloth.

Since post-training typically uses much smaller datasets than pretraining, updating every parameter in the model is wasteful. Instead of retraining all billions of weights, we use Low-rank adaptation (LoRA) — a lightweight fine-tuning method that adds a small number of trainable “adapter” matrices on top of the frozen base model.

This lets us specialize the model for merge tasks without erasing its coding intuition. The base model stays intact, while the adapters learn the merging algorithm we care about.

We ran a series of grid searches to tune the adapter size (rank), scaling factor (alpha), and learning rate, and landed on the configuration below, which produced the best evaluation loss and convergence speed.

Interestingly, we independently validated the optimal hyperparameters for LoRA published in the recent Thinking Machines blog post.

Using LoRA allowed us to train Relace Apply 3 on all ~145k data points using a single Nvidia H200 GPU on Modal with context length up to 64k tokens.

Using Modal allowed us to launch parallel training runs on H200 GPUs without having to wrestle with complicated cloud providers. Even though small model training lets you get way with batch size 1 on a single GPU, running a large hyperparameter sweep would normally take a lot of manual setup. With Modal, we ran our Python scripts and let it handle the scaling for us — no need to ssh to new GPU instances every time.

After training in BF16, we convert the model weights to FP8 using the llm-compressor library from vLLM. This conversion step is crucial — by leveraging the FP8 cores on newer Nvidia GPUs, we achieve a substantial jump in throughput without sacrificing precision.

To confirm that the quantization process was effectively lossless, we evaluated the resulting model against our 500 held-out ground-truth examples to validate that the outputs were the same.

10k tok/s with Speculative Decoding

With the model trained, our goal was to make it feel less like a slow language model and more like a closed-form algorithm for merging code. To reach that user expereince, we needed to push inference speed as far as possible — which meant turning to speculative decoding.

LLMs normally generate tokens sequentially, where each new token depends on the ones that came before it. This dependency makes inference inherently slow, since every step requires a full forward pass through the model.

Speculative decoding takes advantage of the fact that forward passes are heavily memory bound. i.e. Shuttling the LLM weights into the GPU sRAM takes much longer than actually performing the matrix multiplication with the tensor cores. By guessing a sequence of k tokens, we can process many tokens in parallel (like prefill) and move more towards to the compute-bound regime.

After the forward pass, the model checks which of the predicted tokens match the "true" next tokens and keeps the ones that are correct. The better the "guess", the more tokens you accept, and the more you accelerate the model.

Animation demonstrating how speculative decoding parallelizes inference.

For code merging, large sections of the initial_code and diff are nearly identical to what appears in the merged_code. We can use this strong prior to get long, high quality guesses for what tokens the model should output and get huge speed ups.

However, speed and accuracy are tightly coupled for speculative decoding. Each hallucinated token interrupts the guessed sequence, resetting the verification chain and wasting computation.

By training on the meticulously cleaned dataset, we minimized these breaks and were able to push Relace Apply 3 to 10k tok/s:

Distribution of Relace Apply 3 throughput speed after first token (measured in tokens/second)

Results

We tested Relace Apply on two datasets: (1) our manually reviewed benchmark of 500 examples, and (2) a second dataset of pathological merges collected from customer feedback.

Across both, Relace Apply 3 achieves state-of-the-art merge accuracy. Substantial improvements were made over the previous generation of Apply models thanks to targetted dataset tuning based on customer feedback.

Comparison of merge accuracy between Relace Apply 2 and Relace Apply 3 on a dataset of pathological merge requests.

Previous generations of fast apply struggled when edit snippets contained multiple diff formats in one (e.g. combining \\ ... existing code ... format with UDiff edits). We included subsets of UDiff and String Replace edits in the training data for Relace Apply 3 allowing it to act as a universal merger.

Relace Apply 3 also introduces native support for 256k context, allowing it to handle very large files without degradation in performance. Combined with the 10k tok/s throughput, this makes Relace Apply 3 the fastest, most accurate, and longest-context model on the market.

Fast Apply, One Year Later

When we released our first Fast Apply model a year ago, LLMs were notoriously bad at outputting valid diff formats. Deterministic approaches like Search-and-Replace or UDiff were brittle, model-specific, and required extensive prompt engineering.

Diff formatting accuracy became such a bottleneck that Aider’s polyglot leaderboard tracked it as a separate metric — one column for accuracy, another for performance.

Fast Apply changed that. It was the first model to make structured code edits feel reliable, and our customers felt the difference immediately.

Today, frontier models have become much better at diff formatting through heavy reinforcement learning on string-edit tools, but they’re still not perfect. Companies that exclusively use deterministic strategies, like Cline, need a supplmentary merge algorithm to help further boost the diff edit accuracy.

Schematic reconstructed from an X post by the Cline team. Even deterministic approaches require fallback logic to handle edge cases.

The best models now reach roughly 96% edit success, but models without this kind of tuning still fail around 10% of the time.

So while we expect that, over time, apply models may be phased out as diff accuracy improves, the underlying philosophy that drove it remains central to new projects.

Fast Apply proved that small, specialized models can deliver SoTA results when trained with high quality, task-specific datasets. The methods we developed are now guiding our broader work on accelerating codegen with small agentic models for utility tasks like search, merge conflict resolution, and refactoring.

Stay tuned for more releases soon!

We're Hiring

If you have gotten this far, chances are you found this interesting!

We’re hiring pragmatic researchers (Physics/Math/CS/ML) and exceptional engineers to ship models like this that real product teams rely on. Check out our careers page, and join us!