Merging and Combining Models

Average the weights of two fine-tuned models and you often get one that does both jobs. Surprisingly effective, and nobody fully knows why.

Take two models fine-tuned from the same base — one for summarization, one for classification. Average their weights element by element.

The result frequently does both tasks reasonably well. No training, no data, a few minutes of arithmetic.

This should not obviously work, and the fact that it does says something about how fine-tuning changes models.

Why it works at all

The explanation rests on models sharing a base. Fine-tuning from a common starting point moves weights a relatively short distance, and models that begin in the same place tend to stay in a connected region of weight space where interpolation remains sensible.

Merge two models trained from different initializations and you get noise. The shared ancestry is what makes averaging meaningful — the two sets of weights are describing variations on the same solution rather than two unrelated ones.

Methods

Linear averaging. Weighted mean of corresponding parameters. Simplest, works better than expected, and is the right first attempt.

Task arithmetic is the more illuminating framing. Define a task vector as the difference between a fine-tuned model and its base — the change fine-tuning produced. These vectors behave somewhat like directions you can add and subtract:

• Add two task vectors to the base → a model with both capabilities
• Subtract a task vector → suppress that behaviour
• Scale a vector → dial a capability up or down

That behaviour is genuinely surprising and not fully explained. It is also useful, and it reframes merging as arithmetic on changes rather than on models.

TIES merging addresses interference. When task vectors disagree on a parameter’s direction, naive averaging cancels both contributions. TIES trims small changes, resolves sign conflicts by majority, and averages only the agreeing components. Generally outperforms plain averaging when merging several models.

DARE randomly drops a large fraction of task-vector entries and rescales the rest, on the observation that most changes are redundant. Often composes with other methods.

SLERP interpolates along a spherical path rather than a straight line, preserving weight magnitudes better. Commonly used for merging exactly two models.

Where it is useful

Combining capabilities without retraining. The main practical case: several narrow fine-tunes merged into one deployable model, avoiding the cost of serving several.

Recovering general ability. Merging a heavily fine-tuned model back toward its base can restore general capability lost to catastrophic forgetting while keeping much of the task gain. A cheap fix for over-tuning.

Averaging checkpoints. Merging several checkpoints from one training run often outperforms any single checkpoint, and costs nothing.

Building on community fine-tunes. A substantial share of open-weight model releases are merges rather than fresh training runs — which is worth knowing when evaluating them.

Limits

Same architecture, same base. Non-negotiable. Different families cannot be merged.

Quality is unpredictable. Sometimes a merge beats both parents; sometimes it is worse than either. There is little theory to predict which, so it is empirical — merge, evaluate, adjust weights, repeat. Cheap enough that trial and error is viable.

Interference grows with count. Merging two models usually works. Merging eight often degrades everything, as task vectors increasingly conflict.

No new capability. Merging redistributes what exists. It cannot produce ability neither parent had.

MoE is not this. Mixture of experts routes between parameter sets at inference time, keeping them separate. Merging collapses them into one set beforehand. Different mechanisms, different costs — merging adds no inference overhead, MoE adds no interference.

Practical notes

Evaluate on everything, not just the merged tasks. General capability regressions are the common surprise, and they are invisible if you only test what you merged for.

Start with two models and simple averaging. Add complexity only when it fails.

Merging LoRA adapters is cheaper still — they are small, and adapter arithmetic is the same idea at a fraction of the size.

What to remember

• Merging averages weights of models sharing a common base; shared ancestry is what makes it coherent.
• Task vectors (fine-tuned minus base) can be added, subtracted, and scaled — surprisingly and not fully explained.
• TIES and DARE handle interference between conflicting changes; plain averaging is the right first try.
• Useful for combining fine-tunes, recovering forgotten general ability, and averaging checkpoints.
• Quality is unpredictable and interference grows with the number merged — evaluate broadly.

Next: Hybrid Search