What happens when you merge two LoRAs

By June we had two adapters that worked. Expert A put volume back into a saree's pleats. Expert B stopped the pipeline from painting fabric over somebody's hand. Loaded one at a time against the base model, each of them did the thing it was trained to do, and I had the eval renders to prove it. Then I merged them and watched a kimono come out looking like a bathrobe again.

What the naive sum actually does

A LoRA is a low rank update to a weight matrix. You train two small matrices, B and A, and the adapter's contribution to a layer is their product. Applying an adapter means adding that product to the base weight. Applying two means adding both, with a scalar on each:

W_merged = W_base + λ_drape · (B_d A_d) + λ_occ · (B_o A_o)

This is the form in our paper and it's the form nearly everyone uses, for a good reason: it's the only combination that costs nothing at inference. You do the arithmetic once, you get a single set of weights, and the model you serve has no idea it was ever two adapters. The merge is a build step.

The thing that makes it seductive is that it's linear, so it looks like it should compose. Two independent corrections, added independently. And if the two updates wrote into disjoint parts of the weight space, it more or less would.

They don't.

# merge_experts.py
"""Merge two LoRA adapters into the base weights with per-adapter coefficients.
 
This is the whole merge. It is fifty lines and it is why the operation looks
harmless: nothing here knows that the two adapters were trained for different
reasons, or that they might be asking the same layer for opposite things.
"""
 
from pathlib import Path
 
import torch
from safetensors.torch import load_file, save_file
 
def delta_for(lora: dict[str, torch.Tensor], key: str, alpha: float, rank: int):
    """Reconstruct the dense update BA for one layer, scaled by alpha / rank.
 
    Returns None when the adapter did not touch this layer, which is common:
    a LoRA usually targets the attention projections and leaves the rest alone.
    """
    up = lora.get(f"{key}.lora_up.weight")
    down = lora.get(f"{key}.lora_down.weight")
    if up is None or down is None:
        return None
    return (up @ down) * (alpha / rank)
 
 
def merge(base_path: Path, drape_path: Path, occ_path: Path, out_path: Path,
          l_drape: float = 0.6, l_occ: float = 0.4):
    base = load_file(base_path)
    drape = load_file(drape_path)
    occ = load_file(occ_path)
 
    touched_by_both = 0
    merged = {}
 
    for key, weight in base.items():
        stem = key.removesuffix(".weight")
        d = delta_for(drape, stem, alpha=16, rank=32)
        o = delta_for(occ, stem, alpha=16, rank=32)
 
        if d is not None and o is not None:
            touched_by_both += 1
            # Cosine similarity between the two updates, flattened. Near zero
            # means they are writing in roughly orthogonal directions and the sum
            # is close to harmless. Large magnitude, either sign, means they are
            # fighting over the same subspace.
            cos = torch.nn.functional.cosine_similarity(
                d.flatten().float(), o.flatten().float(), dim=0
            ).item()
            if abs(cos) > 0.25:
                print(f"{stem}: cos={cos:+.3f}")
 
        update = torch.zeros_like(weight, dtype=torch.float32)
        if d is not None:
            update += l_drape * d.float()
        if o is not None:
            update += l_occ * o.float()
 
        merged[key] = (weight.float() + update).to(weight.dtype)
 
    print(f"layers touched by both adapters: {touched_by_both}")
    save_file(merged, out_path)
 
 
if __name__ == "__main__":
    merge(
        Path("weights/flux/base.safetensors"),
        Path("weights/lora/expert_a_drape.safetensors"),
        Path("weights/lora/expert_b_occ.safetensors"),
        Path("weights/lora/aurax_merged.safetensors"),
        l_drape=0.6,
        l_occ=0.4,
    )

I added that cosine similarity print to find out how bad the overlap was, and the answer is that both adapters target the attention projections, because that's where LoRA training targets by default. They overlap almost everywhere they exist.

Where the interference comes from

Two things are happening and they're worth separating, because they call for different fixes and I conflated them for weeks.

The first is feature interference. Both adapters have opinions about the same attention layers. Expert A learned to push the model toward generating structured folds inside the masked region. Expert B learned to push the model toward respecting a foreground boundary and generating nothing across it. Summed into one matrix, those pushes partially cancel and partially compound in directions neither adapter was ever evaluated in. The merged model isn't Expert A plus Expert B. It's a third model that nobody trained and nobody tested.

The second is catastrophic forgetting, and it's about the base. Each adapter individually is a small perturbation. Two of them at full strength is a larger one, and past some magnitude you're not adapting the base model's garment prior anymore, you're overwriting it. This is why the kimono went back to being a bathrobe: that particular defect isn't a drape failure or an occlusion failure, it's the base model's semantic prior degrading under a perturbation it wasn't built to absorb.

You can tell them apart by their signature. Interference shows up as the specific behaviours the adapters were trained for weakening. Forgetting shows up as general competence dropping in places neither adapter has anything to do with, hands and faces and backgrounds getting worse in ways that have no connection to the garment at all.

Style drift is what you see first

The observable symptom, before you've worked out which of the above you're looking at, is "Style Drift". The generated garment keeps the reference's colour and loses everything else. A knit sweater comes out as a flat red shirt. A brocade saree keeps the gold and loses the weave. The silhouette is right, the palette is right, and the fabric is wrong in a way that a brand's art director spots in under a second and a similarity metric mostly doesn't.

Drift is what made this expensive rather than just annoying. It's the failure mode that survives your automated checks. A merged model can look fine across an eval sweep and still be unshippable, because "the fabric doesn't look like the fabric" is a judgement that lives in a human's head. It's a large part of why we ended up building a brand-centric aesthetic scorer with a person in the loop at all, since generic aesthetic scorers pull toward darker, moodier imagery rather than toward commercial e-commerce standards.

Choosing lambda by eye

We landed on λ_drape = 0.6 and λ_occ = 0.4. The paper says empirical testing determined this ratio provides the optimal balance, which is true in the narrow sense that we tested a grid and picked the cell whose renders we liked most. It is not true in any sense that would let you derive it.

Here's what that search actually was.

# sweep_lambdas.py
"""Render a fixed prompt set across a grid of merge coefficients.
 
There is no scalar objective at the end of this. It produces a contact sheet and
a person decides. That is the honest description of how 0.6 and 0.4 were chosen.
"""
 
import itertools
from pathlib import Path
 
import torch
from diffusers import FluxFillPipeline
 
from merge_experts import merge
 
GRID = [0.2, 0.4, 0.6, 0.8, 1.0]
 
# Four probes, each targeting a behaviour one of the adapters owns, plus one that
# neither owns so we can see the base prior degrading.
PROBES = [
    ("saree_pleats", "full length saree, pleats gathered at the waist, pallu over shoulder"),
    ("hand_on_hip", "model standing, one hand resting on hip, fitted top"),
    ("brocade_weave", "heavy brocade fabric, visible gold thread, close weave"),
    ("plain_tshirt", "plain cotton t-shirt, studio lighting, front facing"),
]
 
 
def render_all(out_root: Path):
    for l_drape, l_occ in itertools.product(GRID, GRID):
        tag = f"d{l_drape}_o{l_occ}"
        weights = Path(f"tmp/merged_{tag}.safetensors")
 
        merge(
            Path("weights/flux/base.safetensors"),
            Path("weights/lora/expert_a_drape.safetensors"),
            Path("weights/lora/expert_b_occ.safetensors"),
            weights,
            l_drape=l_drape,
            l_occ=l_occ,
        )
        pipe = FluxFillPipeline.from_pretrained(
            "weights/flux/fill", torch_dtype=torch.bfloat16
        ).to("cuda")
        pipe.load_lora_weights(weights)
 
        for name, prompt in PROBES:
            image = pipe(prompt=prompt, num_inference_steps=30,
                         height=1024, width=1024).images[0]
            path = out_root / tag / f"{name}.png"
            path.parent.mkdir(parents=True, exist_ok=True)
            image.save(path)
 
        print(f"{tag}: {len(PROBES)} probes rendered")
        del pipe
        torch.cuda.empty_cache()
 
 
if __name__ == "__main__":
    render_all(Path("sweeps/lambda_grid"))
    print("contact sheet ready, go look at it")

Two things about that coefficient pair bother me and both are still true.

It's one scalar per adapter applied to every layer the adapter touches. There is no reason to believe the right weighting for an early attention block is the same as for a late one, and every reason to suspect it isn't, since the layers are doing different jobs. We're compressing a per-layer question into one number because one number is what the merge API takes.

And the pair is tied to this specific pair of adapters. Train a third expert, for lighting or for a demographic, and the search restarts. The cost of adding an expert grows with the number of experts you already have, which is precisely the property you don't want in a system whose whole premise is that you can patch knowledge gaps with cheap adapters.

The alternatives, and what each one costs

Keep them separate and swap at runtime. Load the base once, apply whichever adapter the request needs. No interference, because there's never more than one adapter in the weights. This is genuinely correct and we ran it during development. It falls apart at serving time for two reasons: applying and unapplying adapters against a resident pipeline is not free, and more importantly it forces you to route, which means deciding per request whether this image is a drape problem or an occlusion problem. A saree worn by someone with their arms crossed is both. There's no branch to take.

Apply them sequentially. This is the one that sounds like it should help and mostly doesn't. If you mean sequential merging, adding one adapter's delta and then the other's, that's algebraically the same sum you started with; addition doesn't care about order, so nothing changes. If you mean sequential inference, one pass with the drape adapter and a second pass with the occlusion adapter over the first pass's output, then you've doubled your inference cost and the second pass is now denoising an image the first pass already committed to. In practice the second pass repairs the hand and softens the pleats the first pass just built. You end up back where you started, having spent twice as long.

Fuse once and serve one model. This is what we needed and what we did. One set of weights, one process, one warm pipeline, no per-request decisions, no VRAM penalty from holding several adapters resident, and a merge step that runs at build time where its cost doesn't matter. In ComfyUI it's a couple of model_merge_lora nodes ahead of everything else in the graph.

{
  "31": {
    "class_type": "LoraLoaderModelOnly",
    "_meta": { "title": "Expert A: draping physics" },
    "inputs": {
      "lora_name": "expert_a_drape.safetensors",
      "strength_model": 0.6,
      "model": ["12", 0]
    }
  },
  "32": {
    "class_type": "LoraLoaderModelOnly",
    "_meta": { "title": "Expert B: occlusion and depth" },
    "inputs": {
      "lora_name": "expert_b_occ.safetensors",
      "strength_model": 0.4,
      "model": ["31", 0]
    }
  },
  "33": {
    "class_type": "CheckpointSave",
    "_meta": { "title": "write fused runtime weights" },
    "inputs": {
      "filename_prefix": "aurax/merged_d06_o04",
      "model": ["32", 0],
      "clip": ["12", 1],
      "vae": ["12", 2]
    }
  }
}

The serving argument won, and I'd make the same call again. What I want to be clear about is that it won on operational grounds, not because the merge is correct. We picked the option whose failure mode we could live with.

Where this sits

The merged model ships. It's better than either baseline we measured against and it's what our clients' images went through. That's the result and I'm not going to undersell it.

It's also built on a constant somebody chose by looking at pictures. I don't have a way to predict, given two new adapters, whether they'll interfere; I find out by merging them and rendering probes. I don't have a per-layer treatment, only the global scalar. And I don't have a story for the fifth adapter, which is the one that matters, because the whole plan for handling demographics and poses and lighting and background styles was more adapters.

That's the thread we've been pulling on since: a composition step that looks at the adapters before they're summed and does something smarter than trusting a human's prior about which one should win. It works well enough now that it's what our scene generation runs on. It's also changing week to week and I'm not going to write up its internals while that's still true, because the version I'd describe today isn't the version that would be running by the time you read it. What I can say is that the problem in this post, two things that are individually correct and jointly wrong, didn't get solved by finding better values for lambda. It got solved by stopping treating lambda as the thing to solve.

For anyone hitting this now: before you spend a week on a coefficient sweep, print the cosine similarity between your two updates layer by layer. If they're near orthogonal, the sum will mostly work and your problem is somewhere else. If they aren't, no value of lambda is going to save you, and you'll have learned that in an afternoon instead of a month.