---
title: "What happens when you merge two LoRAs"
description: "Two adapters that each pass on their own can make each other worse the moment you sum them into the base weights. We shipped a hand-tuned coefficient pair because we had to ship something, and months later I still don't have a principled answer for how those coefficients should be chosen."
date: "July 15, 2025"
url: "https://himanshuat.com/blogs/what-happens-when-you-merge-two-loras"
---
# What happens when you merge two LoRAs

We fine-tuned two specialized LoRA adapters on the same base model. Expert A restored structural volume to saree pleats. Expert B stopped the diffusion pipeline from painting fabric over hands resting on hips.

Loaded individually against the base model, each adapter resolved its target defect. When we merged them into base weights and ran inference, kimonos flattened into generic bathrobes again.

> Two adapters that are individually effective often fail when combined. The direct sum creates a third model that was never explicitly trained.

Summing parameters is arithmetic. It is not composition of behaviors, and the gap between those two things is what this post is about.

### What linear weight fusion actually computes

A LoRA is a low-rank update to a weight matrix. Training learns two low-rank factor matrices, $B$ and $A$, whose outer product $B A$ represents the adapter update. Merging two adapters applies both updates scaled by scalar hyperparameters:

$$
W_{\text{merged}} = W_{\text{base}} + \lambda_{\text{drape}} \cdot (B_d A_d) + \lambda_{\text{occ}} \cdot (B_o A_o)
$$

Linear merging is common because it adds zero computational overhead at inference time: the matrices are summed once during build, producing standard model weights.

Linear combination assumes that the two parameter updates inhabit orthogonal subspaces. When both adapters modify shared attention projections, their updates can directly interfere.

---

### Step 1: Measuring layer-wise cosine similarity before hyperparameter tuning

The standard merge script loads safetensors files and applies updates key by key:

```python
# merge_experts.py
"""Merge two LoRA adapters into base weights with per-adapter coefficients."""

from pathlib import Path

import torch
from safetensors.torch import load_file, save_file


def delta_for(lora: dict[str, torch.Tensor], key: str, alpha: float, rank: int):
    """Reconstruct dense update BA for one layer, scaled by alpha / rank."""
    up = lora.get(f"{key}.lora_up.weight")
    down = lora.get(f"{key}.lora_down.weight")
    if up is None or down is None:
        return None
    return (up @ down) * (alpha / rank)


def merge(base_path: Path, drape_path: Path, occ_path: Path, out_path: Path,
          l_drape: float = 0.6, l_occ: float = 0.4):
    base = load_file(str(base_path))
    drape = load_file(str(drape_path))
    occ = load_file(str(occ_path))

    touched_by_both = 0
    merged = {}

    for key, weight in base.items():
        stem = key.removesuffix(".weight")
        d = delta_for(drape, stem, alpha=16, rank=32)
        o = delta_for(occ, stem, alpha=16, rank=32)

        if d is not None and o is not None:
            touched_by_both += 1
            # Cosine similarity between the two flattened updates.
            # Near zero indicates orthogonal updates; large values indicate subspace conflict.
            cos = torch.nn.functional.cosine_similarity(
                d.flatten().float(), o.flatten().float(), dim=0
            ).item()
            if abs(cos) > 0.25:
                print(f"{stem}: cos={cos:+.3f}")

        update = torch.zeros_like(weight, dtype=torch.float32)
        if d is not None:
            update += l_drape * d.float()
        if o is not None:
            update += l_occ * o.float()

        merged[key] = (weight.float() + update).to(weight.dtype)

    print(f"layers touched by both adapters: {touched_by_both}")
    save_file(merged, str(out_path))


if __name__ == "__main__":
    merge(
        Path("weights/flux/base.safetensors"),
        Path("weights/lora/expert_a_drape.safetensors"),
        Path("weights/lora/expert_b_occ.safetensors"),
        Path("weights/lora/aurax_merged.safetensors"),
        l_drape=0.6,
        l_occ=0.4,
    )
```

Because LoRA training defaults to attention projections, both adapters modified identical projections across almost every block.

---

### Step 2: Distinguishing feature interference from catastrophic forgetting

When outputs degrade following a merge, two separate phenomena may be responsible:

**Feature interference:** Both adapters push shared attention layers in competing directions. Expert A pushes the model to synthesize detailed folds within masked regions. Expert B pushes it to treat foreground pixels as immutable. Linear summation causes these updates to partially cancel out.

**Catastrophic forgetting:** Applying two strong adapter perturbations simultaneously shifts base weights far from their original pre-trained state, degrading foundational capabilities like face rendering or broad garment classification.

| Degradation Type | Impacted Behaviors | Observable Visual Artifacts |
|---|---|---|
| Feature interference | Specialized adapter behaviors | Flattened pleats, clipped fingers at boundaries |
| Catastrophic forgetting | General foundational priors | Distorted faces, corrupted anatomy, misclassified garments |

Kimonos degrading into bathrobes is the forgetting signature. The cumulative update moved the base weights far enough to damage a semantic prior that neither adapter was trained to touch.

---

### Step 3: Detecting style drift in textile weave and material properties

The primary visual indicator of adapter conflict is style drift: the generated garment preserves color and silhouette while losing subtle textile texture.

A knit sweater renders as a flat red shirt, or a brocade saree loses its metallic weave. While automated pixel distance metrics often overlook these shifts, fashion brand directors reject them immediately.

We incorporated brand-specific aesthetic evaluation alongside human review passes to catch texture collapse that automated metrics miss.

---

### Step 4: Sweeping merge coefficients across targeted prompt probes

We swept linear merge coefficients $(\lambda_{\text{drape}}, \lambda_{\text{occ}})$ across a parameter grid, evaluating outputs against specific behavioral probes:

```python
# sweep_lambdas.py
"""Render a fixed prompt set across a grid of merge coefficients."""

import itertools
from pathlib import Path

import torch
from diffusers import FluxFillPipeline

from merge_experts import merge

GRID = [0.2, 0.4, 0.6, 0.8, 1.0]

# Probes targeting individual adapter tasks, plus a neutral probe to check base degradation
PROBES = [
    ("saree_pleats", "full length saree, pleats gathered at the waist, pallu over shoulder"),
    ("hand_on_hip", "model standing, one hand resting on hip, fitted top"),
    ("brocade_weave", "heavy brocade fabric, visible gold thread, close weave"),
    ("plain_tshirt", "plain cotton t-shirt, studio lighting, front facing"),
]


def render_all(out_root: Path):
    for l_drape, l_occ in itertools.product(GRID, GRID):
        tag = f"d{l_drape}_o{l_occ}"
        weights = Path(f"tmp/merged_{tag}.safetensors")

        merge(
            Path("weights/flux/base.safetensors"),
            Path("weights/lora/expert_a_drape.safetensors"),
            Path("weights/lora/expert_b_occ.safetensors"),
            weights,
            l_drape=l_drape,
            l_occ=l_occ,
        )
        pipe = FluxFillPipeline.from_pretrained(
            "weights/flux/fill", torch_dtype=torch.bfloat16
        ).to("cuda")
        pipe.load_lora_weights(str(weights))

        for name, prompt in PROBES:
            image = pipe(prompt=prompt, num_inference_steps=30,
                         height=1024, width=1024).images[0]
            path = out_root / tag / f"{name}.png"
            path.parent.mkdir(parents=True, exist_ok=True)
            image.save(path)

        print(f"{tag}: {len(PROBES)} probes rendered")
        del pipe
        torch.cuda.empty_cache()


if __name__ == "__main__":
    render_all(Path("sweeps/lambda_grid"))
```

The neutral `plain_tshirt` probe verifies that the base model retains general garment priors without unintended distortions.

Our grid sweep landed on `lambda_drape = 0.6` and `lambda_occ = 0.4`. A single global scalar pair is a compromise across layers that do not all need the same compromise, and it held up across our catalog test set.

*I still don't have a principled answer for how those two numbers should be chosen. We shipped them because we had to ship something.*

---

### Step 5: Selecting composition strategies based on serving constraints

We evaluated three methods for combining multiple LoRA adapters:

1. **Dynamic runtime swapping:** Load base weights once, applying individual adapters conditionally per request. This avoids parameter interference, but cannot handle requests requiring both drape and occlusion handling simultaneously.
2. **Sequential multi-pass inference:** Run an initial drape pass followed by a low-denoise occlusion pass. This doubles inference latency and causes the second pass to soften structures created in the first.
3. **Offline weight fusion:** Fuse adapters into base weights prior to deployment. This maintains single-pass inference speed and constant memory usage at the cost of potential subspace interference.

```mermaid
flowchart LR
  R[request] --> C{garment requirements}
  C -->|drape only| A[expert A]
  C -->|occlusion only| B[expert B]
  C -->|complex apparel| F[fused weights]
```

| Strategy | Subspace Interference | Inference Overhead | Per-Request Routing |
|---|---|---|---|
| Dynamic runtime swapping | None | Latency from weight swapping | Required |
| Sequential multi-pass inference | None in weights | 2x compute cost | Not required |
| Offline weight fusion | Present without orthogonalization | Zero overhead | Not required |

We shipped offline fusion. Single-pass inference and constant memory were worth the interference risk.

---

### Failure modes and edge cases

1. **Unchecked cross-attention cancellation:** Linear fusion across opposing weight deltas cancels high-frequency guidance in cross-attention blocks.
2. **Combinatorial explosion of adapter sweeps:** Finding empirical scalars for two adapters requires evaluating a 2D grid; adding a third or fourth adapter makes manual sweeps intractable.
3. **Over-perturbation of base checkpoints:** Setting high merge weights across multiple adapters causes catastrophic forgetting of foundational anatomy and prompt adherence.

---

### When this is the wrong choice

- **Layer-wise cosine similarity is strongly negative.** Below roughly $-0.3$, the two updates are pointing against each other and no pair of global scalars fixes that. Scaling one adapter down to stop the cancellation also scales away the behavior you trained it for. Subspace projection (C-AAC and similar) is the tool, not a finer grid.
- **The two problems never appear in the same request.** If drape and occlusion never co-occur in one input, dynamic routing loads one adapter per request, gets zero interference, and costs you a weight swap. Fusing is solving a composition problem you don't have.
- **You have three or more adapters.** The sweep in Step 4 is a 5x5 grid for two coefficients. A third adapter makes it a 125-point grid, and the interference you are tuning against accumulates across every shared layer at the same time.

---

Source: https://himanshuat.com/blogs/what-happens-when-you-merge-two-loras