Composing adapters that disagree with each other

Merging two LoRA experts with two scalars is a tractable problem, and we shipped it. Merging a library of them is a different animal, and it broke in a way I didn't predict. The failure wasn't that the merged model was bad on average; it was good almost everywhere and wrong in a few specific places, and the knob I had could only make it worse everywhere in exchange for fixing those places. This post is about the shape of that problem and what we ended up serving. It stops short of the merging method itself, and I say why partway down.

Where the two scalar merge ran out

The try-on model fused two experts into the base weights with one coefficient each:

W_merged = W_base + lambda_drape * (B_d A_d) + lambda_occ * (B_o A_o)

We settled on lambda_drape = 0.6 and lambda_occ = 0.4 by testing, which is a polite way of saying we swept a two dimensional grid and looked at pictures until one cell stopped being objectionable. That works when you have two adapters trained on problems that barely touch each other. Fabric draping and hand occlusion overlap a little, not a lot, and 0.6 against 0.4 was enough to keep both.

The base image stage is not that. Before any garment gets composited onto anything, we generate the scene: a model of a particular demographic, in a particular pose, under a particular lighting setup, against a particular backdrop. Each of those is an adapter, some trained in house on our curated corpus and some pulled from good public work. A brand will ask for a combination nobody ever trained together. The grid of coefficients is now large enough that hand sweeping is not a plan, and sweeping it harder wouldn't help anyway, because the coefficient is the wrong instrument.

Conflict is per layer, and it isn't uniform

A LoRA is a set of low rank updates attached to specific modules inside the transformer. Two adapters trained separately land on overlapping sets of modules. Where they overlap, their updates may point in compatible directions or in opposing ones, and that varies module by module inside the same pair of adapters.

Here's what that means in practice. A demographics adapter carries a great deal of information about skin: undertone, pore structure, how highlights sit on a cheekbone. A high key lighting adapter also has opinions about skin, because that's most of what high key lighting is doing to a portrait. Those two agree about the backdrop and fight about the face. A single scalar on the lighting adapter forces you to trade one against the other. Turn it down to protect skin tone and you also lose the backdrop falloff that the same adapter was rendering perfectly well. Turn it up and the face goes waxy.

That is the whole problem in one sentence: a global blend weight is a per adapter instrument applied to a per layer disagreement.

The first useful thing I built was not a merge at all, it was a diagnostic that told me where two adapters even touch:

# tools/inspect_adapter_overlap.py
"""Report which modules two LoRA adapters both write to.
 
Overlap on its own is not evidence of conflict. It tells you where a
conflict is possible, which is the only thing a static check of two
files on disk can honestly tell you. Direction is a separate question
and needs the model in memory.
"""
 
from __future__ import annotations
 
import sys
from collections import OrderedDict
 
from safetensors.torch import load_file
 
 
def target_modules(path: str) -> "OrderedDict[str, int]":
    """Map module name to the rank of its down projection."""
    tensors = load_file(path)
    modules: "OrderedDict[str, int]" = OrderedDict()
    for key, tensor in tensors.items():
        if not key.endswith("lora_down.weight"):
            continue
        name = key.rsplit(".lora_down", 1)[0]
        modules[name] = tensor.shape[0]
    return modules
 
 
def overlap_report(path_a: str, path_b: str, preview: int = 12) -> None:
    """Print the shared and exclusive module sets for two adapters."""
    a = target_modules(path_a)
    b = target_modules(path_b)
 
    shared = [m for m in a if m in b]
    only_a = [m for m in a if m not in b]
    only_b = [m for m in b if m not in a]
 
    print(f"A: {path_a}  modules={len(a)}")
    print(f"B: {path_b}  modules={len(b)}")
    print(f"shared={len(shared)}  a_only={len(only_a)}  b_only={len(only_b)}")
    print()
 
    for name in shared[:preview]:
        print(f"  shared  {name}  rank_a={a[name]} rank_b={b[name]}")
    for name in only_a[:preview]:
        print(f"  a_only  {name}  rank={a[name]}")
 
 
if __name__ == "__main__":
    overlap_report(sys.argv[1], sys.argv[2])

Running that across the adapter library is unglamorous and it changed how I thought about the merge. The overlap is never total and never trivial. Adapters that feel semantically unrelated share a surprising number of attention modules, and adapters that feel like near duplicates diverge in the blocks that carry composition. Once you've seen that laid out per module, a single coefficient stops looking like a simplification and starts looking like a category error.

Priors are the input, and a human writes them

The thing that decides which adapter should win where is not a metric. It's a decision about what the image is for. On an e-commerce shoot, skin follows the demographics adapter and nothing else gets to touch it, because that's the part a brand's legal and marketing teams will both look at. The backdrop follows the background adapter. Garment surface rendering follows lighting. Those are priorities somebody with taste sets, writes down, and defends in a review.

So they live in a file, next to the model they produce:

# priors/editorial_ecom.yaml
# Human authored, reviewed by whoever owns the brand look, versioned with
# the checkpoint it produces. The resolver consumes this file. It never
# writes it, and it is not allowed to reorder the priority block.
 
base: flux
 
adapters:
  - id: demographics_in_female_25_35
    origin: in_house
    rank: 32
  - id: pose_studio_standing
    origin: in_house
    rank: 32
  - id: lighting_softbox_high_key
    origin: public
    rank: 32
  - id: background_studio_sweep
    origin: public
    rank: 32
 
# Which adapter owns which visual concern when two of them disagree.
priority:
  skin_and_hair: demographics_in_female_25_35
  body_proportion: pose_studio_standing
  garment_surface: lighting_softbox_high_key
  frame_and_backdrop: background_studio_sweep
 
# Concerns that no other adapter may override, regardless of what the
# resolver would otherwise prefer.
protected:
  - concern: skin_and_hair
    from: [lighting_softbox_high_key, background_studio_sweep]
 
output:
  name: aurax-v1
  bake: true

This is the artifact I was most wrong about going in. I expected the valuable output of the project to be an algorithm, and the algorithm matters, but the priors file is what makes the algorithm produce something a brand signs off on. Two brands with identical adapter libraries and different priors files get visibly different models. When a shoot came back with notes, nine times out of ten the fix was in the priors and not in the code.

One fused model, not a stack of adapters

The other decision worth writing down is that the output of composition is a single checkpoint, which we called AuraX-V1, and not a set of adapters loaded at request time.

Runtime stacking is the obvious design and it's tempting because it keeps everything composable. It also means every request carries the low rank matmuls for every active adapter in the hot path, holds each adapter resident, and exposes a serving surface where any caller can request any combination, including combinations nobody has ever looked at. That last part is the real cost. If the set of possible models is generated at request time, you cannot review the model, you can only review the code that assembles it.

# pipelines/bake_runtime_model.py
"""Resolve an adapter set against a priors file and bake one checkpoint.
 
The output of this script is the artifact that serving loads: one file,
one version tag, one thing to roll back. Composition happens here, at
build time, never in the request path.
"""
 
from __future__ import annotations
 
import argparse
import json
from pathlib import Path
 
import torch
import yaml
from safetensors.torch import load_file, save_file
 
# Internal. This is the part of the system the blog post does not open up.
from aurax.caac import resolve  # noqa: F401
 
 
def load_priors(path: Path) -> dict:
    """Read and lightly validate a priors file."""
    priors = yaml.safe_load(path.read_text())
    declared = {a["id"] for a in priors["adapters"]}
    referenced = set(priors["priority"].values())
    missing = referenced - declared
    if missing:
        raise ValueError(f"priority references unknown adapters: {sorted(missing)}")
    return priors
 
 
def bake(priors_path: Path, base_path: Path, out_dir: Path) -> Path:
    """Produce the fused runtime checkpoint and its provenance sidecar."""
    priors = load_priors(priors_path)
    base = load_file(str(base_path))
    adapters = {
        a["id"]: load_file(f"adapters/{a['id']}.safetensors")
        for a in priors["adapters"]
    }
 
    merged = resolve(base=base, adapters=adapters, priors=priors)
 
    out_dir.mkdir(parents=True, exist_ok=True)
    ckpt = out_dir / f"{priors['output']['name']}.safetensors"
    save_file(merged, str(ckpt))
 
    sidecar = {
        "name": priors["output"]["name"],
        "base": priors["base"],
        "adapters": [a["id"] for a in priors["adapters"]],
        "priors_sha": priors_path.read_bytes().hex()[:16],
        "torch": torch.__version__,
    }
    (out_dir / "provenance.json").write_text(json.dumps(sidecar, indent=2))
 
    print(f"wrote {ckpt}")
    print(f"fused {len(adapters)} adapters into one checkpoint")
    return ckpt
 
 
if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--priors", type=Path, required=True)
    ap.add_argument("--base", type=Path, required=True)
    ap.add_argument("--out", type=Path, default=Path("build/"))
    args = ap.parse_args()
    bake(args.priors, args.base, args.out)

Baking gives you a model you can review, tag, ship, and revert as one unit, and a cold worker pulls one file instead of assembling a graph. The cost is real: you lose per request composability, and every new priors file means a new bake and a new review. We took that trade because brands don't change their house look between requests. For R&D we kept a stacking path, since being able to hot swap an adapter is worth a lot when you're still deciding what the adapter should do.

The part I'm not opening up

The resolve call above is the method, and I'm not publishing it. That covers what it does internally and the index we use to talk about how much two adapters interfere. It's company IP and it's the reason a brand pays us instead of running model_merge_lora themselves.

I'd rather say that plainly than write a paragraph that gestures at a method without being reproducible. Vague method sketches are worse than an honest omission: they cost the reader time, they can't be checked, and they imply a rigour the writing isn't actually carrying. The problem statement and the outcome are the parts that are useful to anyone outside the company anyway, and both are above.

How we judged the output

We compared the fused model against base Flux-dev, Google's Imagen 4, ChatGPT (August) and Nano-banana, on skin texture and commercial realism, by looking at the images. No score, no leaderboard.

That was a deliberate choice and it's also a weakness, so both halves are worth stating. The reason we didn't score it is that the generic aesthetic scorers available to us were not measuring what we needed. They lean toward darker, moodier imagery, which is a defensible notion of aesthetic and a bad fit for an e-commerce product shot that has to be evenly lit and honest about the garment. We ended up building our own human-in-the-loop scorer for exactly this reason, but for this comparison we used our eyes.

What the images showed: the fused model held natural skin and hair and clean, well lit compositions. Flux-dev's dramatic contrast is good for a campaign and competes with the garment. Imagen 4's photorealism reads slightly too perfect on close inspection. Nano-banana's contrast is strong in a way some brands will love and others will reject on sight. That's a qualitative judgment made by people with a commercial stake in the answer, and you should read it as such. All of this came out of a fine-tune on a comparatively modest corpus of 5,000 curated images, which is the number that surprised me most.

What I'd do differently

Write the priors before training the adapters. We did it in the other order and spent weeks training capability we then had to suppress.

Build the overlap diagnostic on day one. It's forty lines, it needs no GPU, and it reframes the problem faster than any amount of staring at outputs.

Keep the runtime stacking path alive for research even after you commit to baking for production. We nearly deleted it and it's where every subsequent experiment started.

Decide early whether a comparison is going to be qualitative, and then commit to it properly with a blind panel rather than drifting into an informal one. Ours was informal, and that limits how hard I can lean on it.

The next thing on my list is measurement, specifically the try-on side, where we do have numbers and where I've started to suspect the aggregate ones are telling us less than they appear to.