Composing adapters that disagree with each other

Merging two adapters with two scalar weights worked well enough to ship.

Running that same approach across an entire library of adapters produced a model that succeeded in most regions but failed in specific layers. Adjusting a single global blend weight degraded the good regions whenever I tried to correct the broken ones.

A global blend weight is a per-adapter instrument applied to a per-layer disagreement.

This post covers why global blend weights fail on multi-adapter setups and how we structured checkpoint generation for production.

What a merge does when it works

Our virtual try-on model fused two expert LoRAs into the base weights with one coefficient each:

We settled on lambda_drape = 0.6 and lambda_occ = 0.4 through empirical testing, sweeping a two-dimensional grid until the resulting images retained both properties cleanly.

That works when two adapters address mostly orthogonal image regions. Fabric draping and hand occlusion have minimal spatial overlap, so a static 0.6 to 0.4 ratio preserved both.

Scene generation brings a much higher degree of conflict. Before compositing garments, we generate the full scene: a model of a specific demographic, in a target pose, under calibrated studio lighting, against a selected backdrop. Each variable is an adapter, either trained in-house on our curated dataset or pulled from open-source checkpoints. Brand shoots regularly demand combinations that were never trained together.

Sweeping a high-dimensional coefficient grid by hand is impractical. Finding the grid point would not help either: one scalar per adapter cannot resolve conflicting updates across individual transformer blocks.

Step 1: Mapping where adapters overlap

A LoRA applies low-rank updates to specific modules inside the transformer. Two independently trained adapters often write to overlapping attention and feed-forward layers. Within those shared modules, their weight updates can point in compatible directions or directly oppose each other.

Consider a practical example. A demographic adapter modifies skin undertone, pore texture, and cheekbone structure. A high-key lighting adapter also modifies skin values, because lighting directly alters face luminance and specular falloff.

Both adapters agree on the backdrop, but they fight over facial features. Lowering the lighting weight to preserve skin tone flattens the lighting gradient across the rest of the frame. Increasing the lighting weight blows out facial highlights and creates a waxy appearance.

To diagnose this, I wrote a script to inspect which modules two adapters target before running any merges.

# tools/inspect_adapter_overlap.py
"""Report which modules two LoRA adapters both write to.
 
Overlap alone is not proof of conflict. It shows where a conflict is
structurally possible. Update direction requires evaluating the model
weights in memory.
"""
 
from __future__ import annotations
 
import sys
from collections import OrderedDict
 
from safetensors.torch import load_file
 
 
def target_modules(path: str) -> "OrderedDict[str, int]":
    """Map module name to the rank of its down projection."""
    tensors = load_file(path)
    modules: "OrderedDict[str, int]" = OrderedDict()
    for key, tensor in tensors.items():
        if not key.endswith("lora_down.weight"):
            continue
        name = key.rsplit(".lora_down", 1)[0]
        modules[name] = tensor.shape[0]
    return modules
 
 
def overlap_report(path_a: str, path_b: str, preview: int = 12) -> None:
    """Print shared and exclusive module sets for two adapters."""
    a = target_modules(path_a)
    b = target_modules(path_b)
 
    shared = [m for m in a if m in b]
    only_a = [m for m in a if m not in b]
    only_b = [m for m in b if m not in a]
 
    print(f"A: {path_a}  modules={len(a)}")
    print(f"B: {path_b}  modules={len(b)}")
    print(f"shared={len(shared)}  a_only={len(only_a)}  b_only={len(only_b)}")
    print()
 
    for name in shared[:preview]:
        print(f"  shared  {name}  rank_a={a[name]} rank_b={b[name]}")
    for name in only_a[:preview]:
        print(f"  a_only  {name}  rank={a[name]}")
 
 
if __name__ == "__main__":
    overlap_report(sys.argv[1], sys.argv[2])

$ python tools/inspect_adapter_overlap.py \
    adapters/demographics_in_female_25_35.safetensors \
    adapters/lighting_softbox_high_key.safetensors
 
A: demographics_in_female_25_35.safetensors  modules=76
B: lighting_softbox_high_key.safetensors    modules=76
shared=48  a_only=28  b_only=28
 
  shared  double_blocks.0.img_attn.qkv   rank_a=32 rank_b=32 (cos_sim: -0.42)
  shared  double_blocks.0.img_attn.proj  rank_a=32 rank_b=32 (cos_sim: -0.38)
  shared  double_blocks.4.img_attn.qkv   rank_a=32 rank_b=32 (cos_sim: +0.61)
  shared  double_blocks.12.img_mlp.0     rank_a=32 rank_b=32 (cos_sim: -0.55)
  a_only  single_blocks.0.linear1        rank=32
  b_only  single_blocks.14.linear2       rank=32

Running this across our adapter library showed that overlap is neither total nor negligible: 48 of 76 targeted LoRA projections collide. Early attention projections (double_blocks.0) come out at negative cosine similarity ($-0.42$), so the lighting and demographic updates push those weights in opposing directions.

When two adapters share attention modules with opposing gradient directions, a single scalar weight arbitrates a conflict across every layer simultaneously, degrading both properties.

Step 2: Writing explicit priors before merging

Deciding which adapter takes precedence is a product decision dictated by the visual requirements of the catalog shoot.

On an e-commerce shoot, skin characteristics must strictly follow the demographic adapter, because brand and legal guidelines require consistent representation. Backdrop appearance follows the background adapter, while garment surface texture follows lighting.

These rules reflect deliberate visual priorities, so we store them in a versioned configuration file next to the resulting model checkpoint.

# priors/editorial_ecom.yaml
# Reviewed by the brand team and versioned with the target checkpoint.
# The resolver consumes this configuration without modifying priority orders.
 
base: flux
 
adapters:
  - id: demographics_in_female_25_35
    origin: in_house
    rank: 32
  - id: pose_studio_standing
    origin: in_house
    rank: 32
  - id: lighting_softbox_high_key
    origin: public
    rank: 32
  - id: background_studio_sweep
    origin: public
    rank: 32
 
# Which adapter owns which visual domain during conflicts:
priority:
  skin_and_hair: demographics_in_female_25_35
  body_proportion: pose_studio_standing
  garment_surface: lighting_softbox_high_key
  frame_and_backdrop: background_studio_sweep
 
# Concerns protected from overrides regardless of resolver defaults:
protected:
  - concern: skin_and_hair
    from: [lighting_softbox_high_key, background_studio_sweep]
 
output:
  name: aurax-v1
  bake: true

I initially expected the algorithmic resolver to be the central deliverable. In practice, the priors file is what makes the resolver output something a brand signs off on. Two brands using identical adapter libraries with different priors files produce noticeably different models.

Step 3: Baking single checkpoints instead of runtime stacking

Rather than loading and combining multiple adapters dynamically per API request, our composition pipeline bakes a single unified checkpoint (AuraX-V1).

Runtime stacking seems flexible because it allows on-the-fly adapter combinations. However, it forces every inference request to execute multiple low-rank matrix multiplications in the hot path, keeps numerous adapter weights resident in GPU VRAM, and allows client requests to invoke untested adapter combinations.

Reviewability is the decisive factor. When models are assembled dynamically per request, engineers can only review the assembly logic, not the resulting visual model.

The solid path runs in production, while the dotted path remains active for experimental prototyping.

# pipelines/bake_runtime_model.py
"""Resolve adapter conflicts against a priors file and bake a checkpoint.
 
The output is a single deployable artifact: one file, one version tag,
and one unit to roll back. Composition runs entirely at build time.
"""
 
from __future__ import annotations
 
import argparse
import json
from pathlib import Path
 
import torch
import yaml
from safetensors.torch import load_file, save_file
 
# Internal Conflict-Aware Adapter Composition resolver
from aurax.caac import resolve  # noqa: F401
 
 
def load_priors(path: Path) -> dict:
    """Read and validate a priors configuration file."""
    priors = yaml.safe_load(path.read_text())
    declared = {a["id"] for a in priors["adapters"]}
    referenced = set(priors["priority"].values())
    missing = referenced - declared
    if missing:
        raise ValueError(f"Priority references unknown adapters: {sorted(missing)}")
    return priors
 
 
def bake(priors_path: Path, base_path: Path, out_dir: Path) -> Path:
    """Produce the fused runtime checkpoint and provenance metadata."""
    priors = load_priors(priors_path)
    base = load_file(str(base_path))
    adapters = {
        a["id"]: load_file(f"adapters/{a['id']}.safetensors")
        for a in priors["adapters"]
    }
 
    merged = resolve(base=base, adapters=adapters, priors=priors)
 
    out_dir.mkdir(parents=True, exist_ok=True)
    ckpt = out_dir / f"{priors['output']['name']}.safetensors"
    save_file(merged, str(ckpt))
 
    sidecar = {
        "name": priors["output"]["name"],
        "base": priors["base"],
        "adapters": [a["id"] for a in priors["adapters"]],
        "priors_sha": priors_path.read_bytes().hex()[:16],
        "torch": torch.__version__,
    }
    (out_dir / "provenance.json").write_text(json.dumps(sidecar, indent=2))
 
    print(f"Saved {ckpt}")
    print(f"Fused {len(adapters)} adapters into one checkpoint")
    return ckpt
 
 
if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--priors", type=Path, required=True)
    ap.add_argument("--base", type=Path, required=True)
    ap.add_argument("--out", type=Path, default=Path("build/"))
    args = ap.parse_args()
    bake(args.priors, args.base, args.out)

Baking sacrifices per-request adapter selection, requiring a new build and visual review for every updated priors configuration. We accepted this tradeoff because e-commerce brands maintain a fixed catalog aesthetic over time. For R&D, hot-swapping adapters via stacking remained valuable while tuning adapter behavior.

Internal resolver implementation

The resolve function executes C-AAC (Conflict-Aware Adapter Composition), calculating layer-wise interference metrics across weight tensors and enforcing the constraints declared in the priors file. This resolver logic is proprietary to AuraX.

Step 4: Assessing output quality directly

We evaluated AuraX-V1 against base Flux-dev, Google Imagen 4, ChatGPT (August 2025 generation), and Nano-banana on commercial skin fidelity and lighting quality through direct visual side-by-side reviews.

Standard automated aesthetic scoring models favor high-contrast, moody lighting. That aesthetic profile directly conflicts with e-commerce requirements, where products demand uniform lighting and precise fabric depiction. We eventually constructed a dedicated human-in-the-loop scoring tool for this reason, relying on side-by-side visual inspections for initial sign-offs.

These evaluations were conducted directly by our engineering and product teams. The fine-tuning behind this model relied on 5,000 curated fashion images.

Failure modes and edge cases

1. Training adapters without defining target priors first: Developing adapter capability without explicit prior definitions forces the downstream resolver to suppress unneeded features later.
2. Scoring commercial outputs with generic aesthetic metrics: Off-the-shelf aesthetic scorers penalize flat, even commercial lighting. Automated filters must be aligned with catalog requirements.
3. Over-interpreting informal visual inspections: Internal side-by-side reviews suffice for deployment decisions, but rigorous competitive claims require blinded, multi-rater panels.

When this is the wrong choice

• The adapters barely overlap. Run the overlap report first. Fabric drape and hand occlusion share little, and two scalars at 0.6 and 0.4 held that pair well enough to ship.
• Every request needs a different combination. Baking gives up per-request adapter selection. If the product genuinely varies the combination per call, pay the serving overhead and stack at runtime.
• The adapters are still moving. While you are still tuning what an adapter does, stacking lets you swap it without a rebuild and a visual review.
• Nobody has written down the priorities. The resolver enforces a priors file. Without a human decision about which adapter owns skin, backdrop, and garment surface, there is nothing for it to enforce, and per-layer resolution just picks a different arbitrary answer than a scalar would.