One LoRA per failure mode

The bug had a name inside the team before it had a ticket. Put a person in the input image with one hand resting on their hip, run the pipeline, and the generated garment gets painted over the fingers. The hand is just gone, absorbed into the fabric. We called it the "Amputated Hand", and for a few weeks I tried to fix it the way everyone tries to fix it, by adding more of that kind of image to the finetune we already had.

Why the single mixed finetune lost

We started with one LoRA trained on "hard cases". That set was a pile of everything the base model got wrong: sarees, kimonos, hanfus, couture with unusual topology, plus the occlusion images that produced the amputated hands. One adapter, one training run, one number to tune. It's the cheapest thing to build and it's what I'd do again if I hadn't watched it fail.

What happened is that the two problems in that pile want opposite behaviour from the model.

Draping is a generation problem. A saree pallu has to fall over the shoulder with volume, the pleats have to gather at the waist and cast their own shadows, and the adapter's job is to push the model to invent structure inside the masked region that the base model has no prior for. The training signal rewards high frequency detail surviving the denoise.

Occlusion is a suppression problem. When a hand sits on a hip, the correct behaviour inside part of the masked region is to generate nothing and leave the hand alone. The adapter's job is to teach the model that some pixels inside the mask are foreground and immutable.

At rank 32 the adapter has a fixed budget of directions it can write into the base weights. Asking it to simultaneously mean "add structure here" and "add nothing here" spends that budget on an average of the two. When I added occlusion images to the mixed set, the hands started surviving and the pleats went flat again. When I rebalanced toward drape, the hands came back off. I was tuning a dataset mixing ratio, and every value of that ratio cost a full training run to evaluate.

There's a second problem with the single finetune that took longer to notice. When it regressed, I couldn't tell which images caused it. A mixed dataset gives you one loss curve and no attribution.

Triage first, data collection second

So I stopped collecting and started sorting.

The honest starting state was a folder of a few hundred rejected PNGs with timestamps for filenames, and the reasons they'd been rejected living in review threads rather than anywhere structured. That's the condition most of these projects are in at the start, and the tidy version of this story opens with a defect taxonomy nobody actually has.

The first pass was a contact sheet and a person. Print the rejects in a grid, go through them, say out loud what's wrong with each one. That produced about a dozen phrases and the phrases repeated. "Hand gone." "Pleats flat." "Looks like a bathrobe." Nobody agreed on wording, which is why the classifier below matches substrings against a blob of text rather than reading a clean enum. The second pass was making the tagging cheap enough that it kept happening, so the reviewer's job became one line of free text with no dropdown and no schema. Then I ran this over the log.

# triage_failures.py
"""Bucket rejected try-on renders into named failure modes.
 
Reviewers tag a rejected render with short free-text labels ("hand gone",
"pleats flat", "looks like a bathrobe"). This maps those labels onto the failure
modes we actually name, and prints the distribution, so that dataset collection
targets a mode instead of a vague notion of "hard cases".
"""
 
import json
from collections import Counter
from pathlib import Path
 
# Label fragments mapped to the failure mode they belong to. Deliberately not
# exhaustive: anything unmatched lands in "unclassified" and gets read by hand,
# which is how new modes get discovered.
MODE_PATTERNS = {
    "drape": ["pleat", "fold", "flat texture", "pallu", "no volume", "plastered"],
    "occlusion": ["hand", "finger", "arm", "jewel", "necklace", "painted over"],
    "style_drift": ["wrong texture", "lost weave", "flat colour", "not the fabric"],
    "semantic": ["bathrobe", "wrong garment", "looks like a dress"],
}
 
 
def classify(tags: list[str]) -> set[str]:
    """Return every failure mode whose patterns appear in a render's tags.
 
    A render can belong to more than one mode. A saree rendered flat *and* over a
    hand is two separate defects and should be counted twice, otherwise the
    rarer mode gets hidden behind the common one.
    """
    blob = " ".join(tags).lower()
    modes = {mode for mode, pats in MODE_PATTERNS.items()
             if any(p in blob for p in pats)}
    return modes or {"unclassified"}
 
 
def summarise(reject_log: Path) -> Counter:
    counts = Counter()
    for line in reject_log.read_text().splitlines():
        record = json.loads(line)
        for mode in classify(record["tags"]):
            counts[mode] += 1
    return counts
 
 
if __name__ == "__main__":
    counts = summarise(Path("data/rejects.jsonl"))
    total = sum(counts.values())
    for mode, n in counts.most_common():
        print(f"{mode:>14}  {n:>5}  {n / total:6.1%}")
    print(f"{'total defects':>14}  {total:>5}")

The distribution isn't the interesting part; what mattered is that two buckets were big enough to justify their own training run and the other two weren't. That left drape and occlusion.

What earns an adapter

Not every named failure deserves its own training run, and I got this wrong twice before I had a rule I could apply. The bar has four parts.

It has to be describable as a behaviour rather than as a defect. "The hand disappears" is a defect; "treat foreground objects inside the mask as immutable" is a behaviour, and a behaviour is something you can put in a dataset.

You have to be able to collect data where that behaviour is the dominant variable. Every image in the occlusion set has a hand or an arm or a piece of jewelry interacting with the garment, so the thing you want learned is present in nearly every gradient step. A set where it appears in a fifth of the images trains a fifth of an adapter.

It has to conflict with something you're already training, or it doesn't need to be separate. This is the part people skip. Separation is a cost you pay to avoid interference, so only pay it when interference is what you're seeing.

And it has to be a base model gap rather than a conditioning gap. Style drift looked like an adapter problem for about a week. It wasn't: the model knew perfectly well what a weave is, it wasn't receiving the reference's structure, and Flux Redux fixed it without training anything.

Drape and occlusion cleared all four. Kimono-as-bathrobe cleared none of them, since it's the base model's semantic prior and a few thousand images don't move a prior that size.

The case against splitting

One general adapter is a better artifact than two specialists in most of the ways that matter once the research is done, and it deserves a fair hearing before I throw it out. One training run to schedule, one file to version, one thing to load, no merge step. The model you evaluated is the model you serve, which stops being true the moment you fuse two adapters, since the fused weights are a third model nobody trained. No coefficient sitting in the middle of your pipeline for somebody to re-derive when the base checkpoint moves.

The scaling argument is the strongest one. Under a single adapter a new failure mode is more data in an existing set. Under a split, it's a new artifact plus a new interaction to test against everything already there: two adapters is one pair, four is six pairs, and there's no reason to believe pairwise coverage is enough.

I rejected it on the narrow ground that one adapter demonstrably could not hold both behaviours at rank 32, and I had the flat pleats to show for it. If the mixed set had worked I'd have shipped the mixed set. The split isn't the better design in the abstract; it's what the training runs left us with.

The two experts

1. Expert A, draping physics

5,000 images of sarees, hanfus, kimonos and complex haute couture. Everything in it was chosen because the garment's shape is not a function of the body underneath it. A t-shirt is roughly a tube on a torso. A saree is fabric obeying gravity, and where it ends up depends on how it was wrapped.

One thing worth stating plainly because we confused ourselves with it internally more than once: this 5,000-image set is not the same 5,000-image corpus we used to seed the scene generation side, the one organised by garment layer into denim, silk and leather. Same round number, completely different data, different purpose, different model. If you're reading both, keep them apart.

Filtering was blunt: resolution, sharpness, aspect ratio. Anything soft or oddly cropped went out, because a blurred pleat teaches the model that pleats are blurred.

# train_drape_lora.sh
# Expert A: draping physics. Kohya-ss, single L40S (48GB).
# Rank stays at 32 for both experts so the two adapters compose predictably later.
 
accelerate launch --num_cpu_threads_per_process 8 \
  sdxl_train_network.py \
  --pretrained_model_name_or_path "$FLUX_BASE" \
  --train_data_dir  "./datasets/drape_5k" \
  --output_dir      "./out/lora_drape" \
  --output_name     "expert_a_drape" \
  --resolution      1024,1024 \
  --network_module  networks.lora \
  --network_dim     32 \
  --network_alpha   16 \
  --learning_rate   1e-4 \
  --lr_scheduler    cosine \
  --train_batch_size 4 \
  --gradient_accumulation_steps 4 \
  --gradient_checkpointing \
  --optimizer_type  AdamW8bit \
  --mixed_precision bf16 \
  --save_every_n_epochs 1 \
  --caption_extension .txt \
  --keep_tokens 1 \
  --enable_bucket \
  --min_bucket_reso 768 \
  --max_bucket_reso 1536

Rank 32 with alpha 16 was not an optimisation, it was a decision to stop optimising. Both experts share it, which means when I later merge them I'm combining two updates of the same shape and the same effective scale, and any difference in their influence comes from a coefficient I set rather than from a training artifact I forgot about. Bucketing matters here more than it does for square product shots, since a full-length saree image is tall and cropping it to a square throws away the drape you're trying to teach.

2. Expert B, occlusion and depth

3,000 images: hands on hips, arms crossed, jewelry sitting over clothing. The model doesn't have to learn what a hand looks like, it already knows. It has to learn that a hand inside the inpainting mask is not a hole to be filled.

The size difference between the two sets isn't a budget compromise, it follows from how much variation each concept carries. Drape is a product of several axes at once: garment family, how the fabric was wrapped, its weight and stiffness, and where the folds land for a given stance. Crossing those eats 5,000 images quickly. Occlusion is one relation, foreground in front of fabric, over the small number of poses commercial photography actually uses, so once you have hands on hips from several angles and distances the next thousand images are mostly the same lesson again.

Supply matters too. A usable occlusion image needs the interaction clearly visible and the garment boundary clean enough to mask, and the prep script below discards more than I expected on those grounds. We stopped adding when new images stopped changing anything we could see in the eval renders, which is a soft criterion, and I'd rather say it's soft than dress it up.

The data prep did most of the work. For every image we produced a garment mask and a foreground mask, and the caption named the interaction explicitly so the concept had a token to attach to.

# prep_occlusion_set.py
"""Build the Expert B training set: garment masks with foreground held out.
 
For each source image we run SAM2 twice, once for the garment and once for the
occluding body part or accessory, then subtract the second from the first. The
adapter is trained against a mask that already excludes the hand, so the target
behaviour ("leave this alone") is present in the supervision rather than being
something we hope emerges.
"""
 
from pathlib import Path
 
import numpy as np
from PIL import Image
 
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
 
GARMENT_PROMPTS = ["shirt", "tshirt", "top", "saree", "dress"]
FOREGROUND_PROMPTS = ["hand", "fingers", "forearm", "necklace", "bracelet"]
 
DILATE_PX = 9  # inside the 5 to 15 px bleeding zone we use at inference
 
 
def dilate(mask: np.ndarray, radius: int) -> np.ndarray:
    """Grow a boolean mask by `radius` pixels using a square structuring element."""
    from scipy.ndimage import binary_dilation
    kernel = np.ones((radius * 2 + 1, radius * 2 + 1), dtype=bool)
    return binary_dilation(mask, structure=kernel)
 
 
def build_masks(predictor: SAM2ImagePredictor, image: Image.Image):
    predictor.set_image(np.array(image))
    garment, _, _ = predictor.predict(text=GARMENT_PROMPTS, multimask_output=False)
    foreground, _, _ = predictor.predict(text=FOREGROUND_PROMPTS, multimask_output=False)
 
    garment = dilate(garment[0].astype(bool), DILATE_PX)
    # The foreground is dilated harder. An under-sized hand mask leaks skin into
    # the region the model is allowed to repaint, and that is exactly the defect
    # this expert exists to remove.
    foreground = dilate(foreground[0].astype(bool), DILATE_PX * 2)
 
    return garment & ~foreground, foreground
 
 
def main(src: Path, dst: Path):
    predictor = SAM2ImagePredictor(build_sam2("sam2_hiera_l.yaml", "sam2_hiera_large.pt"))
    dst.mkdir(parents=True, exist_ok=True)
    kept = skipped = 0
 
    for path in sorted(src.glob("*.jpg")):
        image = Image.open(path).convert("RGB")
        target, foreground = build_masks(predictor, image)
 
        # If the occluder covers almost nothing, the image teaches nothing.
        if foreground.mean() < 0.005:
            skipped += 1
            continue
 
        Image.fromarray((target * 255).astype(np.uint8)).save(dst / f"{path.stem}_mask.png")
        image.save(dst / path.name)
        (dst / f"{path.stem}.txt").write_text("hand resting over garment, hand in front of fabric")
        kept += 1
 
    print(f"kept {kept} / skipped {skipped} (occluder too small)")
 
 
if __name__ == "__main__":
    main(Path("raw/occlusion"), Path("datasets/occlusion_3k"))

Training config for Expert B is the same as Expert A apart from the data directory and the output name. That's intentional. The only variable I wanted between the two runs was the data.

What broke, and what to watch for

Both adapters passed on their own. Loaded alone against the base, Expert A gives you pleats with volume and shadows that fall the right way, and Expert B keeps hands in front of fabric. Loaded together, they degrade each other, and that turned out to be a much longer story than the one in this post. I'm not going to pretend it was solved in May; it wasn't.

Two other things stayed broken and are worth naming so nobody reads this as a clean win. Sheer fabrics like chiffon and lace still render opaque, because getting them right means blending skin tone with fabric rather than replacing one with the other, and neither expert was trained to do that. Extreme poses still fail, since both datasets skew toward standing and seated subjects, which is what commercial fashion photography mostly is.

Where the split did pay off was the eventual eval. On our 500-image set, split evenly between Western casual and global traditional, the full pipeline reached 92% occlusion accuracy against 70% for base Flux Fill, and 0.85 SSIM on the global traditional half against 0.58. Those are pipeline numbers rather than adapter numbers, but the occlusion column in particular is the direct descendant of a 3,000-image dataset that exists only because we bothered to name the failure first.

What I learned

Name the failure before you collect the data. "Hard cases" is not a category, and a dataset built from it trains an adapter that is mediocre at several things instead of good at one.

Separate adapters give you separate eval. When one regresses, you know which training run to look at and which images to blame. A single mixed finetune gives you a loss curve and a shrug.

Keep the shapes identical across experts. Same rank, same alpha, same optimiser. It costs nothing at training time and it's what makes the adapters comparable when you go to combine them.

The next thing was combining them into one model we could actually serve, since loading two adapters and switching between them per request was not a serving story I wanted. That's where the interesting part starts, and it did not go the way I expected.