One LoRA per failure mode

When a model in the input photo rested a hand on her hip, our early try-on pipeline painted garment fabric directly over her fingers. The hand disappeared into the cloth.

We called this defect the "Amputated Hand." I initially tried to fix it by pooling all failure cases (sarees, kimonos, jewelry occlusions, and hand poses) into a single dataset for joint fine-tuning.

Pooling disparate failure modes into one training set creates conflicting optimization targets within a fixed adapter rank budget, yielding averaged behavior that fixes neither problem.

Capacity limits of single low-rank adapters

A single LoRA trained on all difficult edge cases failed because the two primary defects demand opposing model behaviors:

Fabric draping is a generative task: A saree pallu must fall across the shoulder with natural volume, and pleats must gather at the waist casting directional shadows. The adapter must push the diffusion model to synthesize new geometric folds within the masked region.

Hand occlusion is a suppression task: When fingers rest over a torso, the model must suppress generation across those specific pixels, treating foreground anatomy as immutable.

At rank 32, a LoRA possesses a constrained subspace of parameter updates. Forcing it to simultaneously synthesize complex folds and suppress generation over foreground pixels spends that capacity on a compromised average.

Adding occlusion samples restored hands but flattened pleats. Rebalancing toward draping samples restored pleat depth but caused fabric to overpaint fingers again. Evaluating each iteration required a complete training run with no layer-wise attribution.

Step 1: Triage the failure logs before expanding the dataset

Rather than scraping more uncategorized data, we classified rejected renders by failure type.

Reviewers tagged rejected outputs with concise notes ("hand clipped", "pleats flat", "kimono looks like bathrobe"). We parsed these logs to map recurring terms into concrete failure categories:

# triage_failures.py
"""Bucket rejected try-on renders into named failure modes.
 
Reviewers tag rejected renders with short labels. This script maps
labels to specific failure categories to focus dataset collection.
"""
 
import json
from collections import Counter
from pathlib import Path
 
MODE_PATTERNS = {
    "drape": ["pleat", "fold", "flat texture", "pallu", "no volume", "plastered"],
    "occlusion": ["hand", "finger", "arm", "jewel", "necklace", "painted over"],
    "style_drift": ["wrong texture", "lost weave", "flat colour", "not the fabric"],
    "semantic": ["bathrobe", "wrong garment", "looks like a dress"],
}
 
 
def classify(tags: list[str]) -> set[str]:
    """Return every failure mode matching a render's tags.
 
    A render displaying both flat pleats and occluded hands counts toward
    both categories to prevent hiding secondary failure modes.
    """
    blob = " ".join(tags).lower()
    modes = {mode for mode, pats in MODE_PATTERNS.items()
             if any(p in blob for p in pats)}
    return modes or {"unclassified"}
 
 
def summarise(reject_log: Path) -> Counter:
    counts = Counter()
    for line in reject_log.read_text().splitlines():
        record = json.loads(line)
        for mode in classify(record["tags"]):
            counts[mode] += 1
    return counts
 
 
if __name__ == "__main__":
    counts = summarise(Path("data/rejects.jsonl"))
    total = sum(counts.values())
    for mode, n in counts.most_common():
        print(f"{mode:>14}  {n:>5}  {n / total:6.1%}")
    print(f"{'total defects':>14}  {total:>5}")

Two defect modes dominated the logs: drape collapse and foreground occlusion.

The tell: if the reject tags do not cluster into a handful of named buckets, you have a data quality problem, not a set of failure modes to split on.

Step 2: Test each failure mode against four criteria

Before allocating compute to train an isolated adapter, a candidate failure mode must satisfy four conditions:

1. Formulated as an actionable behavior: "Treat foreground hands as immutable" can be supervised in a dataset; "hands look broken" cannot.
2. Isolatable in training data: The target behavior must represent the primary variable across training pairs.
3. Presents conflicting gradients with existing adapters: Separate adapters are only needed when joint training causes parameter interference.
4. Represents a base weight gap rather than a conditioning issue: Style drift was resolved by adding Flux Redux image conditioning without training new adapter weights.

Draping physics and occlusion boundaries satisfied all four criteria.

Step 3: Weigh single-adapter simplicity against specialists

Maintaining a single unified adapter offers operational advantages: one training schedule, one artifact to version, and no merge step.

However, when a single rank-32 adapter cannot simultaneously learn orthogonal generative and suppressive behaviors, splitting the task into specialized models becomes necessary.

Step 4: Train the drape specialist with aspect bucketing

Expert A (draping physics) was trained on 5,000 curated images of sarees, hanfus, kimonos, and structured couture. These garments do not conform to body contours like t-shirts; their folds are dictated by wrapping technique and gravity.

# train_drape_lora.sh
# Expert A: draping physics. Single NVIDIA L40S (48GB).
# Rank 32 and alpha 16 match Expert B for predictable weight composition.
 
accelerate launch --num_cpu_threads_per_process 8 \
  sdxl_train_network.py \
  --pretrained_model_name_or_path "$FLUX_BASE" \
  --train_data_dir  "./datasets/drape_5k" \
  --output_dir      "./out/lora_drape" \
  --output_name     "expert_a_drape" \
  --resolution      1024,1024 \
  --network_module  networks.lora \
  --network_dim     32 \
  --network_alpha   16 \
  --learning_rate   1e-4 \
  --lr_scheduler    cosine \
  --train_batch_size 4 \
  --gradient_accumulation_steps 4 \
  --gradient_checkpointing \
  --optimizer_type  AdamW8bit \
  --mixed_precision bf16 \
  --save_every_n_epochs 1 \
  --caption_extension .txt \
  --keep_tokens 1 \
  --enable_bucket \
  --min_bucket_reso 768 \
  --max_bucket_reso 1536

Aspect ratio bucketing is essential for full-length garments: cropping vertical saree shots to square frames removes the lower pleat cascades that the adapter is meant to learn.

Step 5: Supervise occlusion boundaries explicitly

Expert B (occlusion and depth) was trained on 3,000 images showing hands on hips, crossed arms, and jewelry over fabric.

We prepared masks by segmenting both the garment and the occluding foreground object with SAM2, then subtracting the foreground mask from the garment mask. The adapter learns to leave foreground elements untouched because the supervision explicitly holds them out.

# prep_occlusion_set.py
"""Build Expert B training set: garment masks with foreground held out.
 
For each image, SAM2 segments the garment and the occluding limb or accessory.
Subtracting the foreground mask ensures the model learns to preserve foreground
structures rather than repainting over them.
"""
 
from pathlib import Path
 
import numpy as np
from PIL import Image
 
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
 
GARMENT_PROMPTS = ["shirt", "tshirt", "top", "saree", "dress"]
FOREGROUND_PROMPTS = ["hand", "fingers", "forearm", "necklace", "bracelet"]
 
DILATE_PX = 9
 
 
def dilate(mask: np.ndarray, radius: int) -> np.ndarray:
    """Expand boolean mask using a square structuring element."""
    from scipy.ndimage import binary_dilation
    kernel = np.ones((radius * 2 + 1, radius * 2 + 1), dtype=bool)
    return binary_dilation(mask, structure=kernel)
 
 
def build_masks(predictor: SAM2ImagePredictor, image: Image.Image):
    predictor.set_image(np.array(image))
    garment, _, _ = predictor.predict(text=GARMENT_PROMPTS, multimask_output=False)
    foreground, _, _ = predictor.predict(text=FOREGROUND_PROMPTS, multimask_output=False)
 
    garment = dilate(garment[0].astype(bool), DILATE_PX)
    foreground = dilate(foreground[0].astype(bool), DILATE_PX * 2)
 
    return garment & ~foreground, foreground
 
 
def main(src: Path, dst: Path):
    predictor = SAM2ImagePredictor(build_sam2("sam2_hiera_l.yaml", "sam2_hiera_large.pt"))
    dst.mkdir(parents=True, exist_ok=True)
    kept = skipped = 0
 
    for path in sorted(src.glob("*.jpg")):
        image = Image.open(path).convert("RGB")
        target, foreground = build_masks(predictor, image)
 
        if foreground.mean() < 0.005:
            skipped += 1
            continue
 
        Image.fromarray((target * 255).astype(np.uint8)).save(dst / f"{path.stem}_mask.png")
        image.save(dst / path.name)
        (dst / f"{path.stem}.txt").write_text("hand resting over garment, hand in front of fabric")
        kept += 1
 
    print(f"Kept {kept} / skipped {skipped} (occluder too small)")
 
 
if __name__ == "__main__":
    main(Path("raw/occlusion"), Path("datasets/occlusion_3k"))

Both adapters share identical training hyperparameter shapes (rank 32, alpha 16, cosine learning rate schedule), making them directly comparable when merged.

Failure modes and edge cases

1. Inter-adapter interference upon merging: While each adapter succeeds independently, direct linear addition into base weights can introduce interference across overlapping attention layers.
2. Sheer fabric transparency: Neither drape nor occlusion adapters model alpha transparency in sheer lace or chiffon.
3. Dataset distribution bias: Both datasets primarily cover standing and seated catalog poses; unconventional poses continue to exhibit geometry warping.

When this is the wrong choice

• The tasks do not conflict. If adding the new training data leaves existing behavior intact, criterion 3 fails and there is nothing to split. Keep one adapter and skip the merge step entirely.
• The failure is a conditioning gap. Missing reference texture or a pattern repeat is fixed by image conditioning, which is how style drift got resolved here. Training new weights for it spends compute on a problem that has no weight component.
• Serving cannot load more than one checkpoint. If the infrastructure supports neither multi-adapter builds nor offline weight fusion, a single compromised adapter is the only thing you can actually ship, and the compromise is real: hands or pleats, not both.