Virtual try-on is not image generation

Generating a picture of someone wearing a shirt is a solved problem. Generating a picture of someone wearing that shirt, the one already photographed on a white background in the brand's catalogue, with the same weave and the same logo placement and the same button spacing, is not. Most of the published work slides between those two problems without saying so, and I did too for about a week. This post is the correction.

Why a good text-to-image model is the wrong tool

The thing that makes text-to-image impressive is exactly what makes it useless here. You give it a description, it gives you a plausible instance of that description. Plausible is the whole product. In try-on, the customer is looking at the garment on the next tab over. They already know what it looks like. If the render gives them a slightly different collar, the render is wrong, and no amount of aesthetic quality fixes it.

So garment fidelity is a constraint, not a suggestion. There is one correct answer and the model either hits it or doesn't.

The second constraint is identity. The person in the output has to be the same person who was in the input, down to the face, the hands, the hair, the skin tone at the collarbone. Brands are shipping these images next to a buy button. A model that shifts a face by five percent produces something that looks fine in isolation and deeply wrong next to the original. We had brand teams reject renders for reasons they couldn't articulate beyond "that's not her", and they were always right.

Two hard boundaries, then, and text-to-image respects neither by construction. It has no mechanism to be told "this region is immutable" and no mechanism to be told "this texture is the ground truth, not a hint".

Western bias, and what it does to a saree

Once you accept that try-on is a constrained edit rather than a generation, the next problem shows up in the weights themselves.

The open checkpoints everyone starts from, Stable Diffusion 1.5 and SDXL, and the try-on datasets built around them, are overwhelmingly Western casual wear. T-shirts, jeans, hoodies, jackets. Those garments share a topology: they are approximately tubes that sit close to the body, and the body underneath determines the silhouette. A model that has seen a million t-shirts has learned a very good prior for "fabric conforming to a torso".

That prior is actively wrong for a lot of what our clients sell. A saree isn't determined by the body. It's determined by pleats, by the specific drape of the pallu over one shoulder, and by gravity acting on several metres of fabric that is not attached to anything. The model, having only the tube prior, renders it as a flat printed texture painted onto the skin. The pleats become a pattern rather than a set of folds with their own shadows. It looks, at a glance, like a saree. It looks nothing like a saree to anyone who wears one.

The failure gets funnier and worse at the semantic level. Ask a stock inpainting model to put a kimono on someone and there is a good chance it produces a bathrobe. That isn't a rendering bug. The model's nearest concept to "loose belted robe" is drawn from a training distribution where that shape is a bathrobe, so it confidently gives you a bathrobe with slightly nicer fabric. You cannot prompt your way out of a missing concept.

We started calling this Western Bias internally, and it stopped being a talking point and became a number the moment we had an eval set. Five hundred images, split evenly between Western casual and global traditional. On SDXL with a ControlNet baseline we got SSIM 0.82 against the reference garment on the Western half and 0.45 on the global half. Moving to Flux Fill as the base inpainter helped a lot on the easy half and much less on the hard one: 0.91 and 0.58.

The complexity gap

That spread between the two halves is the thing worth naming, so we named it the Complexity Gap. It's not that global garments are harder in some general sense. It's that every improvement we made to the base pipeline improved the Western half faster than the global half, so the gap widened as the system got better. A team optimizing for an average score would have shipped a product that got progressively worse at the thing our market actually needed.

There is a second axis of the same problem that has nothing to do with culture. Complex poses involve hands touching the garment: a hand on a hip, a hand holding a lapel, arms crossed, a bangle sitting on top of a sleeve. Standard inpainting has no notion of depth ordering inside the masked region, so it paints garment texture straight over the hand. We called that one the Amputated Hand, and it is the single most reliable way to make a person recoil from an otherwise good image.

Inpainting, not img2img

The first thing I tried was the obvious thing, and I want to keep it here because the way it fails points directly at the fix.

# tryon_img2img_naive.py
"""First attempt at try-on: img2img over the model photo with a garment prompt.
 
Kept in the repo because the failure mode is instructive, not because it works.
"""
import torch
from diffusers import StableDiffusionXLImg2ImgPipeline
from PIL import Image
 
MODEL_PHOTO = "data/model_0412.png"
PROMPT = (
    "a woman wearing a deep green silk saree with a gold zari border, "
    "studio lighting, plain background, full body"
)
 
pipe = StableDiffusionXLImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
).to("cuda")
 
 
def sweep_strength(image_path: str, prompt: str, values: list[float]) -> None:
    """Run img2img at several denoise strengths and write each result to disk.
 
    `strength` is the fraction of the diffusion trajectory we re-noise before
    denoising back. It is a single global knob applied to every pixel.
    """
    init = Image.open(image_path).convert("RGB").resize((1024, 1024))
    for s in values:
        out = pipe(
            prompt=prompt,
            image=init,
            strength=s,
            guidance_scale=6.0,
            num_inference_steps=30,
        ).images[0]
        path = f"out/img2img_s{int(s * 100):03d}.png"
        out.save(path)
        print(f"strength={s:.2f} -> {path}")
 
 
sweep_strength(MODEL_PHOTO, PROMPT, [0.35, 0.55, 0.75, 0.95])

Run that sweep and look at the four outputs side by side. At low strength the original garment is still visible through the new one, because you haven't destroyed enough of the original signal for the model to replace it. At high strength you get a convincing saree on a woman who is not the woman you started with. Her face has drifted, her arms are a different length, and the background studio seam has moved.

There is no value of strength that works, and the reason is structural. img2img gives you one policy for the entire frame, and try-on needs two: destroy everything inside the garment region, preserve everything outside it exactly. That is the definition of inpainting. So the pipeline is an inpainting pipeline, and the interesting engineering moves into the mask.

The mask does most of the work

If the mask is wrong, nothing downstream can save you. Too tight and you get a halo of the old garment around the collar and cuffs. Too loose and the model starts inventing shoulders. We ended up on SAM2 for the boundary, with a hierarchical prompt set, and a morphological dilation on top.

# build_garment_mask.py
"""Produce the inpainting mask for a single try-on job.
 
SAM2 gives us the garment boundary. Dilation gives the diffusion model a few
pixels of adjacent skin and background to blend against, which is what stops
the composite from showing a hard seam at the neckline.
"""
import cv2
import numpy as np
 
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
 
GARMENT_PROMPTS = ["shirt", "tshirt", "top", "saree", "dress"]
DILATION_PX = 11           # the bleeding zone; we ran anywhere from 5 to 15
MIN_MASK_AREA_FRAC = 0.04  # below this the segmentation has clearly missed
 
predictor = SAM2ImagePredictor(
    build_sam2("sam2_hiera_l.yaml", "checkpoints/sam2_hiera_large.pt")
)
 
 
def garment_mask(image_bgr: np.ndarray, prompts: list[str]) -> np.ndarray:
    """Return a dilated binary mask covering the garment region.
 
    Raises if the winning mask is implausibly small, which in practice means
    the subject is turned away or the garment is mostly out of frame.
    """
    predictor.set_image(cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB))
    masks, scores, _ = predictor.predict(prompts=prompts, multimask_output=True)
 
    best = masks[int(np.argmax(scores))].astype(np.uint8)
    frac = float(best.sum()) / best.size
    if frac < MIN_MASK_AREA_FRAC:
        raise ValueError(f"mask covers {frac:.3f} of the frame, refusing to inpaint")
 
    kernel = cv2.getStructuringElement(
        cv2.MORPH_ELLIPSE, (DILATION_PX, DILATION_PX)
    )
    dilated = cv2.dilate(best, kernel, iterations=1)
 
    grown = (float(dilated.sum()) / dilated.size) - frac
    print(f"mask frac={frac:.3f}, grew by {grown:.3f} at {DILATION_PX}px dilation")
    return dilated * 255

The prompts argument there is our own wrapper, not upstream SAM2's signature; the resolver that turns a garment label into concrete box prompts is a small piece of glue we wrote and it is not the interesting part. The interesting part is DILATION_PX. That band of dilated pixels is the only place where the model can see both fabric and skin at once, and it is where the transition between the two gets decided. Too thin and the neckline reads as a sticker. Too thick and you have handed the model permission to modify the collarbone, which it will happily take.

Conditioning on the garment, not on a description of it

The mask tells the model where to paint. Something else has to tell it what to paint, and a text prompt is a terrible channel for that. "Green silk saree with gold zari border" is maybe forty bits of information about an object that has thousands of bits of relevant structure.

This is where the dual-stream setup came from. Flux Fill handles the inpainting. Flux Redux runs alongside it and injects structural features from the reference garment image itself into cross-attention.

# tryon_dual_stream.py
"""Dual-stream try-on: Flux Fill inpaints, Flux Redux conditions on structure.
 
The Redux stream is what separates this from prompt-driven inpainting. A CLIP
vision encoder would compress the reference to something like "a red knit
sweater"; Redux keeps enough structure to reproduce the knit itself.
"""
from dataclasses import dataclass
 
 
@dataclass(frozen=True)
class TryOnConfig:
    """Inference settings for a single garment swap."""
 
    steps_primary: int = 30
    steps_refine: int = 10
    sampler: str = "euler_ancestral"
    scheduler: str = "beta"
    native_resolution: int = 1024
    redux_weight: float = 1.0
 
 
def run_tryon(model_image, garment_image, mask, cfg: TryOnConfig):
    """Composite `garment_image` onto `model_image` inside `mask`.
 
    Returns the inpainted frame at the model's native resolution. Compositing
    back into the original 4K frame happens one layer up.
    """
    style = redux_encoder(sigclip(garment_image), weight=cfg.redux_weight)
 
    latents = flux_fill(
        image=model_image,
        mask=mask,
        conditioning=style,
        steps=cfg.steps_primary,
        sampler=cfg.sampler,
        scheduler=cfg.scheduler,
        width=cfg.native_resolution,
        height=cfg.native_resolution,
    )
    refined = flux_fill(
        image=latents,
        mask=mask,
        conditioning=style,
        steps=cfg.steps_refine,
        sampler=cfg.sampler,
        scheduler=cfg.scheduler,
        denoise=0.35,
    )
    print(f"primary {cfg.steps_primary} steps, refine {cfg.steps_refine} steps")
    return refined

The named failure this fixes is Style Drift: the output matches the reference in colour and silhouette but loses the material. A cable-knit sweater arrives as a flat red top. A brocade arrives as printed fabric. Colour survives the compression to a semantic vector and weave does not, so if your only conditioning path is semantic, weave is what you lose.

What still breaks

Sheer fabrics. Lace and chiffon need the model to blend skin tone through the fabric rather than paint over it, and we render them opaque. I don't have a fix for this that doesn't involve a separate alpha-aware path.

Extreme poses. Anything outside standing and sitting degrades, because both the base model and everything we trained on top of it is biased toward catalogue poses. Yoga and dance shoots fail.

And the gap itself is still open at this point. Getting from 0.58 on global traditional to something a brand will sign off on isn't a mask problem or a conditioning problem. It's a missing prior, and priors have to be trained.

What I learned

The constraint set defines the architecture, and I lost time by not writing the constraints down first. Once "the garment is ground truth" and "the person is immutable" were on a whiteboard, inpainting was the only option left and the img2img sweep was obviously a dead end.

Averages hide the only problem that matters. A single quality score across our eval set went up every week while the half of the set we actually cared about barely moved. Split your metrics along the axis where you expect the model to be biased, before you have evidence that it is.

Semantic conditioning and structural conditioning are different channels, and text is the weakest structural channel available. Anything you can hand the model as pixels, hand it as pixels.

The failures are worth naming. Western Bias, the Complexity Gap, Style Drift, the Amputated Hand: once a failure has a name, a person can file a bug against it, and a brand reviewer can tell you which one they're looking at. That did more for our review cycle than any single model change.

Next up is the missing prior. I'm training a LoRA on nothing but garments whose shape comes from gravity rather than from the body underneath, sarees and hanfus and kimonos and haute couture, to see whether the drape can be taught as a specialisation rather than baked into a foundation model none of us can afford to retrain.