Virtual try-on is not image generation

Our early saree render looked plausible at a glance. Up close the pleats were flat printed texture, not folds with geometry casting directional shadows. On the global traditional half of our evaluation set, that baseline scored SSIM 0.45 against the reference garment.

On the Western casual half of the exact same set, it scored 0.82.

A try-on pipeline has one valid solution: the exact physical SKU rendered on the designated model. Every alternate output represents a defect, regardless of aesthetic appeal.

Generating a convincing photo of someone wearing a generic shirt is straightforward. Generating that specific inventory SKU with the correct textile weave, seam placement, and button spacing while preserving the model's identity is an entirely different engineering challenge.

Dual constraints of garment fidelity and identity preservation

Virtual try-on requires managing two distinct image regions under strict invariant policies:

Garment fidelity: The customer views the try-on output alongside a product catalog photo. Any alteration to neckline cut, print scale, or button spacing makes the render incorrect.

Identity preservation: The person in the output image must match the input photo exactly: facial anatomy, skin tone, hands, hair, and collarbones. A model that shifts facial features by even five percent produces uncanny outputs that retail brands reject.

Text-to-image synthesis respects neither constraint out of the box. It has no mechanism for treating unmasked regions as immutable, and none for holding a reference image as ground truth.

Step 1: Formulating try-on as a constraint satisfaction problem

Treat try-on as a model selection problem and you will pick the wrong pipeline. It is a constraint satisfaction problem.

Establishing "the garment is ground truth" and "the person is immutable" as hard constraints dictates the downstream architecture: localized inpainting over full-frame image-to-image, precise boundary masking, and dedicated visual conditioning streams.

Step 2: Measuring the complexity gap across garment categories

Standard foundation models (SDXL, Flux) and public try-on benchmarks skew heavily toward Western casual wear: fitted t-shirts, jeans, hoodies, and structured blazers. These garments conform to the body like cylindrical tubes, allowing foundation models with human anatomy priors to render them effectively.

Traditional global garments (sarees, kimonos, hanfus) behave differently. A saree is governed by pleat cascades, shoulder drapes, and gravity acting on unstitched fabric. Relying solely on cylindrical body priors causes diffusion models to paint traditional apparel as flat decals stuck to skin.

Foundation inpainting models also carry semantic biases: prompting for a kimono often generates a generic bathrobe because the training distribution associates loose belted garments with loungewear.

We stratified our evaluation suite into 500 image pairs split evenly between Western Casual and Global Traditional:

That delta is the Complexity Gap. Moving from SDXL to Flux Fill lifted both distributions, but global garments needed dedicated drape and depth adapters before they reached production quality.

A related failure mode occurs during complex poses: when models rest hands on hips or cross their arms, standard inpainting lacks depth ordering and repaints fabric over fingers (the "Amputated Hand" defect).

Step 3: Global latent distortion in img2img versus localized inpainting

We initially tested standard image-to-image (img2img) conditioning with descriptive prompts:

# tryon_img2img_naive.py
"""Illustrative example: img2img over a model photo with a text prompt."""
 
import torch
from diffusers import StableDiffusionXLImg2ImgPipeline
from PIL import Image
 
MODEL_PHOTO = "data/model_0412.png"
PROMPT = (
    "a woman wearing a deep green silk saree with a gold zari border, "
    "studio lighting, plain background, full body"
)
 
pipe = StableDiffusionXLImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
).to("cuda")
 
 
def sweep_strength(image_path: str, prompt: str, values: list[float]) -> None:
    """Run img2img at varying denoise strengths."""
    init = Image.open(image_path).convert("RGB").resize((1024, 1024))
    for s in values:
        out = pipe(
            prompt=prompt,
            image=init,
            strength=s,
            guidance_scale=6.0,
            num_inference_steps=30,
        ).images[0]
        path = f"out/img2img_s{int(s * 100):03d}.png"
        out.save(path)
        print(f"strength={s:.2f} -> {path}")

Running this sweep reveals a fundamental trade-off:

• At low denoise strength ($s = 0.35$), the original garment remains visible beneath the new texture.
• At high denoise strength ($s = 0.75$), a convincing garment appears, but the model's face, limb proportions, and background lighting drift.

Applying a single global denoise value across the entire frame cannot simultaneously replace clothing while preserving unmasked anatomy.

Step 4: Mask boundary engineering with SAM2 and morphological dilation

In an inpainting architecture, the mask determines where synthesis occurs. Tight masks leave halos of the original clothing along necklines, while loose masks permit unintended anatomical alterations.

We combine SAM2 segmentation with morphological dilation:

# build_garment_mask.py
"""Generate inpainting masks using SAM2 and morphological dilation."""
 
import cv2
import numpy as np
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
 
GARMENT_PROMPTS = ["shirt", "tshirt", "top", "saree", "dress"]
DILATION_PX = 11           # bleeding zone for boundary blending
MIN_MASK_AREA_FRAC = 0.04  # validation threshold for detected garment area
 
predictor = SAM2ImagePredictor(
    build_sam2("sam2_hiera_l.yaml", "checkpoints/sam2_hiera_large.pt")
)
 
 
def garment_mask(image_bgr: np.ndarray, prompts: list[str]) -> np.ndarray:
    """Return dilated binary mask covering the target garment region."""
    predictor.set_image(cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB))
    masks, scores, _ = predictor.predict(prompts=prompts, multimask_output=True)
 
    best = masks[int(np.argmax(scores))].astype(np.uint8)
    frac = float(best.sum()) / best.size
    if frac < MIN_MASK_AREA_FRAC:
        raise ValueError(f"Mask covers {frac:.3f} of frame; segmentation failed")
 
    kernel = cv2.getStructuringElement(
        cv2.MORPH_ELLIPSE, (DILATION_PX, DILATION_PX)
    )
    dilated = cv2.dilate(best, kernel, iterations=1)
 
    grown = (float(dilated.sum()) / dilated.size) - frac
    print(f"Mask frac={frac:.3f}, grew by {grown:.3f} with {DILATION_PX}px dilation")
    return dilated * 255

The dilated boundary gives the diffusion model room to blend fabric shadow against skin tone, which is what removes the hard collar seam.

Step 5: Dual-stream visual conditioning with Flux Redux

Text prompts provide insufficient bandwidth for detailed textile patterns: "green silk saree with gold border" contains roughly forty bits of semantic information for an object possessing complex structural weaves.

Our dual-stream architecture splits conditioning responsibilities: Flux Fill performs spatial inpainting, while Flux Redux extracts and injects dense visual feature tokens from the reference garment into cross-attention layers.

# tryon_dual_stream.py
"""Dual-stream inference with Flux Fill and Flux Redux conditioning."""
 
from dataclasses import dataclass
 
 
@dataclass(frozen=True)
class TryOnConfig:
    steps_primary: int = 30
    steps_refine: int = 10
    sampler: str = "euler_ancestral"
    scheduler: str = "beta"
    native_resolution: int = 1024
    redux_weight: float = 1.0
 
 
def run_tryon(model_image, garment_image, mask, cfg: TryOnConfig):
    """Composite garment reference onto model image within mask boundaries."""
    style = redux_encoder(sigclip(garment_image), weight=cfg.redux_weight)
 
    latents = flux_fill(
        image=model_image,
        mask=mask,
        conditioning=style,
        steps=cfg.steps_primary,
        sampler=cfg.sampler,
        scheduler=cfg.scheduler,
        width=cfg.native_resolution,
        height=cfg.native_resolution,
    )
    refined = flux_fill(
        image=latents,
        mask=mask,
        conditioning=style,
        steps=cfg.steps_refine,
        sampler=cfg.sampler,
        scheduler=cfg.scheduler,
        denoise=0.35,
    )
    return refined

Dual-stream conditioning is what stops style drift. Knit texture and jacquard weave survive synthesis instead of collapsing into flat color.

Failure modes and edge cases

1. Sheer fabric transparency: Rendering chiffon or lace requires alpha-aware blending of underlying skin tones rather than binary inpainting.
2. Extreme athletic poses: Non-standard poses (such as yoga or dance) challenge both segmentation and foundational diffusion priors.
3. Complex multi-layer occlusions: Multiple overlapping accessories (such as necklaces over scarves) require multi-stage depth masks to preserve layer ordering.

When this is the wrong choice

• There is no SKU to match. Concept art and moodboards have no ground-truth garment, so the constraint that drives this whole architecture does not exist. Unconstrained generation gives you more range for the same compute.
• Nobody's identity has to survive. If the model in the frame is fictional, the identity-preservation constraint disappears with them, and full-frame synthesis skips the segmentation, dilation, and boundary blending entirely.
• The garment is a plain t-shirt or a hoodie. Foundation priors already render body-conforming basics well. Custom adapter composition adds build steps and merge coefficients to a case that did not need them.