Two streams: why one diffusion pass couldn't do virtual try-on

Masking a torso and prompting for a red saree in Flux Fill produces a plausible red saree.

However, it will not reproduce the specific garment shipped by a brand. For e-commerce catalog production, that discrepancy invalidates the asset.

Text prompts cannot carry weave density or exact pattern alignment. Describing complex embroidery in text still leads the model toward whatever generic fabric priors dominated its pre-training data.

One conditioning stream cannot simultaneously anchor body anatomy and enforce reference garment fidelity, which led us to design the dual-stream architecture in Flux-VTON+.

Division of responsibilities across conditioning streams

A conditioning stream supplies guidance to the diffusion sampler. Flux Fill takes an image, an inpainting mask, and a text prompt, filling the masked area with content consistent with surrounding pixels. Given a model's torso and the prompt "red saree", it generates a generic saree. It lacks any representation of the specific gold zari border, thread count, or motif repeat.

Flux Redux provides the second conditioning stream. It encodes the reference garment directly, feeding high-dimensional structural tokens into the transformer's cross-attention layers.

The division of labor is separated by function:

Neither stream can fulfill the other's role.

Step 1: Identifying style drift and spatial distortion

Single-stream pipelines fail in two distinct ways:

Style drift happens under inpainting-only conditioning. The output matches the reference in base color and rough silhouette but loses structural detail. A cable-knit sweater flattens into a plain red top, and a hand-blocked motif degrades into a generic floral print. Low-frequency color persists, but high-frequency fabric structure is lost.

Spatial distortion and amputation happens when you condition heavily on reference structure without enough spatial anchoring. The model renders an accurate garment texture that disregards the subject's physical anatomy. Drapes ignore shoulder positions, hem lengths shift arbitrarily, and hands resting on hips get painted over with fabric.

The tell: if the color survived and the structure didn't, the reference stream is too weak. If the texture is right but it sits in the wrong place on the body, the mask is the problem.

Step 2: Segmenting with SAM2 and building the transition zone

Because inpainting relies on precise spatial boundaries, faulty masks degrade both conditioning streams. We use Segment Anything 2 (SAM2) with hierarchical prompts, followed by morphological dilation to create a transition zone.

# mask_garment.py
"""Produce the inpainting mask for a source image.
 
SAM2 provides a tight garment boundary. Morphological dilation creates
a transition zone that the sampler repaints, allowing new fabric to meet
skin without a visible seam.
"""
 
import cv2
import numpy as np
import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
 
GARMENT_PROMPTS = ["shirt", "tshirt", "top", "saree", "dress"]
 
# Transition zone radius in pixels at inference crop resolution.
# 5 for tight-fitting tops, up to 15 for flowing drapes.
DILATION_PX = 9
 
# Minimum mask fraction to filter out false segmentation hits.
MIN_MASK_FRACTION = 0.04
 
 
def build_predictor(checkpoint: str, config: str) -> SAM2ImagePredictor:
    """Load SAM2 once per worker; the encoder pass is compute-heavy."""
    model = build_sam2(config, checkpoint, device="cuda")
    return SAM2ImagePredictor(model)
 
 
def segment_garment(predictor, image_bgr: np.ndarray, prompts=GARMENT_PROMPTS):
    """Run hierarchical prompts and select the highest-scoring mask."""
    image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
    predictor.set_image(image_rgb)
 
    best_mask, best_score, best_prompt = None, -1.0, None
    for prompt in prompts:
        masks, scores, _ = predictor.predict(text=prompt, multimask_output=True)
        top = int(np.argmax(scores))
        if scores[top] > best_score:
            best_mask, best_score, best_prompt = masks[top], float(scores[top]), prompt
 
    return best_mask.astype(np.uint8), best_score, best_prompt
 
 
def dilate(mask: np.ndarray, radius_px: int = DILATION_PX) -> np.ndarray:
    """Expand mask by radius_px using an elliptical structuring element."""
    k = 2 * radius_px + 1
    element = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (k, k))
    return cv2.dilate(mask, element, iterations=1)
 
 
def prepare_mask(predictor, image_bgr, radius_px=DILATION_PX):
    raw, score, prompt = segment_garment(predictor, image_bgr)
    coverage = float(raw.mean())
    if coverage < MIN_MASK_FRACTION:
        raise ValueError(f"Mask too small for '{prompt}': coverage={coverage:.3f}")
 
    final = dilate(raw, radius_px)
    grown = float(final.mean()) - coverage
    print(f"prompt={prompt} sam2_score={score:.3f} coverage={coverage:.3f} "
          f"radius={radius_px}px grown_by={grown:.3f}")
    return final

Dilation radius directly affects edge quality. If the transition zone is too narrow, the sampler cannot observe surrounding skin pixels, preventing natural lighting integration and creating a pasted-on look. If the zone is too wide, the mask erases anatomical references like necklines and shoulders, encouraging the model to synthesize distorted anatomy.

A range of 5 to 15 pixels accommodates most garments, tuned per category based on drape flexibility.

Step 3: Extracting structural tokens with Flux Redux

Standard CLIP vision encoders compress reference imagery into high-level semantic embeddings, effectively reducing an image to a text-like caption. Flux Redux preserves intermediate spatial token representations, allowing micro-textures, print repeats, and logo typography to pass through the bottleneck.

# reference_stream.py
"""Encode the reference garment into structural conditioning tokens.
 
C_style = ReduxEncoder(SigCLIP(I_ref))
 
Garment backgrounds are removed before encoding to prevent studio sweeps
or mannequin textures from polluting cross-attention tokens.
"""
 
from dataclasses import dataclass
 
import torch
from PIL import Image
 
 
@dataclass
class ReferenceStreamConfig:
    redux_strength: float = 0.85   # scales C_style before cross-attention
    cutout_feather_px: int = 3     # softens RMBG alpha edge
    square_pad: bool = True        # pads rather than crops to prevent pattern distortion
 
 
class ReferenceStream:
    def __init__(self, siglip, redux, rmbg, config: ReferenceStreamConfig):
        self.siglip = siglip
        self.redux = redux
        self.rmbg = rmbg
        self.config = config
 
    def cutout(self, ref: Image.Image) -> Image.Image:
        """Isolate garment on transparent background using RMBG."""
        alpha = self.rmbg(ref)
        garment = ref.convert("RGBA")
        garment.putalpha(alpha.filter_edge(self.config.cutout_feather_px))
        if self.config.square_pad:
            garment = pad_to_square(garment)
        return garment
 
    @torch.inference_mode()
    def encode(self, ref: Image.Image) -> torch.Tensor:
        """Return C_style tokens for injection into cross-attention."""
        garment = self.cutout(ref)
        pixel_values = self.siglip.preprocess(garment).to("cuda", torch.bfloat16)
 
        # SigCLIP extracts visual features; Redux maps them into transformer tokens
        features = self.siglip(pixel_values)
        c_style = self.redux(features)
 
        c_style = c_style * self.config.redux_strength
        print(f"c_style tokens={tuple(c_style.shape)} strength={self.config.redux_strength}")
        return c_style

redux_strength balances reference fidelity against spatial flexibility. Increasing strength improves texture accuracy until it begins overriding body pose, imprinting flat-lay geometry onto standing subjects. Decreasing strength reintroduces style drift.

Background isolation with RMBG is mandatory: background studio props or hanger artifacts in reference photos otherwise bleed into cross-attention conditioning.

Step 4: Two-pass sampling with refinement

Both conditioning paths feed into a unified sampling pipeline, followed by a localized low-denoise refinement pass over the garment mask.

# run_vton.py
"""Dual-stream try-on pass: Fill conditioning on the masked body,
Redux conditioning on the reference garment, unified sampler with refinement.
"""
 
SAMPLER = {
    "sampler_name": "euler_ancestral",
    "scheduler": "beta",
    "cfg": 3.5,
}
 
PRIMARY = {**SAMPLER, "steps": 30, "denoise": 1.0}
REFINE = {**SAMPLER, "steps": 10, "denoise": 0.28}
 
# FP16 VAE decoding prevents color blotching on large solid-color fabrics
VAE_DTYPE = "fp16"
 
 
def try_on(pipe, source, mask, reference, ref_stream, prompt: str):
    """Execute dual-stream inpainting and return composited image."""
    c_style = ref_stream.encode(reference)
 
    latents = pipe.fill(
        image=source,
        mask_image=mask,
        prompt=prompt,
        style_conditioning=c_style,
        **PRIMARY,
    )
 
    # Low-denoise refinement pass settles boundary transitions and fine threads
    latents = pipe.fill(
        image=latents,
        mask_image=mask,
        prompt=prompt,
        style_conditioning=c_style,
        **REFINE,
    )
 
    image = pipe.decode(latents, vae_dtype=VAE_DTYPE)
    print(f"primary_steps={PRIMARY['steps']} refine_steps={REFINE['steps']} "
          f"sampler={SAMPLER['sampler_name']}/{SAMPLER['scheduler']}")
    return image

We select Euler Ancestral with a Beta scheduler because ancestral noise injection preserves the stochastic micro-roughness of natural textiles, whereas deterministic samplers tend to smooth out subtle weaves.

Thirty primary steps combined with ten refinement steps consistently settled boundary artifacts.

Step 5: Disentangling architectural gains from baselines

In our benchmark comparisons against ControlNet and T2I-Adapter baselines on SDXL, spatial conditioning alone proved insufficient for texture synthesis. While pose and depth control maintain body geometry, they do not resolve fabric texture transfer.

On our 500-image evaluation dataset, SDXL with ControlNet achieved an SSIM of 0.45 on Global Traditional garments, compared to 0.85 for the dual-stream Flux-VTON+ pipeline. For Western Casual garments, the baseline gap was considerably narrower (0.82 vs 0.94), confirming that standard diffusion priors already handle simple fitted apparel.

Failure modes and edge cases

1. Sheer and semi-transparent fabrics: Chiffon and lace require alpha blending of underlying skin tones rather than solid inpainting. Dual-stream diffusion currently renders sheer fabrics mostly opaque.
2. Dynamic athletic poses: Severe articulation (such as yoga or dance) stresses both SAM2 segmentation and spatial conditioning, resulting in local geometry warping.
3. Hyperparameter coupling: Dilation radius and Redux conditioning strength interact. Expanding the dilation zone gives the sampler more freedom, increasing the likelihood that strong Redux conditioning distorts body anatomy.

When this is the wrong choice

• There is no specific garment to reproduce. Mood boards and concept apparel have no manufacturing reference to be faithful to. "Red saree" as a prompt is the whole requirement, and single-stream Flux Fill answers it.
• The fabric is flat and solid. A plain cotton has no weave, no print repeat, and no motif alignment. Redux tokens are carrying nothing, and you pay for the encode anyway.
• The reference photo will not segment cleanly. RMBG has to isolate the garment before encoding. If the reference is a hanger shot against a busy background, the second stream feeds studio props into cross-attention and makes the output worse than one stream would have.