Fitting a saree into 24GB: the context window trick

At low resolution, a saree pallu flattens against the skin instead of draping over the shoulder, and waist pleats blur into a solid smear.

Resolving fine pleats requires high pixel density on the garment. Running the entire full-body frame at that resolution exceeds the VRAM limits of a standard 24GB card.

The unit of inference is the garment, not the frame. Crop to the region that has to be right, and a 24GB card runs the pipeline that a 48GB card was buying you.

Latent token scaling and frame waste

In a full-body e-commerce catalog photo, a model stands in the middle of a tall frame. The garment occupies roughly a third of the canvas. The remaining pixels consist of neutral backdrops, floor surfaces, and empty headroom: regions that a diffusion model still has to encode, denoise across every step, and decode through the VAE.

At native resolution, the model distributes compute evenly across the frame. Upscaling the full image to resolve pleat details means allocating most of that compute to background studio sweeps.

Memory consumption in diffusion transformers scales with the count of latent tokens, which grows proportionally with surface area. Doubling linear resolution quadruples the latent token footprint. That difference pushes memory requirements from a comfortable 24GB consumer GPU (such as an RTX 4090) up into 48GB enterprise hardware (such as an NVIDIA L40S) primarily to process empty backdrop pixels.

We prioritized 24GB compatibility because it represents the standard hardware tier across our self-hosted inference clusters.

Why naive tiling breaks continuous pleats

A common approach to high-resolution diffusion is spatial tiling: splitting the image into overlapping sub-blocks, denoising each independently, and blending the seams. While this works for landscapes, it destroys structured garments.

A saree pleat is a continuous physical structure. It originates at the waist and falls to the floor, maintaining consistent angle, shadow falloff, and spacing dictated by gravity.

When a pleat is split across two independent tiles, each tile is denoised without visibility into the adjacent latent field. The model's fabric prior generates plausible folds in both tiles, but with no phase alignment. When stitched back together, two disconnected sets of pleats collide at the boundary.

Alpha feathering across tile boundaries softens the seam but blends out-of-phase geometric lines into a blurry patch that reads as defective tailoring. Tiling fails whenever pixel values depend on structures extending beyond the tile window.

Because continuous draping requires long-range structural coherence, we crop directly around the garment region with surrounding context, run inference at native resolution, and composite the result back into the full frame.

Step 1: Cropping and padding the mask

We compute the bounding box of the garment mask and expand it with proportional padding. Sufficient surrounding context is necessary so the model can align the garment against body anatomy, such as shoulders and hips. We use rho = 0.2.

# pipeline/context_window.py
"""Crop, upscale, inpaint and composite: the context window mechanism.
 
The unit of inference is the garment region, not the frame. The model
processes a high-density crop while the delivered asset maintains full
source resolution.
"""
 
from __future__ import annotations
 
from dataclasses import dataclass
 
import numpy as np
 
 
NATIVE_RESOLUTIONS = ((1024, 1024), (1344, 1344))
ROI_PADDING = 0.2
 
 
@dataclass(frozen=True)
class Window:
    """A crop rectangle in the coordinate space of the source image."""
 
    x0: int
    y0: int
    x1: int
    y1: int
 
    @property
    def size(self) -> tuple[int, int]:
        return self.x1 - self.x0, self.y1 - self.y0
 
 
def roi_from_mask(mask: np.ndarray, rho: float = ROI_PADDING) -> Window:
    """Bounding box of the mask, padded by rho and clamped to the frame.
 
    rho is a fraction of the box dimension, applied on every side, so a
    tall narrow mask gets proportionally more horizontal context than a
    fixed pixel pad would provide.
    """
    ys, xs = np.nonzero(mask)
    if ys.size == 0:
        raise ValueError("empty mask: nothing to inpaint")
 
    y0, y1 = int(ys.min()), int(ys.max())
    x0, x1 = int(xs.min()), int(xs.max())
 
    pad_y = int((y1 - y0) * rho)
    pad_x = int((x1 - x0) * rho)
 
    h, w = mask.shape[:2]
    window = Window(
        x0=max(0, x0 - pad_x),
        y0=max(0, y0 - pad_y),
        x1=min(w, x1 + pad_x),
        y1=min(h, y1 + pad_y),
    )
 
    print(f"mask box: {(x0, y0, x1, y1)}")
    print(f"window:   {(window.x0, window.y0, window.x1, window.y1)} size={window.size}")
    return window
 
 
def pick_native(window: Window) -> tuple[int, int]:
    """Choose the inference resolution whose aspect is closest to the crop."""
    crop_w, crop_h = window.size
    crop_aspect = crop_w / crop_h
    return min(
        NATIVE_RESOLUTIONS,
        key=lambda r: abs((r[0] / r[1]) - crop_aspect),
    )

Padding by a fraction of the bounding box rather than a fixed pixel count accommodates vertical silhouettes. Saree masks are narrow and tall; fixed pixel padding starves the sides of torso context, which the diffusion model requires to anchor the drape against the shoulders.

Step 2: Upscaling the crop to native resolution

The cropped region is resized to match standard model dimensions (1024x1024 or 1344x1344), chosen via pick_native by aspect ratio proximity.

The garment now fills the majority of the latent space rather than a third of the canvas, giving delicate pleat lines sufficient latent tokens to avoid collapsing during denoising.

Step 3: Inpainting only the garment window

The inpainting model runs directly on the isolated crop using the specified mask, conditioning, and adapter weights.

# pipeline/context_window.py (continued)
 
INFERENCE = {
    "primary_steps": 30,
    "refinement_steps": 10,
    "sampler": "euler_ancestral",
    "scheduler": "beta",
    "vae_dtype": "fp16",
    "mask_dilation_px": 12,   # bleeding zone, tuned per garment between 5 and 15
}
 
 
def run_window(
    image: np.ndarray,
    mask: np.ndarray,
    pipe,
    conditioning,
) -> np.ndarray:
    """Inpaint the garment region at native resolution and return the crop."""
    window = roi_from_mask(mask)
    target = pick_native(window)
 
    crop = resize(image[window.y0:window.y1, window.x0:window.x1], target)
    crop_mask = resize(mask[window.y0:window.y1, window.x0:window.x1], target)
    crop_mask = dilate(crop_mask, radius=INFERENCE["mask_dilation_px"])
 
    generated = pipe(
        image=crop,
        mask_image=crop_mask,
        num_inference_steps=INFERENCE["primary_steps"],
        **conditioning,
    ).images[0]
 
    generated = pipe(
        image=generated,
        mask_image=crop_mask,
        num_inference_steps=INFERENCE["refinement_steps"],
        strength=0.25,
        **conditioning,
    ).images[0]
 
    print(f"inference at {target}, window {window.size}")
    return np.asarray(generated)

We execute 30 primary diffusion steps followed by 10 refinement steps at 0.25 denoise strength. Dilating the mask by 5 to 15 pixels creates a boundary transition zone where the model sees adjacent skin tones, generating a natural boundary rather than a hard composite line.

Profiling VRAM allocation per stage

Tracking torch.cuda.memory_allocated() across each stage on an RTX 4090 (24,576 MB total) reveals where the memory savings occur:

[INFO] mask box detected: (y0=312, y1=1840, x0=420, x1=890) size=(470, 1528)
[INFO] padding rho=0.2 applied: window=(y0=0, y1=2145, x0=326, x1=984)
[INFO] allocated before inpaint: 12,240.50 MB | max cached: 14,880.00 MB
[INFO] running diffusion at (1024, 1024), 30 primary + 10 refinement steps
[INFO] pass completed in 11.84s | peak memory: 15,360.25 MB

Step 4: Compositing across the mask boundary

The generated crop is downscaled back to its source bounding box dimensions and composited into the original high-resolution frame. The surrounding background remains untouched.

Compositing generated pixels into a photographic plate introduces two specific boundary challenges:

Downscaling generated latents acts as a low-pass filter, leaving the output slightly smoother than the surrounding photograph. Alpha blending over the dilated segmentation mask prevents rectangular boundary artifacts from appearing.

Failure modes and edge cases

1. Unchecked boundary seams: Visual inspection by raters catches boundary artifacts late in the pipeline. Automated edge-variance checks should validate transitions before assets leave inference workers.
2. Low-resolution reference photography: The context window preserves existing reference detail; it cannot synthesize clarity from blurry input photos. 2K or 4K source assets produce significantly crisper outputs.
3. Multi-window garments: Garments with sprawling dimensions that exceed a single aspect crop require multi-window coordination, which introduces cross-window blending complexity.

When this is the wrong choice

• The garment already fills the frame. On a tight portrait or torso crop there is almost no backdrop to skip. You get no memory back and you have added a compositing boundary to defend.
• You have 48GB cards. On an L40S or an A100, the full frame fits and never OOMs. Every seam problem in Step 4 exists only because we chose to crop.
• The garment has no long-range structure. Plain t-shirts and shorts have no pleat phase to keep aligned across a boundary, so an ordinary tiled upscaler works and is less code.