---
title: "We beat the benchmarks and ran out of runway"
description: "Flux-VTON+ scores 0.85 SSIM on global traditional garments where SDXL manages 0.45, and we have signed deals with real apparel groups. We're winding the company down anyway. This is what I think the research was worth, written while both of those things are true at once."
date: "September 25, 2025"
url: "https://himanshuat.com/blogs/we-beat-the-benchmarks-and-ran-out-of-runway"
---
# We beat the benchmarks and ran out of runway

We founded AuraX in December 2024, incubated at IIIT-Hyderabad's CIE. Today, we are winding down operations.

The technology works. Flux-VTON+ improved SSIM on global traditional garments from 0.45 to 0.85, and we signed commercial contracts with Aditya Birla Fashion Group and multiple Mensa Brands labels.

> Strong benchmark performance confirms algorithmic capability. It does not validate go-to-market distribution, sales cycle resilience, or enterprise runway.

### Defining the complexity gap in virtual try-on

Open-source diffusion models and standard public benchmarks are overwhelmingly trained on Western casual wear: fitted t-shirts, jeans, and hoodies. These garments feature simple topology conforming closely to the human body.

Traditional global garments (sarees, hanfus, kimonos) behave according to different physical laws. A saree depends on pleat cascades, gravity, and the pallu draping across the shoulder. Without explicit structural priors, diffusion models render sarees as flat textures painted directly onto skin, and misclassify kimonos as bathrobes.

We defined the Complexity Gap as the difference between a pipeline's performance on Western casual garments and its performance on global traditional garments on identical test sets:

$$
\text{Gap} = \text{SSIM}_{\text{Western}} - \text{SSIM}_{\text{Global}}
$$

SDXL with ControlNet carries a gap of 0.37. Our architecture reduced this gap to 0.09.

---

### Step 1: Stratifying evaluation datasets by cultural apparel topology

We evaluated our models against a balanced 500-image dataset split evenly between Western Casual and Global Traditional categories.

Reporting a single aggregate score across all 500 images would have masked structural failures on traditional apparel behind strong baseline performance on Western basics.

---

### Step 2: Evaluating performance spread across difficulty tiers

We benchmarked three model configurations across our stratified evaluation suite:

| Method | FID | SSIM (Western) | SSIM (Global) | Occlusion Accuracy | Complexity Gap |
|---|---|---|---|---|---|
| SDXL + ControlNet | 28.4 | 0.82 | 0.45 | 62% | 0.37 |
| Base Flux Fill | 22.1 | 0.91 | 0.58 | 70% | 0.33 |
| Flux-VTON+ (Ours) | 18.5 | 0.94 | 0.85 | 92% | 0.09 |

Occlusion accuracy measures whether human hands and jewelry resting on clothing survive generation intact rather than being overpainted by fabric.

```python
# eval/report.py
"""Tabulate Flux-VTON+ evaluation benchmarks across stratified categories."""

from dataclasses import dataclass


@dataclass
class MethodResult:
    name: str
    fid: float            # lower indicates better distribution match
    ssim_western: float   # higher indicates better structural fidelity
    ssim_global: float
    occlusion_acc: float  # human review metric


RESULTS = [
    MethodResult("SDXL + ControlNet", 28.4, 0.82, 0.45, 0.62),
    MethodResult("Base Flux Fill", 22.1, 0.91, 0.58, 0.70),
    MethodResult("Flux-VTON+ (Ours)", 18.5, 0.94, 0.85, 0.92),
]


def complexity_gap(result: MethodResult) -> float:
    """Calculate the performance spread between Western and Global apparel."""
    return result.ssim_western - result.ssim_global


def report(results: list[MethodResult]) -> None:
    for r in results:
        print(
            f"{r.name:<22} "
            f"FID {r.fid:>5.1f}  "
            f"SSIM-W {r.ssim_western:.2f}  "
            f"SSIM-G {r.ssim_global:.2f}  "
            f"Occ {r.occlusion_acc:.0%}  "
            f"gap {complexity_gap(r):.2f}"
        )


if __name__ == "__main__":
    report(RESULTS)
```

On traditional garments, our pipeline achieved an 85% production acceptance rate compared to 15% for baseline diffusion inpainters.

---

### Step 3: Closing the gap with specialized low-rank adapters

Rather than undertaking a full foundation model retraining, we closed the complexity gap using two specialized LoRA adapters:

1. **Expert A (Draping Physics):** Trained on 5,000 curated images of sarees, hanfus, and kimonos to teach gravity and fold dynamics.
2. **Expert B (Occlusion and Depth):** Trained on 3,000 images of hands resting over clothing and layered accessories.

```mermaid
flowchart LR
  B[Flux Fill] --> M[weight fusion]
  D[drape LoRA: 0.6] --> M
  O[occlusion LoRA: 0.4] --> M
  M --> W[Flux-VTON+ checkpoint]
```

Fusing both adapters into base weights at build time (`lambda_drape = 0.6`, `lambda_occ = 0.4`) provided single-pass inference without runtime weight-swapping overhead.

---

### Step 4: Translating technical failure modes into enterprise terminology

Giving technical issues clear names helped communicate with apparel executives:

- "The Complexity Gap" clearly articulated why general Western models failed on Indian ethnic wear.
- "The Amputated Hand" clearly described foreground occlusion failures where cloth painted over fingers.

Once a failure had a name, buyers could point at the exact one they needed fixed.

---

### Step 5: Enterprise procurement realities and runway dynamics

Enterprise fashion sales operate on long timelines. Large apparel groups often require six-to-nine-month evaluation and procurement cycles, custom SDK integrations, and extensive workflow adaptations.

Then well-capitalized competitors entered the broader diffusion space. Our runway ran out before the integration timelines did.

---

### Failure modes and edge cases

1. **Equating benchmark leads with enterprise defensibility:** Technical leads in SSIM or FID do not protect against competitors with pre-built e-commerce plugins and large enterprise sales organizations.
2. **Qualitative competitor comparisons:** Evaluating commercial competitors through qualitative side-by-side renders rather than standardized automated benchmarks creates blind spots regarding competitors' real-world capabilities.
3. **Over-investing research capacity in solved domains:** Allocating engineering time to push Western casual SSIM from 0.91 to 0.94 delivered negligible business value compared to solving enterprise workflow integrations.

---

### When this is the wrong choice

- **Procurement takes six months.** Nothing you ship to the model weights arrives before the evaluation cycle closes. Developer tooling and integration work do.
- **A competitor already has distribution.** An adequate model behind a turn-key Shopify or Magento integration beats a better model that needs custom API wiring, and the buyer is comparing time-to-first-render, not SSIM.
- **Runway is short.** Research velocity has to be measured against the date the cash runs out. Ours went into decimal-point metric gains rather than onboarding friction, and the integration timelines turned out to be the only clock that mattered.

---

Source: https://himanshuat.com/blogs/we-beat-the-benchmarks-and-ran-out-of-runway