FID was the wrong metric for the problem we had

We built a try-on pipeline to fix a specific failure: models that render a saree as a flat texture stuck to a torso, or classify a kimono as a bathrobe. Then we evaluated it with a single distributional score computed over a mixed eval set, watched that score move a little, and nearly concluded the work hadn't paid off. The number was correct. The way we were reading it was not, and fixing that took one afternoon and changed what we built next.

The number that barely moved

Our evaluation set is 500 images split evenly between two categories we called Western Casual and Global Traditional. The first is t-shirts and jeans. The second is sarees, kimonos, hanfus and structured couture. We ran three systems over it: SDXL with a ControlNet baseline, base Flux Fill with no adapters and no structural conditioning, and our pipeline.

Aggregate FID went 28.4, then 22.1, then 18.5.

Look at just that column and here's the story you get: the baseline was bad, plain Flux was a big jump, and everything we did on top of Flux bought a modest additional improvement. Several months of expert adapter training, structural conditioning and merge work, worth about four FID points. If I'd been reporting that column to an investor I would have had a hard afternoon, and if I'd been reporting it to myself I might have started questioning the roadmap.

What a mixed eval set hides

FID is a distributional distance. You embed the generated set, embed the reference set, and compare the two distributions. That single number is computed over everything you fed it, which means the composition of the eval set is baked into the result and invisible in the output.

Our set was half Western Casual. On that half, base Flux Fill was already close to good. There was very little headroom, so almost nothing we did could move it much. The other half is where every failure we cared about lived. When you average a category with no headroom against a category with enormous headroom, you get a number that moves by roughly half of what actually happened, and you cannot tell from the number which half it came from.

There's a sharper version of this. If we had shipped an improvement that fixed Global Traditional completely and slightly regressed Western Casual, the aggregate could have stayed flat. A metric that can stay flat while the thing you built the company around gets solved is not measuring your problem.

The fix is dull. Split the eval set by the axis you're actually trying to improve, and report per category.

The split

Reported per category, with SSIM against the reference garment and a human-judged occlusion score:

| Method | FID | SSIM (Western) | SSIM (Global) | Occlusion accuracy | | --- | --- | --- | --- | --- | | SDXL + ControlNet | 28.4 | 0.82 | 0.45 | 62% | | Base Flux Fill | 22.1 | 0.91 | 0.58 | 70% | | Ours | 18.5 | 0.94 | 0.85 | 92% |

Now read across the rows instead of down the FID column. Western SSIM goes 0.82, 0.91, 0.94. That is a good baseline getting slightly better, and it confirms the thing we already believed, which is that Western casual wear is close to solved by any competent diffusion inpainter. Global SSIM goes 0.45, 0.58, 0.85. That is a different graph entirely. The baseline is failing outright, plain Flux is failing more politely, and the adapters are doing the thing they were trained to do.

The gap between 0.94 and 0.85 is also worth sitting with. Even after the work, structurally complex garments score below simple ones. We closed most of the gap and did not close it. Reporting the split makes that honest in a way the aggregate never could, because the aggregate had no way to express that a gap existed at all.

Here is the shape of the harness that produces it:

# eval/run_split_eval.py
"""Score a checkpoint on the VTON eval set, split by garment category.
 
The aggregate FID is kept because it is what everyone else reports and
dropping it makes comparison impossible. It is reported alongside the
per category structural scores, never on its own.
"""
 
from __future__ import annotations
 
import argparse
import json
from collections import defaultdict
from pathlib import Path
 
from skimage.metrics import structural_similarity
 
from aurax.eval.fid import frechet_distance
from aurax.eval.io import load_pair, load_manifest
 
CATEGORIES = ("western_casual", "global_traditional")
 
 
def ssim_by_category(manifest: list[dict], run_dir: Path) -> dict[str, float]:
    """Mean SSIM between generated and reference garment, per category."""
    scores: dict[str, list[float]] = defaultdict(list)
 
    for item in manifest:
        generated, reference = load_pair(run_dir / item["id"], item["reference"])
        score = structural_similarity(
            generated,
            reference,
            channel_axis=-1,
            data_range=1.0,
        )
        scores[item["category"]].append(score)
 
    return {cat: sum(v) / len(v) for cat, v in scores.items()}
 
 
def evaluate(manifest_path: Path, run_dir: Path) -> dict:
    """Return the full report: aggregate FID plus per category SSIM."""
    manifest = load_manifest(manifest_path)
 
    counts = {c: sum(1 for i in manifest if i["category"] == c) for c in CATEGORIES}
    if len(set(counts.values())) != 1:
        raise ValueError(f"eval set is not balanced across categories: {counts}")
 
    report = {
        "n": len(manifest),
        "per_category_n": counts,
        "fid_aggregate": frechet_distance(manifest, run_dir),
        "ssim": ssim_by_category(manifest, run_dir),
    }
 
    print(json.dumps(report, indent=2))
    return report
 
 
if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--manifest", type=Path, default=Path("eval/manifest.json"))
    ap.add_argument("--run", type=Path, required=True)
    evaluate(ap.parse_args().manifest, ap.parse_args().run)

The balance check in the middle is there because an unbalanced eval set silently reweights the aggregate, and once you've been burned by a number that averaged away your result you start asserting the things you assumed. If somebody adds forty more t-shirts to the manifest next quarter, the harness should refuse to run rather than quietly report a slightly different FID.

Occlusion accuracy is a human number and I should have written down how

The fourth column is the one I'd defend least, and it's also the one that predicted product acceptance best. Occlusion accuracy is the rate at which a rater judged that hand and body occlusions were preserved correctly: the garment tucking behind a hand resting on a hip instead of being painted over it. We called the failure the Amputated Hand, and 62% to 70% to 92% is the single most convincing sequence in the table for anyone who has looked at a lot of try-on output.

It was collected as a blind A/B test. That's genuinely all my notes record. I don't have the rater count, whether they were internal, how many images each one saw, how disagreements were resolved, or whether the same people scored all three systems. I believe the comparison is directionally right because the effect is large and visible without training, but I can't hand you a protocol, and a number without a protocol is not one you should port into your own comparison. Write the protocol down before you collect the data. I didn't, and the number is permanently weaker for it.

About the 85 percent

The other figure attached to this work is that we hit an 85% success rate on complex global garments against about 15% for baseline models. It's the headline claim and it's the one I'd caveat hardest.

Success was never given a precise rubric. In practice it meant somebody looked at the output and decided whether it was usable in a catalogue, which is a real and commercially meaningful judgment and is not a measurement. Different people draw that line differently, and the line probably moved as we got used to better output. The ratio is large enough that I don't think a stricter rubric would flip the conclusion, but the honest statement is narrower than the headline: on the garments where baselines fall over, our pipeline usually produces something a person would ship, and baselines usually don't.

One more caveat, on FID itself. 500 images is a small sample for a distributional metric, and FID is biased upward at small sample sizes in a way that depends on the sample size. My notes also don't record which feature extractor we used. That combination means the FID column is fine for ranking three systems evaluated identically by us, and close to useless for comparing against a number in someone else's paper. I'd treat every cross-paper FID comparison you read with the same suspicion.

What the table doesn't show

Two failure modes survive at 0.85 Global SSIM and neither of them is legible in any column above.

Extreme poses still break. Anything acrobatic or unusually articulated, yoga positions being the case we hit most, and the warping goes wrong. That's a data problem rather than an architecture problem: the base model and both of our adapters are trained on people standing or sitting, so the model has no prior for the pose and improvises. Our eval set inherits the same bias, which is why a category split by garment doesn't catch it. A split by pose would.

Sheer fabrics come out opaque. Lace and chiffon need the skin tone underneath to blend with the fabric on top, and the model renders them as solid cloth. SSIM against a reference garment is comparatively forgiving here, because the structure is roughly right and only the transparency is wrong, so the metric under-penalises a failure a customer would reject immediately.

Both of these are arguments for the same thing: the axis you split on is a claim about which variation matters. We split on garment category because that was our thesis, and that split was right. It was also incomplete, and every uncaught failure mode above is a category we didn't think to separate.

What I learned, and where this goes next

Choose the metric after you write down the failure you're trying to fix, not before. We had a precise failure statement, in the form of sarees rendering as flat texture, and then reached for the metric the field reports instead of the one that could see it.

Any aggregate over a mixed population is a weighted average with the weights hidden. If the population contains an already solved subset, that subset is not neutral, it's actively suppressing your signal.

A crude metric with a written protocol beats a sophisticated one without. The occlusion column moved product decisions and the FID column didn't, and the difference is that people trusted the thing they could picture.

Numbers you can't define, define your ceiling. Being unable to state what success meant is why the 85% figure sits in a blog post with three sentences of hedging around it instead of in a paper.

The next thing to fix is upstream of all of this. The Global Traditional half is exactly where fine structure matters most, pleats and embroidery, and that's a resolution problem before it's a modelling problem. Which means confronting how much of a full frame we can actually afford to run at native resolution.