---
title: "FID was the wrong metric for the problem we had"
description: "Aggregate FID went 22.1 to 18.5 and I nearly read that as months of adapter work buying four points. Half the eval set was already close to solved and it dragged the average toward nothing. Splitting by garment category was the change that made the result visible."
date: "August 26, 2025"
url: "https://himanshuat.com/blogs/fid-was-the-wrong-metric"
---
# FID was the wrong metric for the problem we had

Aggregate Fréchet Inception Distance (FID) dropped from 22.1 to 18.5 across our evaluation runs. I initially read that as months of adapter training and structural conditioning yielding a modest four-point improvement, and nearly treated it as evidence of diminishing returns.

The calculation was mathematically correct, but using a single aggregate number obscured where the model was actually improving.

> Any aggregate metric calculated over a heterogeneous population is an implicit weighted average. If half that population is already solved, the solved subset dilutes the performance signal of the hard subset.

## Limitations of single distributional scores

FID calculates the Wasserstein-2 distance between feature distributions in Inception-v3 latent space. It produces a single scalar comparing a generated image set against reference photographs, baking the composition of the evaluation set into the final output.

Our evaluation suite contains 500 image pairs split evenly between Western Casual (fitted t-shirts and jeans) and Global Traditional (sarees, kimonos, hanfus, and draped couture).

We benchmarked three configurations across this dataset: SDXL with ControlNet, base Flux Fill without custom conditioning, and our dual-stream adapter pipeline. Aggregate FID scored 28.4, 22.1, and 18.5 respectively.

On Western casual wear, base Flux Fill was already strong, leaving little headroom for further numerical gains. The Global Traditional category contained virtually all the structural failures we aimed to resolve.

```mermaid
flowchart LR
  E[eval set] --> W[western half]
  E --> G[global half]
  W -->|no headroom| A[aggregate FID]
  G -->|all headroom| A
  A --> R[reads as modest]
```

Averaging a solved category with low headroom against an unsolved category with high headroom halves the visible magnitude of any real improvement.

---

## Step 1: Defining failure modes before selecting metrics

We designed our virtual try-on architecture to address specific visual defects: sarees rendering as flat printed decals on skin, and kimonos flattening into bathrobes.

Those are precise failure conditions. Pick the metric that moves when they are fixed, not the one that is standard in the literature.

---

## Step 2: Stratifying evaluation datasets by garment complexity

Splitting the evaluation dataset by garment complexity and measuring structural similarity (SSIM) alongside human occlusion ratings revealed the true performance profile:

| Method | Aggregate FID | SSIM (Western) | SSIM (Global) | Occlusion Accuracy |
|---|---|---|---|---|
| SDXL + ControlNet | 28.4 | 0.82 | 0.45 | 62% |
| Base Flux Fill | 22.1 | 0.91 | 0.58 | 70% |
| Flux-VTON+ (Ours) | 18.5 | 0.94 | 0.85 | 92% |

Examining the split columns clarifies the difference:

On Western apparel, SSIM moved 0.82 to 0.91 to 0.94. Standard diffusion inpainters already handle fitted garments.

On Global Traditional garments, SSIM moved 0.45 (SDXL failure) to 0.58 (Flux baseline) to 0.85 (Flux-VTON+). That is the drape and depth adapters doing the work the aggregate was hiding.

The remaining gap between 0.94 (Western) and 0.85 (Global) says complex drapes are still harder to synthesize than simple tubes. Aggregate FID reports none of this.

---

## Step 3: Enforcing balanced category manifests in the eval runner

Stratified evaluation remains reliable only when category distributions are strictly controlled. We enforce parity checks directly in the evaluation runner:

```python
# eval/run_split_eval.py
"""Score a checkpoint on the VTON eval set, split by garment category.

Aggregate FID is retained for external benchmark comparison, but is
always reported alongside category-stratified SSIM metrics.
"""

from __future__ import annotations

import argparse
import json
from collections import defaultdict
from pathlib import Path

from skimage.metrics import structural_similarity

from aurax.eval.fid import frechet_distance
from aurax.eval.io import load_pair, load_manifest

CATEGORIES = ("western_casual", "global_traditional")


def ssim_by_category(manifest: list[dict], run_dir: Path) -> dict[str, float]:
    """Mean SSIM between generated and reference garment, per category."""
    scores: dict[str, list[float]] = defaultdict(list)

    for item in manifest:
        generated, reference = load_pair(run_dir / item["id"], item["reference"])
        score = structural_similarity(
            generated,
            reference,
            channel_axis=-1,
            data_range=1.0,
        )
        scores[item["category"]].append(score)

    return {cat: sum(v) / len(v) for cat, v in scores.items()}


def evaluate(manifest_path: Path, run_dir: Path) -> dict:
    """Return full report: aggregate FID plus per-category SSIM."""
    manifest = load_manifest(manifest_path)

    counts = {c: sum(1 for i in manifest if i["category"] == c) for c in CATEGORIES}
    if len(set(counts.values())) != 1:
        raise ValueError(f"Eval set is not balanced across categories: {counts}")

    report = {
        "n": len(manifest),
        "per_category_n": counts,
        "fid_aggregate": frechet_distance(manifest, run_dir),
        "ssim": ssim_by_category(manifest, run_dir),
    }

    print(json.dumps(report, indent=2))
    return report


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--manifest", type=Path, default=Path("eval/manifest.json"))
    ap.add_argument("--run", type=Path, required=True)
    evaluate(ap.parse_args().manifest, ap.parse_args().run)
```

The category count validation prevents silent skewing of evaluation sets. If someone adds 50 additional t-shirts to the manifest, the script halts execution.

---

## Step 4: Structuring human evaluation protocols

Occlusion accuracy measures whether human hands and jewelry resting on garments survive generation rather than being overpainted by fabric (the "Amputated Hand" defect).

Occlusion accuracy went 62% to 70% to 92%, and that ordering matched which renders clients accepted.

Human perceptual data needs a documented rubric. It also needs a recorded panel size, an inter-rater reliability score such as Cohen's kappa, and a dispute-resolution protocol. Without those three, an occlusion number is directional validation, not a benchmark you can put in a paper.

---

## Failure modes and edge cases

1. **Sample size bias in distributional metrics:** 500 images represents a relatively small sample for FID calculations, which are subject to upward bias on smaller sets.
2. **Hidden orthogonal failure modes:** Stratifying exclusively by garment category fails to catch pose-related distortions (such as seated or athletic poses) or sheer fabric opaqueness.
3. **Subjective success definitions:** Converting visual inspection into binary pass/fail rates requires explicit quality rubrics defining acceptable boundary tolerances.

---

## When this is the wrong choice

- **Every input is equally hard.** Stratification pays off because one half of our eval set had no headroom left. If difficulty and headroom are uniform across the set, the splits produce the same number twice.
- **You are comparing against a published paper.** Papers report aggregate FID. Report it too, alongside the stratified numbers, or the comparison stops being a comparison.
- **The buckets get too small.** 500 images is already small enough that FID carries upward bias. Splitting a set that size into several narrow buckets drops per-bucket counts below anything you should quote.

---

Source: https://himanshuat.com/blogs/fid-was-the-wrong-metric