Generic aesthetic scorers hate e-commerce photography

The first time we ran an off-the-shelf aesthetic scorer over a batch of generated product images, it put the dramatic ones on top. Deep shadows, warm rim light, a face half in darkness, a background you couldn't read. Genuinely nice images, and completely unusable as catalogue assets, which is when it became obvious that we were optimising against somebody else's definition of good. This is about the scorer we built instead, what it's actually used for, and the parts of it I can't put a number on.

Why the open scorers point the wrong way

Aesthetic scorers get trained on human ratings of art and photography. That's the data that exists, and the people rating it are answering a question like "is this a good photograph". A good photograph, by the standards of most people rating photographs, tends to have mood: low-key lighting, strong contrast, shallow depth of field, a bit of vignette, colour that's been graded rather than corrected. The scorer learns that correlation honestly. It isn't broken.

A catalogue product shot wants close to the opposite. Even lighting, because a shadow across a garment reads as a stain or a fold that isn't there. A neutral background, because the product page has its own design and the image has to sit inside it. Colour that matches the physical garment, because the return rate is the metric that eventually pays for all of this. Nothing cropped or occluded, because a customer needs to see the hem, the sleeve, and the neckline. Every one of those is a property the general scorer treats as flat, boring, or under-directed.

So you get a scorer that is confidently wrong in a consistent direction. It doesn't produce noise, which would be easy to spot. It produces a systematic pull toward darker, moodier images, and if you use it as a training filter you build that pull into everything downstream.

I want to be precise about the failure. The scorer isn't making a mistake about photographs. It's answering a different question from the one we're asking, and its outputs are legible enough that you can go a long way before noticing the mismatch.

What a scorer is actually for

Two jobs, and they have different tolerance for error.

Filtering the training corpus. After the mechanical pass (resolution, sharpness, aspect ratio), there's a much harder question: is this image the kind of image we want the model to produce? Mechanical filters can't answer that. A perfectly sharp, well-exposed image of a garment shot on a beach at golden hour passes every technical check and is the wrong training example for a catalogue model. The aesthetic scorer is what makes that call at a scale where a human can't.

Ranking generation candidates. At inference you generate more than one candidate and you have to pick. This is the job where the scorer earns its keep most obviously, because the alternative is showing every candidate to a person, and that doesn't scale past a demo.

Those two uses interact in a way that took me a while to see clearly. If the scorer filters the training corpus, the model learns to produce images the scorer likes. If the scorer then ranks that model's candidates, it's grading work that was shaped by its own preferences. That's a closed loop, and closed loops narrow. Whatever the scorer slightly over-rewards gets amplified twice, once through what the model was trained on and once through what gets selected at generation time.

We kept a holdout the scorer never touched, and human spot checks on candidate ranking rather than trusting the top-1 blindly. That's a mitigation, not a solution.

The loop

The labelling side is deliberately dull. Pairwise comparison rather than an absolute rating, because a human asked to score an image from one to ten will drift across a session, anchor on whatever they saw last, and use a different part of the scale after lunch. Asked which of two images is better for this brand's catalogue, the same person is far more consistent.

# scorer/collect_preferences.py
"""Pairwise preference collection for the brand-centric aesthetic model.
 
Raters see two images and answer one question: which of these belongs in
this brand's catalogue? Absolute 1-to-10 scoring was tried first and
abandoned; the same rater would not reproduce their own scores across
sessions, and two raters would not agree on what a 7 meant.
"""
 
import json
import random
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from pathlib import Path
 
 
@dataclass
class Comparison:
    brand_id: str
    left: str
    right: str
    winner: str          # "left", "right", or "skip"
    rater_id: str
    flipped: bool        # whether display order was reversed
    recorded_at: str
 
 
def sample_pair(pool: list[str], seen: set[frozenset]) -> tuple[str, str] | None:
    """Draw an unseen pair. Returns None once the pool is exhausted."""
    for _ in range(200):
        left, right = random.sample(pool, 2)
        if frozenset((left, right)) not in seen:
            return left, right
    return None
 
 
def present(left: str, right: str) -> tuple[str, str, bool]:
    """Randomise display order so position bias averages out."""
    flipped = random.random() < 0.5
    return (right, left, True) if flipped else (left, right, False)
 
 
def record(out_path: Path, comparison: Comparison) -> None:
    """Append one comparison. Skips are kept: they mark ambiguous pairs."""
    with out_path.open("a") as handle:
        handle.write(json.dumps(asdict(comparison)) + "\n")
 
 
def session(brand_id: str, rater_id: str, pool: list[str], out_path: Path,
            target: int = 100) -> None:
    seen: set[frozenset] = set()
    collected = 0
 
    while collected < target:
        pair = sample_pair(pool, seen)
        if pair is None:
            break
 
        left, right = pair
        shown_a, shown_b, flipped = present(left, right)
        choice = prompt_rater(shown_a, shown_b)   # UI layer, elided
 
        seen.add(frozenset(pair))
        record(out_path, Comparison(
            brand_id=brand_id,
            left=left,
            right=right,
            winner=choice,
            rater_id=rater_id,
            flipped=flipped,
            recorded_at=datetime.now(timezone.utc).isoformat(),
        ))
        collected += 1
 
    print(f"brand={brand_id} rater={rater_id} collected={collected} "
          f"pairs_exhausted={pair is None}")

Skips are stored rather than dropped. A pair that a rater refuses to call is information: it's usually two images that differ along an axis the brand doesn't care about, and knowing which axes those are is useful when you're deciding what the scorer should be blind to.

What sits behind the scoring call itself is not something I'm writing up here. The interface is the part that matters for anyone building this, and the interface is small.

# scorer/rank_candidates.py
"""Rank generation candidates with the brand-centric scorer.
 
The scorer is a black box at this boundary on purpose. Everything below
depends only on score(image, brand_id) -> float.
"""
 
from dataclasses import dataclass
from typing import Protocol
 
import yaml
 
 
class BrandScorer(Protocol):
    def score(self, image_path: str, brand_id: str) -> float:
        """Higher is more catalogue-appropriate for this brand."""
        ...
 
 
@dataclass(frozen=True)
class BrandPolicy:
    brand_id: str
    accept_threshold: float     # below this, do not surface the candidate
    review_threshold: float     # between the two, route to human review
    max_surfaced: int
 
 
def load_policy(path: str, brand_id: str) -> BrandPolicy:
    with open(path) as handle:
        config = yaml.safe_load(handle)[brand_id]
    return BrandPolicy(brand_id=brand_id, **config)
 
 
def rank(scorer: BrandScorer, candidates: list[str],
         policy: BrandPolicy) -> dict[str, list[str]]:
    """Split candidates into surfaced, review, and rejected buckets."""
    scored = sorted(
        ((path, scorer.score(path, policy.brand_id)) for path in candidates),
        key=lambda item: item[1],
        reverse=True,
    )
 
    surfaced, review, rejected = [], [], []
    for path, value in scored:
        if value >= policy.accept_threshold and len(surfaced) < policy.max_surfaced:
            surfaced.append(path)
        elif value >= policy.review_threshold:
            review.append(path)
        else:
            rejected.append(path)
 
    print(f"brand={policy.brand_id} candidates={len(candidates)} "
          f"surfaced={len(surfaced)} review={len(review)} rejected={len(rejected)}")
    return {"surfaced": surfaced, "review": review, "rejected": rejected}

The three-bucket split rather than a straight top-N is the part I'd keep in any version of this. A single threshold forces the scorer to be right, and it isn't reliably right near the boundary. The middle band is where the human effort goes, and it stays small enough to be affordable because most candidates fall clearly on one side or the other.

The thresholds themselves live in per-brand config rather than in code, and I'm deliberately not printing values here. They aren't comparable across brands: the scorer is calibrated separately for each one, so the same number means a different thing for two different clients, and publishing a pair of them would suggest a shared scale that doesn't exist. They also moved often enough that any value I quoted would be a snapshot of one week.

The subjectivity problem, stated plainly

Here's the part that doesn't resolve. "Clean" is not one thing.

One brand means a white cyclorama, hard even light, a bright high-key look, and skin that's been smoothed. Another means daylight through a window, visible shadow under the jaw, warm neutrals, and skin with texture left in it. Both will tell you, in the same words, that they want clean commercial imagery, and an image that one of them signs off on will get rejected by the other. The scorer has to be calibrated per brand, or at minimum per brand profile, and that means the labelling cost doesn't amortise the way you'd hope. Every new client is partly a new labelling problem.

Individual raters aren't stable either. Order effects are real. Fatigue is real. Somebody who has looked at four hundred images in a row starts rewarding novelty, which is exactly wrong for catalogue work.

And I'm not going to put numbers on any of this. No rater counts, no agreement statistics, no accuracy figures for the scorer. We ran this as a working process rather than as a study, and quoting a number would imply a level of rigour the setup didn't have. I'd rather say that plainly than publish a figure that sounds authoritative and isn't. If you're building the same thing and you want to know whether your raters agree, you'll have to run that properly yourself, and it's worth doing.

What to watch for

Reward hacking is the obvious risk and it arrives quietly. If a scorer rewards even lighting and neutral backgrounds, and you select generations against it, the model drifts toward flat and safe. Every individual image passes. The catalogue as a whole gets duller, and nobody can point at the image where it went wrong. The only defence I found was keeping images the scorer never selected in front of a human periodically.

Staleness is the other one. A brand's art direction changes seasonally, and a scorer trained on last season's approved assets will start rejecting the new direction with total confidence. The scorer needs a refresh path from the beginning, not as a later feature.

There's also a boundary worth defending: the scorer answers whether an image fits a brand's catalogue, and it does not answer whether the try-on is correct. Garment fidelity, occlusion, drape, all of that is measured separately. Letting an aesthetic score stand in for correctness would mean a beautifully lit image of the wrong garment scoring well, and that is the single worst output this system can produce.

What I'd tell someone starting this

Build the scorer around a question you can actually state. "Does this belong in this brand's catalogue" is answerable by a human in two seconds. "Is this aesthetic" is not, and a scorer trained on the second question will quietly answer a third one you never asked.

Pairwise beats absolute scoring for anything a human labels by eye, and the gap is larger than I expected.

Keep the scorer's output as a routing decision rather than a verdict. Three buckets, with a middle band that goes to a person, absorbs most of the error the model is going to make.

Say out loud which parts you haven't measured. It's more useful to a reader than an invented number, and it keeps you honest with yourself about which of your components you actually trust.

The concrete next step is a per-brand calibration set collected at onboarding: a short pairwise session run by the brand's own art director before we generate anything for them, so the scorer starts from their definition of clean instead of ours. It costs the client part of an afternoon and it removes the argument that otherwise happens three weeks later, over images we've already delivered.