From a ComfyUI graph to an API a brand's team can call

Running a diffusion workflow yourself is easy to get away with. You keep the graph open in a browser tab, you nudge a denoise value when the fabric looks plastic, you re-run a node until the mask stops eating the model's hand, and nobody ever finds out how much of the final image came from you sitting there. An API takes you out of the loop. Once the first beta keys went out to GettoIndia and EcomBuddha, the workflow had to behave the same way on a Tuesday afternoon as it did the night I tuned it.

Why wrapping the graph directly failed

The first version took about a day. ComfyUI already exposes an HTTP endpoint that accepts a workflow in its API JSON form, so I wrote a small Python service that loaded our saved graph, substituted the model image and the garment image into the two LoadImage nodes, posted it, and polled for the result. In a demo this is indistinguishable from a real product. As something another team depends on, it fell apart in four separate ways.

Custom node versions floated. Our graph used community nodes for SAM2 masking, RMBG, and the LoRA merge, and they were installed the way everyone installs them, by cloning from git. Updating one node to fix a mask edge case quietly changed how dilation was applied somewhere else in the graph. Nothing in the request told you which code had produced your image.

The server held state we weren't tracking. A running ComfyUI process keeps models resident in VRAM, including a merged model that was built by model_merge_lora at some point in the past. If you edited the merge weights and didn't restart cleanly, a request could be served by weights that no longer matched anything on disk. This is fine when you're the only user and you know what you just changed. It is not fine when the person calling you is debugging their own storefront.

Failures came back as prose. A short-side resolution the graph didn't like, an alpha channel where we expected RGB, an image where SAM2 found no garment at all: all of these surfaced as a Python traceback in a terminal I was watching, and as a timeout to the caller.

And there was no statement anywhere of what a caller was allowed to send. We got a 4K studio flatlay of a lehenga, a phone photo of a kurta on a hanger, and a PNG with a transparent background, all in the same week, all from people who reasonably assumed a try-on API takes a picture of clothing.

The wrapper wasn't wrong so much as it was honest about the wrong thing. It exposed our tool. What an engineering team on the other side needs is a contract.

What the contract had to pin down

I ended up thinking about this as four separate promises: which code ran, which random draw was used, when the work happens, and what we agree not to attempt.

1. A manifest, not a graph

The thing we versioned was never the workflow JSON on its own. It was the workflow plus every artifact the workflow reaches for, written down as a single file with content hashes.

// manifests/vton-2025-09-a.json
{
  "manifest_id": "vton-2025-09-a",
  "created": "2025-09-02",
  "graph": {
    "file": "graphs/flux_vton_plus.api.json",
    "sha256": "9f2c1e0b7a4d3f88c5b6e2a1d0c9f4b3e8a7d6c5b4a39281706f5e4d3c2b1a09"
  },
  "nodes": [
    { "repo": "comfyui-sam2", "commit": "4b1e9c2" },
    { "repo": "comfyui-rmbg", "commit": "a07d331" },
    { "repo": "comfyui-essentials", "commit": "c19f4ab" }
  ],
  "models": [
    { "role": "base", "file": "flux_fill.safetensors", "sha256": "1a2b3c4d" },
    { "role": "redux", "file": "flux_redux.safetensors", "sha256": "5e6f7a8b" },
    { "role": "vae", "file": "ae.safetensors", "sha256": "9c0d1e2f", "dtype": "fp16" },
    { "role": "lora_drape", "file": "drape_physics_r32.safetensors", "sha256": "3a4b5c6d" },
    { "role": "lora_occlusion", "file": "occlusion_depth_r32.safetensors", "sha256": "7e8f9a0b" }
  ],
  "merge": {
    "node": "model_merge_lora",
    "lambda_drape": 0.6,
    "lambda_occlusion": 0.4
  },
  "sampling": {
    "sampler": "euler_ancestral",
    "scheduler": "beta",
    "steps_primary": 30,
    "steps_refine": 10
  },
  "context_window": {
    "roi_padding": 0.2,
    "mask_dilation_px": 9,
    "inference_resolution": [1344, 1344]
  }
}

Two things follow from writing it this way. The merge ratio stops being folklore that lives in my head and becomes part of the published surface, which matters because 0.6 for draping and 0.4 for occlusion was an empirical choice and someone would eventually ask why their output changed when we touched it. And the API version is no longer a number I increment when I feel like it: a request either names a manifest or gets the current default, and the response always echoes back the manifest it actually ran. When a caller reports that "the model got worse", the first question has an answer in the payload.

We kept old manifests servable for as long as the weights fit on disk. That turned out to matter more than any other decision in this post.

2. Seeds, and the limits of what reproducible means

ComfyUI gives every sampler node a seed and a control mode, and in interactive use the useful mode is randomize, because you're rolling for a good result. Through an API that behaviour is unacceptable: two identical requests returning two different images means a caller cannot cache, cannot diff, and cannot write a regression test against you.

# api/seeding.py
import hashlib
import json
from dataclasses import dataclass
 
 
@dataclass
class TryOnRequest:
    """A single try-on job as it arrives from a caller."""
    request_id: str
    model_image_key: str
    garment_image_key: str
    category: str
    manifest_id: str
    seed: int | None = None
 
 
def derive_seed(req: TryOnRequest) -> int:
    """Return the seed for this request.
 
    If the caller supplied a seed we use it untouched, so that replaying a
    request replays the draw. If they didn't, we derive one deterministically
    from the request contents rather than calling random(). Retrying the same
    job after a worker crash then lands on the same seed instead of quietly
    producing a different garment drape.
    """
    if req.seed is not None:
        return req.seed % (2 ** 32)
 
    payload = json.dumps(
        {
            "model": req.model_image_key,
            "garment": req.garment_image_key,
            "category": req.category,
            "manifest": req.manifest_id,
        },
        sort_keys=True,
    ).encode()
    return int.from_bytes(hashlib.sha256(payload).digest()[:4], "big")
 
 
def apply_seed(graph: dict, seed: int) -> dict:
    """Write the seed into every sampler node and disable randomisation."""
    for node in graph.values():
        if node.get("class_type") in {"KSampler", "KSamplerAdvanced"}:
            node["inputs"]["seed"] = seed
            node["inputs"]["noise_seed"] = seed
            node["inputs"]["control_after_generate"] = "fixed"
    return graph
 
 
if __name__ == "__main__":
    req = TryOnRequest(
        request_id="req_01J9",
        model_image_key="in/abf/model_204.png",
        garment_image_key="in/abf/saree_881.png",
        category="dresses_one_pieces",
        manifest_id="vton-2025-09-a",
    )
    print("derived seed:", derive_seed(req))
    print("stable across calls:", derive_seed(req) == derive_seed(req))

The honest caveat goes in the docs next to this, not in a footnote. Same seed plus same manifest plus the same class of GPU gets you the same image. Move between GPU generations or driver versions and floating point accumulation order shifts, so you get an image that is visually the same and not bit identical. I'd rather write that down than let a caller discover it while diffing PNGs.

The other half of seed handling is that the seed goes in the response. Brands wanted a specific generated look approved by their creative team and then reused across a set of SKUs, and without the seed in hand they were approving something they couldn't ask for again.

3. Queueing, because one request owns a GPU

A full-body try-on took roughly 12 seconds on an RTX 4090 with the model already resident. That figure is a lab measurement of a single image on one machine, and I'm deliberately not turning it into a production latency claim, because we never ran the study that would let me. What it does tell you is the shape of the system: the unit of work is seconds, and while it runs, a GPU is doing nothing else.

That rules out request and response. The API accepts a job, returns a 202 with an id, and the caller either polls or gives us a webhook.

# api/queue.py
import json
import time
import uuid
 
import redis
 
r = redis.Redis(decode_responses=True)
 
QUEUE_KEY = "vton:pending"
LEASE_TTL_SECONDS = 180
MAX_INFLIGHT_PER_KEY = 4
 
 
def submit(api_key: str, payload: dict) -> dict:
    """Admit a job or reject it, then return the caller-visible job record."""
    inflight = r.scard(f"vton:inflight:{api_key}")
    if inflight >= MAX_INFLIGHT_PER_KEY:
        return {"status": "rejected", "reason": "concurrency_limit"}
 
    job_id = f"job_{uuid.uuid4().hex[:12]}"
    record = {
        "job_id": job_id,
        "api_key": api_key,
        "status": "queued",
        "manifest_id": payload["manifest_id"],
        "submitted_at": time.time(),
        "payload": payload,
    }
    r.hset(f"vton:job:{job_id}", mapping={"data": json.dumps(record)})
    r.sadd(f"vton:inflight:{api_key}", job_id)
    r.lpush(QUEUE_KEY, job_id)
    return {"job_id": job_id, "status": "queued", "queue_depth": r.llen(QUEUE_KEY)}
 
 
def claim(worker_id: str) -> dict | None:
    """Block until a job is available, then take a lease on it.
 
    The lease is what makes a worker crash survivable: if the worker dies
    mid-inference the lease expires and a sweeper puts the job back on the
    queue. Because the seed is derived from the request, the retry produces
    the same image the first attempt was going to produce.
    """
    job_id = r.brpoplpush(QUEUE_KEY, f"vton:leased:{worker_id}", timeout=30)
    if job_id is None:
        return None
    r.setex(f"vton:lease:{job_id}", LEASE_TTL_SECONDS, worker_id)
    raw = r.hget(f"vton:job:{job_id}", "data")
    record = json.loads(raw)
    record["status"] = "running"
    r.hset(f"vton:job:{job_id}", mapping={"data": json.dumps(record)})
    return record
 
 
if __name__ == "__main__":
    print(submit("key_gettoindia", {"manifest_id": "vton-2025-09-a"}))
    print(submit("key_ecombuddha", {"manifest_id": "vton-2025-09-a"}))

Everything about the product side fell into place around that. Next.js on the front, NestJS holding the REST surface and the job records in Postgres, Redis for the queue and the leases, S3 for inputs, outputs and logs, and the Python services doing nothing but pulling work and running the graph. Making the inference layer a queue consumer rather than an HTTP server also meant it stopped mattering where it ran, which is a separate post.

Queue depth became the number we watched, and it was the only number worth putting on a dashboard, because every backend problem surfaces there first.

4. Saying no, in writing

The strategy was always specialized models per garment family rather than one generic try-on, with five categories planned: Dresses and One-Pieces, Tops, Bottoms, Intimates and Swimwear, and Co-ords and Sets. The API had to be honest about which of those actually had a model behind it at any given moment, and refuse the rest rather than produce something a brand would have to reject.

# api/surface.yaml
categories:
  supported:
    - dresses_one_pieces
    - tops
    - bottoms
  planned:
    - intimates_swimwear
    - coords_sets
 
input_requirements:
  model_image:
    color_space: rgb        # alpha channels rejected, not flattened
    min_short_side: 1024
    max_faces: 1
  garment_image:
    background: any         # RMBG runs before conditioning
    on_body_reference: preferred
 
rejections:
  category_unsupported:
    http: 422
    hint: "This category has no specialised model yet. See categories.planned."
  no_garment_found:
    http: 422
    hint: "Segmentation returned an empty mask for the requested category."
  sheer_fabric_unsupported:
    http: 422
    hint: "Lace and chiffon currently render opaque. Not accepted."
  pose_out_of_distribution:
    http: 422
    hint: "Non-standing, non-seated poses are outside the trained range."

The last two rejections are the ones I'd argue for hardest. Sheer fabrics and unusual poses were known failure modes from our own evaluation, and an API that accepts them anyway is choosing to bill someone for an image its authors already know is wrong. Refusing a request costs you a conversation once. Shipping a bad render costs you the account.

What broke anyway

We hadn't defined idempotency, so a caller retrying a network timeout got a second job, a second GPU-second, and eventually a support email asking why one upload produced two images. Adding an optional idempotency key on submit fixed it in an afternoon and should have been there on day one.

Webhooks were worse than polling for a while, because our first implementation retried on any non-200 and a partner's staging endpoint was returning 500 for a week. We were hammering them politely and constantly.

And one partner pinned an old manifest, upgraded nothing for a month, then filed what they described as a quality regression. It was our own older behaviour, reproduced exactly as promised. That was simultaneously the system working and a lesson that pinning without an expiry date is a way of preserving your worst output forever.

What the contract taught me

Version the artifacts, not the code. The graph file was the least interesting thing in the manifest; the model hashes and the merge ratios were what actually determined the image.

A refusal is a feature, and the failure modes you found during evaluation are the cheapest possible source of refusal rules. We knew about the amputated hand and the opaque chiffon long before we had callers, and every one of those known failures should have become a 422 before it became a complaint.

Deterministic seeding is worth more than fast seeding. Deriving the seed from the request rather than a RNG made retries safe, made caching possible for the caller, and made bug reports reproducible, all from about fifteen lines of code.

Design for the unit of work you actually have. Twelve seconds of exclusive GPU time is a job, and pretending it's a request produces a system that fails under exactly the load you wanted.

The next piece of this is per-category manifests. Once each garment family has its own merged model, one manifest per API version stops making sense, and the routing decision (which model, which LoRA pair, which resolution) has to move into the contract instead of living inside the worker. I'd rather make that change while we have two partners than after we have twenty.