Three clouds, one inference path

At some point in 2025 our GPU work was spread across dedicated Amazon EC2 nodes, Modal.com, and ShaktiCloud, an India-hosted GPU cloud. Whenever I drew that on a whiteboard for someone, the first question back was always some version of "why not pick one". It's a fair question, and the answer isn't a spreadsheet. We never built the measurement harness that would let me put a cost comparison in front of you, so this post is about the reasoning, and I'll say clearly where the reasoning ran ahead of the evidence.

The workload had two shapes, and they wanted different machines

Training a LoRA expert is a long, stateful, messy job. You want a machine you can leave dirty: datasets on local disk, a Kohya-ss config you're editing between runs, checkpoints accumulating, a tmux session you reattach to the next morning. Rank 32 at alpha 16 on a few thousand curated images is not something you want to re-provision from scratch every time you change one number. The same goes for R&D on the graph itself, where half the work is watching an intermediate mask in a browser and deciding the dilation radius is wrong. That work lived on dedicated EC2 nodes, and dedicated is the operative word. It was capacity we had already decided to hold.

Inference is a different animal. A brand doesn't send a steady trickle of garments. They send nothing for four days and then a catalogue drop, because someone finished a shoot and wants the whole set through the pipeline this afternoon. Between bursts the useful amount of GPU capacity is close to zero, and holding dedicated nodes for that pattern means paying for idle silicon most of the week and still being short when the drop lands. That's the case for a serverless GPU runtime, and that's what Modal was doing for us: capacity that appears when a job is queued and goes away when the queue drains.

If it had stopped there, this would be a two-line post about baseline versus burst.

The third one was about where the images live

The thing that made it three was not technical. Indian fashion brands care where unreleased campaign imagery sits.

Think about what we were being handed: a model image and a garment image for a collection that hasn't launched, from a brand whose commercial advantage that quarter is that nobody has seen it yet. In conversations with larger apparel groups, the questions we got were never about FID. They were about where the file goes, whose hardware it touches, and whether the output could be produced without leaving the country.

ShaktiCloud is an India-hosted GPU cloud, and we used it for diffusion inference. That was the reason. Not price, not a benchmark we ran against the alternatives, but the ability to say that a given brand's inference happens in-country and to have that be true rather than a claim we'd have to walk back under scrutiny.

It's worth being concrete about what that requirement forces, because "data residency" sounds like a checkbox and behaves like an architecture. The weights are the easy part: the merged model and the LoRA experts are ours, they contain no customer pixels, and we can copy them anywhere. What's constrained is what the brand hands us and what we hand back. S3 was holding three classes of object under one label, the input images, the generated outputs, and the logs, and the third is the one teams forget, because a diffusion pipeline's logs accumulate image keys, mask previews, and sometimes a downscaled render of whatever went wrong. If the constraint covers the picture, it covers the thumbnail of the picture sitting in your error tracker. So a constrained tenant needs its own bucket in the right place, the worker has to be told which one rather than knowing a default, and I couldn't pull a failing image down to my laptop to look at it.

There was a second, quieter benefit. GPU availability in-region is genuinely uneven, and having a provider whose capacity is not competing with every other workload on the planet meant that when we needed machines, sometimes we could get machines. I can't quantify that for you. It's an operational impression from months of trying to get GPUs, not a study.

Making the choice in exactly one place

Three providers is survivable only if the decision about which one runs a job is made once, explicitly, and never leaks into the worker code. The queue from our API already gave us the right seam: workers pull jobs, so the placement decision is just a routing rule about which pool a job goes to.

# infra/placement.py
from dataclasses import dataclass
from enum import Enum
 
 
class Pool(str, Enum):
    """Where a unit of GPU work can be scheduled."""
    EC2_DEDICATED = "ec2_dedicated"      # long-lived nodes, training and R&D
    MODAL_BURST = "modal_burst"          # serverless GPU, scales with the queue
    SHAKTI_IN_REGION = "shakti_in_region"  # India-hosted, diffusion inference
 
 
@dataclass
class Tenant:
    tenant_id: str
    residency_required: str | None   # e.g. "IN", or None if unconstrained
 
 
@dataclass
class Job:
    job_id: str
    kind: str            # "train" | "inference" | "eval"
    tenant: Tenant
    manifest_id: str
    priority: str = "standard"   # "standard" | "interactive"
 
 
def place(job: Job, queue_depth: int) -> Pool:
    """Pick the pool for a job.
 
    Order matters. Residency is a constraint, not a preference, so it is
    checked before anything about capacity or cost. Everything after it is
    a judgement call we can revisit without breaking a promise to a brand.
    """
    if job.kind in {"train", "eval"}:
        # Long, stateful, checkpoint-heavy. Never worth a cold container.
        return Pool.EC2_DEDICATED
 
    if job.tenant.residency_required == "IN":
        return Pool.SHAKTI_IN_REGION
 
    if job.priority == "interactive" and queue_depth == 0:
        # A human is waiting on this one and there is warm capacity.
        return Pool.MODAL_BURST
 
    return Pool.MODAL_BURST
 
 
def explain(job: Job, queue_depth: int) -> str:
    """Human-readable reason, written into the job record for auditing."""
    pool = place(job, queue_depth)
    if pool is Pool.EC2_DEDICATED:
        reason = "training or evaluation workload"
    elif pool is Pool.SHAKTI_IN_REGION:
        reason = f"residency constraint {job.tenant.residency_required}"
    else:
        reason = "burst inference"
    return f"{job.job_id} -> {pool.value} ({reason})"
 
 
if __name__ == "__main__":
    abfg = Tenant("abfg", residency_required="IN")
    partner = Tenant("api_partner", residency_required=None)
    print(explain(Job("job_a1", "inference", abfg, "vton-2025-09-a"), queue_depth=12))
    print(explain(Job("job_b2", "inference", partner, "vton-2025-09-a"), queue_depth=0))
    print(explain(Job("job_c3", "train", partner, "drape-r32"), queue_depth=0))

The part I care about in that file is explain. Every job record carried the pool it ran on and the reason it went there. When a brand asked us where their images had been processed, the answer was a database query rather than an engineer's recollection. That single habit paid for itself more than once.

Notice also what isn't in there: no cost term, no bidding on spot capacity, no attempt to be clever. A placement policy that optimises something you aren't measuring is a policy that will surprise you.

One artifact layout, wherever the work ran

The other half of making this tolerable was refusing to let storage fragment. Everything, from every pool, wrote to S3 with the same key layout, and nothing in the worker was allowed to invent a path.

# infra/artifacts.py
import json
import io
from datetime import datetime
 
import boto3
 
s3 = boto3.client("s3")
BUCKET = "aurax-artifacts"
 
 
def key_for(tenant_id: str, job_id: str, kind: str, ext: str) -> str:
    """Build the one canonical key for an artifact.
 
    Layout is tenant first so that a per-brand export or deletion is a
    prefix operation, and date second so lifecycle rules can expire
    intermediates without touching delivered outputs.
    """
    day = datetime.utcnow().strftime("%Y/%m/%d")
    return f"{tenant_id}/{day}/{job_id}/{kind}.{ext}"
 
 
def put_output(tenant_id: str, job_id: str, image_bytes: bytes, meta: dict) -> dict:
    """Write a generated image plus its provenance sidecar."""
    image_key = key_for(tenant_id, job_id, "output", "png")
    meta_key = key_for(tenant_id, job_id, "provenance", "json")
 
    s3.upload_fileobj(io.BytesIO(image_bytes), BUCKET, image_key)
    s3.put_object(
        Bucket=BUCKET,
        Key=meta_key,
        Body=json.dumps(meta, indent=2).encode(),
        ContentType="application/json",
    )
    return {"image": image_key, "provenance": meta_key}
 
 
if __name__ == "__main__":
    provenance = {
        "job_id": "job_a1",
        "pool": "shakti_in_region",
        "manifest_id": "vton-2025-09-a",
        "seed": 2847193044,
        "lambda_drape": 0.6,
        "lambda_occlusion": 0.4,
        "steps_primary": 30,
        "steps_refine": 10,
    }
    print(json.dumps(provenance, indent=2))
    print(key_for("abfg", "job_a1", "output", "png"))

A provenance sidecar next to every image sounds like bookkeeping until the week you need it. Ours recorded the pool, the manifest, and the seed, which meant any output could be traced back to a specific set of weights on a specific kind of machine. When you're running the same graph in three places, that file is the only thing standing between you and a guessing game.

The worker itself was built to be uninteresting. One container image, model weights synced from object storage on start rather than baked in, no provider SDK imported anywhere in the inference path, and no assumption about the local filesystem beyond a scratch directory.

#!/usr/bin/env bash
# infra/bootstrap_worker.sh
# Prepare a GPU host to run the try-on graph. Identical on every provider.
set -euo pipefail
 
MANIFEST_ID="${1:?usage: bootstrap_worker.sh <manifest_id>}"
SCRATCH="${SCRATCH_DIR:-/scratch}"
MODELS="${SCRATCH}/models"
 
mkdir -p "${MODELS}" "${SCRATCH}/comfy"
 
# 1. Pull the manifest that defines this worker's behaviour.
aws s3 cp "s3://aurax-artifacts/manifests/${MANIFEST_ID}.json" "${SCRATCH}/manifest.json"
 
# 2. Fetch exactly the weights it names, verifying each hash.
python3 - "${SCRATCH}/manifest.json" "${MODELS}" <<'PY'
import hashlib, json, pathlib, subprocess, sys
 
manifest = json.load(open(sys.argv[1]))
dest = pathlib.Path(sys.argv[2])
 
for model in manifest["models"]:
    target = dest / model["file"]
    if not target.exists():
        subprocess.check_call([
            "aws", "s3", "cp",
            f"s3://aurax-artifacts/weights/{model['file']}", str(target),
        ])
    digest = hashlib.sha256(target.read_bytes()).hexdigest()
    print(f"{model['role']:<12} {model['file']:<32} {digest[:8]}")
PY
 
# 3. Check out the custom nodes at the commits the manifest pins.
python3 infra/sync_nodes.py "${SCRATCH}/manifest.json" "${SCRATCH}/comfy"
 
# 4. Hand over to the queue consumer. No HTTP server, no provider specifics.
exec python3 -m worker.consume --manifest "${SCRATCH}/manifest.json"

Every provider difference that survived after this lived in about thirty lines of glue per provider: how you get credentials, how you request a machine, how logs come back. That was the target, and mostly we hit it.

Baseline and burst, not primary and fallback

One thing I want to be precise about, because three providers reads as redundancy: these pools were not backups for each other. A residency-constrained job can't fail over to a pool outside the country, which is the point of the constraint. The dedicated EC2 nodes were sized for training runs we'd already committed to, so pointing burst inference at them during an outage would have cost us the capacity we were holding them for. The split is about workload shape, and redundancy would have been a different design with spare capacity we didn't have.

The condition that would collapse this to one provider is easy to state and wasn't on offer: in-country capacity, scale-to-zero for the bursty half, and machines we could hold for a week, all from the same vendor. If that existed we'd have moved the same afternoon.

The bill you pay for three of anything

Here is the part I'd want to read if someone else wrote this post.

Running inference in three places cost us real time, and I don't have a number for how much. Three sets of credentials and three permission models. Three ways a machine can fail to appear, with three different error messages meaning "no capacity right now". Three driver and CUDA combinations, which is where the genuinely annoying evenings went, because a graph that runs on one host and produces a black image on another is a bad way to spend a Thursday. And three mental models to keep loaded, which for a team our size is the expensive part even though it never shows up anywhere you can point at.

The concrete version is the question "why hasn't this image come back yet", which we got often enough that I can still recite the sequence. Check the job record in Postgres for status and pool. If it's queued, check Redis for depth and whether any worker is claiming. If it's running, go to that pool's console, which is a different console with a different login and a different idea of what a log line looks like. There was no single dashboard, and building one meant normalising three log formats, so we kept not doing it.

There's a subtler cost too. Splitting inference across pools means your traffic is split, which means every operational instinct you build is built on partial data. Warm capacity behaves differently when a pool is handling most of your jobs than when it's handling a slice, and by dividing the work we made our own signal noisier.

There's a counterargument I can't wave away. A team our size probably should have picked one provider and lived with it: serve the brands we could serve, and tell the ones with a residency requirement we'd be ready for them later. I went the other way because those questions were coming from exactly the accounts that would have made the company work, so simplicity would have meant choosing to stay small on purpose. I still think the call was right, and I hold it less confidently than I did in January.

I'm not going to produce a cost comparison, because we never ran one properly. We had invoices and impressions, and the honest summary is that we chose placement on constraints we could state (this brand's images stay in India, this training run needs a machine that persists) rather than on a per-image figure we could defend. If someone tells you they know the cost-per-image of a diffusion pipeline across three providers, ask them how they attributed idle time on the dedicated nodes. We couldn't have answered that.

What I'd do differently

Write the residency requirement into the tenant record before you need it, not after a brand asks. We retrofitted that field, and retrofitting a constraint is always worse than starting with it.

Build the accounting harness at the same time as the second provider. The moment you have two pools you have a comparison question, and the data to answer it has to be collected from the beginning; you cannot reconstruct it later from invoices. Our whole inability to show numbers in this post traces back to skipping that step for one more sprint, repeatedly.

Keep the provider surface tiny and treat it as an asset. The bootstrap script above is unglamorous and it is the reason moving a workload was a config change rather than a project.

And be suspicious of your own justifications. Two of our three pools had a constraint behind them that I can still defend today. The third was defensible when we set it up and became partly habit afterwards, and I didn't revisit it as often as I should have.

The concrete next step, if the work continues, is to make the placement policy read utilisation instead of hard-coding it: have the router consult live queue depth and warm-capacity signals per pool, and log what it would have chosen against what it did choose. Run that in shadow mode for a month and you finally have the dataset that makes the cost conversation possible instead of theoretical.