Three clouds, one inference path

A diffusion graph that ran cleanly on one host produced blank outputs on another. The model weights and graph manifest were identical; the underlying NVIDIA driver and CUDA minor version differed.

Managing cross-cloud environment drift is the ongoing cost of running GPU inference across multiple providers simultaneously.

Distributing workloads across three providers was driven by data residency and workload elasticity requirements rather than raw cost arbitrage.

Our infrastructure spanned dedicated Amazon EC2 nodes, serverless GPU workers on Modal, which autoscales GPU containers from zero, and Shakti Cloud, Yotta Data Services' GPU cloud, which is built and hosted entirely in India.

Chapter 0: Defining execution pools

An execution pool is a compute target defined by operational characteristics:

1. Dedicated compute (Amazon EC2): Long-lived instances for training LoRA adapters and interactive graph development.
2. Serverless compute (Modal): Elastic GPU workers that scale from zero during catalog rendering bursts.
3. In-region compute (ShaktiCloud): Dedicated regional GPU capacity satisfying contractual data residency requirements.

Step 1: Segment workloads by statefulness and elasticity

Keeping dedicated instances alive for sporadic catalog drops means paying for GPU hours nobody uses. Running stateful training sweeps on ephemeral serverless containers means fighting the container every time you want to look at an intermediate checkpoint.

Step 2: Enforce data residency as a hard routing constraint

Apparel and luxury fashion clients frequently impose strict data residency constraints on pre-launch campaign imagery: unreleased garment assets cannot leave national boundaries prior to public disclosure.

While model weights and LoRA adapters are non-sensitive internal intellectual property, client input images, intermediate masked latents, and preview thumbnails are subject to strict residency controls.

Residency requires isolated Amazon S3 buckets in specific geographical regions and dedicated in-region GPU execution nodes.

Step 3: Centralize placement logic at the API gateway

Routing decisions execute in a single module at job ingestion time, isolating worker containers from cluster topology details:

# infra/placement.py
from dataclasses import dataclass
from enum import Enum
 
class Pool(str, Enum):
    EC2_DEDICATED = "ec2_dedicated"        # Long-lived nodes, training and R&D
    MODAL_BURST = "modal_burst"            # Serverless GPU, queue-driven scaling
    SHAKTI_IN_REGION = "shakti_in_region"  # India-hosted, diffusion inference
 
@dataclass
class Tenant:
    tenant_id: str
    residency_required: str | None         # e.g., "IN", or None
 
@dataclass
class Job:
    job_id: str
    kind: str                              # "train", "inference", "eval"
    tenant: Tenant
    manifest_id: str
    priority: str = "standard"
 
def place_job(job: Job, queue_depth: int) -> Pool:
    """
    Selects compute pool based on strict residency constraints and workload type.
    """
    if job.kind in {"train", "eval"}:
        return Pool.EC2_DEDICATED
 
    if job.tenant.residency_required == "IN":
        return Pool.SHAKTI_IN_REGION
 
    return Pool.MODAL_BURST
 
def explain_placement(job: Job, queue_depth: int) -> str:
    """Generates structured explanation for audit logs."""
    pool = place_job(job, queue_depth)
    if pool is Pool.EC2_DEDICATED:
        reason = "Stateful training or interactive evaluation workload"
    elif pool is Pool.SHAKTI_IN_REGION:
        reason = f"Data residency constraint: {job.tenant.residency_required}"
    else:
        reason = "Elastic burst inference"
    return f"{job.job_id} -> {pool.value} ({reason})"

The reason string is what makes the audit question answerable: which regional host processed this customer's images, and on what grounds.

Step 4: Enforce deterministic artifact schemas and provenance sidecars

To prevent data fragmentation across providers, all execution pools write outputs to centralized object storage using a unified key layout and structured metadata sidecars:

# infra/artifacts.py
import json
import io
from datetime import datetime
import boto3
 
s3 = boto3.client("s3")
BUCKET = "aurax-artifacts"
 
def generate_key(tenant_id: str, job_id: str, artifact_type: str, extension: str) -> str:
    """Generates partitioned canonical storage keys for export and lifecycle policies."""
    day = datetime.utcnow().strftime("%Y/%m/%d")
    return f"{tenant_id}/{day}/{job_id}/{artifact_type}.{extension}"
 
def store_output_with_provenance(tenant_id: str, job_id: str, image_bytes: bytes, provenance_meta: dict) -> dict:
    """Writes rendered image alongside immutable provenance metadata."""
    image_key = generate_key(tenant_id, job_id, "output", "png")
    meta_key = generate_key(tenant_id, job_id, "provenance", "json")
 
    s3.upload_fileobj(io.BytesIO(image_bytes), BUCKET, image_key)
    s3.put_object(
        Bucket=BUCKET,
        Key=meta_key,
        Body=json.dumps(provenance_meta, indent=2).encode(),
        ContentType="application/json",
    )
    return {"image_key": image_key, "provenance_key": meta_key}

Provenance records track exact random seeds, checkpoint SHA-256 hashes, adapter weights, and host execution pools to guarantee that any output image can be audited and reproduced.

Step 5: Minimize provider-specific SDK coupling

Worker processes operate inside standardized container environments that download explicit weight manifests upon startup:

#!/usr/bin/env bash
# infra/bootstrap_worker.sh
set -euo pipefail
 
MANIFEST_ID="${1:?usage: bootstrap_worker.sh <manifest_id>}"
SCRATCH="${SCRATCH_DIR:-/scratch}"
MODELS="${SCRATCH}/models"
 
mkdir -p "${MODELS}" "${SCRATCH}/comfy"
 
# 1. Download execution manifest
aws s3 cp "s3://aurax-artifacts/manifests/${MANIFEST_ID}.json" "${SCRATCH}/manifest.json"
 
# 2. Synchronize model weights and verify checksums
python3 - "${SCRATCH}/manifest.json" "${MODELS}" <<'PY'
import hashlib, json, pathlib, subprocess, sys
 
manifest = json.load(open(sys.argv[1]))
dest = pathlib.Path(sys.argv[2])
 
for model in manifest["models"]:
    target = dest / model["file"]
    if not target.exists():
        subprocess.check_call([
            "aws", "s3", "cp",
            f"s3://aurax-artifacts/weights/{model['file']}", str(target),
        ])
    digest = hashlib.sha256(target.read_bytes()).hexdigest()
    print(f"{model['role']:<12} {model['file']:<32} {digest[:8]}")
PY
 
# 3. Start decoupled queue consumer
exec python3 -m worker.consume --manifest "${SCRATCH}/manifest.json"

Key operational trade-offs

• Driver and runtime disparity: Discrepancies between NVIDIA driver versions, CUDA toolkits, and PyTorch builds across cloud vendors can introduce non-deterministic tensor outputs or silent pipeline failures. Enforce strict containerization.
• Fragmented telemetry: Without a centralized logging plane (such as Datadog or Grafana Loki) aggregating metrics across AWS, Modal, and bare-metal nodes, debugging queue stalls requires checking multiple vendor dashboards.
• Capacity isolation: Multi-cloud routing accommodates hard compliance boundaries, but it does not give you failover unless data replication and compute pools are pre-provisioned in both regions.

When this is the wrong choice

• No client contract names a jurisdiction. The residency branch of the router never fires, so place_job collapses to a workload-type switch you could have written inside one provider's SDK. You still pay the drift tax: mismatched driver and CUDA versions, checksum verification on every worker boot, and outputs split across three vendor consoles.
• Load is steady rather than bursty. The serverless pool exists because catalog inference sits at zero for days and then arrives in thousand-image bursts. Under continuous throughput, the scale-from-zero behavior buys nothing and you keep the cold-start penalty.
• You have no aggregated logging plane yet. Build that first. Multi-cloud placement without it turns every queue stall into a dashboard hunt across AWS, Modal, and the in-region cluster, which is a worse problem than the one the routing solved.