ComfyUI as a research lab

ComfyUI has a reputation as the thing you use when you don't want to write code. That was not why we used it. We used it because a try-on pipeline has two conditioning streams, a segmentation stage, a mask post-process, two merged adapters and a two-pass sampler, and the honest question every morning was "which of those did I change yesterday". A graph answers that question. A notebook does not.

Why the notebook lost

I started in a notebook, like everyone does. The problem showed up in about two weeks.

A diffusion experiment isn't one function you tune. It's a directed graph of components where the interesting variables live at the edges: which encoder feeds which conditioning input, whether the mask is dilated before or after the ROI crop, whether the refinement pass sees the original mask or a shrunk one. In a notebook, all of that structure is implicit in the order of cells and the contents of variables that were reassigned six cells ago. When a run comes out good, you have a .ipynb where cell 14 was executed after cell 22 and the diff against yesterday is forty lines of JSON with an embedded PNG in it.

I could have written a proper config-driven trainer with a YAML per experiment, and for the training side we did. But the inference pipeline was changing shape rather than values, several times a day. A config file is the right artifact when the topology is fixed and you're sweeping values. It's the wrong artifact when the question is "what if Redux fed the refinement pass but not the primary one".

A node graph makes topology the first-class thing. You can't accidentally leave a node connected. The picture on screen is the pipeline, with no gap between what you think is running and what is running.

The graph is the experiment record

The part that actually changed how we worked is that a ComfyUI workflow serialises to JSON, and JSON goes in git.

That sounds trivial and it is not. It means an experiment is a commit. It means "the version that produced the saree renders the client liked" is a SHA rather than a folder called final_v3_USE_THIS. It means I can ask what changed between two runs and get an answer in text.

The default export fights you a little. ComfyUI writes node positions, canvas zoom and monotonically increasing node ids into the same file as the semantics, so dragging a node two centimetres to the left produces a diff. We ended up normalising before committing.

# normalize_workflow.py
"""Strip canvas noise out of a ComfyUI workflow export before committing it.
 
Node positions, sizes and the canvas viewport carry no semantics but change
constantly, which buries the real diff. This drops them and sorts everything
into a stable order so `git diff` shows only what the pipeline does.
"""
import json
import sys
from pathlib import Path
 
DROP_KEYS = {"pos", "size", "flags", "order", "extra", "color", "bgcolor"}
 
 
def normalize(workflow: dict) -> dict:
    """Return a canvas-independent view of a ComfyUI workflow."""
    nodes = []
    for node in sorted(workflow.get("nodes", []), key=lambda n: n["id"]):
        nodes.append(
            {k: v for k, v in sorted(node.items()) if k not in DROP_KEYS}
        )
 
    links = sorted(workflow.get("links", []), key=lambda l: (l[1], l[3], l[2]))
 
    return {
        "nodes": nodes,
        "links": links,
        "widget_defaults": workflow.get("extra", {}).get("ds", None) and None,
    }
 
 
def main(paths: list[str]) -> None:
    for p in paths:
        src = Path(p)
        dst = src.with_suffix(".norm.json")
        data = json.loads(src.read_text())
        dst.write_text(json.dumps(normalize(data), indent=2, sort_keys=True))
        print(f"{src.name} -> {dst.name}")
 
 
if __name__ == "__main__":
    main(sys.argv[1:])

We ran that as a pre-commit hook, kept the normalised file as the tracked artifact and treated the raw export as a build output. After that, a pull request against the pipeline was readable by a person who hadn't been sitting next to me.

What the dual-stream pipeline looks like as a graph

Here is a stripped-down version of the shape. I want to be clear about what this is: I hand-wrote this fragment for this post. It is not an export from our workflow, our workflow is not published anywhere, and a real ComfyUI file carries a lot more scaffolding than this. Read it as a diagram that happens to be in JSON.

{
  "_file": "flux_vton_plus.illustrative.json",
  "_note": "hand-written for this post, simplified, not an export",
  "nodes": [
    { "id": 1, "type": "LoadImage",
      "widgets_values": ["model_0412.png"] },
    { "id": 2, "type": "LoadImage",
      "widgets_values": ["garment_ref_saree_green.png"] },
    { "id": 3, "type": "RMBG",
      "widgets_values": ["u2net", 0.5],
      "inputs": [{ "name": "image", "link": [2, 0] }] },
    { "id": 4, "type": "SAM2Segment",
      "widgets_values": ["shirt, tshirt, top, saree, dress"],
      "inputs": [{ "name": "image", "link": [1, 0] }] },
    { "id": 5, "type": "GrowMask",
      "widgets_values": [11, true],
      "inputs": [{ "name": "mask", "link": [4, 0] }] },
    { "id": 6, "type": "StyleModelLoader",
      "widgets_values": ["flux1-redux-dev.safetensors"] },
    { "id": 7, "type": "ReduxApply",
      "inputs": [
        { "name": "style_model", "link": [6, 0] },
        { "name": "reference", "link": [3, 0] }
      ] },
    { "id": 8, "type": "UNETLoader",
      "widgets_values": ["flux1-fill.safetensors", "default"] },
    { "id": 9, "type": "LoraLoaderModelOnly",
      "widgets_values": ["drape_physics_r32.safetensors", 0.6],
      "inputs": [{ "name": "model", "link": [8, 0] }] },
    { "id": 10, "type": "ModelMergeLoRA",
      "widgets_values": ["occlusion_depth_r32.safetensors", 0.4],
      "inputs": [{ "name": "model", "link": [9, 0] }] },
    { "id": 11, "type": "KSampler",
      "widgets_values": [30, 3.5, "euler_ancestral", "beta", 1.0],
      "inputs": [
        { "name": "model", "link": [10, 0] },
        { "name": "conditioning", "link": [7, 0] },
        { "name": "mask", "link": [5, 0] }
      ] },
    { "id": 12, "type": "KSampler",
      "widgets_values": [10, 3.5, "euler_ancestral", "beta", 0.35],
      "inputs": [
        { "name": "model", "link": [10, 0] },
        { "name": "latent", "link": [11, 0] }
      ] },
    { "id": 13, "type": "VAEDecode",
      "widgets_values": ["fp16"],
      "inputs": [{ "name": "samples", "link": [12, 0] }] }
  ]
}

Two things in there are worth pointing at. The RMBG node on the reference garment matters more than it looks: Redux encodes structure from whatever pixels you hand it, so if the reference photo has a mannequin or a studio backdrop in it, some of that leaks into the conditioning and shows up as texture on the output garment. Cutting the garment to a transparent channel first removed a whole category of "why does the sleeve have a shadow that isn't in the reference" bugs.

The other is that there are two KSampler nodes, at thirty steps and ten steps, and the second one runs at low denoise. Fine detail like embroidery and stitching comes out of the second pass. Running the whole thing at forty steps in one pass does not produce the same image and is slower.

Merging the experts before anything samples

The adapters get merged into the base UNet weights rather than swapped at runtime. In the graph that's model_merge_lora, and the merge happens once at initialisation, upstream of both samplers.

The arithmetic is a linear combination of the low-rank updates:

W_merged = W_base + λ_drape · (B_d A_d) + λ_occ · (B_o A_o)

with λ_drape at 0.6 and λ_occ at 0.4, which is where empirical testing landed. Expert A is draping physics, Expert B is occlusion and depth.

The reason this is a graph node rather than a script is that the coefficients are the most-tuned numbers in the entire pipeline, and having them as two widget values on a visible node meant anyone could change one, run four images and see the result without touching Python. Push λ_occ too high and hands get preserved beautifully on garments that now hang like cardboard. Push λ_drape too high and the fabric moves correctly right over the top of somebody's fingers. The graph made that trade legible in a way a config file didn't, because you can see both numbers and the sampler they feed in one screen.

I'd add one caution. Merging is destructive in the sense that you can no longer attribute a behaviour to one expert once the weights are combined. When output quality regressed, the first debugging step was always to rebuild the graph with a single adapter at a time, which is fast to do in a graph and easy to forget to do.

Versioning workflows without pretending it's a codebase

Our convention was one directory per experiment family, normalised JSON tracked, checkpoints referenced by filename and pinned in a lockfile alongside.

# scripts/snapshot_workflow.sh
# Freeze the current pipeline: normalised graph + the exact weights it names.
set -euo pipefail
 
WORKFLOW="${1:?usage: snapshot_workflow.sh <workflow.json> <tag>}"
TAG="${2:?missing tag}"
OUT="experiments/${TAG}"
 
mkdir -p "${OUT}"
python scripts/normalize_workflow.py "${WORKFLOW}"
cp "${WORKFLOW%.json}.norm.json" "${OUT}/workflow.json"
 
# Record which checkpoint each loader node actually resolved to, by hash.
python - "${OUT}/workflow.json" <<'PY' > "${OUT}/weights.lock"
import hashlib, json, pathlib, sys
 
MODEL_ROOT = pathlib.Path("/models")
graph = json.loads(pathlib.Path(sys.argv[1]).read_text())
 
for node in graph["nodes"]:
    for value in node.get("widgets_values", []):
        if isinstance(value, str) and value.endswith(".safetensors"):
            path = next(MODEL_ROOT.rglob(value), None)
            if path is None:
                print(f"MISSING {value}")
                continue
            digest = hashlib.sha256(path.read_bytes()).hexdigest()[:16]
            print(f"{value} sha256:{digest}")
PY
 
# Node pack versions, because a ComfyUI update can silently change behaviour.
pip freeze | grep -Ei "torch|diffusers|transformers" > "${OUT}/python.lock"
(cd custom_nodes && for d in */; do
   printf "%s %s\n" "${d%/}" "$(git -C "$d" rev-parse --short HEAD 2>/dev/null || echo unpinned)"
 done) > "${OUT}/nodes.lock"
 
git add "${OUT}" && git commit -m "snapshot: ${TAG}"
echo "snapshotted ${TAG}"

That nodes.lock file exists because of a specific bad afternoon. A custom node pack updated, a mask node changed its default behaviour on inverted masks, and outputs shifted with no change on our side. Nothing in the workflow JSON records which version of a third-party node interpreted it. Pinning the node repos by commit is the only defence, and I'd do it on day one next time rather than after losing a day.

Where the graph stops being enough

I want to be straight about the limits, because "ComfyUI is a real research environment" is a claim I'll defend and "ComfyUI is a good engineering environment" is one I won't.

Repeatability is partial. The workflow JSON pins the topology and the widget values. It does not pin the node implementations, the ComfyUI version, the model file contents or the CUDA and PyTorch stack underneath. Everything I wrote above about lockfiles is scaffolding bolted on around a format that was not designed to carry that information.

Review does not work. A pull request that changes a normalised workflow JSON is technically diffable, and I said earlier that this beat a notebook, which it did. It is still a bad review artifact. A reviewer looking at "widgets_values": [11, true] changing to "widgets_values": [7, true] has no idea that this is the mask dilation radius unless they already know the pipeline. There are no function names, no types, no comments and nowhere to put one. Every meaningful review we did happened by standing at a screen together, which does not scale past the people in the room and leaves no record.

Composition is weak. You cannot easily write the graph equivalent of a helper function that you call from four places. Subgraphs help and are not the same thing. Our workflow files had duplicated sections that drifted apart, which is exactly the failure mode you'd predict from a language without abstraction.

And graphs have no tests. There is no assertion you can attach to a node saying the mask must cover between four and sixty percent of the frame. That check lives in Python in the serving path, which means the research pipeline and the production pipeline validate different things, which is its own problem.

What I'd do differently

Treat the graph as the research surface and export it to code before anything it produces reaches a customer. We eventually reimplemented the settled parts of the pipeline as a Python service and kept ComfyUI for exploration. Doing that split deliberately, early, with a defined moment where a workflow graduates, would have saved a rewrite.

Pin third-party node packs from the first commit. The failure mode where nothing you changed changed the output is expensive to diagnose and trivial to prevent.

Write a one-paragraph README next to every experiment directory. The graph records what ran. It records nothing about why, and three weeks later "why" is the only thing you actually want.

Accept that the artifact is a diagram, not a program. Once I stopped expecting the workflow JSON to behave like source code and started treating it as an executable diagram with a lockfile stapled to it, my expectations matched reality and I stopped being annoyed at it.

The next thing I want to build is the bridge: a small exporter that walks a normalised workflow and emits the equivalent diffusers calls, so that graduating an experiment to the serving path is a mechanical step rather than a careful manual transcription that quietly drops a mask dilation somewhere.