ComfyUI as a research lab

A custom node pack updated, a mask node changed its default on inverted masks, and our outputs moved with nothing changed on our side. Running a try-on diffusion pipeline as a node graph made every experiment a commit, and made very clear where a graph stops being a program.

HimanshuFebruary 04, 2025Updated September 08, 2026

A community node pack updated silently overnight. A mask utility inverted its default boolean flag. Our renders shifted, with zero code changes on our side. We lost an afternoon finding out why.

Standard ComfyUI workflow JSON records node connections, but remains silent about third-party node package revisions.

ComfyUI is an effective visual prototyping environment and a difficult production engineering environment. Problems arise when treating the same artifact as both.

What node graphs represent in generative R&D

Our virtual try-on architecture combines two conditioning streams, a segmentation network, morphological mask operations, two merged LoRAs, and a two-pass sampler. During development, tracking what changed day-to-day is critical.

A visual node graph makes topology explicit. There is no hidden variable left connected in background memory. What is on the canvas is what runs.

Step 1: Separating topological changes from hyperparameter sweeps

Diffusion prototyping in Jupyter notebooks quickly becomes messy. Execution state depends implicitly on cell execution order and mutated global variables. When a run produces a strong render, you are often left with a dirty .ipynb file where cell 14 ran after cell 22, making git diffs unreadable.

We separated our experimental workflows based on whether an experiment altered pipeline topology or scalar values:

Experiment Type	Optimal Artifact	Rationale
Fixed topology, scalar tuning	YAML configuration per run	Clean text diffs across numerical sweeps
Daily structural rewiring	Visual node graph	The experiment consists of rewiring connections
Unstructured notebook exploration	Jupyter notebook	Interactive scratchpad, poor version control

When testing whether Flux Redux tokens should condition only the primary sampling pass or both passes, rewiring nodes in a visual graph provides immediate feedback.

Step 2: Normalizing workflow JSON exports for git versioning

ComfyUI workflows serialize natively to JSON, allowing them to be tracked in git.

However, default ComfyUI exports include transient UI state: screen coordinates, canvas zoom levels, and auto-incrementing internal IDs. Dragging a node across the screen produces large git diffs that obscure actual architectural changes. We built a pre-commit normalizer to strip canvas noise before saving.

python

# normalize_workflow.py
"""Strip canvas layout data from a ComfyUI workflow export.
 
Node positions, dimensions, and canvas viewports carry no execution
semantics. This script removes layout fields and sorts nodes deterministically
so git diffs reflect only pipeline changes.
"""
import json
import sys
from pathlib import Path
 
DROP_KEYS = {"pos", "size", "flags", "order", "extra", "color", "bgcolor"}
 
 
def normalize(workflow: dict) -> dict:
    """Return a layout-independent representation of a ComfyUI workflow."""
    nodes = []
    for node in sorted(workflow.get("nodes", []), key=lambda n: n["id"]):
        nodes.append(
            {k: v for k, v in sorted(node.items()) if k not in DROP_KEYS}
        )
 
    links = sorted(workflow.get("links", []), key=lambda l: (l[1], l[3], l[2]))
 
    return {
        "nodes": nodes,
        "links": links,
        "widget_defaults": workflow.get("extra", {}).get("ds", None) and None,
    }
 
 
def main(paths: list[str]) -> None:
    for p in paths:
        src = Path(p)
        dst = src.with_suffix(".norm.json")
        data = json.loads(src.read_text())
        dst.write_text(json.dumps(normalize(data), indent=2, sort_keys=True))
        print(f"{src.name} -> {dst.name}")
 
 
if __name__ == "__main__":
    main(sys.argv[1:])

Tracking normalized workflow files turned pipeline pull requests into clean, reviewable diffs.

Step 3: Graph topology and branch convergence

Here is an illustrative schema showing the convergence of segmentation, background removal, and dual conditioning streams:

json

{
  "_file": "flux_vton_plus.illustrative.json",
  "_note": "Simplified reference schema",
  "nodes": [
    { "id": 1, "type": "LoadImage",
      "widgets_values": ["model_0412.png"] },
    { "id": 2, "type": "LoadImage",
      "widgets_values": ["garment_ref_saree_green.png"] },
    { "id": 3, "type": "RMBG",
      "widgets_values": ["u2net", 0.5],
      "inputs": [{ "name": "image", "link": [2, 0] }] },
    { "id": 4, "type": "SAM2Segment",
      "widgets_values": ["shirt, tshirt, top, saree, dress"],
      "inputs": [{ "name": "image", "link": [1, 0] }] },
    { "id": 5, "type": "GrowMask",
      "widgets_values": [11, true],
      "inputs": [{ "name": "mask", "link": [4, 0] }] },
    { "id": 6, "type": "StyleModelLoader",
      "widgets_values": ["flux1-redux-dev.safetensors"] },
    { "id": 7, "type": "ReduxApply",
      "inputs": [
        { "name": "style_model", "link": [6, 0] },
        { "name": "reference", "link": [3, 0] }
      ] },
    { "id": 8, "type": "UNETLoader",
      "widgets_values": ["flux1-fill.safetensors", "default"] },
    { "id": 9, "type": "LoraLoaderModelOnly",
      "widgets_values": ["drape_physics_r32.safetensors", 0.6],
      "inputs": [{ "name": "model", "link": [8, 0] }] },
    { "id": 10, "type": "ModelMergeLoRA",
      "widgets_values": ["occlusion_depth_r32.safetensors", 0.4],
      "inputs": [{ "name": "model", "link": [9, 0] }] },
    { "id": 11, "type": "KSampler",
      "widgets_values": [30, 3.5, "euler_ancestral", "beta", 1.0],
      "inputs": [
        { "name": "model", "link": [10, 0] },
        { "name": "conditioning", "link": [7, 0] },
        { "name": "mask", "link": [5, 0] }
      ] },
    { "id": 12, "type": "KSampler",
      "widgets_values": [10, 3.5, "euler_ancestral", "beta", 0.35],
      "inputs": [
        { "name": "model", "link": [10, 0] },
        { "name": "latent", "link": [11, 0] }
      ] },
    { "id": 13, "type": "VAEDecode",
      "widgets_values": ["fp16"],
      "inputs": [{ "name": "samples", "link": [12, 0] }] }
  ]
}

Isolating the reference garment with RMBG before applying Redux prevents background studio elements from corrupting cross-attention tokens. The two-pass sampler splits generation between full 30-step spatial synthesis and a 10-step low-denoise pass for fine fabric texture.

Step 4: Surfacing critical merge coefficients on the canvas

We fuse LoRA adapters directly into base weights via model_merge_lora nodes upstream of inference:

W_{\text{merged}} = W_{\text{base}} + \lambda_{\text{drape}} \cdot (B_d A_d) + \lambda_{\text{occ}} \cdot (B_o A_o)

With lambda_drape = 0.6 (Expert A, draping physics) and lambda_occ = 0.4 (Expert B, occlusion depth):

Parameter Over-weighting	Visual Artifact	Underlying Cause
High $\lambda_{\text{occ}}$	Hands preserved cleanly	Garment fabric loses natural drape and hangs stiffly
High $\lambda_{\text{drape}}$	Folds and pleats render with depth	Fabric paints over foreground fingers and jewelry

Placing these coefficients as visible node inputs allowed team members to inspect trade-offs directly on the canvas without modifying underlying Python scripts.

Step 5: Pinning dependencies and weight hashes in lockfiles

To ensure repeatability, we snapshot normalized workflow JSON alongside explicit lockfiles capturing checkpoint SHA-256 hashes and exact git commits for all custom node packages.

bash

# scripts/snapshot_workflow.sh
# Freeze the current pipeline: normalized graph and exact weights.
set -euo pipefail
 
WORKFLOW="${1:?usage: snapshot_workflow.sh <workflow.json> <tag>}"
TAG="${2:?missing tag}"
OUT="experiments/${TAG}"
 
mkdir -p "${OUT}"
python scripts/normalize_workflow.py "${WORKFLOW}"
cp "${WORKFLOW%.json}.norm.json" "${OUT}/workflow.json"
 
# Record exact checkpoint SHA-256 hashes
python - "${OUT}/workflow.json" <<'PY' > "${OUT}/weights.lock"
import hashlib, json, pathlib, sys
 
MODEL_ROOT = pathlib.Path("/models")
graph = json.loads(pathlib.Path(sys.argv[1]).read_text())
 
for node in graph["nodes"]:
    for value in node.get("widgets_values", []):
        if isinstance(value, str) and value.endswith(".safetensors"):
            path = next(MODEL_ROOT.rglob(value), None)
            if path is None:
                print(f"MISSING {value}")
                continue
            digest = hashlib.sha256(path.read_bytes()).hexdigest()[:16]
            print(f"{value} sha256:{digest}")
PY
 
# Lock custom node commits to prevent silent upstream breaking changes
pip freeze | grep -Ei "torch|diffusers|transformers" > "${OUT}/python.lock"
(cd custom_nodes && for d in */; do
   printf "%s %s\n" "${d%/}" "$(git -C "$d" rev-parse --short HEAD 2>/dev/null || echo unpinned)"
 done) > "${OUT}/nodes.lock"
 
git add "${OUT}" && git commit -m "snapshot: ${TAG}"
echo "snapshotted ${TAG}"

Failure modes and edge cases

Unpinned third-party node repositories: Node packages cloned directly from git main introduce silent breaking changes during automated worker rebuilds.
Loss of individual adapter attribution: Once two LoRAs are merged into base weights, diagnosing single-expert regressions requires rebuilding graphs with isolated adapters.
Subgraph duplication drift: Duplicating node clusters across multiple workflows leads to configuration drift when parameters are updated in one workflow and forgotten in another.

When this is the wrong choice

The topology is fixed and only the numbers move. Sweeping learning rates or step counts across a stable pipeline is a CLI script and a YAML file. Clicking through canvas edits to change a scalar is slower and leaves no clean sweep record.
The change has to survive asynchronous review. A normalized JSON diff of a node array is still a node array. Reviewers outside the research group cannot see from it what moved in the graph.
You are serving requests. Production backends need type checking, queue leasing, and error handling that names the failing stage. None of that belongs in a canvas.
You need to attribute a regression to one adapter. Merging LoRAs into base weights on the graph is convenient and destroys per-adapter attribution. Diagnosing a single expert means rebuilding the graph with isolated adapters.