LoRA finetuning: the hyperparameters settled in a week, the data never did

I went into LoRA training expecting a long tuning phase. Rank sweeps, learning rate schedules, the whole ritual. What actually happened is that the config settled after a handful of runs and then sat unchanged for months while I rebuilt the dataset more than once. That's the honest shape of the work, so this post is mostly about the second part, with the first part written down properly because people keep asking for it.

The config, and why it stopped moving

Both experts used the same settings. Expert A is Draping Physics, trained on sarees, hanfus, kimonos and complex haute couture. Expert B is Occlusion and Depth, trained on hands on hips, arms crossed, and jewelry sitting over clothing. Same rank, same alpha, same schedule, different data, and that symmetry is deliberate: if the two adapters are going to be merged into one backbone later, you want their update magnitudes to be comparable rather than accidentally scaled apart by their training configs.

Rank 32, alpha 16. The ratio matters more than either number on its own. A LoRA computes its weight update as a low-rank product scaled by alpha over rank:

ΔW = (α / r) · B A

With alpha 16 and rank 32 the scale is 0.5. The common conventions are alpha equal to rank, which gives a scale of 1.0, or alpha at twice the rank, which gives 2.0. Running at 0.5 damps every update the adapter makes to the base weights, and the practical effect is roughly the same as halving the learning rate for the adapter's contribution while leaving the optimizer's own step size alone.

That's the behaviour I wanted. These adapters exist to add a physics prior to a model that's already very good at images. If the adapter shouts, you get the thing everyone gets on their first LoRA: the output stops looking like the base model and starts looking like the training set, including its backgrounds, its colour grading, and its favourite camera angle. Rank 32 gives enough capacity to represent something as structured as pleat geometry, and the low alpha keeps that capacity from bleeding into everything else. Raising alpha to 32 made the drape more emphatic and the images more obviously overfit at the same time, which is not a trade I wanted.

Learning rate 1e-4 with cosine annealing. Standard, and it stayed standard. Cosine matters more than the peak value here because the last stretch of training on a small, visually consistent dataset is where an adapter picks up the dataset's incidental style, and annealing down keeps those late steps small.

Batch size 4 with gradient accumulation. There's room on a 48GB L40S to raise the real batch instead, and I deliberately didn't, because of bucketing. Images of different shapes can't be stacked into one tensor, so a physical batch is drawn from inside a single aspect bucket, and in this dataset a bucket is effectively a shot type: tall full-length shots in one, wide flat-lays in another. A large physical batch means a long run of gradient signal from one shot type. Accumulating small batches lets the effective batch span buckets, so a step averages across shapes rather than specialising to whichever bucket it landed in. It also keeps the number of optimizer steps per epoch high enough that a cosine schedule has something to anneal over, which a few thousand images at a large true batch would not.

AdamW8bit. Eight-bit optimizer states. The memory it frees goes into resolution rather than batch size, which for garment work is the better place to spend it. Fabric detail is the entire point, and a pleat rendered at low resolution in training is a pleat the adapter never learns.

UNet only, attention projections only. The frozen text encoder is a decision about what kind of adapter this is. Training the text encoder too lets the model redefine words, which is right when you're teaching a new token: make this made-up word mean this person, or this illustration style. Draping physics isn't a word-meaning problem. The base model knows what a saree is; what it doesn't know is what the fabric does under gravity, and that lives in the denoiser's spatial attention rather than in a text embedding.

Leaving the text encoder trainable also hands the optimizer a cheaper route to the same loss. It can rebind the caption tokens to the training set's overall look, reconstruct better, and learn nothing about folds. Freezing it removes that option. The same reasoning keeps the adapter on the attention projections rather than every linear layer in the network: query, key, value and output projections are where spatial relationships between regions get decided, and a fold is a spatial relationship.

The training config

# configs/drape_lora.toml
# Expert A: Draping Physics. Kohya-ss network trainer.
# Expert B (occlusion) uses this file with the dataset paths swapped.
 
[model]
pretrained_model_name_or_path = "/models/flux/base"
vae = "/models/flux/vae"
mixed_precision = "bf16"
save_precision = "fp16"
 
[network]
network_module = "networks.lora_flux"
network_dim = 32                  # rank
network_alpha = 16                # scale = alpha / dim = 0.5
network_train_unet_only = true    # text encoder stays frozen
 
[optimizer]
optimizer_type = "AdamW8bit"
learning_rate = 1e-4
lr_scheduler = "cosine"
lr_warmup_steps = 100
max_grad_norm = 1.0
 
[training]
train_batch_size = 4
gradient_accumulation_steps = 4
gradient_checkpointing = true
cache_latents = true
cache_latents_to_disk = true
seed = 42
max_train_epochs = 12
save_every_n_epochs = 1           # keep every epoch; the best one is chosen by eye
 
[dataset]
resolution = "1024,1024"
enable_bucket = true              # sarees and full-body shots are not square
bucket_reso_steps = 64
min_bucket_reso = 768
max_bucket_reso = 1536
caption_extension = ".txt"
shuffle_caption = false           # caption order is load-bearing, see below
keep_tokens = 1                   # the trigger token stays in position one
 
[[dataset.subsets]]
image_dir = "/data/drape/saree"
num_repeats = 1
 
[[dataset.subsets]]
image_dir = "/data/drape/hanfu_kimono"
num_repeats = 1
 
[[dataset.subsets]]
image_dir = "/data/drape/couture"
num_repeats = 1
 
[logging]
logging_dir = "/logs/drape"
log_with = "tensorboard"

Two lines in there are doing more work than they look like they are. shuffle_caption = false with keep_tokens = 1 pins the trigger token to the front of every caption, which makes the adapter's activation predictable at inference instead of something that partially fires whenever a related word appears. And bucketing is not optional for this domain. A full-length saree shot is tall, a flat-lay couture reference is wide, and if you square-crop everything to fit a fixed resolution you cut off precisely the pallu drape you're trying to teach.

save_every_n_epochs = 1 is there because no validation loss on this task correlates with "the pleats look right", so epoch selection stays a human judgement and every epoch has to survive to be judged.

The invocation

# scripts/train_drape_lora.sh
# Single L40S node (48GB). Run from the kohya-ss/sd-scripts checkout.
 
set -euo pipefail
 
RUN_NAME="drape_r32a16_$(date +%Y%m%d_%H%M)"
DATA_ROOT="/data/drape"
OUT_DIR="/artifacts/lora/${RUN_NAME}"
 
mkdir -p "${OUT_DIR}"
 
# Fail here rather than deep into a run: every image needs a caption file.
missing=$(find "${DATA_ROOT}" -name '*.jpg' | while read -r img; do
  [ -f "${img%.jpg}.txt" ] || echo "${img}"
done | wc -l)
 
if [ "${missing}" -gt 0 ]; then
  echo "aborting: ${missing} images have no caption file" >&2
  exit 1
fi
 
accelerate launch \
  --num_cpu_threads_per_process 8 \
  --mixed_precision bf16 \
  flux_train_network.py \
  --config_file configs/drape_lora.toml \
  --output_dir "${OUT_DIR}" \
  --output_name "${RUN_NAME}" \
  --save_state
 
# Every epoch is kept. Selection happens by generating the fixed evaluation
# prompt set against each checkpoint and looking at the results.
ls -1 "${OUT_DIR}"/*.safetensors
echo "checkpoints written to ${OUT_DIR}"

The caption check at the top exists because I lost a run to it. Kohya will happily train on an image with an empty caption and the result is an adapter that has learned to associate the trigger with nothing in particular.

I'm not going to quote wall-clock times, cost, or a loss curve here. I don't have numbers from those runs that I'd stand behind publishing, and a training curve on this task wouldn't tell you much anyway, since the loss goes down smoothly through epochs that are visibly getting worse.

What actually moved the needle

Once the config was fixed, every real improvement came from the dataset. Four things, roughly in order of how much they mattered.

Filtering before anything else. Public images go through a mechanical pass first: resolution, sharpness, aspect ratio. This throws away a lot and it should. An image that's soft at the garment boundary teaches the adapter that garment boundaries are soft, and there's no later stage that undoes that.

# data/filter_corpus.py
"""First-pass technical filter over a candidate image corpus.
 
Nothing here is about whether an image is *good*. It's about whether the
image is technically usable as a training example for fabric structure.
"""
 
from dataclasses import dataclass
from pathlib import Path
 
import cv2
import numpy as np
 
 
@dataclass(frozen=True)
class FilterConfig:
    min_short_side: int = 1024      # below this, fabric texture is already gone
    min_sharpness: float = 120.0    # variance of Laplacian
    min_aspect: float = 0.5         # portrait limit
    max_aspect: float = 1.6         # wide-crop limit
    max_clipped_fraction: float = 0.04  # blown highlights eat white fabric
 
 
def sharpness(gray: np.ndarray) -> float:
    """Variance of the Laplacian. Crude, fast, and good enough at scale."""
    return float(cv2.Laplacian(gray, cv2.CV_64F).var())
 
 
def clipped_fraction(gray: np.ndarray) -> float:
    """Fraction of pixels at or near the top of the range."""
    return float((gray >= 250).mean())
 
 
def inspect(path: Path, cfg: FilterConfig):
    """Return (accepted, reason). Reason is None when accepted."""
    image = cv2.imread(str(path))
    if image is None:
        return False, "unreadable"
 
    h, w = image.shape[:2]
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    aspect = w / h
 
    if min(h, w) < cfg.min_short_side:
        return False, f"resolution {w}x{h}"
    if not cfg.min_aspect <= aspect <= cfg.max_aspect:
        return False, f"aspect {aspect:.2f}"
    if sharpness(gray) < cfg.min_sharpness:
        return False, "soft"
    if clipped_fraction(gray) > cfg.max_clipped_fraction:
        return False, "clipped highlights"
 
    return True, None
 
 
def run(root: Path, cfg: FilterConfig = FilterConfig()):
    kept, rejected = [], {}
    for path in sorted(root.rglob("*.jpg")):
        ok, reason = inspect(path, cfg)
        if ok:
            kept.append(path)
        else:
            rejected[reason] = rejected.get(reason, 0) + 1
 
    print(f"kept {len(kept)} of {len(kept) + sum(rejected.values())}")
    for reason, count in sorted(rejected.items(), key=lambda kv: -kv[1]):
        print(f"  rejected {count:>6}  {reason}")
    return kept

What that filter can't catch is the sample type that hurt most: catalogue shots where the drape is fake. Studio garments get clipped, pinned, or taped at the back so the front hangs the way the stylist wants. Those images are sharp, well lit, high resolution, and they pass every check in that script, while being photographs of fabric under a tension gravity never applied. It's the exact opposite of what a draping adapter should learn.

I found it by accident. An early Draping expert kept producing the same suspiciously flat plane across the back of the shoulder on prompts that had nothing in common, which is the signature of memorisation rather than a learned prior. It traced back to a cluster of images from one shoot where the fabric had been pulled tight and pinned out of frame. The check that came out of that has stayed useful: repeated style across unrelated prompts is an alpha problem, and repeated geometry is almost always a specific pile of images you need to go find.

Caption discipline. Captions have to describe what varies and stay silent about what doesn't. If every saree image says "saree" and every one of them also happens to be shot against a grey backdrop, and the caption never mentions the backdrop, the adapter learns that grey backdrop is part of what "saree" means. Writing captions that name the incidental properties is how you tell the model those properties are separable. This was more work than all the hyperparameter tuning combined and it produced more improvement.

Pose diversity inside the occlusion set. The Occlusion and Depth expert is trained on hand-garment interaction, and the first version of that dataset had a lot of near-duplicates: the same hand-on-hip pose from slightly different angles. That teaches a pose, not a relationship. What the adapter needs is the same relationship, a hand in front of fabric, across genuinely different hands, garments, and fabric behaviours behind the hand.

Being willing to throw a run away. The cheapest debugging tool was accepting that a dataset was wrong and rebuilding it, rather than trying to rescue it with a different learning rate. I resisted this for longer than I should have because retraining feels expensive and tuning feels productive.

Deciding an adapter was done

There was no automated metric for drape quality at the time, and "we looked at the outputs" is only half the story of what replaced it.

Every saved epoch gets generated against the same frozen prompt set with the same fixed seeds. Frozen and fixed are the load-bearing words: change the prompts and you're comparing two things at once, change the seeds and you're mostly looking at sampling noise. The result is a grid, prompts down one axis and epochs across the other, and the eye reads that grid well even though it scores any single image badly. What you're watching for is two failures moving in opposite directions. Undercooked is mushy pleats and a pallu that won't commit to a shoulder. Overcooked shows up nowhere near the garment: backgrounds converging, colour grading drifting toward the training set's, faces getting less varied. You want the last epoch where drape improved and the second kind of drift hadn't started, which is usually not the final one.

Two other checks mattered more than they sound. Look at the adapter at reduced strength as well as at full weight, because it will be merged at a coefficient below one alongside the occlusion expert, and an adapter whose drape only appears at full strength is useless to that merge. I threw one away that looked excellent in isolation for exactly this reason. Then generate things well outside the training domain, a plain t-shirt or a jacket, anything the Draping expert has no business touching. Catastrophic forgetting announces itself outside the domain first, and a checkpoint that renders a beautiful saree and a broken t-shirt has learned the dataset rather than the physics.

What I'd do differently

I'd build the evaluation prompt set before the first training run rather than partway in. Epoch selection is subjective, and a frozen prompt set at least makes it consistently subjective. We ended up with one, and having it from the start would have saved the early runs that I now can't compare against anything.

I'd stop treating hyperparameters as the interesting surface. Rank and alpha have a real effect and it's a small, bounded one. The dataset has an unbounded one.

I'd version datasets as strictly as code. Half of my confusion about why one adapter behaved differently from another came down to not knowing exactly which images went into it.

And I'd separate "the adapter learned the concept" from "the adapter learned the dataset" as an explicit check rather than a vibe. That distinction is the actual outcome of the alpha-to-rank choice, and it deserved a deliberate test rather than my eye on a grid of samples.

The next thing worth doing is a proper ablation on the alpha ratio, holding the dataset fixed and training at 0.5, 1.0, and 2.0 scale, then evaluating for style contamination rather than for drape quality. I picked 0.5 by reasoning and confirmed it by looking, which isn't the same as measuring it.