---
title: "LoRA finetuning: the hyperparameters settled in a week, the data never did"
description: "An early draping adapter kept rendering identical flat planes across shoulder folds, regardless of prompt. This is memorization rather than learned geometric priors, traced back to pinned studio photos. Hyperparameters stabilize quickly in Low-Rank Adaptation; dataset curation and caption isolation determine generalization."
date: "April 08, 2025"
url: "https://himanshuat.com/blogs/lora-finetuning-what-moved-the-needle"
---
# LoRA finetuning: the hyperparameters settled in a week, the data never did

An early draping adapter kept rendering the same flat plane across the back of the shoulder, under prompts that shared no vocabulary at all.

That artifact is the signature of dataset memorization, not a learned spatial prior. It traced back to a cluster of studio photographs where stylists had pinned the fabric tight behind the subject.

In Low-Rank Adaptation (LoRA) fine-tuning, the hyperparameters settle in a handful of sweeps. Dataset quality, caption token isolation, and aspect ratio bucketing decide whether the adapter generalizes.

> Rank and alpha define capacity and update magnitude. The dataset distribution defines the upper bound on generation quality.

### Chapter 0: Mechanics of alpha over rank scaling

[LoRA](https://arxiv.org/abs/2106.09685), introduced by Edward Hu and co-authors in 2021, freezes base model weights $W_0$ and injects trainable rank decomposition matrices:

$$
W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} (B \cdot A)
$$

where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ with rank $r \ll \min(d, k)$.

Rank ($r$) sets parameter capacity. The scaling factor $\frac{\alpha}{r}$ regulates how strongly adapter updates modify the frozen base representations. Section 4.1 of the paper puts it plainly: "We then scale $\Delta W x$ by $\alpha/r$, where $\alpha$ is a constant in $r$."

---

### Step 1: Fix the scale ratio below 1.0 to prevent style collapse

Setting $r = 32$ and $\alpha = 16$ yields a scaling factor of:

$$
\text{scale} = \frac{\alpha}{r} = \frac{16}{32} = 0.5
$$

| Alpha ($\alpha$) | Scale Factor ($\frac{\alpha}{r}$) | Observed Empirical Behavior |
|---|---|---|
| 16 | 0.5 | Target physical priors learned; base model style and photorealism preserved |
| 32 | 1.0 | Stronger drape effect; noticeable background and contrast overfitting |
| 64 | 2.0 | Severe style collapse; output mimics training set color grading and lighting |

A 0.5 ratio damps adapter updates relative to base attention activations, so the network picks up the fold geometry without overwriting what it already knows.

Use a base learning rate of `1e-4` with cosine decay. The cosine schedule lowers step sizes in later epochs, preventing the optimizer from memorizing spurious high-frequency noise from small image sets.

**The diagnostic check:** if increasing $\alpha$ amplifies the target concept while simultaneously degrading lighting diversity across unrelated prompts, the adapter update scale is too high.

---

### Step 2: Freeze text encoders and isolate attention projections

Train attention projection layers in the denoiser (Transformer blocks or UNet) while keeping text encoders frozen.

If the base foundation model already understands the semantic concept (for example, the noun "saree" or "tuxedo"), fine-tuning text encoders introduces shortcut paths: the optimizer rebinds text embeddings directly to training image artifacts rather than learning spatial cross-attention relationships.

Confining updates to the Query, Key, Value, and Output projections forces the adapter to learn relationships between spatial image patches instead.

**The diagnostic check:** if the adapter only produces the target concept when the trigger token appears verbatim, and ignores paraphrases the base model understands, the text encoder learned a shortcut.

---

### Step 3: Configure aspect ratio bucketing and gradient accumulation

Small batch sizes (such as `train_batch_size = 4`) paired with `gradient_accumulation_steps = 4` preserve bucket diversity.

[Aspect ratio bucketing](https://github.com/NovelAI/novelai-aspect-ratio-bucketing), released by NovelAI under MIT after they found square-crop training was generating "humans... without feet or heads", groups non-square images (such as tall full-body portraits or wide flat-lays) into discrete resolution bins. Large physical batches from a single bucket skew gradients toward specific aspect ratios. Accumulating gradients across multiple small batches averages updates across varied compositions.

The config below is [`kohya-ss/sd-scripts`](https://github.com/kohya-ss/sd-scripts) TOML, the training scripts most FLUX and SDXL LoRA work runs on.

```toml
# configs/drape_lora.toml
# Low-Rank Adaptation configuration for diffusion models

[model]
pretrained_model_name_or_path = "/models/flux/base"
vae = "/models/flux/vae"
mixed_precision = "bf16"
save_precision = "fp16"

[network]
network_module = "networks.lora_flux"
network_dim = 32                  # Rank (r)
network_alpha = 16                # Alpha (a) -> scale = 0.5
network_train_unet_only = true    # Keep text encoder frozen

[optimizer]
optimizer_type = "AdamW8bit"
learning_rate = 1e-4
lr_scheduler = "cosine"
lr_warmup_steps = 100
max_grad_norm = 1.0

[training]
train_batch_size = 4
gradient_accumulation_steps = 4
gradient_checkpointing = true
cache_latents = true
cache_latents_to_disk = true
seed = 42
max_train_epochs = 12
save_every_n_epochs = 1

[dataset]
resolution = "1024,1024"
enable_bucket = true
bucket_reso_steps = 64
min_bucket_reso = 768
max_bucket_reso = 1536
caption_extension = ".txt"
shuffle_caption = false
keep_tokens = 1                   # Trigger token fixed at position 0
```

Using 8-bit AdamW ([`AdamW8bit`](https://huggingface.co/docs/bitsandbytes/main/en/optimizers), from Tim Dettmers and co-authors' [8-bit Optimizers via Block-wise Quantization](https://arxiv.org/abs/2110.02861)) reduces optimizer memory footprint from 8 bytes to 2 bytes per parameter, freeing VRAM for higher training resolutions (1024x1024+).

---

### Step 4: Technical filtering and automated image validation

Mechanical filters reject low-quality inputs prior to training:

```python
# data/filter_corpus.py
from dataclasses import dataclass
from pathlib import Path
import cv2
import numpy as np

@dataclass(frozen=True)
class FilterConfig:
    min_short_side: int = 1024
    min_sharpness: float = 120.0        # Variance of Laplacian
    min_aspect: float = 0.5             # Portrait limit
    max_aspect: float = 1.6             # Landscape limit
    max_clipped_fraction: float = 0.04  # Blown highlights threshold

def calculate_sharpness(gray: np.ndarray) -> float:
    return float(cv2.Laplacian(gray, cv2.CV_64F).var())

def calculate_clipped_fraction(gray: np.ndarray) -> float:
    return float((gray >= 250).mean())

def inspect_image(path: Path, cfg: FilterConfig):
    image = cv2.imread(str(path))
    if image is None:
        return False, "unreadable"

    h, w = image.shape[:2]
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    aspect = w / h

    if min(h, w) < cfg.min_short_side:
        return False, f"resolution {w}x{h}"
    if not (cfg.min_aspect <= aspect <= cfg.max_aspect):
        return False, f"aspect {aspect:.2f}"
    if calculate_sharpness(gray) < cfg.min_sharpness:
        return False, "soft_focus"
    if calculate_clipped_fraction(gray) > cfg.max_clipped_fraction:
        return False, "clipped_highlights"

    return True, None
```

```mermaid
flowchart TD
  A[Repeated Artifact in Generation] --> B{Style or Geometry?}
  B -->|Style / Color Bleed| C[Scale Alpha / Rank Too High]
  B -->|Fixed Geometric Distortion| D[Isolate Bad Sub-cluster in Dataset]
  C --> E[Lower Alpha to 0.5 Ratio]
  D --> F[Prune / Recaption Dataset]
  F --> G[Retrain Checkpoint]
  E --> G
```

---

### Step 5: Caption token isolation

Captions must explicitly describe background, lighting, and camera perspective so the model does not entangle those traits with the trigger token.

If every training image sits on a solid grey studio background and no caption says so, the trigger token absorbs the grey studio into its concept. Naming the non-target elements is what decouples them from the adapter weights.

**The diagnostic check:** generate with the trigger token and a prompt that specifies a different background. If the grey studio comes back anyway, the captions did not isolate it.

---

### Step 6: Evaluation across fixed seed grids and out-of-domain prompts

Select optimal training checkpoints using a standardized validation suite:

1. **Fixed seed grids:** Generate samples for each saved epoch using identical prompt sets and constant seeds.
2. **Partial adapter weights:** Evaluate checkpoints at merged strengths ($0.5$ to $0.7$) to verify interoperability with secondary adapters.
3. **Out-of-domain regression checks:** Generate non-target subjects (such as basic denim or leather jackets) to detect whether the adapter induces catastrophic forgetting on general attire.

---

### When this is the wrong choice

- **The base model already renders the concept.** Step 2 turns on exactly this. If the base understands the noun, an adapter mostly rebinds the token to your training artifacts, and prompt work costs nothing and is reversible.
- **Most of your images fail the mechanical filters.** The filters in Step 4 reject soft focus, blown highlights, and off-aspect frames. If the set thins out badly under them, no rank or alpha setting recovers the missing distribution. You get the flat shoulder plane at the top of this post.
- **The defect is a caption problem, not a capacity problem.** Step 5's failure mode is entanglement. When the trigger token has absorbed the background or the lighting, re-captioning the set you already have fixes it, and retraining at a different rank does not.

---

Source: https://himanshuat.com/blogs/lora-finetuning-what-moved-the-needle