# Himanshu — full site text Every page on https://himanshuat.com as Markdown, concatenated. Each section keeps the canonical URL it came from, so anything quoted here can be attributed. Any single page is also available on its own by appending `.md` to its URL. ======================================================================== # Himanshu I co-founded **AuraX**, a try-on startup whose diffusion models outperformed Google's and Stable Diffusion's in our benchmarks. Before that I built **SynthAiLabs**, a 500-student learning community that juniors restarted after I left. Most recently I built search and agents at **Tejas AI (YC W25)**. I left in August 2026 and I'm **looking for what's next**. ## Availability Left Tejas AI (YC W25) in August 2026. Open to full-time roles in AI infrastructure, search and quantitative research; contract engagements; and research collaborations. Book 15 minutes: https://cal.com/himanshuat/makecustom12?duration=15 — or email hyattherate2005@gmail.com ## Sections - [About](https://himanshuat.com/about) — Markdown: https://himanshuat.com/about.md - [Experience](https://himanshuat.com/experience) — Markdown: https://himanshuat.com/experience.md - [Projects](https://himanshuat.com/projects) — Markdown: https://himanshuat.com/projects.md - [Research](https://himanshuat.com/research) — Markdown: https://himanshuat.com/research.md - [Writing](https://himanshuat.com/blogs) — Markdown: https://himanshuat.com/blogs.md - [Books](https://himanshuat.com/books) — Markdown: https://himanshuat.com/books.md - [Skills](https://himanshuat.com/skills) — Markdown: https://himanshuat.com/skills.md - [Changelog](https://himanshuat.com/changelog) — Markdown: https://himanshuat.com/changelog.md - [Work with me](https://himanshuat.com/hire) — Markdown: https://himanshuat.com/hire.md --- Source: https://himanshuat.com/ ======================================================================== # About Himanshu I co-founded **AuraX**, a try-on startup whose diffusion models outperformed Google's and Stable Diffusion's in our benchmarks. Before that I built **SynthAiLabs**, a 500-student learning community that juniors restarted after I left. Most recently I built search and agents at **Tejas AI (YC W25)**. I left in August 2026 and I'm **looking for what's next**. ## 01 · Built a community that outlived me. *2023 – 2024* In my 2nd year at NIT Kurukshetra, I started **SynthAiLabs (OpenEdu)** — part startup, part movement. I hand-selected 500 students from 2,000 and built an accelerated learning community. Students were completing 6 months of curriculum in just 2. The revenue model was B2B: companies could distribute coupons to students, and I'd sell outsourced learning materials and software to companies training their employees. But funds were always the bottleneck. What happened after I stopped was the real validation — **juniors restarted and continued it on their own.** It had given them something worth continuing. ## 02 · Beat the benchmarks. Ran out of runway. *2024 – 2025* I co-founded **AuraX**, a fashion-tech startup incubated at **IIIT-Hyderabad's CIE**. We researched **diffusion models for virtual try-on** and built a pipeline that, in our quality benchmarks, outperformed Google's and Stable Diffusion's published results — along with the Indian competitors we tested against. We signed a deal with **Aditya Birla Fashion Group** and several brands under **Mensa Brands**. Real revenue. Real contracts. In October 2025, **Google DeepMind's vision team in Bangalore** came to me for a research engineer role, the loop cut to one technical interview. I said no. We were mid-roadmap, and I would rather build my own thing than take a research seat in someone else's lab. But the space moved fast. Larger players with deeper pockets entered. We ran out of runway. I made the call to close AuraX — not because the tech failed, but because the timing wasn't on our side. ## 03 · Pause. Reset. Rebuild. *early 2026* After closing AuraX, I took intentional time off. I studied finance to understand the business mistakes I'd made — and started building on [MQL5](https://www.mql5.com/en/users/himanshuat) along the way. I focused on health — something I'd ignored for two years of building. > Sometimes, the most productive thing that you can do is to step outside and do nothing... relax and enjoy nature. > > — Melanie Charlene ## 04 · Shipped agents at scale. Then left. *2026* For five months I was an SDE at **Tejas AI (YC W25)** — WhatsApp-based AI agents with cross-session memory, a scraping fleet indexing roughly 60 ATS platforms, and OpenSearch-backed hybrid retrieval doing the matching. The lesson was that none of the hard parts were model problems. They were memory, deterministic routing, and knowing when a user's requirement is a filter rather than a similarity score. I left in **August 2026**. ## 05 · Open to what's next. *Now* I'm looking for the next thing, and I'm deliberately open about the shape it takes: a **full-time role** in AI infrastructure, search, or quantitative research; **contract work** on inference cost, agent reliability, or retrieval that misses; or a **research collaboration** — I'm currently working on measurement studies in systems and finance. If any of that is a thing you need, the fastest route is [email](mailto:hyattherate2005@gmail.com). The productized versions of the contract work are on the [hire page](/hire). --- Source: https://himanshuat.com/about ======================================================================== # Experience Every role, newest first. Append `.md` to any detail URL for its Markdown. ## Software Development Engineer (SDE) — Tejas AI (YC W25) *Mar 2026 – Aug 2026 · SF, California, USA* Tejas AI builds Elip, an AI career coach that lives where job seekers already are: WhatsApp, with a web app, a Chrome extension and a mobile app around it. The pitch is that looking for work is mostly unglamorous logistics, and most of that can be handled by an agent that remembers you between conversations. Detail: https://himanshuat.com/experience/tejas-ai-yc-w25-software-development-engineer-sde.md ## AI Researcher & CTO — AuraX *Dec 2024 – Dec 2025 · Hyderabad, India* AuraX was a fashion-tech company incubated at IIIT-Hyderabad's CIE. Fashion e-commerce needs on-model photography for every garment, in every size, on every body type, and a photoshoot per SKU does not scale. We generated that imagery with diffusion models instead. Detail: https://himanshuat.com/experience/aurax-ai-researcher-cto.md ## Founder — SynthAiLabs (formerly OpenEdu) *Dec 2023 – Feb 2025 · Remote* SynthAiLabs, which started as OpenEdu, was an attempt to compress how long it takes a student to become employable. Part edtech product, part community, aimed at Indian engineering students who had the ability but not the ramp. Detail: https://himanshuat.com/experience/synthailabs-formerly-openedu-founder.md ## AI Dev — Tublian *July 2024 – Dec 2024 · Columbus, Ohio* Tublian is a developer platform, with TublianOS as the product and an open-source collaboration programme around it. I came back to it in a second, more engineering-focused stint. Detail: https://himanshuat.com/experience/tublian-ai-dev.md ## QM Researcher — Stealth *March 2024 – June 2024 · Kyoto, Japan* A stealth research group in Kyoto working at the intersection of quantum computing and machine learning. The specifics are under NDA, so this entry stays deliberately thin. Detail: https://himanshuat.com/experience/stealth-qm-researcher.md ## Open Source Collaborator | Project Mentor — Tublian *Dec. 2023 – April 2024 · Columbus, Ohio* Tublian's open-source programme pairs developers with real projects and with each other. I joined as a contributor and ended up mentoring. Detail: https://himanshuat.com/experience/tublian-open-source-collaborator-project-mentor.md ## Project Manager — Excelerate *April 2023 – Sept 2023 · Remote* Excelerate runs global education and experience programmes for students. I led delivery on one of their events. Detail: https://himanshuat.com/experience/excelerate-project-manager.md --- Source: https://himanshuat.com/experience ======================================================================== # Research Published and ongoing work. Append `.md` to any detail URL for its Markdown. ## It's the Fault Policy, Not the Page Size: Why mmap Loses to read() for Out-of-Core Scans on Apple Silicon *ongoing · Systems* Re-measuring mmap vs read() on Apple Silicon: a working-set crossover, attributed to OS fault-population policy rather than page size. Detail: https://himanshuat.com/research/mmap-vs-read-apple-silicon.md ## Conflict-Aware Adapter Composition (C-AAC) *completed · Computer Vision* Outperforms base Flux, Imagen 4, and ChatGPT in skin quality and aesthetic appeal. Detail: https://himanshuat.com/research/c-aac.md ## Flux-VTON+: Hybrid Flux Inpainting and Multi-LoRA Expert Fusion *completed · Generative AI* State-of-the-art VTON pipeline synergizing Flux Fill and Redux with Multi-LoRA Fusion, achieving an 85% success rate on complex garments. Detail: https://himanshuat.com/research/renu-virtual-try-on.md ## MoE: Brain-Inspired LLM Architecture *ongoing · Artificial Intelligence* A whitepaper advancing MoE as a computational analogue to the brain's functional specialization. Detail: https://himanshuat.com/research/moe-brain.md ## High-Dimensional Lattice-Based Quantum Encryption (LWE) *stopped · Cryptography* A PQC scheme using the LWE problem with detailed mathematical formulations. Detail: https://himanshuat.com/research/lattice-based-quantum-encryption.md ## GPT-Neo: Transformer Implementation *completed · Natural Language Processing* Implementing 'Attention is All You Need' from scratch. Detail: https://himanshuat.com/research/gpt-neo.md ## Custom Neural Network for MNIST *completed · Machine Learning* A custom-built Neural Net for the MNIST digit recognizer dataset. Detail: https://himanshuat.com/research/custom-n-n-mnist.md --- Source: https://himanshuat.com/research ======================================================================== # Projects ## CCProxy *ongoing · AI Tools · Mar 2026 -- ongoing* - A universal proxy that lets Claude Code talk to any AI provider: OpenAI, Gemini, or local Ollama models. Stack: AI Tools, TypeScript Site: https://ccproxy.himanshuat.com/ Source: https://github.com/Himasnhu-AT/ccproxy ## OpenCrow *ongoing · AI Platform · Jan 2026 -- ongoing* - Open-source AI agent platform for embedding intelligent customer support assistants into web applications. - Features an embeddable React chat widget with dual-layer tooling: server-side tools via OpenAPI specs and client-side tools executed in the browser. - Dynamic tool merging lets the LLM pick between server and client actions per request. - Includes a management dashboard for agents, products, and integrations. Inspired by usecrow.ai. Stack: React, TypeScript, Node.js, OpenAPI, LLM Site: https://opencrow.himanshuat.com Source: https://github.com/Himasnhu-AT/opencrow ## AI Inference Platform *completed · Infrastructure · Dec 2025 -- Jan 2026* - High-performance AI inference platform with intelligent model routing — classifies query complexity via embeddings and heuristics, routing simple queries to fast/cheap models and complex ones to GPT-4 class models. - Multi-level caching: L1 in-memory (Moka) + L2 distributed (Redis) with semantic similarity matching via cosine similarity on embeddings. - Full observability stack with Prometheus metrics and pre-configured Grafana dashboards for inference latency, cache hit rates, and model health. - Fully async Rust architecture with horizontal scaling via Docker and Kubernetes. Stack: Rust, Axum, Tokio, Redis, Prometheus, Grafana, Docker Site: https://infer.himanshuat.com ## TuneSLM *ongoing · AI Tools · Jan 2026 -- ongoing* - Drop your dataset and answer a few questions — get an ipynb notebook ready for Google Colab to fine-tune your SLM of choice, powered by Unsloth. - Future: AI-driven dataset analysis to automatically choose optimal training strategies and hyperparameters. Stack: Python, Unsloth, Google Colab, LLM Site: https://tuneslm.himanshuat.com ## MockServer *ongoing · DevTools · Nov 2025 -- ongoing* - A CLI tool that spins up a mock backend so you can build the frontend before the real one exists, with a GUI to configure it. - Inject latency and errors on demand to test loading and error states in the frontend. Stack: TypeScript, Next.js, Express Site: https://mockserver.himanshuat.com Source: https://github.com/Himasnhu-AT/mockserver ## FastSearch *paused · Software · April 2024 -- June 2024* - A privacy-focused search engine written in Rust, as an alternative to Google Search. - Ranks results with TF-IDF over an inverted index built for fast lookups. Stack: Rust, JavaScript Source: https://github.com/Himasnhu-AT/FastSearch ## FallenAngel-SE/CY *archived · AI tools · 2024 -- Present [Paused because of funds issue]* - A suite of AI tools for development and security, built as a coding assistant. - Runs on the LLaMA 3.1 70B model. - Includes interactive learning features. Stack: AI, Python, TypeScript, Cybersecurity demo1: https://www.linkedin.com/posts/himanshu-at_testing-out-new-product-revealed-soon-activity-7232783015360221185-t-Dr demo2: https://www.linkedin.com/posts/himanshu-at_llm-really-matters-a-lot-in-my-earlier-activity-7233145551096012800-d87p demo3: https://www.linkedin.com/posts/himanshu-at_edtech-machinelearning-pythonprogramming-activity-7233733311326474241-FbZK ## Dynamo-Prisma *completed · package · 2024* - Generates Prisma models from a JSON schema, so you don't hand-write them. Stack: TypeScript, Prisma, Node.js Source: https://github.com/techsavvyash/dynamo-prisma ## ScratchAI-Algo *in-progress · Learning · SynthAiLabs · April 2024 -- May 2024* - AI/ML algorithms implemented from scratch, each with a written explanation. - Covers KNN, logistic regression, K-means, decision trees, SVM, and more; 10+ so far. - Built for the SynthAiLabs education work, open to contributions. Stack: Python, NumPy, Pandas Source: https://github.com/Himasnhu-AT/ScratchML-Algo ## QM - Research *in-progress · Learning · April 2024 -- June 2024* - A research repo on using quantum computing against encryption like RSA, SHA-256, and SHA-128. - Built quantum circuits in Qiskit: Quantum Fourier Transform, GHZ state, and commutation-relation experiments. - Wrote OpenQASM for quantum registers and entanglement experiments. Stack: Python, Qiskit, OpenQASM Source: https://github.com/Himasnhu-AT/QM-res ## RustyExplorer *completed · Application · Oct 2023 -- Dec 2023* - A cross-platform file explorer built with Rust and TypeScript on Tauri. - Searches files noticeably faster than Windows Explorer, with cached directory listings to keep navigation snappy. Stack: Rust, TypeScript, Tauri, Tailwind CSS, Vite, React Source: https://github.com/Himasnhu-AT/rusty-explorer ## Linux Mini-Firewall *completed · Learning · July 2023* - A mini firewall application that allows users to add, delete, and print rules based on criteria like source IP, destination IP, and ports. Stack: C Source: https://github.com/Himasnhu-AT/Linux_mini-Firewall ## indieOS *paused · Learning · May 2024* - A demo operating system written from scratch for the x86 architecture. - Includes a bootloader, shell, ports, and custom memory management. Stack: Assembly, C, Bash, OS Source: https://github.com/Himasnhu-AT/indieOS ## Codeforces Scraper *completed · scrapper · June 2024* - A scraper to extract problems from Codeforces HTML pages and return them in JSON format. Stack: Python Source: https://github.com/Himasnhu-AT/codeforce_scrapper ## Log Ingestor *archived · package · Aug 2024 -- Sep 2024* - A high-performance log ingestor that stores logs in a database or prints them to the terminal in a clean format. Stack: Rust Source: https://github.com/Himasnhu-AT/LogInjestor-depreciated- ## ZenStore (In-memory DB) *paused · Application · Oct 2024* - A minimalistic in-memory database inspired by Redis, built entirely in Rust. Stack: Rust Source: https://github.com/Himasnhu-AT/ZenStore ## frm (Fast Delete) *completed · package · Dec 2024* - A high-performance, safe alternative to the `rm` command for fast file deletion, written in Rust. Stack: Rust Source: https://github.com/Himasnhu-AT/frm ## roadmap.ai *archived · Application · Oct 2024 -- Nov 2024* - An AI-powered tool that generates personalized learning roadmaps for various technical topics. Stack: Next.js, AI Site: https://roadmap-ai-xi.vercel.app ## biv *completed · package · Mar 2025* - An experimental, high-performance binary data format designed as a faster alternative to JSON. Stack: Rust Source: https://github.com/Himasnhu-AT/biv ## ruspy *paused · Learning · Nov 2024 -- Mar 2025* - A hobby programming language combining Rust's efficiency with Python's simplicity, built to learn compiler design. Stack: Rust, Compiler Design Source: https://github.com/Himasnhu-AT/ruspy ## PoW-Shield (DDoS Prevention) *completed · Application · Aug 2023* - Prevents DDoS attacks using a Proof of Work system to validate requests before forwarding them to the server. Stack: C++, Rust, C, TypeScript, Shell Source: https://github.com/Himasnhu-AT/PoW-Shield ## git-history *completed · package · Sep 2024* - A utility to convert a repository's entire Git commit history into a structured JSON file. Stack: Rust, Python Source: https://github.com/Himasnhu-AT/git-history ## localai *completed · Application · Apr 2025* - A desktop app to chat with and compare various local AI models deployed via Ollama, with an integrated web search tool. Stack: Tauri, Rust, Next.js, Ollama Site: https://github.com/Himasnhu-AT/localai ## draw (Excalidraw for Mac) *completed · Application · Feb 2025 -- Mar 2025* - A native macOS and tablet application that replicates the functionality of Excalidraw for offline use. Stack: Tauri, Rust, Next.js Source: https://github.com/Himasnhu-AT/draw ## Syntaxia (learncode) *paused · Application · Apr 2025* - An interactive desktop app for learning to code through hands-on practice and AI-driven guidance. Stack: Tauri, Rust, Next.js, Ollama, Gemini API Source: https://github.com/Himasnhu-AT/learncode ## dbquery *completed · Application · July 2025* - An AI agent that translates natural language queries into SQL commands to interact with databases. Stack: Tauri, Rust, Next.js, Ollama, Gemini API Source: https://github.com/Himasnhu-AT/dbquery ## codehaven.org *in-progress · Application · Nov 2024 -- Dec 2024* - A minimalistic, memory-friendly IDE built with Tauri, inspired by Neovim, as a GUI-based alternative to VSCode. Stack: Tauri, Rust, Next.js, Ollama Source: https://github.com/Himasnhu-AT/codehaven.org --- Source: https://himanshuat.com/projects ======================================================================== # Skills ## Languages - **TypeScript** — Daily driver for frontend, Next.js apps, CLI tools, and most production services. - **Python** — Go-to for ML, data pipelines, scraping, and quick experiments. - **Rust** — Reach for it when latency and memory matter — inference platforms, CLIs, file tools. - **Swift** — Native iOS/macOS work — SwiftUI for clean app surfaces. - **Dart** — Cross-platform mobile builds via Flutter. - **C/C++** — Systems-level work, OS experiments, performance-critical bits. - **Go** — Backend services and concurrent network tooling. - **Assembly x86** — Bootloaders and kernel work in indieOS. ## Frontend - **React** — Default UI library for every web product I ship. - **Next.js** — Full-stack framework of choice — App Router, RSC, edge-ready. - **TailwindCSS** — Styling layer for every Next.js / React surface I build. - **SwiftUI** — Declarative UI for native macOS and iOS apps. - **Flutter** — Cross-platform mobile when Swift isn't the right tool. - **Tauri** — Lightweight Rust-backed desktop shell — used for RustyExplorer, draw, dbquery, codehaven. ## Backend - **Nest.js** — Structured Node.js backend framework — used at scale for SynthAiLabs services. - **Flask** — Lightweight Python web framework for quick AI inference endpoints. - **http (rust)** — Low-level HTTP primitives in Rust for custom servers. - **Axum** — Async Rust web framework powering the AI Inference Platform. - **Tokio (rust)** — Async runtime underpinning every Rust service I write. - **Node.js** — Backend runtime for TypeScript services, CLIs, and dev tooling. - **FastAPI** — Async Python framework for ML serving and rapid API prototyping. - **Express** — Minimal Node.js framework — used in MockServer's mock-engine. - **Prisma** — Type-safe ORM for PostgreSQL-backed Next.js apps. ## Database - **PostgreSQL** — Primary relational store for production apps. - **MongoDB** — Document store for flexible-schema workloads. - **TimescaleDB** — Time-series Postgres extension for metrics and event data. - **Milvus** — Vector database for high-recall semantic search at scale. - **Redis** — Cache, message queue, and L2 layer for the AI Inference Platform. - **MySQL** — Used in legacy/contract setups where Postgres isn't the default. - **pgvector** — Embedding storage and similarity search inside Postgres. - **OpenSearch** — Powered low-latency profile retrieval for the WhatsApp AI agents at Tejas AI. ## DevOps & Cloud - **Docker** — Default packaging for every backend service I deploy. - **Kubernetes** — Container orchestration for horizontally scaled inference workloads. - **AWS** — Primary cloud — EC2, S3, RDS, Lambda for production deploys. - **Azure** — Used selectively for client engagements requiring Microsoft stack. - **GCP** — BigQuery, Cloud Run, and Vertex AI when the workload fits. - **ShaktiCloud** — India-hosted GPU cloud — used at AuraX for diffusion inference. - **CI/CD** — Build pipelines, automated testing, and zero-touch deploys. - **GitHub Actions** — Workflow engine of choice — covers every repo I ship. - **Prometheus** — Metrics scraping for the AI Inference Platform observability stack. - **Grafana** — Dashboards on top of Prometheus for latency, cache, and model health. ## AI/ML/QM - **PyTorch** — Core deep-learning framework for VTO research and custom models. - **TensorFlow** — Used selectively where the ecosystem fits (Keras, TFLite). - **scikit-learn** — Classical ML for baselines and feature engineering. - **diffusers** — HuggingFace library used heavily for Flux-based VTO pipelines. - **CUDA** — Custom kernels and inference tuning on NVIDIA GPUs. - **Unsloth** — Memory-efficient fine-tuning library — powers TuneSLM. - **Qiskit** — IBM's quantum SDK — used for entanglement and QFT experiments. - **OpenQASM** — Quantum assembly for circuit-level optimization in QM research. - **LLM** — General large language model stack — OpenAI, Anthropic, Gemini, local Ollama. - **LoRA** — Low-rank adapters — central to AuraX's C-AAC composition algorithm. - **Ollama** — Local LLM runtime — used in localai, codehaven, and dbquery. - **Gemini API** — Google's LLM API — used in dbquery and Syntaxia. - **OpenAPI** — Spec-driven backend tooling — powers OpenCrow's server-side tool layer. - **Mem0** — Persistent conversational memory layer built for the Tejas AI WhatsApp agents. --- Source: https://himanshuat.com/skills ======================================================================== # Changelog ## Started a systems measurement study on mmap versus read() *2026-04-01 · research* Re-measuring the CIDR'22 result that mmap underperforms read() for large scans, on 16 KiB-page Apple Silicon, which that literature does not cover. Still in progress: bare-metal Linux confirmation is outstanding. Tags: Systems, Performance, Rust, Research Link: [object Object] ## Released MockServer v1.3.0 *2025-12-14 · project* Released version 1.3.0 of MockServer. Tags: npm package, mock server, release Link: [object Object] ## Wound down AuraX *2025-12-01 · career* Closed AuraX after a year. The research held up, reaching FID 18.5 and SSIM 0.85 on complex global garments, and we had signed deals with Aditya Birla Fashion Group and Mensa Brands labels. Larger players entered the space and the runway ran short. Tags: AuraX, Startup ## Released MockServer *2025-11-22 · project* Generate realistic mock data, simulate network chaos, and build robust UIs with a single CLI command to run backend. Tags: npm package, mock server Link: [object Object] ## Released Fintracker *2025-11-15 · project* Published my first Flutter application to the Play Store. Fintracker is a privacy-first expense manager that keeps all data local. Built with Flutter and Hive. Tags: Flutter, Mobile, Release Link: [object Object] ## Co-founded AuraX, joined as CTO *2024-12-01 · career* Co-founded AuraX, a fashion-tech startup incubated at IIIT-Hyderabad's CIE, leading research on diffusion models for virtual try-on and building the GPU inference infrastructure behind it. Tags: AuraX, Diffusion Models, CTO --- Source: https://himanshuat.com/changelog ======================================================================== # Writing 32 posts, newest first. Append `.md` to any post URL for its Markdown source. - **The backtest is the experiment, and most of them aren't controlled** (August 12, 2026) — Self-taught quantitative research primarily fails on statistical biases and uncorrected multiple testing rather than mathematical complexity. Front-loading the pitfall literature dictates building reproducibility infrastructure before writing strategy code. https://himanshuat.com/blogs/the-backtest-is-the-experiment.md - **554 million ticks and the wrong reason mmap loses** (August 12, 2026) — Industry lore states that memory mapping is inefficient for large sequential database scans. On Apple Silicon, empirical profiling shows mmap is faster, slower, and faster again based on working-set ratios relative to physical RAM, governed by kernel fault-population policies rather than base page size. https://himanshuat.com/blogs/the-fault-policy-not-the-page-size.md - **LLM as judge, what the numbers actually say** (June 23, 2026) — I let an LLM judge gate whether changes shipped because public benchmarks cited 80% human agreement. Six weeks in, an apparent performance improvement turned out to be judge variance across reruns. This is what that 80% figure measures and why chance-corrected agreement deflates headline metrics. https://himanshuat.com/blogs/llm-as-judge-what-the-numbers-actually-say.md - **Loop Engineering** (June 02, 2026) — Agents fail in production not because models lack reasoning capacity, but because unmonitored variance causes identical tasks to succeed once and fail on subsequent runs. Here is how to engineer verification ladders, context budgets, and deterministic checklists. https://himanshuat.com/blogs/loop-engineering.md - **Graph Agents and When the Graph Earns Its Keep** (May 12, 2026) — A graph framework introduces an additional control flow abstraction. Here is when cyclic state machines, checkpoint durability, and human approval justify graph infrastructure over plain loops. https://himanshuat.com/blogs/graph-agents-when-the-graph-earns-its-keep.md - **When the documents disagree** (March 17, 2026) — An answer cited a specification section by number and was wrong, because an addendum had already revised that clause and the vector index had no model of temporal precedence. A technical account of hierarchical document routing, supersession graphs, and NLI verification. https://himanshuat.com/blogs/when-the-documents-disagree.md - **Database Agents That Know When They're Wrong** (January 20, 2026) — A database agent that returns a clean, confident, incorrect result set is a liability. Here is how to build AST validation guards, database-level security boundaries, and ambiguity classifiers to catch silent query errors. https://himanshuat.com/blogs/db-agents-that-know-when-theyre-wrong.md - **Production-Grade RAG: A Blueprint for Scalable, Real-Time Architecture** (November 27, 2025) — A RAG system fails quietly when it answers from documents that were modified or deleted upstream. Real-time accuracy requires event-driven change data capture, explicit vector index configuration, tiered semantic caching, and telemetry that monitors silent retrieval drift. https://himanshuat.com/blogs/production-grade-rag-architecture-blueprint.md - **The Self-Correcting RAG: Implementing Agentic and Recursive Retrieval Loops** (November 18, 2025) — Static RAG retrieves once and assumes top-k results contain complete context. Recursive retrieval loops inspect accumulated context, identify missing facts, synthesize targeted follow-up queries, and terminate on explicit criteria or iteration bounds. https://himanshuat.com/blogs/self-correcting-rag-agentic-recursive-retrieval.md - **We beat the benchmarks and ran out of runway** (September 25, 2025) — Flux-VTON+ scores 0.85 SSIM on global traditional garments where SDXL manages 0.45, and we have signed deals with real apparel groups. We're winding the company down anyway. This is what I think the research was worth, written while both of those things are true at once. https://himanshuat.com/blogs/we-beat-the-benchmarks-and-ran-out-of-runway.md - **Three clouds, one inference path** (September 17, 2025) — Inference executed across three cloud providers because client data residency required in-region processing, interactive research needed persistent state, and catalog rendering arrived in sporadic batch spikes. A technical review of the multi-cloud placement router, artifact provenance, and operational tradeoffs. https://himanshuat.com/blogs/three-clouds-one-inference-path.md - **From a ComfyUI graph to an API a brand's team can call** (September 09, 2025) — Our try-on pipeline lived as a ComfyUI graph with implicit state, hand-picked seeds, and custom nodes tracking whatever was on main that week. Turning it into something another company's engineers could call meant writing down which code ran, which seed, when the work happens, and what we refuse. https://himanshuat.com/blogs/from-comfyui-graph-to-an-api.md - **Fitting a saree into 24GB: the context window trick** (September 02, 2025) — A saree needs pixel density to hold its pleats, and a full-frame high-resolution pass doesn't fit on a 24GB card. The fix was to stop treating the frame as the unit of inference and hand the model only the region that matters. https://himanshuat.com/blogs/fitting-a-saree-into-24gb.md - **FID was the wrong metric for the problem we had** (August 26, 2025) — Aggregate FID went 22.1 to 18.5 and I nearly read that as months of adapter work buying four points. Half the eval set was already close to solved and it dragged the average toward nothing. Splitting by garment category was the change that made the result visible. https://himanshuat.com/blogs/fid-was-the-wrong-metric.md - **Composing adapters that disagree with each other** (August 05, 2025) — Two LoRA experts merged with two scalars shipped fine. A whole library of them for demographics, poses, lighting, and backgrounds did not, because adapters conflict per layer and one blend weight per adapter cannot express that. https://himanshuat.com/blogs/conflict-aware-adapter-composition.md - **Production Agent Architecture: Memory, Approval Gates, and Durable Checkpointing** (August 05, 2025) — What actually breaks when you put a LangGraph agent in production: memory interfaces for vector stores, human-in-the-loop approval routing for critical operations, TypeScript orchestrators with transactional state persistence, and the deployment considerations that don't appear in tutorials. https://himanshuat.com/blogs/production-grade-agent-architecture-memory-human-in-the-loop.md - **Architecting Multi-Agent Teams with LangGraph: Supervisor Patterns, Worker Isolation, and Sub-Graph Composition** (July 22, 2025) — Two patterns for multi-agent coordination: a Python supervisor that routes between specialist worker nodes via explicit state evaluation, and a typed graph engine that composes child sub-graphs with isolated state boundaries. When each earns its complexity, and the failure modes that make both patterns break. https://himanshuat.com/blogs/architecting-multi-agent-teams-with-langgraph.md - **What happens when you merge two LoRAs** (July 15, 2025) — Two adapters that each pass on their own can make each other worse the moment you sum them into the base weights. We shipped a hand-tuned coefficient pair because we had to ship something, and months later I still don't have a principled answer for how those coefficients should be chosen. https://himanshuat.com/blogs/what-happens-when-you-merge-two-loras.md - **Serverless GPUs for bursty training** (June 17, 2025) — Our fine-tuning load arrived in bursts: intensive compute sprints followed by quiet intervals. Rented static GPU instances handle this pattern poorly. An architectural breakdown of splitting workloads between dedicated EC2 nodes and serverless Modal GPU workers. https://himanshuat.com/blogs/serverless-gpus-for-bursty-training.md - **RAG Foundations: From First Query to a Retriever You Can Actually Trust** (June 12, 2025) — Building a RAG system is straightforward. Building one whose retrieval you can measure, debug, and improve is not. This post covers chunking strategies with real Hit Rate and MRR comparisons, embedding model selection against concrete constraints, and the diagnostic checks that matter before you touch the generation side. https://himanshuat.com/blogs/rag-foundations.md - **One LoRA per failure mode** (May 20, 2025) — Put a hand on a hip and our try-on pipeline painted the garment straight over the fingers. The obvious fix, one bigger dataset covering every hard case at once, quietly made the draping worse. Splitting the finetune into two adapters, one per named failure mode, is what actually worked. https://himanshuat.com/blogs/one-lora-per-failure-mode.md - **Generic aesthetic scorers hate e-commerce photography** (April 29, 2025) — We pointed an open aesthetic scorer at a batch of generated product shots and it ranked the moody, low-key ones highest. That's a defensible opinion about photographs and the wrong answer for a catalogue. This is the scorer we built instead, and the parts of it I can't put a number on. https://himanshuat.com/blogs/teaching-a-model-what-a-catalogue-looks-like.md - **LoRA finetuning: the hyperparameters settled in a week, the data never did** (April 08, 2025) — An early draping adapter kept rendering identical flat planes across shoulder folds, regardless of prompt. This is memorization rather than learned geometric priors, traced back to pinned studio photos. Hyperparameters stabilize quickly in Low-Rank Adaptation; dataset curation and caption isolation determine generalization. https://himanshuat.com/blogs/lora-finetuning-what-moved-the-needle.md - **Two streams: why one diffusion pass couldn't do virtual try-on** (March 18, 2025) — Masking a torso and inpainting a garment demos well in an afternoon. Getting back the exact garment, with its weave and its print placement intact, is a different problem. One conditioning stream could not hold both the body and the reference, so Flux-VTON+ runs two. https://himanshuat.com/blogs/flux-fill-and-redux-two-streams.md - **What 5,000 images taught me about curation** (February 25, 2025) — More data made the model worse in a way no automatic metric could see. The seed adapters behind AuraX-V1 ended up trained on 5,000 images a person had looked at one by one. Here is what the filter caught, and what it kept missing. https://himanshuat.com/blogs/what-5000-images-taught-me-about-curation.md - **ComfyUI as a research lab** (February 04, 2025) — A custom node pack updated, a mask node changed its default on inverted masks, and our outputs moved with nothing changed on our side. Running a try-on diffusion pipeline as a node graph made every experiment a commit, and made very clear where a graph stops being a program. https://himanshuat.com/blogs/comfyui-as-a-research-lab.md - **Virtual try-on is not image generation** (January 14, 2025) — A text-to-image model invents a plausible shirt. Try-on has to reproduce the exact shirt the customer is already looking at, on a person whose face has to survive the process intact. Getting that wrong on a saree is the failure that started all of this. https://himanshuat.com/blogs/virtual-try-on-is-not-image-generation.md - **Transformer Architectures Compared: BERT, GPT, Mamba, and Mixture of Experts** (May 31, 2024) — A single triangular mask separates BERT from GPT. That structural choice determines pre-training objectives, task alignment, and inference mechanics. This post covers the masking math, MLM vs CLM training objectives, PyTorch implementations of both, then the architectures pushing beyond attention: SSMs, Mamba, and sparse MoE routing. https://himanshuat.com/blogs/understanding-transformers-architectures-bert-gpt.md - **Building a Transformer from Scratch: Attention, Architecture, and Memory** (May 10, 2024) — A ground-up implementation of the Transformer: scaled dot-product attention in NumPy and PyTorch, the full decoder block with RoPE, RMSNorm, and causal masking, then the memory optimizations that make large models practical: FlashAttention, KV caching, GQA, and LoRA. https://himanshuat.com/blogs/understanding-transformers-implementation.md - **Part 3: Scaling LangGraph: State Persistence, Checkpointing, and Parallelism** (May 03, 2024) — Long-running multi-agent workflows require durable state persistence and concurrency management. Here is how to configure SQLite and Redis checkpointers alongside Pregel super-step parallel branching in LangGraph. https://himanshuat.com/blogs/scaling-langgraph-persistence-and-parallelism.md - **StyleX Architecture: Compile-Time CSS Extraction, Deterministic Resolution, and Atomic Bundling** (December 16, 2023) — An architectural breakdown of StyleX, examining how compile-time Babel AST transforms eliminate CSS-in-JS runtime overhead, guarantee deterministic class merges, and generate atomic stylesheets. https://himanshuat.com/blogs/stylex-new-frontend-styling-framework.md - **Modern Neovim from Scratch: Lua Modular Architecture, Lazy.nvim, LSP, and Treesitter** (September 12, 2023) — A practical guide to configuring a fast, modular Neovim development environment using Lua, Lazy.nvim, native LSP with Mason, nvim-cmp, and Treesitter. https://himanshuat.com/blogs/my-nvim-setup.md --- Source: https://himanshuat.com/blogs ======================================================================== # Books 35 books with notes. - **80/20 Your Life!** — Damon Zahariades (2018): Damon Zahariades applies the Pareto principle to daily routines, arguing that focusing on a vital few priorities and dropping the rest is what genuinely buys back time. https://himanshuat.com/books/80-20-your-life.md - **How Doctors Think** — Jerome Groopman (2007): Jerome Groopman, a physician at Harvard Medical School, investigates the cognitive errors behind medical misdiagnosis (anchoring, attribution bias, premature closure), and argues for restoring genuine conversation between doctor and patient as the primary diagnostic tool. https://himanshuat.com/books/how-doctors-think.md - **The Stranger** — Albert Camus (1942): A man buries his mother, kills an Arab on a beach, and faces a court that wants him to lie about both. Camus's clearest account of the absurd. https://himanshuat.com/books/the-stranger.md - **The Body Keeps the Score** — Bessel van der Kolk (2014): A psychiatrist's three-decade investigation into how trauma reshapes the body and brain, and the surprisingly effective treatments that follow once we take that seriously. https://himanshuat.com/books/the-body-keeps-the-score.md - **The Death of Ivan Ilyich** — Leo Tolstoy (1886): A nineteenth-century judge realizes across a hundred pages that his respectable life was merely a costume, and that nobody around him will acknowledge the truth. https://himanshuat.com/books/the-death-of-ivan-ilych.md - **Factfulness** — Hans Rosling (2018): Hans Rosling documents the ten systematic cognitive instincts that cause educated people to hold a more pessimistic picture of the world than the data supports. His remedy, factfulness, replaces dramatic instinct with a set of rules. https://himanshuat.com/books/factfulness.md - **The Celestine Prophecy** — James Redfield (1993): A 1990s bestseller that disguised a self-help manual inside a Peruvian adventure novel, prompting mainstream readers to reconsider the meaning of everyday coincidences. https://himanshuat.com/books/the-celestine-prophecy.md - **Moonwalking with Einstein** — Joshua Foer (2011): Joshua Foer, a journalist covering the US Memory Championship, ends up training for and winning it, using his experience to investigate the history of memory techniques, the science of expertise, and why humans outsourced their memories to external devices. https://himanshuat.com/books/moonwalking-with-einstein.md - **No Exit** — Jean-Paul Sartre (1944): Three people arrive in a Second Empire drawing room furnished as a hell with no mirrors, no windows, and no sleep, and gradually discover they are each other's torturers. https://himanshuat.com/books/no-exit.md - **The Sense of an Ending** — Julian Barnes (2011): Tony Webster, retired and content, receives a bequest from his first girlfriend's mother that forces him to revisit a friendship and a letter from his youth, discovering that his memory has been systematically editing the record in his favor for forty years. https://himanshuat.com/books/the-sense-of-an-ending.md - **The Analects** — Confucius (-479): A collection of Confucius's conversations with his students and contemporaries, compiled after his death, addressing virtue, political legitimacy, ritual propriety, and the relationship between education, character, and governance. https://himanshuat.com/books/the-analects.md - **Invisible Man** — Ralph Ellison (1952): An unnamed Black narrator recounts his journey from a Southern college through Harlem's political organizations to a basement where he lives surrounded by 1,369 light bulbs, tracing how American institutions, white liberal and Black nationalist alike, refused to see him as a person. https://himanshuat.com/books/invisible-man.md - **Flowers for Algernon** — Daniel Keyes (1966): Told through the progress reports of Charlie Gordon, a 32-year-old man with an intellectual disability who undergoes experimental surgery to increase his intelligence, watches it peak above genius level, and then watches it decline again. https://himanshuat.com/books/flowers-for-algernon.md - **Prometheus Rising** — Robert Anton Wilson (1983): A practical manual for examining and reprogramming your own nervous system, organized around Timothy Leary's eight-circuit model of consciousness, with exercises at the end of each chapter designed to make the reader question their default reality tunnel. https://himanshuat.com/books/prometheus-rising.md - **The Republic** — Plato (-375): Plato's central dialogue, in which Socrates constructs an ideal city in argument to answer the question of why a just person is happier than an unjust one, producing along the way the Theory of Forms, the Allegory of the Cave, and a political philosophy that has been debated and condemned for 2,400 years. https://himanshuat.com/books/the-republic.md - **The Trial** — Franz Kafka (1925): Josef K., a bank clerk, is arrested one morning by an opaque authority for a crime no one will name, and spends the rest of the novel attempting to navigate a court system that is entirely inaccessible to anyone who tries to engage it directly. https://himanshuat.com/books/the-trial.md - **Life Is Elsewhere** — Milan Kundera (1973): A satirical novel about Jaromil, a Czech poet of revolutionary ardor whose mother has designed his entire emotional life around his artistic destiny, traced from his conception through his death at twenty. https://himanshuat.com/books/life-is-elsewhere.md - **Hunger** — Knut Hamsun (1890): A nameless narrator wanders Christiania in a state of near-starvation, refusing charity and employment out of a pride he cannot fully explain, producing one of the first sustained literary portraits of consciousness destabilized by deprivation. https://himanshuat.com/books/hunger.md - **The Brothers Karamazov** — Fyodor Dostoevsky (1880): Dostoevsky's final novel, in which three brothers of radically different temperament are implicated in their father's murder, and which he built to carry his arguments for and against the existence of God. https://himanshuat.com/books/the-brothers-karamazov.md - **The Magic Mountain** — Thomas Mann (1924): Hans Castorp arrives at a Swiss tuberculosis sanatorium for a three-week visit to see his cousin and stays for seven years, drawn into the suspended time and intellectual ferment of the mountain while Europe descends toward the war below. https://himanshuat.com/books/the-magic-mountain.md - **The Age of Reason** — Jean-Paul Sartre (1945): The first volume of Sartre's Roads to Freedom trilogy, following philosophy lecturer Mathieu Delarue through forty-eight hours in 1938 Paris as he tries to raise money for an abortion and confronts his inability to commit to anything without feeling it constrains his freedom. https://himanshuat.com/books/the-age-of-reason.md - **Strange Case of Dr Jekyll and Mr Hyde** — Robert Louis Stevenson (1886): A Victorian horror novella in which a scientist chemically separates his respectable identity from his suppressed impulses, only to find the second self growing stronger until it no longer needs the chemical to emerge. https://himanshuat.com/books/jekyll-and-hyde.md - **We** — Yevgeny Zamyatin (1924): The Soviet dystopia that Orwell read before writing 1984: a glass-walled One State where every hour is scheduled, citizens are addressed by number rather than name, and D-503, a spacecraft engineer, begins to develop an interior life after meeting a woman from the underground. https://himanshuat.com/books/we.md - **The Handmaid's Tale** — Margaret Atwood (1985): Margaret Atwood's portrait of a near-future American theocracy assembled entirely from historical precedents, told through Offred, a fertile woman assigned as a reproductive vessel to a high-ranking commander. https://himanshuat.com/books/the-handmaids-tale.md - **A Clockwork Orange** — Anthony Burgess (1962): Set in near-future Britain, Burgess follows Alex, a teenage gang leader who speaks in Nadsat slang and commits crimes for sport, until the state captures and conditions him into passivity. The novel's moral debate is whether good behavior is meaningful if it is forced rather than chosen. https://himanshuat.com/books/a-clockwork-orange.md - **Animal Farm** — George Orwell (1945): A fable in which the animals of Manor Farm overthrow their human farmer and build a society of animal equality, which deteriorates steadily into a pig dictatorship indistinguishable from what it replaced. https://himanshuat.com/books/animal-farm.md - **Anthem** — Ayn Rand (1938): A short dystopia in which the first-person singular pronoun has been abolished and a young man rediscovers the concept of the individual self after finding an electric light in a forbidden underground tunnel. https://himanshuat.com/books/anthem.md - **Eat, Pray, Love** — Elizabeth Gilbert (2006): A memoir of the year Elizabeth Gilbert spent in Italy, India, and Bali after her marriage collapsed: eating well, learning meditation, and falling in love again. https://himanshuat.com/books/eat-pray-love.md - **Essays in Love** — Alain de Botton (1993): A philosophical novel about falling in love and its aftermath, structured as a series of analytical essays embedded in a first-person romantic narrative. https://himanshuat.com/books/essays-in-love.md - **Man's Search for Meaning** — Viktor Frankl (1946): Viktor Frankl's account of his years in Nazi concentration camps, followed by logotherapy, the framework he derived from the observation that prisoners who found meaning survived longer than those who did not. https://himanshuat.com/books/mans-search-for-meaning.md - **The Screwtape Letters** — C.S. Lewis (1942): A senior demon named Screwtape writes to his nephew Wormwood with detailed tactical advice on how to corrupt a single human soul, producing an inverted moral handbook in which every recommended sin illuminates the corresponding virtue by its absence. https://himanshuat.com/books/the-screwtape-letters.md - **Meditations** — Marcus Aurelius (180): Private journal entries from a Roman emperor who was also a Stoic philosopher, written as self-instruction rather than publication, in which Marcus Aurelius reminds himself daily of what he actually believes about duty, impermanence, and the control of attention. https://himanshuat.com/books/meditations.md - **On the Shortness of Life** — Seneca (49): A letter from a Roman Stoic to his father-in-law, arguing that life is not short, only wasted on activities that don't constitute living, and that most people reach old age without ever having fully inhabited any of their time. https://himanshuat.com/books/on-the-shortness-of-life.md - **Wind, Sand and Stars** — Antoine de Saint-Exupéry (1939): A lyrical memoir of flying mail routes over the Sahara and Andes in the 1920s and 30s, near-fatal crashes in the desert, and what those extremities revealed about solidarity, craft, and human dignity. https://himanshuat.com/books/wind-sand-and-stars.md - **Zorba the Greek** — Nikos Kazantzakis (1946): A bookish narrator meets an old miner named Zorba on a boat to Crete and spends a year being slowly dismantled by a man who treats life as something to be consumed entirely rather than observed from a safe distance. https://himanshuat.com/books/zorba-the-greek.md --- Source: https://himanshuat.com/books ======================================================================== # Work with me I build the system that does the work nobody should still be doing by hand, and hand over the drawing, the credentials, and the thing already running. Two of them are described below. Book fifteen minutes: https://cal.com/himanshuat/makecustom12?duration=15 ## What changes after ### Your clients stop waiting for someone to be free Every enquiry answered in seconds, on WhatsApp or email, with that client's own filing status and outstanding documents already in the reply. Strangers get qualified and scored instead of sitting in an inbox. ### Deadlines stop depending on somebody remembering Filing dates chase themselves at seven days, three days, one day, and then daily once they are late. Documents and unpaid invoices chase themselves too. A partner is paged only when something is genuinely stuck. ### Nobody needs to know SQL to know the numbers Ask in plain English, get the answer, the chart, and the query itself. Anything that writes stops at an approval gate first, so the person asking can never break what they are asking about. ### The bill for running all of it stops being a surprise Cheap questions stop reaching an expensive model, paraphrased repeats never reach the API at all, and there is enough instrumentation to show the spend actually moved rather than asserting it. ## CA Firm Automation Suite *Sold to a chartered accountancy practice in Hyderabad* A practice's whole client-facing surface — questions, documents, deadlines, invoices — running as one workflow instead of as somebody's afternoon. 152 nodes · 238 links · 9 entry points · 5 use cases - **Reaches clients on** — WhatsApp · Email · Telegram - **Drafts with** — Groq llama-3.3-70b - **Store of record** — Google Sheets - **Failure design** — every fallible node into one log chain ### What it does while nobody is watching - **09:00 — Compliance deadlines.** GST, ITR, TDS and ROC dates checked against every client. Reminders go out at 7, 3 and 1 days; tone escalates on its own once a date passes. More than three days overdue pages the partner. - **09:30 — Document follow-up.** Anything still marked pending for three days gets asked for again, worded a little more firmly each time. The third ask copies the partner in. - **10:00 — Lead follow-up.** Open leads get nudged on their due date and rescheduled three days out. After four attempts with no movement they close themselves as cold. - **10:30 — Invoice follow-up.** Unpaid invoices step through upcoming, due, and progressively later reminders. Past fifteen days it flags as escalated and pages the partner. The drawing is generated from the workflow file itself — every node sits at its real position, every line is a real connection. It cannot drift from what was delivered. ## DBQuery *Natural language in, checked SQL out* A database anyone in the room can interrogate, and nobody in the room can damage. 01. **Ask** — A question in plain English. No schema knowledge, no query builder. 02. **Generate** — The agent reads the live schema, reasons, and writes the SQL — shown in full, not hidden behind a summary. 03. **Approve** — Anything that could write stops here and states its impact in a sentence. Nothing runs until a human says yes. 04. **Read** — The answer in words, the rows in a table, and the shape as a chart — with the timing and token cost printed underneath. - **Loop** — ReAct — thought, action, observation, summary - **Transport** — server-sent events, streamed as it thinks - **Credentials** — AES-256-CBC at rest, JWT and role-gated - **Ships as** — web app and Tauri desktop ## Revision history ### Inference cost, in Rust — Open source An open-source inference platform that classifies query complexity by embedding and routes on it, with an L1 Moka and L2 Redis semantic cache and a full Prometheus and Grafana stack behind it. The code is public. ### Search and agents in production — Tejas AI (YC W25) Hybrid keyword and vector retrieval on OpenSearch over a corpus indexed from around sixty ATS platforms, and the WhatsApp agent in front of it — a router with specialist sub-agents and memory that survives across sessions. ### Research taken to production — AuraX Co-founded AuraX as CTO. Diffusion-based virtual try-on incubated at IIIT-Hyderabad's CIE, carried from research through to production inference and into commercial deals with Aditya Birla Fashion Group and brands under Mensa Brands. ### Teaching at scale — SynthAiLabs Founded SynthAiLabs and OpenEdu, which reached roughly eleven thousand registered students, and ran the OpenCodeDevelopers society for eight hundred more. ## Contact - Book fifteen minutes: https://cal.com/himanshuat/makecustom12?duration=15 - Email: hyattherate2005@gmail.com - Takes on contract, retained and research work; also open to full-time engineering roles. --- Source: https://himanshuat.com/hire ======================================================================== # Knowledge graph An interactive force-directed graph of everything on this site — 111 nodes and 250 edges connecting projects, research, roles and skills. The rendered page is a canvas; this is the same structure as text. ## Root (1) - **~/himanshu** (root): Knowledge graph of everything I've built — projects, research, companies, and the skills threading them together. Click any node to inspect it. ## Projects (33) - **#projects** (domain): Personal and OSS projects, grouped by category. - **#ai** (hub): Project category: ai. - **CCProxy** (project): (Mar 2026 -- ongoing) A universal proxy that lets Claude Code talk to any AI provider: OpenAI, Gemini, or local Ollama models. — https://ccproxy.himanshuat.com/ - **OpenCrow** (project): (Jan 2026 -- ongoing) Open-source AI agent platform for embedding intelligent customer support assistants into web applications. Features an embeddable React chat widget with dual-layer tooling: server-side tools via OpenAPI specs and client-side tools executed in the browser. Dynamic tool merging lets the LLM pick between server and client actions per request. Includes a management dashboard for agents, products, and integrations. Inspired by usecrow.ai. — https://opencrow.himanshuat.com - **#infra** (hub): Project category: infra. - **AI Inference Platform** (project): (Dec 2025 -- Jan 2026) High-performance AI inference platform with intelligent model routing — classifies query complexity via embeddings and heuristics, routing simple queries to fast/cheap models and complex ones to GPT-4 class models. Multi-level caching: L1 in-memory (Moka) + L2 distributed (Redis) with semantic similarity matching via cosine similarity on embeddings. Full observability stack with Prometheus metrics and pre-configured Grafana dashboards for inference latency, cache hit rates, and model health. Fully async Rust architecture with horizontal scaling via Docker and Kubernetes. — https://infer.himanshuat.com - **TuneSLM** (project): (Jan 2026 -- ongoing) Drop your dataset and answer a few questions — get an ipynb notebook ready for Google Colab to fine-tune your SLM of choice, powered by Unsloth. Future: AI-driven dataset analysis to automatically choose optimal training strategies and hyperparameters. — https://tuneslm.himanshuat.com - **#devtools** (hub): Project category: devtools. - **MockServer** (project): (Nov 2025 -- ongoing) A CLI tool that spins up a mock backend so you can build the frontend before the real one exists, with a GUI to configure it. Inject latency and errors on demand to test loading and error states in the frontend. — https://mockserver.himanshuat.com - **FastSearch** (project): (April 2024 -- June 2024) A privacy-focused search engine written in Rust, as an alternative to Google Search. Ranks results with TF-IDF over an inverted index built for fast lookups. — https://github.com/Himasnhu-AT/FastSearch - **FallenAngel-SE/CY** (project): (2024 -- Present [Paused because of funds issue]) A suite of AI tools for development and security, built as a coding assistant. Runs on the LLaMA 3.1 70B model. Includes interactive learning features. — https://www.linkedin.com/posts/himanshu-at_testing-out-new-product-revealed-soon-activity-7232783015360221185-t-Dr - **Dynamo-Prisma** (project): (2024) Generates Prisma models from a JSON schema, so you don't hand-write them. — https://github.com/techsavvyash/dynamo-prisma - **#learning** (hub): Project category: learning. - **ScratchAI-Algo** (project): (April 2024 -- May 2024) AI/ML algorithms implemented from scratch, each with a written explanation. Covers KNN, logistic regression, K-means, decision trees, SVM, and more; 10+ so far. Built for the SynthAiLabs education work, open to contributions. — https://github.com/Himasnhu-AT/ScratchML-Algo - **QM - Research** (project): (April 2024 -- June 2024) A research repo on using quantum computing against encryption like RSA, SHA-256, and SHA-128. Built quantum circuits in Qiskit: Quantum Fourier Transform, GHZ state, and commutation-relation experiments. Wrote OpenQASM for quantum registers and entanglement experiments. — https://github.com/Himasnhu-AT/QM-res - **#apps** (hub): Project category: apps. - **RustyExplorer** (project): (Oct 2023 -- Dec 2023) A cross-platform file explorer built with Rust and TypeScript on Tauri. Searches files noticeably faster than Windows Explorer, with cached directory listings to keep navigation snappy. — https://github.com/Himasnhu-AT/rusty-explorer - **Linux Mini-Firewall** (project): (July 2023) A mini firewall application that allows users to add, delete, and print rules based on criteria like source IP, destination IP, and ports. — https://github.com/Himasnhu-AT/Linux_mini-Firewall - **indieOS** (project): (May 2024) A demo operating system written from scratch for the x86 architecture. Includes a bootloader, shell, ports, and custom memory management. — https://github.com/Himasnhu-AT/indieOS - **Codeforces Scraper** (project): (June 2024) A scraper to extract problems from Codeforces HTML pages and return them in JSON format. — https://github.com/Himasnhu-AT/codeforce_scrapper - **Log Ingestor** (project): (Aug 2024 -- Sep 2024) A high-performance log ingestor that stores logs in a database or prints them to the terminal in a clean format. — https://github.com/Himasnhu-AT/LogInjestor-depreciated- - **ZenStore (In-memory DB)** (project): (Oct 2024) A minimalistic in-memory database inspired by Redis, built entirely in Rust. — https://github.com/Himasnhu-AT/ZenStore - **frm (Fast Delete)** (project): (Dec 2024) A high-performance, safe alternative to the `rm` command for fast file deletion, written in Rust. — https://github.com/Himasnhu-AT/frm - **roadmap.ai** (project): (Oct 2024 -- Nov 2024) An AI-powered tool that generates personalized learning roadmaps for various technical topics. — https://roadmap-ai-xi.vercel.app - **biv** (project): (Mar 2025) An experimental, high-performance binary data format designed as a faster alternative to JSON. — https://github.com/Himasnhu-AT/biv - **ruspy** (project): (Nov 2024 -- Mar 2025) A hobby programming language combining Rust's efficiency with Python's simplicity, built to learn compiler design. — https://github.com/Himasnhu-AT/ruspy - **PoW-Shield (DDoS Prevention)** (project): (Aug 2023) Prevents DDoS attacks using a Proof of Work system to validate requests before forwarding them to the server. — https://github.com/Himasnhu-AT/PoW-Shield - **git-history** (project): (Sep 2024) A utility to convert a repository's entire Git commit history into a structured JSON file. — https://github.com/Himasnhu-AT/git-history - **localai** (project): (Apr 2025) A desktop app to chat with and compare various local AI models deployed via Ollama, with an integrated web search tool. — https://github.com/Himasnhu-AT/localai - **draw (Excalidraw for Mac)** (project): (Feb 2025 -- Mar 2025) A native macOS and tablet application that replicates the functionality of Excalidraw for offline use. — https://github.com/Himasnhu-AT/draw - **Syntaxia (learncode)** (project): (Apr 2025) An interactive desktop app for learning to code through hands-on practice and AI-driven guidance. — https://github.com/Himasnhu-AT/learncode - **dbquery** (project): (July 2025) An AI agent that translates natural language queries into SQL commands to interact with databases. — https://github.com/Himasnhu-AT/dbquery - **codehaven.org** (project): (Nov 2024 -- Dec 2024) A minimalistic, memory-friendly IDE built with Tauri, inspired by Neovim, as a GUI-based alternative to VSCode. — https://github.com/Himasnhu-AT/codehaven.org ## Research (8) - **#research** (domain): Research output — VTO, MoE, quantum, transformers. - **It's the Fault Policy, Not the Page Size: Why mmap Loses to read() for Out-of-Core Scans on Apple Silicon** (research): Re-measuring mmap vs read() on Apple Silicon: a working-set crossover, attributed to OS fault-population policy rather than page size. — https://himanshuat.com/research/mmap-vs-read-apple-silicon - **Conflict-Aware Adapter Composition (C-AAC)** (research): Outperforms base Flux, Imagen 4, and ChatGPT in skin quality and aesthetic appeal. — https://himanshuat.com/research/c-aac - **Flux-VTON+: Hybrid Flux Inpainting and Multi-LoRA Expert Fusion** (research): State-of-the-art VTON pipeline synergizing Flux Fill and Redux with Multi-LoRA Fusion, achieving an 85% success rate on complex garments. — https://himanshuat.com/research/renu-virtual-try-on - **MoE: Brain-Inspired LLM Architecture** (research): A whitepaper advancing MoE as a computational analogue to the brain's functional specialization. — https://himanshuat.com/research/moe-brain - **High-Dimensional Lattice-Based Quantum Encryption (LWE)** (research): A PQC scheme using the LWE problem with detailed mathematical formulations. — https://himanshuat.com/research/lattice-based-quantum-encryption - **GPT-Neo: Transformer Implementation** (research): Implementing 'Attention is All You Need' from scratch. — https://himanshuat.com/research/gpt-neo - **Custom Neural Network for MNIST** (research): A custom-built Neural Net for the MNIST digit recognizer dataset. — https://himanshuat.com/research/custom-n-n-mnist ## Experience (7) - **#experience** (domain): Companies and roles across the timeline. - **Tejas AI** (company): Software Development Engineer (SDE) · Mar 2026 -- Aug 2026. Built Elip, an AI career coach on WhatsApp (with web, a Chrome extension, and a mobile app) that helps job seekers fix their resume, build a search profile, and get matched to jobs. The core was a FastAPI WhatsApp agent, a router model with specialist sub-agents, that parsed resumes and LinkedIn profiles, ran resume reviews, and returned ranked jobs, with cross-session memory across Redis, Postgres, and S3. Built the search layer over OpenSearch (hybrid keyword and vector search with Bedrock embeddings) and a fleet of Go services that index jobs from around 60 ATS and job boards. Built the Go backend behind the web and mobile apps, plus a Chrome extension that autofills multi-step job applications and tailors a resume to each posting. Stack: TypeScript, Go, Node.js, Nest.js, PostgreSQL, Redis, OpenSearch, Mem0, LLM, Docker, AWS. — https://himanshuat.com/experience - **AuraX** (company): AI Researcher & CTO · Dec 2024 -- Dec 2025. Co-founded AuraX, a fashion-tech startup funded by IIIT-Hyderabad's CIE incubator. Led research on diffusion models for virtual try-on (VTO), achieving quality that outperformed Google, Stable Diffusion, and leading Indian startups in benchmark comparisons. Signed commercial deals with Aditya Birla Fashion Group and multiple brands under Mensa Brands. Built the backend and deployment infrastructure on AWS and ShaktiCloud, and tuned inference latency for production. Wound the company down as bigger players entered the space and our runway ran short. Stack: Python, PyTorch, diffusers, LoRA, CUDA, LLM, FastAPI, PostgreSQL, Redis, Docker, AWS, ShaktiCloud, Next.js, Nest.js. — https://himanshuat.com/experience - **SynthAiLabs** (company): Founder · Dec 2023 -- Feb 2025. Researched Mixture-of-Experts (MoE) LLM architectures. Built learncode (interactive AI lessons) and eduAI (an agent that answers questions about course material). Grew techos.synthailabs to 11,000 registered students across India. Co-founded the OpenCodeDevelopers (OCD) society, which grew by word of mouth and helped 800+ students cut their CP/DSA ramp from six months to two. Stack: TypeScript, Next.js, Node.js, Express, MongoDB, PostgreSQL, Redis, Docker, CI/CD, React, TailwindCSS. — https://himanshuat.com/experience - **Tublian** (company): AI Dev · July 2024 -- Dec 2024. Set up Docker-based build and deployment workflows. Added code obfuscation to protect the source before public deployment. Contributed to TublianOS, working on performance and the user-facing experience. Completed a system-design masterclass that used Google's search architecture as its worked example. Open Source Collaborator | Project Mentor · Dec. 2023 -- April 2024. Contributed to several open-source projects and mentored 20+ newer developers. Helped build DevDocGenie, a RAG-based chatbot for searching documentation. Reverse-engineered features like Samsung's live call translation as build exercises. Stack: Python, LLM, Ollama, FastAPI, Docker, TypeScript, React, Node.js, GitHub Actions. — https://himanshuat.com/experience - **Stealth** (company): QM Researcher · March 2024 -- June 2024. [NDA], tested ML model deployment on quantum computers using qiskit Stack: Python, Qiskit, OpenQASM, C/C++. — https://himanshuat.com/experience - **Excelerate** (company): Project Manager · April 2023 -- Sept 2023. Led a team of six on a global education event with a $30,000 budget. Owned the project management docs, including the RACI matrix and risk register. Delivered on schedule. Stack: TypeScript, React, Next.js. — https://himanshuat.com/experience ## Skills (62) - **#skills** (domain): Languages, frameworks, infra, and AI/ML tools. - **#languages** (hub): 8 skills in Languages. - **TypeScript** (skill): Daily driver for frontend, Next.js apps, CLI tools, and most production services. - **Python** (skill): Go-to for ML, data pipelines, scraping, and quick experiments. - **Rust** (skill): Reach for it when latency and memory matter — inference platforms, CLIs, file tools. - **Swift** (skill): Native iOS/macOS work — SwiftUI for clean app surfaces. - **Dart** (skill): Cross-platform mobile builds via Flutter. - **C/C++** (skill): Systems-level work, OS experiments, performance-critical bits. - **Go** (skill): Backend services and concurrent network tooling. - **Assembly x86** (skill): Bootloaders and kernel work in indieOS. - **#frontend** (hub): 6 skills in Frontend. - **React** (skill): Default UI library for every web product I ship. - **Next.js** (skill): Full-stack framework of choice — App Router, RSC, edge-ready. - **TailwindCSS** (skill): Styling layer for every Next.js / React surface I build. - **SwiftUI** (skill): Declarative UI for native macOS and iOS apps. - **Flutter** (skill): Cross-platform mobile when Swift isn't the right tool. - **Tauri** (skill): Lightweight Rust-backed desktop shell — used for RustyExplorer, draw, dbquery, codehaven. - **#backend** (hub): 9 skills in Backend. - **Nest.js** (skill): Structured Node.js backend framework — used at scale for SynthAiLabs services. - **Flask** (skill): Lightweight Python web framework for quick AI inference endpoints. - **http (rust)** (skill): Low-level HTTP primitives in Rust for custom servers. - **Axum** (skill): Async Rust web framework powering the AI Inference Platform. - **Tokio (rust)** (skill): Async runtime underpinning every Rust service I write. - **Node.js** (skill): Backend runtime for TypeScript services, CLIs, and dev tooling. - **FastAPI** (skill): Async Python framework for ML serving and rapid API prototyping. - **Express** (skill): Minimal Node.js framework — used in MockServer's mock-engine. - **Prisma** (skill): Type-safe ORM for PostgreSQL-backed Next.js apps. - **#database** (hub): 8 skills in Database. - **PostgreSQL** (skill): Primary relational store for production apps. - **MongoDB** (skill): Document store for flexible-schema workloads. - **TimescaleDB** (skill): Time-series Postgres extension for metrics and event data. - **Milvus** (skill): Vector database for high-recall semantic search at scale. - **Redis** (skill): Cache, message queue, and L2 layer for the AI Inference Platform. - **MySQL** (skill): Used in legacy/contract setups where Postgres isn't the default. - **pgvector** (skill): Embedding storage and similarity search inside Postgres. - **OpenSearch** (skill): Powered low-latency profile retrieval for the WhatsApp AI agents at Tejas AI. - **#devops** (hub): 10 skills in DevOps & Cloud. - **Docker** (skill): Default packaging for every backend service I deploy. - **Kubernetes** (skill): Container orchestration for horizontally scaled inference workloads. - **AWS** (skill): Primary cloud — EC2, S3, RDS, Lambda for production deploys. - **Azure** (skill): Used selectively for client engagements requiring Microsoft stack. - **GCP** (skill): BigQuery, Cloud Run, and Vertex AI when the workload fits. - **ShaktiCloud** (skill): India-hosted GPU cloud — used at AuraX for diffusion inference. - **CI/CD** (skill): Build pipelines, automated testing, and zero-touch deploys. - **GitHub Actions** (skill): Workflow engine of choice — covers every repo I ship. - **Prometheus** (skill): Metrics scraping for the AI Inference Platform observability stack. - **Grafana** (skill): Dashboards on top of Prometheus for latency, cache, and model health. - **#ai** (hub): 14 skills in AI/ML/QM. - **PyTorch** (skill): Core deep-learning framework for VTO research and custom models. - **TensorFlow** (skill): Used selectively where the ecosystem fits (Keras, TFLite). - **scikit-learn** (skill): Classical ML for baselines and feature engineering. - **diffusers** (skill): HuggingFace library used heavily for Flux-based VTO pipelines. - **CUDA** (skill): Custom kernels and inference tuning on NVIDIA GPUs. - **Unsloth** (skill): Memory-efficient fine-tuning library — powers TuneSLM. - **Qiskit** (skill): IBM's quantum SDK — used for entanglement and QFT experiments. - **OpenQASM** (skill): Quantum assembly for circuit-level optimization in QM research. - **LLM** (skill): General large language model stack — OpenAI, Anthropic, Gemini, local Ollama. - **LoRA** (skill): Low-rank adapters — central to AuraX's C-AAC composition algorithm. - **Ollama** (skill): Local LLM runtime — used in localai, codehaven, and dbquery. - **Gemini API** (skill): Google's LLM API — used in dbquery and Syntaxia. - **OpenAPI** (skill): Spec-driven backend tooling — powers OpenCrow's server-side tool layer. - **Mem0** (skill): Persistent conversational memory layer built for the Tejas AI WhatsApp agents. ## Edges - ~/himanshu → #projects - ~/himanshu → #research - ~/himanshu → #experience - ~/himanshu → #skills - #skills → #sk-languages - #sk-languages → skill:TypeScript - #sk-languages → skill:Python - #sk-languages → skill:Rust - #sk-languages → skill:Swift - #sk-languages → skill:Dart - #sk-languages → skill:C/C++ - #sk-languages → skill:Go - #sk-languages → skill:Assembly x86 - #skills → #sk-frontend - #sk-frontend → skill:React - #sk-frontend → skill:Next.js - #sk-frontend → skill:TailwindCSS - #sk-frontend → skill:SwiftUI - #sk-frontend → skill:Flutter - #sk-frontend → skill:Tauri - #skills → #sk-backend - #sk-backend → skill:Nest.js - #sk-backend → skill:Flask - #sk-backend → skill:http (rust) - #sk-backend → skill:Axum - #sk-backend → skill:Tokio (rust) - #sk-backend → skill:Node.js - #sk-backend → skill:FastAPI - #sk-backend → skill:Express - #sk-backend → skill:Prisma - #skills → #sk-database - #sk-database → skill:PostgreSQL - #sk-database → skill:MongoDB - #sk-database → skill:TimescaleDB - #sk-database → skill:Milvus - #sk-database → skill:Redis - #sk-database → skill:MySQL - #sk-database → skill:pgvector - #sk-database → skill:OpenSearch - #skills → #sk-devops-cloud - #sk-devops-cloud → skill:Docker - #sk-devops-cloud → skill:Kubernetes - #sk-devops-cloud → skill:AWS - #sk-devops-cloud → skill:Azure - #sk-devops-cloud → skill:GCP - #sk-devops-cloud → skill:ShaktiCloud - #sk-devops-cloud → skill:CI/CD - #sk-devops-cloud → skill:GitHub Actions - #sk-devops-cloud → skill:Prometheus - #sk-devops-cloud → skill:Grafana - #skills → #sk-ai-ml-qm - #sk-ai-ml-qm → skill:PyTorch - #sk-ai-ml-qm → skill:TensorFlow - #sk-ai-ml-qm → skill:scikit-learn - #sk-ai-ml-qm → skill:diffusers - #sk-ai-ml-qm → skill:CUDA - #sk-ai-ml-qm → skill:Unsloth - #sk-ai-ml-qm → skill:Qiskit - #sk-ai-ml-qm → skill:OpenQASM - #sk-ai-ml-qm → skill:LLM - #sk-ai-ml-qm → skill:LoRA - #sk-ai-ml-qm → skill:Ollama - #sk-ai-ml-qm → skill:Gemini API - #sk-ai-ml-qm → skill:OpenAPI - #sk-ai-ml-qm → skill:Mem0 - #experience → co:tejas-ai - co:tejas-ai → skill:TypeScript - co:tejas-ai → skill:Go - co:tejas-ai → skill:Node.js - co:tejas-ai → skill:Nest.js - co:tejas-ai → skill:PostgreSQL - co:tejas-ai → skill:Redis - co:tejas-ai → skill:OpenSearch - co:tejas-ai → skill:Mem0 - co:tejas-ai → skill:LLM - co:tejas-ai → skill:Docker - co:tejas-ai → skill:AWS - #experience → co:aurax - co:aurax → skill:Python - co:aurax → skill:PyTorch - co:aurax → skill:diffusers - co:aurax → skill:LoRA - co:aurax → skill:CUDA - co:aurax → skill:LLM - co:aurax → skill:FastAPI - co:aurax → skill:PostgreSQL - co:aurax → skill:Redis - co:aurax → skill:Docker - co:aurax → skill:AWS - co:aurax → skill:ShaktiCloud - co:aurax → skill:Next.js - co:aurax → skill:Nest.js - co:aurax → res:c-aac - co:aurax → res:renu-virtual-try-on - #experience → co:synthailabs - co:synthailabs → skill:TypeScript - co:synthailabs → skill:Next.js - co:synthailabs → skill:Node.js - co:synthailabs → skill:Express - co:synthailabs → skill:MongoDB - co:synthailabs → skill:PostgreSQL - co:synthailabs → skill:Redis - co:synthailabs → skill:Docker - co:synthailabs → skill:CI/CD - co:synthailabs → skill:React - co:synthailabs → skill:TailwindCSS - co:synthailabs → res:moe-brain - #experience → co:tublian - co:tublian → skill:Python - co:tublian → skill:LLM - co:tublian → skill:Ollama - co:tublian → skill:FastAPI - co:tublian → skill:Docker - co:tublian → skill:TypeScript - co:tublian → skill:React - co:tublian → skill:Node.js - co:tublian → skill:GitHub Actions - #experience → co:stealth - co:stealth → skill:Python - co:stealth → skill:Qiskit - co:stealth → skill:OpenQASM - co:stealth → skill:C/C++ - #experience → co:excelerate - co:excelerate → skill:TypeScript - co:excelerate → skill:React - co:excelerate → skill:Next.js - #projects → #pr-ai - #pr-ai → proj:CCProxy - proj:CCProxy → skill:TypeScript - #pr-ai → proj:OpenCrow - proj:OpenCrow → skill:React - proj:OpenCrow → skill:TypeScript - proj:OpenCrow → skill:Node.js - proj:OpenCrow → skill:OpenAPI - proj:OpenCrow → skill:LLM - #projects → #pr-infra - #pr-infra → proj:AI Inference Platform - proj:AI Inference Platform → skill:Rust - proj:AI Inference Platform → skill:Axum - proj:AI Inference Platform → skill:Tokio (rust) - proj:AI Inference Platform → skill:Redis - proj:AI Inference Platform → skill:Prometheus - proj:AI Inference Platform → skill:Grafana - proj:AI Inference Platform → skill:Docker - #pr-ai → proj:TuneSLM - proj:TuneSLM → skill:Python - proj:TuneSLM → skill:Unsloth - proj:TuneSLM → skill:LLM - #projects → #pr-devtools - #pr-devtools → proj:MockServer - proj:MockServer → skill:TypeScript - proj:MockServer → skill:Next.js - proj:MockServer → skill:Express - #pr-devtools → proj:FastSearch - proj:FastSearch → skill:Rust - proj:FastSearch → skill:TypeScript - #pr-ai → proj:FallenAngel-SE/CY - proj:FallenAngel-SE/CY → skill:LLM - proj:FallenAngel-SE/CY → skill:Python - proj:FallenAngel-SE/CY → skill:TypeScript - #pr-devtools → proj:Dynamo-Prisma - proj:Dynamo-Prisma → skill:TypeScript - proj:Dynamo-Prisma → skill:Prisma - proj:Dynamo-Prisma → skill:Node.js - #projects → #pr-learning - #pr-learning → proj:ScratchAI-Algo - proj:ScratchAI-Algo → skill:Python - co:synthailabs → proj:ScratchAI-Algo - #pr-learning → proj:QM - Research - proj:QM - Research → skill:Python - proj:QM - Research → skill:Qiskit - proj:QM - Research → skill:OpenQASM - #projects → #pr-apps - #pr-apps → proj:RustyExplorer - proj:RustyExplorer → skill:Rust - proj:RustyExplorer → skill:TypeScript - proj:RustyExplorer → skill:Tauri - proj:RustyExplorer → skill:TailwindCSS - proj:RustyExplorer → skill:React - #pr-learning → proj:Linux Mini-Firewall - proj:Linux Mini-Firewall → skill:C/C++ - #pr-learning → proj:indieOS - proj:indieOS → skill:Assembly x86 - proj:indieOS → skill:C/C++ - #pr-devtools → proj:Codeforces Scraper - proj:Codeforces Scraper → skill:Python - #pr-devtools → proj:Log Ingestor - proj:Log Ingestor → skill:Rust - #pr-apps → proj:ZenStore (In-memory DB) - proj:ZenStore (In-memory DB) → skill:Rust - #pr-devtools → proj:frm (Fast Delete) - proj:frm (Fast Delete) → skill:Rust - #pr-apps → proj:roadmap.ai - proj:roadmap.ai → skill:Next.js - proj:roadmap.ai → skill:LLM - #pr-devtools → proj:biv - proj:biv → skill:Rust - #pr-learning → proj:ruspy - proj:ruspy → skill:Rust - #pr-apps → proj:PoW-Shield (DDoS Prevention) - proj:PoW-Shield (DDoS Prevention) → skill:C/C++ - proj:PoW-Shield (DDoS Prevention) → skill:Rust - proj:PoW-Shield (DDoS Prevention) → skill:TypeScript - #pr-devtools → proj:git-history - proj:git-history → skill:Rust - proj:git-history → skill:Python - #pr-apps → proj:localai - proj:localai → skill:Tauri - proj:localai → skill:Rust - proj:localai → skill:Next.js - proj:localai → skill:Ollama - #pr-apps → proj:draw (Excalidraw for Mac) - proj:draw (Excalidraw for Mac) → skill:Tauri - proj:draw (Excalidraw for Mac) → skill:Rust - proj:draw (Excalidraw for Mac) → skill:Next.js - #pr-apps → proj:Syntaxia (learncode) - proj:Syntaxia (learncode) → skill:Tauri - proj:Syntaxia (learncode) → skill:Rust - proj:Syntaxia (learncode) → skill:Next.js - proj:Syntaxia (learncode) → skill:Ollama - proj:Syntaxia (learncode) → skill:Gemini API - #pr-apps → proj:dbquery - proj:dbquery → skill:Tauri - proj:dbquery → skill:Rust - proj:dbquery → skill:Next.js - proj:dbquery → skill:Ollama - proj:dbquery → skill:Gemini API - #pr-apps → proj:codehaven.org - proj:codehaven.org → skill:Tauri - proj:codehaven.org → skill:Rust - proj:codehaven.org → skill:Next.js - proj:codehaven.org → skill:Ollama - #research → res:mmap-vs-read-apple-silicon - res:mmap-vs-read-apple-silicon → skill:C/C++ - res:mmap-vs-read-apple-silicon → skill:Rust - res:mmap-vs-read-apple-silicon → skill:PostgreSQL - #research → res:c-aac - res:c-aac → skill:LLM - res:c-aac → skill:diffusers - res:c-aac → skill:LoRA - #research → res:renu-virtual-try-on - res:renu-virtual-try-on → skill:diffusers - res:renu-virtual-try-on → skill:LoRA - #research → res:moe-brain - res:moe-brain → skill:LLM - #research → res:lattice-based-quantum-encryption - #research → res:gpt-neo - res:gpt-neo → skill:PyTorch - #research → res:custom-n-n-mnist - res:custom-n-n-mnist → skill:PyTorch --- Source: https://himanshuat.com/graph ======================================================================== # Fintracker A minimal, offline-first personal finance app built with Flutter. Track income, categorise spending and manage accounts, with everything stored locally on the device — no accounts, no cloud sync, no servers. Status: in verification on Android. ## Why it exists - **Privacy first.** Data never leaves the device. - **Smart analytics.** Spending visualised as charts. - **Local only.** No sign-up, no backend to trust. Privacy policy: https://himanshuat.com/app/fintracker/privacy --- Source: https://himanshuat.com/app/fintracker ======================================================================== # Fintracker — Privacy Policy Fintracker stores all financial data locally on your device. There are no user accounts, no cloud servers and no transmission of your data off the device. > This is a plain-text summary for convenience. The page below is the > authoritative version of the policy — read it there before relying on it. https://himanshuat.com/app/fintracker/privacy --- Source: https://himanshuat.com/app/fintracker/privacy ======================================================================== --- title: "Architecting Multi-Agent Teams with LangGraph: Supervisor Patterns, Worker Isolation, and Sub-Graph Composition" description: "Two patterns for multi-agent coordination: a Python supervisor that routes between specialist worker nodes via explicit state evaluation, and a typed graph engine that composes child sub-graphs with isolated state boundaries. When each earns its complexity, and the failure modes that make both patterns break." date: "July 22, 2025" url: "https://himanshuat.com/blogs/architecting-multi-agent-teams-with-langgraph" --- # Architecting Multi-Agent Teams with LangGraph: Supervisor Patterns, Worker Isolation, and Sub-Graph Composition Every hop in a multi-agent graph is a network round trip, a fresh set of input tokens, and one more place for state to go stale. > A graph earns that cost only when the problem needs iterative self-correction, role-isolated tool permissions, or state you can inspect and resume at each step. Otherwise it is one LLM call with extra hops. Two patterns cover most real cases. The first is a Python supervisor that routes between specialist workers using state evaluation at `temperature=0.0`. The second is a typed graph engine that composes child sub-graphs with isolated execution boundaries. Each has specific failure modes the other avoids. --- ## Pattern 1: Star topology with a central supervisor ### Step 1: Define the shared state schema The state schema is the single source of truth for routing decisions. Every field must be explicitly typed. The supervisor reads structured dictionary keys, not natural language. ```python from typing import TypedDict, Optional from langgraph.graph import StateGraph, END class AgentState(TypedDict): query: str research_results: list[str] draft: Optional[str] review_comments: list[str] iterations: int max_iterations: int next_action: Optional[str] messages: list[dict] ``` ### Step 2: Implement the supervisor and worker nodes The supervisor runs at `temperature=0.0` and returns exactly one token from a closed vocabulary. If the model returns anything else, routing fails with a missing key exception. ```python def call_llm(prompt: str, model: str = "gpt-4o", temperature: float = 0.7) -> str: """Direct LLM call. Replace with your actual client.""" raise NotImplementedError class Agents: def __init__(self, llm_model: str = "gpt-4o"): self.llm_model = llm_model def supervisor_agent(self, state: AgentState) -> dict: """Routes to the correct worker based on current state content.""" supervisor_prompt = f""" Analyze the current pipeline state and decide the next action. State summary: - Research results: {len(state.get('research_results', []))} items collected - Draft exists: {bool(state.get('draft'))} - Review comments: {len(state.get('review_comments', []))} items - Iterations completed: {state.get('iterations', 0)} / {state.get('max_iterations', 2)} Rules: 1. If research is empty or insufficient, return 'research'. 2. If research exists and no draft is written, return 'draft'. 3. If a draft exists without review feedback, return 'review'. 4. If review comments exist and iterations < max_iterations, return 'revise'. 5. If draft is reviewed and validated, or max_iterations is reached, return 'finish'. Output EXACTLY one word: research, draft, review, revise, finish. """ action = call_llm(supervisor_prompt, model=self.llm_model, temperature=0.0).strip().lower() return {"next_action": action} def researcher_agent(self, state: AgentState) -> dict: """Gathers domain evidence for the topic.""" research_prompt = f""" Find 3-5 concise technical facts about: "{state['query']}". Focus on concrete architecture details and code-level constraints. Existing findings: {state.get('research_results', [])} """ new_research = call_llm(research_prompt, model=self.llm_model).strip() updated_results = state.get("research_results", []) + [new_research] return {"research_results": updated_results} def writer_agent(self, state: AgentState) -> dict: """Drafts from gathered research.""" research_summary = "\n".join(state.get("research_results", [])) writer_prompt = f""" Draft a 200-300 word technical summary on: "{state['query']}". Research context: {research_summary} Target: Senior software engineers. """ draft = call_llm(writer_prompt, model=self.llm_model).strip() return {"draft": draft} def reviewer_agent(self, state: AgentState) -> dict: """Evaluates draft for accuracy and clarity gaps.""" comments = call_llm(f""" Critically review this draft for: "{state['query']}". If sufficient, state "No specific changes needed." Otherwise, provide bulleted critique. Draft: {state.get('draft', '')} """, model=self.llm_model).strip() return {"review_comments": state.get("review_comments", []) + [comments]} def reviser_agent(self, state: AgentState) -> dict: """Applies review feedback and increments iteration count.""" revised = call_llm(f""" Revise this draft to address all feedback while preserving factual precision. Query: "{state['query']}" Original: {state.get('draft', '')} Feedback: {"\n".join(state.get('review_comments', []))} """, model=self.llm_model).strip() return { "draft": revised, "review_comments": [], "iterations": state.get("iterations", 0) + 1, } ``` ### Step 3: Connect nodes in a star topology All coordination flows through the supervisor. No worker routes directly to another worker. ```mermaid flowchart TD S{supervisor} S -->|research| R[researcher] S -->|draft| W[writer] S -->|review| V[reviewer] S -->|revise| X[reviser] S -->|finish| E([END]) R --> S W --> S V --> S X --> S ``` ```python team_agents = Agents() workflow = StateGraph(AgentState) workflow.add_node("supervisor", team_agents.supervisor_agent) workflow.add_node("researcher", team_agents.researcher_agent) workflow.add_node("writer", team_agents.writer_agent) workflow.add_node("reviewer", team_agents.reviewer_agent) workflow.add_node("reviser", team_agents.reviser_agent) workflow.set_entry_point("supervisor") workflow.add_conditional_edges( "supervisor", lambda state: state["next_action"], { "research": "researcher", "draft": "writer", "review": "reviewer", "revise": "reviser", "finish": END, } ) for node in ["researcher", "writer", "reviewer", "reviser"]: workflow.add_edge(node, "supervisor") app = workflow.compile() ``` ### Step 4: Stream state transitions ```python initial_state: AgentState = { "query": "Explain transformer attention in distributed inference.", "research_results": [], "draft": None, "review_comments": [], "iterations": 0, "max_iterations": 2, "next_action": None, "messages": [], } for event in app.stream(initial_state): for node_name, state_update in event.items(): print(f"Node: {node_name} | Delta: {list(state_update.keys())}") ``` --- ## Pattern 2: Typed sub-graph engine with isolated worker state When workers need distinct execution contexts, the star topology breaks down: workers contaminate each other's state through shared fields. The alternative is a graph registry where each worker runs as an isolated sub-graph with its own state boundary. ### Step 1: Define the graph engine ```typescript // core/graphEngine.ts import { WorkflowState } from './workflowState'; export type NodeFunction = (state: WorkflowState) => Promise; export type EdgeTarget = string | null | ((state: WorkflowState) => string | null); export interface WorkflowGraph { id: string; startNode: string; nodes: Map; edges: Map; } export const graphRegistry = new Map(); export async function runWorkflowGraph(graphId: string, state: WorkflowState): Promise { const graph = graphRegistry.get(graphId); if (!graph) throw new Error(`Graph '${graphId}' not registered.`); let currentNode: string | null = graph.startNode; while (currentNode !== null) { const nodeFn = graph.nodes.get(currentNode); if (!nodeFn) throw new Error(`Node '${currentNode}' not found in graph '${graphId}'.`); state = await nodeFn(state); const edgeTarget = graph.edges.get(currentNode); currentNode = typeof edgeTarget === 'function' ? edgeTarget(state) : edgeTarget ?? null; } return state; } ``` ### Step 2: Orchestrator with bounded sub-task queue ```typescript // graphs/mainOrchestratorGraph.ts const decomposeTaskNode: NodeFunction = async (state) => { const prompt = `Break down this task into 2-4 concrete sub-tasks: "${state.overallTask}" Return JSON: {"sub_tasks": [{"id": "...", "description": "...", "requiredWorkerGraph": "..."}]}`; const response = await llmCall(prompt, state.contextData); const parsed = JSON.parse(response); state.subTaskQueue = parsed.sub_tasks; return state; }; const expertRouterNode: NodeFunction = async (state) => { if (state.subTaskQueue.length > 0) { state.currentSubTask = state.subTaskQueue.shift()!; state.metadata.nextManagerAction = 'invokeWorker'; } else { state.metadata.nextManagerAction = 'synthesize'; } return state; }; const invokeWorkerGraphNode: NodeFunction = async (state) => { if (!state.currentSubTask) throw new Error("No sub-task to execute."); const workerGraphId = state.currentSubTask.requiredWorkerGraph; if (!graphRegistry.has(workerGraphId)) { throw new Error(`Worker graph '${workerGraphId}' not registered.`); } // Construct isolated sub-state: worker cannot modify parent queue or metadata let workerState: WorkflowState = { overallTask: state.currentSubTask.description, currentSubTask: { ...state.currentSubTask, output: null, status: 'PENDING' }, subTaskQueue: [], subTaskHistory: [], metadata: {}, contextData: state.contextData, }; workerState = await runWorkflowGraph(workerGraphId, workerState); state.currentSubTask.output = workerState.currentSubTask?.output ?? null; state.currentSubTask.status = workerState.currentSubTask?.status ?? 'FAILED'; state.subTaskHistory.push({ ...state.currentSubTask }); state.currentSubTask = null; return state; }; ``` ### Step 3: Worker isolation boundary Only `contextData` and the input payload cross the parent-worker boundary. Workers cannot modify the parent's queue or pollute state used by sibling nodes: | Field | Orchestrator state | Worker sub-state | |---|---|---| | `overallTask` | User prompt | Sub-task instruction | | `subTaskHistory` | Global execution log | Fresh empty array | | `metadata` | Orchestrator queue control | Fresh empty dictionary | | `contextData` | Inherited from caller | Passed down directly | ## What breaks both patterns **Unstructured supervisor output.** If the supervisor prompt fails to enforce a closed vocabulary and the LLM returns "I think we should research more", the conditional edge lookup fails with a missing key exception. Enforce with JSON schema validation or response_format. **Shared state contamination in the star topology.** When workers write output into unstructured message lists rather than explicit dictionary keys, downstream nodes must parse natural language to find parameters, and dropped context is silent. **Unbounded revision cycles.** Without a programmatic ceiling on `max_iterations`, a reviewer that always returns "needs improvement" runs until API quota is exhausted. Check iteration bounds inside the edge function, not just inside the LLM prompt. **Registry dispatch errors.** The sub-graph engine must validate that `requiredWorkerGraph` identifiers exist in `graphRegistry` before invocation. A missing graph at runtime produces an opaque error that doesn't point to the decomposition step that hallucinated the graph name. --- ## When this is the wrong choice - **The task is sequential and deterministic.** Plain functions are faster, cheaper, and easier to test. A supervisor that always routes research, draft, review, finish in that order is a `for` loop paying LLM prices per iteration. - **One call with structured output already solves it.** If JSON schema enforcement on a single call gets the answer, the graph adds a state schema, a router, and four failure modes for nothing. - **You are on an interactive latency budget.** Every hop is a model round trip. Two hops at 300ms each already exceed most interactive SLAs, and the supervisor pattern spends a hop on routing before any work happens. - **The workers share one state object anyway.** If nothing needs isolated execution context, the sub-graph engine buys you a registry, a boundary table, and dispatch errors that point at the wrong step. --- Source: https://himanshuat.com/blogs/architecting-multi-agent-teams-with-langgraph ======================================================================== --- title: "ComfyUI as a research lab" description: "A custom node pack updated, a mask node changed its default on inverted masks, and our outputs moved with nothing changed on our side. Running a try-on diffusion pipeline as a node graph made every experiment a commit, and made very clear where a graph stops being a program." date: "February 04, 2025" url: "https://himanshuat.com/blogs/comfyui-as-a-research-lab" --- # ComfyUI as a research lab A community node pack updated silently overnight. A mask utility inverted its default boolean flag. Our renders shifted, with zero code changes on our side. We lost an afternoon finding out why. Standard ComfyUI workflow JSON records node connections, but remains silent about third-party node package revisions. > ComfyUI is an effective visual prototyping environment and a difficult production engineering environment. Problems arise when treating the same artifact as both. ## What node graphs represent in generative R&D Our virtual try-on architecture combines two conditioning streams, a segmentation network, morphological mask operations, two merged LoRAs, and a two-pass sampler. During development, tracking what changed day-to-day is critical. A visual node graph makes topology explicit. There is no hidden variable left connected in background memory. What is on the canvas is what runs. --- ## Step 1: Separating topological changes from hyperparameter sweeps Diffusion prototyping in Jupyter notebooks quickly becomes messy. Execution state depends implicitly on cell execution order and mutated global variables. When a run produces a strong render, you are often left with a dirty `.ipynb` file where cell 14 ran after cell 22, making git diffs unreadable. We separated our experimental workflows based on whether an experiment altered pipeline topology or scalar values: | Experiment Type | Optimal Artifact | Rationale | |---|---|---| | Fixed topology, scalar tuning | YAML configuration per run | Clean text diffs across numerical sweeps | | Daily structural rewiring | Visual node graph | The experiment consists of rewiring connections | | Unstructured notebook exploration | Jupyter notebook | Interactive scratchpad, poor version control | When testing whether Flux Redux tokens should condition only the primary sampling pass or both passes, rewiring nodes in a visual graph provides immediate feedback. --- ## Step 2: Normalizing workflow JSON exports for git versioning ComfyUI workflows serialize natively to JSON, allowing them to be tracked in git. However, default ComfyUI exports include transient UI state: screen coordinates, canvas zoom levels, and auto-incrementing internal IDs. Dragging a node across the screen produces large git diffs that obscure actual architectural changes. We built a pre-commit normalizer to strip canvas noise before saving. ```python # normalize_workflow.py """Strip canvas layout data from a ComfyUI workflow export. Node positions, dimensions, and canvas viewports carry no execution semantics. This script removes layout fields and sorts nodes deterministically so git diffs reflect only pipeline changes. """ import json import sys from pathlib import Path DROP_KEYS = {"pos", "size", "flags", "order", "extra", "color", "bgcolor"} def normalize(workflow: dict) -> dict: """Return a layout-independent representation of a ComfyUI workflow.""" nodes = [] for node in sorted(workflow.get("nodes", []), key=lambda n: n["id"]): nodes.append( {k: v for k, v in sorted(node.items()) if k not in DROP_KEYS} ) links = sorted(workflow.get("links", []), key=lambda l: (l[1], l[3], l[2])) return { "nodes": nodes, "links": links, "widget_defaults": workflow.get("extra", {}).get("ds", None) and None, } def main(paths: list[str]) -> None: for p in paths: src = Path(p) dst = src.with_suffix(".norm.json") data = json.loads(src.read_text()) dst.write_text(json.dumps(normalize(data), indent=2, sort_keys=True)) print(f"{src.name} -> {dst.name}") if __name__ == "__main__": main(sys.argv[1:]) ``` Tracking normalized workflow files turned pipeline pull requests into clean, reviewable diffs. --- ## Step 3: Graph topology and branch convergence Here is an illustrative schema showing the convergence of segmentation, background removal, and dual conditioning streams: ```json { "_file": "flux_vton_plus.illustrative.json", "_note": "Simplified reference schema", "nodes": [ { "id": 1, "type": "LoadImage", "widgets_values": ["model_0412.png"] }, { "id": 2, "type": "LoadImage", "widgets_values": ["garment_ref_saree_green.png"] }, { "id": 3, "type": "RMBG", "widgets_values": ["u2net", 0.5], "inputs": [{ "name": "image", "link": [2, 0] }] }, { "id": 4, "type": "SAM2Segment", "widgets_values": ["shirt, tshirt, top, saree, dress"], "inputs": [{ "name": "image", "link": [1, 0] }] }, { "id": 5, "type": "GrowMask", "widgets_values": [11, true], "inputs": [{ "name": "mask", "link": [4, 0] }] }, { "id": 6, "type": "StyleModelLoader", "widgets_values": ["flux1-redux-dev.safetensors"] }, { "id": 7, "type": "ReduxApply", "inputs": [ { "name": "style_model", "link": [6, 0] }, { "name": "reference", "link": [3, 0] } ] }, { "id": 8, "type": "UNETLoader", "widgets_values": ["flux1-fill.safetensors", "default"] }, { "id": 9, "type": "LoraLoaderModelOnly", "widgets_values": ["drape_physics_r32.safetensors", 0.6], "inputs": [{ "name": "model", "link": [8, 0] }] }, { "id": 10, "type": "ModelMergeLoRA", "widgets_values": ["occlusion_depth_r32.safetensors", 0.4], "inputs": [{ "name": "model", "link": [9, 0] }] }, { "id": 11, "type": "KSampler", "widgets_values": [30, 3.5, "euler_ancestral", "beta", 1.0], "inputs": [ { "name": "model", "link": [10, 0] }, { "name": "conditioning", "link": [7, 0] }, { "name": "mask", "link": [5, 0] } ] }, { "id": 12, "type": "KSampler", "widgets_values": [10, 3.5, "euler_ancestral", "beta", 0.35], "inputs": [ { "name": "model", "link": [10, 0] }, { "name": "latent", "link": [11, 0] } ] }, { "id": 13, "type": "VAEDecode", "widgets_values": ["fp16"], "inputs": [{ "name": "samples", "link": [12, 0] }] } ] } ``` ```mermaid flowchart LR G[garment ref] --> R[RMBG] R --> RX[Redux apply] SM[style model] --> RX M[model photo] --> S[SAM2 segment] S --> GM[grow mask] U[base UNet] --> L1[drape LoRA] L1 --> L2[occlusion merge] RX --> K1[sampler 30] GM --> K1 L2 --> K1 K1 --> K2[sampler 10] L2 --> K2 K2 --> D[VAE decode] ``` Isolating the reference garment with RMBG before applying Redux prevents background studio elements from corrupting cross-attention tokens. The two-pass sampler splits generation between full 30-step spatial synthesis and a 10-step low-denoise pass for fine fabric texture. --- ## Step 4: Surfacing critical merge coefficients on the canvas We fuse LoRA adapters directly into base weights via `model_merge_lora` nodes upstream of inference: $$ W_{\text{merged}} = W_{\text{base}} + \lambda_{\text{drape}} \cdot (B_d A_d) + \lambda_{\text{occ}} \cdot (B_o A_o) $$ With `lambda_drape = 0.6` (Expert A, draping physics) and `lambda_occ = 0.4` (Expert B, occlusion depth): | Parameter Over-weighting | Visual Artifact | Underlying Cause | |---|---|---| | High $\lambda_{\text{occ}}$ | Hands preserved cleanly | Garment fabric loses natural drape and hangs stiffly | | High $\lambda_{\text{drape}}$ | Folds and pleats render with depth | Fabric paints over foreground fingers and jewelry | Placing these coefficients as visible node inputs allowed team members to inspect trade-offs directly on the canvas without modifying underlying Python scripts. --- ## Step 5: Pinning dependencies and weight hashes in lockfiles To ensure repeatability, we snapshot normalized workflow JSON alongside explicit lockfiles capturing checkpoint SHA-256 hashes and exact git commits for all custom node packages. ```bash # scripts/snapshot_workflow.sh # Freeze the current pipeline: normalized graph and exact weights. set -euo pipefail WORKFLOW="${1:?usage: snapshot_workflow.sh }" TAG="${2:?missing tag}" OUT="experiments/${TAG}" mkdir -p "${OUT}" python scripts/normalize_workflow.py "${WORKFLOW}" cp "${WORKFLOW%.json}.norm.json" "${OUT}/workflow.json" # Record exact checkpoint SHA-256 hashes python - "${OUT}/workflow.json" <<'PY' > "${OUT}/weights.lock" import hashlib, json, pathlib, sys MODEL_ROOT = pathlib.Path("/models") graph = json.loads(pathlib.Path(sys.argv[1]).read_text()) for node in graph["nodes"]: for value in node.get("widgets_values", []): if isinstance(value, str) and value.endswith(".safetensors"): path = next(MODEL_ROOT.rglob(value), None) if path is None: print(f"MISSING {value}") continue digest = hashlib.sha256(path.read_bytes()).hexdigest()[:16] print(f"{value} sha256:{digest}") PY # Lock custom node commits to prevent silent upstream breaking changes pip freeze | grep -Ei "torch|diffusers|transformers" > "${OUT}/python.lock" (cd custom_nodes && for d in */; do printf "%s %s\n" "${d%/}" "$(git -C "$d" rev-parse --short HEAD 2>/dev/null || echo unpinned)" done) > "${OUT}/nodes.lock" git add "${OUT}" && git commit -m "snapshot: ${TAG}" echo "snapshotted ${TAG}" ``` --- ## Failure modes and edge cases 1. **Unpinned third-party node repositories:** Node packages cloned directly from git `main` introduce silent breaking changes during automated worker rebuilds. 2. **Loss of individual adapter attribution:** Once two LoRAs are merged into base weights, diagnosing single-expert regressions requires rebuilding graphs with isolated adapters. 3. **Subgraph duplication drift:** Duplicating node clusters across multiple workflows leads to configuration drift when parameters are updated in one workflow and forgotten in another. --- ## When this is the wrong choice - **The topology is fixed and only the numbers move.** Sweeping learning rates or step counts across a stable pipeline is a CLI script and a YAML file. Clicking through canvas edits to change a scalar is slower and leaves no clean sweep record. - **The change has to survive asynchronous review.** A normalized JSON diff of a node array is still a node array. Reviewers outside the research group cannot see from it what moved in the graph. - **You are serving requests.** Production backends need type checking, queue leasing, and error handling that names the failing stage. None of that belongs in a canvas. - **You need to attribute a regression to one adapter.** Merging LoRAs into base weights on the graph is convenient and destroys per-adapter attribution. Diagnosing a single expert means rebuilding the graph with isolated adapters. --- Source: https://himanshuat.com/blogs/comfyui-as-a-research-lab ======================================================================== --- title: "Composing adapters that disagree with each other" description: "Two LoRA experts merged with two scalars shipped fine. A whole library of them for demographics, poses, lighting, and backgrounds did not, because adapters conflict per layer and one blend weight per adapter cannot express that." date: "August 05, 2025" url: "https://himanshuat.com/blogs/conflict-aware-adapter-composition" --- # Composing adapters that disagree with each other Merging two adapters with two scalar weights worked well enough to ship. Running that same approach across an entire library of adapters produced a model that succeeded in most regions but failed in specific layers. Adjusting a single global blend weight degraded the good regions whenever I tried to correct the broken ones. > A global blend weight is a per-adapter instrument applied to a per-layer disagreement. This post covers why global blend weights fail on multi-adapter setups and how we structured checkpoint generation for production. ## What a merge does when it works Our virtual try-on model fused two expert LoRAs into the base weights with one coefficient each: $$ W_{\text{merged}} = W_{\text{base}} + \lambda_{\text{drape}}(B_d A_d) + \lambda_{\text{occ}}(B_o A_o) $$ We settled on `lambda_drape = 0.6` and `lambda_occ = 0.4` through empirical testing, sweeping a two-dimensional grid until the resulting images retained both properties cleanly. That works when two adapters address mostly orthogonal image regions. Fabric draping and hand occlusion have minimal spatial overlap, so a static 0.6 to 0.4 ratio preserved both. Scene generation brings a much higher degree of conflict. Before compositing garments, we generate the full scene: a model of a specific demographic, in a target pose, under calibrated studio lighting, against a selected backdrop. Each variable is an adapter, either trained in-house on our curated dataset or pulled from open-source checkpoints. Brand shoots regularly demand combinations that were never trained together. Sweeping a high-dimensional coefficient grid by hand is impractical. Finding the grid point would not help either: one scalar per adapter cannot resolve conflicting updates across individual transformer blocks. --- ## Step 1: Mapping where adapters overlap A LoRA applies low-rank updates to specific modules inside the transformer. Two independently trained adapters often write to overlapping attention and feed-forward layers. Within those shared modules, their weight updates can point in compatible directions or directly oppose each other. Consider a practical example. A demographic adapter modifies skin undertone, pore texture, and cheekbone structure. A high-key lighting adapter also modifies skin values, because lighting directly alters face luminance and specular falloff. Both adapters agree on the backdrop, but they fight over facial features. Lowering the lighting weight to preserve skin tone flattens the lighting gradient across the rest of the frame. Increasing the lighting weight blows out facial highlights and creates a waxy appearance. To diagnose this, I wrote a script to inspect which modules two adapters target before running any merges. ```python # tools/inspect_adapter_overlap.py """Report which modules two LoRA adapters both write to. Overlap alone is not proof of conflict. It shows where a conflict is structurally possible. Update direction requires evaluating the model weights in memory. """ from __future__ import annotations import sys from collections import OrderedDict from safetensors.torch import load_file def target_modules(path: str) -> "OrderedDict[str, int]": """Map module name to the rank of its down projection.""" tensors = load_file(path) modules: "OrderedDict[str, int]" = OrderedDict() for key, tensor in tensors.items(): if not key.endswith("lora_down.weight"): continue name = key.rsplit(".lora_down", 1)[0] modules[name] = tensor.shape[0] return modules def overlap_report(path_a: str, path_b: str, preview: int = 12) -> None: """Print shared and exclusive module sets for two adapters.""" a = target_modules(path_a) b = target_modules(path_b) shared = [m for m in a if m in b] only_a = [m for m in a if m not in b] only_b = [m for m in b if m not in a] print(f"A: {path_a} modules={len(a)}") print(f"B: {path_b} modules={len(b)}") print(f"shared={len(shared)} a_only={len(only_a)} b_only={len(only_b)}") print() for name in shared[:preview]: print(f" shared {name} rank_a={a[name]} rank_b={b[name]}") for name in only_a[:preview]: print(f" a_only {name} rank={a[name]}") if __name__ == "__main__": overlap_report(sys.argv[1], sys.argv[2]) ``` ```text $ python tools/inspect_adapter_overlap.py \ adapters/demographics_in_female_25_35.safetensors \ adapters/lighting_softbox_high_key.safetensors A: demographics_in_female_25_35.safetensors modules=76 B: lighting_softbox_high_key.safetensors modules=76 shared=48 a_only=28 b_only=28 shared double_blocks.0.img_attn.qkv rank_a=32 rank_b=32 (cos_sim: -0.42) shared double_blocks.0.img_attn.proj rank_a=32 rank_b=32 (cos_sim: -0.38) shared double_blocks.4.img_attn.qkv rank_a=32 rank_b=32 (cos_sim: +0.61) shared double_blocks.12.img_mlp.0 rank_a=32 rank_b=32 (cos_sim: -0.55) a_only single_blocks.0.linear1 rank=32 b_only single_blocks.14.linear2 rank=32 ``` Running this across our adapter library showed that overlap is neither total nor negligible: 48 of 76 targeted LoRA projections collide. Early attention projections (`double_blocks.0`) come out at negative cosine similarity ($-0.42$), so the lighting and demographic updates push those weights in opposing directions. When two adapters share attention modules with opposing gradient directions, a single scalar weight arbitrates a conflict across every layer simultaneously, degrading both properties. --- ## Step 2: Writing explicit priors before merging Deciding which adapter takes precedence is a product decision dictated by the visual requirements of the catalog shoot. On an e-commerce shoot, skin characteristics must strictly follow the demographic adapter, because brand and legal guidelines require consistent representation. Backdrop appearance follows the background adapter, while garment surface texture follows lighting. These rules reflect deliberate visual priorities, so we store them in a versioned configuration file next to the resulting model checkpoint. ```yaml # priors/editorial_ecom.yaml # Reviewed by the brand team and versioned with the target checkpoint. # The resolver consumes this configuration without modifying priority orders. base: flux adapters: - id: demographics_in_female_25_35 origin: in_house rank: 32 - id: pose_studio_standing origin: in_house rank: 32 - id: lighting_softbox_high_key origin: public rank: 32 - id: background_studio_sweep origin: public rank: 32 # Which adapter owns which visual domain during conflicts: priority: skin_and_hair: demographics_in_female_25_35 body_proportion: pose_studio_standing garment_surface: lighting_softbox_high_key frame_and_backdrop: background_studio_sweep # Concerns protected from overrides regardless of resolver defaults: protected: - concern: skin_and_hair from: [lighting_softbox_high_key, background_studio_sweep] output: name: aurax-v1 bake: true ``` I initially expected the algorithmic resolver to be the central deliverable. In practice, the priors file is what makes the resolver output something a brand signs off on. Two brands using identical adapter libraries with different priors files produce noticeably different models. --- ## Step 3: Baking single checkpoints instead of runtime stacking Rather than loading and combining multiple adapters dynamically per API request, our composition pipeline bakes a single unified checkpoint (AuraX-V1). Runtime stacking seems flexible because it allows on-the-fly adapter combinations. However, it forces every inference request to execute multiple low-rank matrix multiplications in the hot path, keeps numerous adapter weights resident in GPU VRAM, and allows client requests to invoke untested adapter combinations. | Characteristic | Runtime Stacking | Build-Time Bake | |---|---|---| | Composition stage | Request path | Build path | | Serving memory footprint | Every active adapter resident | Single checkpoint | | Cold worker startup | Assemble graph | Load single file | | Review surface | Assembly logic | Model artifact | | Rollback mechanism | Code deployment | Version tag | | Request-level combinations | Yes | No | Reviewability is the decisive factor. When models are assembled dynamically per request, engineers can only review the assembly logic, not the resulting visual model. ```mermaid flowchart LR A[adapters] --> R[resolve] P[priors file] --> R B[base weights] --> R R --> C[one checkpoint] C --> S[serving] A -.-> K[runtime stack] K -.-> D[R&D only] ``` The solid path runs in production, while the dotted path remains active for experimental prototyping. ```python # pipelines/bake_runtime_model.py """Resolve adapter conflicts against a priors file and bake a checkpoint. The output is a single deployable artifact: one file, one version tag, and one unit to roll back. Composition runs entirely at build time. """ from __future__ import annotations import argparse import json from pathlib import Path import torch import yaml from safetensors.torch import load_file, save_file # Internal Conflict-Aware Adapter Composition resolver from aurax.caac import resolve # noqa: F401 def load_priors(path: Path) -> dict: """Read and validate a priors configuration file.""" priors = yaml.safe_load(path.read_text()) declared = {a["id"] for a in priors["adapters"]} referenced = set(priors["priority"].values()) missing = referenced - declared if missing: raise ValueError(f"Priority references unknown adapters: {sorted(missing)}") return priors def bake(priors_path: Path, base_path: Path, out_dir: Path) -> Path: """Produce the fused runtime checkpoint and provenance metadata.""" priors = load_priors(priors_path) base = load_file(str(base_path)) adapters = { a["id"]: load_file(f"adapters/{a['id']}.safetensors") for a in priors["adapters"] } merged = resolve(base=base, adapters=adapters, priors=priors) out_dir.mkdir(parents=True, exist_ok=True) ckpt = out_dir / f"{priors['output']['name']}.safetensors" save_file(merged, str(ckpt)) sidecar = { "name": priors["output"]["name"], "base": priors["base"], "adapters": [a["id"] for a in priors["adapters"]], "priors_sha": priors_path.read_bytes().hex()[:16], "torch": torch.__version__, } (out_dir / "provenance.json").write_text(json.dumps(sidecar, indent=2)) print(f"Saved {ckpt}") print(f"Fused {len(adapters)} adapters into one checkpoint") return ckpt if __name__ == "__main__": ap = argparse.ArgumentParser() ap.add_argument("--priors", type=Path, required=True) ap.add_argument("--base", type=Path, required=True) ap.add_argument("--out", type=Path, default=Path("build/")) args = ap.parse_args() bake(args.priors, args.base, args.out) ``` Baking sacrifices per-request adapter selection, requiring a new build and visual review for every updated priors configuration. We accepted this tradeoff because e-commerce brands maintain a fixed catalog aesthetic over time. For R&D, hot-swapping adapters via stacking remained valuable while tuning adapter behavior. --- ## Internal resolver implementation The `resolve` function executes C-AAC (Conflict-Aware Adapter Composition), calculating layer-wise interference metrics across weight tensors and enforcing the constraints declared in the priors file. This resolver logic is proprietary to AuraX. --- ## Step 4: Assessing output quality directly We evaluated AuraX-V1 against base Flux-dev, Google Imagen 4, ChatGPT (August 2025 generation), and Nano-banana on commercial skin fidelity and lighting quality through direct visual side-by-side reviews. Standard automated aesthetic scoring models favor high-contrast, moody lighting. That aesthetic profile directly conflicts with e-commerce requirements, where products demand uniform lighting and precise fabric depiction. We eventually constructed a dedicated human-in-the-loop scoring tool for this reason, relying on side-by-side visual inspections for initial sign-offs. | Model | Visual Characteristics | |---|---| | Fused (AuraX-V1) | Natural skin pores and hair detail with neutral studio lighting | | Flux-dev | Strong cinematic contrast that overpowers delicate garment details | | Imagen 4 | High overall photorealism, though skin smoothing reads slightly synthetic on close crops | | Nano-banana | Sharp contrast suitable for editorial campaigns but rejected by standard catalog guidelines | These evaluations were conducted directly by our engineering and product teams. The fine-tuning behind this model relied on 5,000 curated fashion images. --- ## Failure modes and edge cases 1. **Training adapters without defining target priors first:** Developing adapter capability without explicit prior definitions forces the downstream resolver to suppress unneeded features later. 2. **Scoring commercial outputs with generic aesthetic metrics:** Off-the-shelf aesthetic scorers penalize flat, even commercial lighting. Automated filters must be aligned with catalog requirements. 3. **Over-interpreting informal visual inspections:** Internal side-by-side reviews suffice for deployment decisions, but rigorous competitive claims require blinded, multi-rater panels. --- ## When this is the wrong choice - **The adapters barely overlap.** Run the overlap report first. Fabric drape and hand occlusion share little, and two scalars at 0.6 and 0.4 held that pair well enough to ship. - **Every request needs a different combination.** Baking gives up per-request adapter selection. If the product genuinely varies the combination per call, pay the serving overhead and stack at runtime. - **The adapters are still moving.** While you are still tuning what an adapter does, stacking lets you swap it without a rebuild and a visual review. - **Nobody has written down the priorities.** The resolver enforces a priors file. Without a human decision about which adapter owns skin, backdrop, and garment surface, there is nothing for it to enforce, and per-layer resolution just picks a different arbitrary answer than a scalar would. --- Source: https://himanshuat.com/blogs/conflict-aware-adapter-composition ======================================================================== --- title: "Database Agents That Know When They're Wrong" description: "A database agent that returns a clean, confident, incorrect result set is a liability. Here is how to build AST validation guards, database-level security boundaries, and ambiguity classifiers to catch silent query errors." date: "January 20, 2026" url: "https://himanshuat.com/blogs/db-agents-that-know-when-theyre-wrong" --- # Database Agents That Know When They're Wrong A SQL query runs without errors, returns a scalar number, and the number is wrong because an unindexed join fanned out and multiplied an aggregate sum. No database alert triggers. The number flows straight into an analytics report. Somebody catches it weeks later, after decisions have already been made on it. > An agent that returns a clean, confident, wrong result set is worse than one that crashes. The crash at least tells you something is broken. So the design goal is not accuracy. It is making the failures that stay silent become loud. ## Chapter 0: Defining silent failure in query agents A database query fails in three ways: 1. **Syntax or execution failure:** the query is invalid SQL. The database engine returns an error code, which is easy to catch and retry. 2. **Intent mismatch:** the SQL is valid, but answers a different question than intended due to ambiguous schema definitions. 3. **Semantic shape error:** the SQL runs against the intended schema, but unexpected null semantics, fan-out joins, or missing filters silently distort row counts or aggregates. Only the first failure mode signals itself. The other two return clean tables. Guarding against silent failure requires deterministic application scaffolding, not longer system prompts. --- ## Step 1: Benchmark metrics and annotation noise On the [BIRD leaderboard](https://bird-bench.github.io/), read on 8 September 2026, the human baseline is 92.96% execution accuracy (measured with database engineers and students). Top automated pipelines reach roughly 82% on the test set: | BIRD system | Dev set | Test set | |---|---|---| | Human baseline | N/A | 92.96% | | AskData + GPT-4o | 77.64% | 81.95% | | Agentar-Scale-SQL | 74.90% | 81.67% | These benchmarks provide external evidence strings alongside queries. Real production environments rarely provide pre-annotated schema hints. On the [Spider 2.0 leaderboard](https://spider2-sql.github.io/), read the same day, standard models show significant degradation: | Spider 2.0 baseline | Snow | Lite | |---|---|---| | Spider-Agent + o1-preview | 23.58% | 23.03% | | Spider-Agent + GPT-4o | 12.98% | 13.16% | Tengjun Jin, Yoojin Choi, Yuxuan Zhu and Daniel Kang's CIDR 2026 paper [Text-to-SQL Benchmarks are Broken](https://www.vldb.org/cidrdb/2026/text-to-sql-benchmarks-are-broken-an-in-depth-analysis-of-annotation-errors.html), circulated as the preprint [Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards](https://arxiv.org/abs/2601.08778) in January 2026, found annotation error rates of 52.8% on BIRD Mini-Dev and 66.1% on Spider 2.0-Snow. The Spider figure is worth reading twice: the earlier preprint put it at 62.8% and the published version at 66.1%, which is to say two careful passes over the same benchmark disagreed by three points about how wrong its labels are. When benchmark labels contain flawed assumptions around timestamp casting and null handling, leaderboard ranks measure annotation noise alongside genuine capability. Select architectural patterns by evaluating ablation studies and failure modes within specific domain schemas rather than leaderboard rankings. --- ## Step 2: High-value guardrails for query generation Ablation studies on published Text-to-SQL pipelines point at which architectural layers return the most per unit of engineering effort: 1. **AST parsing and syntax verification:** immediate, local validation before database execution. 2. **Execution error feedback loops:** capturing database engine errors and supplying them to a repair agent. 3. **Empty result heuristics:** treating zero-row outputs as potential filter mismatches rather than definitive answers. 4. **Candidate sampling and selection:** generating multiple candidate queries and scoring them against schema constraints. 5. **Assertion checks over output rows:** validating row ranges, uniqueness, and aggregate boundaries. AST parsing provides an effective first gate against unsafe queries. [`sqlglot`](https://github.com/tobymao/sqlglot) parses SQL into a dialect-aware AST: ```python # sql_guard.py """AST-level checks over model-generated SQL.""" from sqlglot import exp, parse_one from sqlglot.errors import ParseError ALLOWED_TABLES = { "analytics.v_orders", "analytics.v_customer", "analytics.v_product", } FORBIDDEN = ( exp.Insert, exp.Update, exp.Delete, exp.Drop, exp.Create, exp.Alter, exp.Merge, exp.Grant, exp.Command, ) MAX_ROWS = 5_000 class RejectedQueryError(Exception): """Exception containing feedback for model query revision.""" pass def guard_sql(sql: str, dialect: str = "postgres") -> str: """Parses SQL and enforces structural safety rules via AST analysis.""" try: tree = parse_one(sql, read=dialect) except ParseError as err: raise RejectedQueryError(f"SQL parsing failed: {err}") from err if not isinstance(tree, exp.Select): raise RejectedQueryError(f"Root AST node is {type(tree).__name__}, expected Select.") for node in tree.walk(): if isinstance(node, FORBIDDEN): raise RejectedQueryError(f"Prohibited operation detected: {type(node).__name__}") # CTE names are valid identifiers that will not exist in the base table allowlist cte_names = {c.alias_or_name for c in tree.find_all(exp.CTE)} for table in tree.find_all(exp.Table): name = table.sql(dialect=dialect, identify=False) if name in cte_names: continue if name not in ALLOWED_TABLES: raise RejectedQueryError(f"Table '{name}' is not in the allowed schema list.") limit = tree.args.get("limit") if limit is None or int(limit.expression.this) > MAX_ROWS: tree = tree.limit(MAX_ROWS) return tree.sql(dialect=dialect) ``` Walking the AST and checking an explicit allowlist prevents circumvention techniques that bypass naive regex matchers. --- ## Step 3: Enforce safety at the database engine layer Application-level guards have edge cases. Put the security boundary where the model cannot reach it: PostgreSQL role permissions and connection parameters. ```sql -- roles.sql CREATE ROLE llm_agent LOGIN PASSWORD 'secure_agent_password'; ALTER ROLE llm_agent SET default_transaction_read_only = on; ALTER ROLE llm_agent SET statement_timeout = '10s'; ALTER ROLE llm_agent SET idle_in_transaction_session_timeout = '15s'; ALTER ROLE llm_agent SET lock_timeout = '2s'; ALTER ROLE llm_agent CONNECTION LIMIT 8; REVOKE ALL ON SCHEMA public FROM llm_agent; GRANT USAGE ON SCHEMA analytics TO llm_agent; -- Grant access to explicitly curated analytical views only GRANT SELECT ON analytics.v_orders TO llm_agent; GRANT SELECT ON analytics.v_customer TO llm_agent; GRANT SELECT ON analytics.v_product TO llm_agent; -- Pre-execution cost check: -- Run EXPLAIN before running the real query. Reject queries exceeding cost thresholds. EXPLAIN (FORMAT JSON) SELECT c.region, sum(o.amount) FROM analytics.v_orders o JOIN analytics.v_customer c ON c.customer_id = o.customer_id GROUP BY 1; ``` With `default_transaction_read_only = on`, any write operation emitted by the model triggers PostgreSQL error [`SQLSTATE 25006`](https://www.postgresql.org/docs/current/errcodes-appendix.html) (`read_only_sql_transaction`) regardless of prompt phrasing. Running an `EXPLAIN` query before execution catches cartesian products and runaway table scans before they consume database resources. --- ## Step 4: Classify query ambiguity before code generation Most wrong answers come from ambiguous business terms, not invalid SQL: - **Metric definitions:** whether "revenue" refers to gross bookings or net settled funds. - **Grain and deduplication:** using `COUNT(*)` instead of `COUNT(DISTINCT user_id)` across joined tables. - **Time boundaries:** inclusive versus exclusive date ranges and timezone conversions. - **Soft deletes:** querying underlying tables without filtering for `deleted_at IS NULL`. Define an explicit output schema that allows the agent to request clarification rather than guessing: ```python # contract.py from dataclasses import dataclass from typing import Literal, Optional, List, Dict, Any Outcome = Literal["answer", "clarification_request", "refusal"] @dataclass(frozen=True) class Answer: kind: Outcome = "answer" sql: str = "" rows: Optional[List[Dict[str, Any]]] = None row_count: int = 0 checks_passed: Optional[List[str]] = None @dataclass(frozen=True) class ClarificationRequest: kind: Outcome = "clarification_request" question: str = "" ambiguity_class: str = "" # metric, grain, time_window, soft_delete plausible_interpretations: Optional[List[str]] = None @dataclass(frozen=True) class Refusal: kind: Outcome = "refusal" reason: str = "" attempted_sql: Optional[str] = None QueryResult = Answer | ClarificationRequest | Refusal ``` Track the rate of clarification requests and refusals over time. If an agent answers 100% of user queries without ever requesting clarification, its ambiguity detection is non-functional. --- ## Step 5: Screen result sets against data sanity assertions Run lightweight programmatic assertions over returned data before passing outputs to downstream consumers: ```python # plausibility.py from dataclasses import dataclass from typing import List, Dict, Any @dataclass class AnomalyWarning: check_name: str message: str def screen_result_rows( rows: List[Dict[str, Any]], table_total_rows: int, expected_scalar: bool = False ) -> List[AnomalyWarning]: """Evaluates result set shapes for common analytical failure modes.""" warnings: List[AnomalyWarning] = [] n = len(rows) if n == 0: warnings.append(AnomalyWarning("empty_set", "Query returned 0 rows. Verify filter values.")) if n == table_total_rows and table_total_rows > 100: warnings.append(AnomalyWarning("unfiltered", f"Row count ({n}) equals full table size.")) for row in rows: for col, val in row.items(): if any(term in col for term in ["_pct", "_rate", "_share"]) and val is not None: try: num = float(val) if not (0.0 <= num <= 100.0 or 0.0 <= num <= 1.0): warnings.append(AnomalyWarning("range_violation", f"Column '{col}' value {val} out of percentage bounds.")) except ValueError: pass if expected_scalar and n > 1: warnings.append(AnomalyWarning("cardinality_mismatch", f"Expected scalar value, received {n} rows.")) return warnings ``` Connecting these components creates an execution graph with verification loops: ```mermaid flowchart TD Q[User Question] --> A{Ambiguous Intent?} A -->|Yes| C[Clarification Request] A -->|No| G[Generate SQL] G --> V{AST Safety Check} V -->|Reject| G V -->|Pass| E{EXPLAIN Cost Gate} E -->|Budget Exceeded| R[Refusal] E -->|Pass| X[Execute on Read Replica] X -->|Engine Error| G X -->|Result Rows| S{Screen Data Sanity} S -->|Suspicious Shape| C S -->|Verified| ANS[Deliver Answer] ``` --- ## Step 6: Avoid treating document stores as relational SQL Generating queries for document stores like MongoDB presents distinct failure modes: 1. **Implicit schemas:** missing relational foreign keys makes identifying join paths harder. 2. **Polymorphic documents:** fields can vary across records, causing aggregation operations to drop data unexpectedly. 3. **Pipeline stage order sensitivity:** `$unwind` on a non-existent field drops documents silently without throwing an exception. 4. **Negative query semantics:** `{status: {$ne: "inactive"}}` matches documents where `status` is missing entirely. --- ## When this is the wrong choice - **A human reads every result before anyone acts on it.** In exploratory analysis the analyst is already the sanity check. The ambiguity classifier costs a clarification round trip on questions they would have refined anyway. - **The question set is fixed.** If the queries are a known set of reports, write the SQL once and let the model pick a report ID. AST allowlists and `EXPLAIN` gates exist to contain generated SQL. Do not generate SQL you did not need to generate. - **Nobody has written down the metric definitions.** The clarification path only works when someone can answer "gross bookings or net settled funds". Ship the ambiguity classifier without that, and it stalls on every question instead of guessing on some. - **The latency budget is tight.** The `EXPLAIN` pass, the sanity screen, and any repair loop each add a round trip before the caller sees a row. --- ## Implementation checklist 1. Configure read-only database users with strict connection limits and statement timeouts. 2. Parse model-generated queries using AST libraries like `sqlglot` against table allowlists. 3. Run `EXPLAIN` queries to reject runaway cartesian products before execution. 4. Catch zero-row and full-table result anomalies before presenting data to users. 5. Provide explicit schema options for query refusal and clarification. --- Source: https://himanshuat.com/blogs/db-agents-that-know-when-theyre-wrong ======================================================================== --- title: "FID was the wrong metric for the problem we had" description: "Aggregate FID went 22.1 to 18.5 and I nearly read that as months of adapter work buying four points. Half the eval set was already close to solved and it dragged the average toward nothing. Splitting by garment category was the change that made the result visible." date: "August 26, 2025" url: "https://himanshuat.com/blogs/fid-was-the-wrong-metric" --- # FID was the wrong metric for the problem we had Aggregate Fréchet Inception Distance (FID) dropped from 22.1 to 18.5 across our evaluation runs. I initially read that as months of adapter training and structural conditioning yielding a modest four-point improvement, and nearly treated it as evidence of diminishing returns. The calculation was mathematically correct, but using a single aggregate number obscured where the model was actually improving. > Any aggregate metric calculated over a heterogeneous population is an implicit weighted average. If half that population is already solved, the solved subset dilutes the performance signal of the hard subset. ## Limitations of single distributional scores FID calculates the Wasserstein-2 distance between feature distributions in Inception-v3 latent space. It produces a single scalar comparing a generated image set against reference photographs, baking the composition of the evaluation set into the final output. Our evaluation suite contains 500 image pairs split evenly between Western Casual (fitted t-shirts and jeans) and Global Traditional (sarees, kimonos, hanfus, and draped couture). We benchmarked three configurations across this dataset: SDXL with ControlNet, base Flux Fill without custom conditioning, and our dual-stream adapter pipeline. Aggregate FID scored 28.4, 22.1, and 18.5 respectively. On Western casual wear, base Flux Fill was already strong, leaving little headroom for further numerical gains. The Global Traditional category contained virtually all the structural failures we aimed to resolve. ```mermaid flowchart LR E[eval set] --> W[western half] E --> G[global half] W -->|no headroom| A[aggregate FID] G -->|all headroom| A A --> R[reads as modest] ``` Averaging a solved category with low headroom against an unsolved category with high headroom halves the visible magnitude of any real improvement. --- ## Step 1: Defining failure modes before selecting metrics We designed our virtual try-on architecture to address specific visual defects: sarees rendering as flat printed decals on skin, and kimonos flattening into bathrobes. Those are precise failure conditions. Pick the metric that moves when they are fixed, not the one that is standard in the literature. --- ## Step 2: Stratifying evaluation datasets by garment complexity Splitting the evaluation dataset by garment complexity and measuring structural similarity (SSIM) alongside human occlusion ratings revealed the true performance profile: | Method | Aggregate FID | SSIM (Western) | SSIM (Global) | Occlusion Accuracy | |---|---|---|---|---| | SDXL + ControlNet | 28.4 | 0.82 | 0.45 | 62% | | Base Flux Fill | 22.1 | 0.91 | 0.58 | 70% | | Flux-VTON+ (Ours) | 18.5 | 0.94 | 0.85 | 92% | Examining the split columns clarifies the difference: On Western apparel, SSIM moved 0.82 to 0.91 to 0.94. Standard diffusion inpainters already handle fitted garments. On Global Traditional garments, SSIM moved 0.45 (SDXL failure) to 0.58 (Flux baseline) to 0.85 (Flux-VTON+). That is the drape and depth adapters doing the work the aggregate was hiding. The remaining gap between 0.94 (Western) and 0.85 (Global) says complex drapes are still harder to synthesize than simple tubes. Aggregate FID reports none of this. --- ## Step 3: Enforcing balanced category manifests in the eval runner Stratified evaluation remains reliable only when category distributions are strictly controlled. We enforce parity checks directly in the evaluation runner: ```python # eval/run_split_eval.py """Score a checkpoint on the VTON eval set, split by garment category. Aggregate FID is retained for external benchmark comparison, but is always reported alongside category-stratified SSIM metrics. """ from __future__ import annotations import argparse import json from collections import defaultdict from pathlib import Path from skimage.metrics import structural_similarity from aurax.eval.fid import frechet_distance from aurax.eval.io import load_pair, load_manifest CATEGORIES = ("western_casual", "global_traditional") def ssim_by_category(manifest: list[dict], run_dir: Path) -> dict[str, float]: """Mean SSIM between generated and reference garment, per category.""" scores: dict[str, list[float]] = defaultdict(list) for item in manifest: generated, reference = load_pair(run_dir / item["id"], item["reference"]) score = structural_similarity( generated, reference, channel_axis=-1, data_range=1.0, ) scores[item["category"]].append(score) return {cat: sum(v) / len(v) for cat, v in scores.items()} def evaluate(manifest_path: Path, run_dir: Path) -> dict: """Return full report: aggregate FID plus per-category SSIM.""" manifest = load_manifest(manifest_path) counts = {c: sum(1 for i in manifest if i["category"] == c) for c in CATEGORIES} if len(set(counts.values())) != 1: raise ValueError(f"Eval set is not balanced across categories: {counts}") report = { "n": len(manifest), "per_category_n": counts, "fid_aggregate": frechet_distance(manifest, run_dir), "ssim": ssim_by_category(manifest, run_dir), } print(json.dumps(report, indent=2)) return report if __name__ == "__main__": ap = argparse.ArgumentParser() ap.add_argument("--manifest", type=Path, default=Path("eval/manifest.json")) ap.add_argument("--run", type=Path, required=True) evaluate(ap.parse_args().manifest, ap.parse_args().run) ``` The category count validation prevents silent skewing of evaluation sets. If someone adds 50 additional t-shirts to the manifest, the script halts execution. --- ## Step 4: Structuring human evaluation protocols Occlusion accuracy measures whether human hands and jewelry resting on garments survive generation rather than being overpainted by fabric (the "Amputated Hand" defect). Occlusion accuracy went 62% to 70% to 92%, and that ordering matched which renders clients accepted. Human perceptual data needs a documented rubric. It also needs a recorded panel size, an inter-rater reliability score such as Cohen's kappa, and a dispute-resolution protocol. Without those three, an occlusion number is directional validation, not a benchmark you can put in a paper. --- ## Failure modes and edge cases 1. **Sample size bias in distributional metrics:** 500 images represents a relatively small sample for FID calculations, which are subject to upward bias on smaller sets. 2. **Hidden orthogonal failure modes:** Stratifying exclusively by garment category fails to catch pose-related distortions (such as seated or athletic poses) or sheer fabric opaqueness. 3. **Subjective success definitions:** Converting visual inspection into binary pass/fail rates requires explicit quality rubrics defining acceptable boundary tolerances. --- ## When this is the wrong choice - **Every input is equally hard.** Stratification pays off because one half of our eval set had no headroom left. If difficulty and headroom are uniform across the set, the splits produce the same number twice. - **You are comparing against a published paper.** Papers report aggregate FID. Report it too, alongside the stratified numbers, or the comparison stops being a comparison. - **The buckets get too small.** 500 images is already small enough that FID carries upward bias. Splitting a set that size into several narrow buckets drops per-bucket counts below anything you should quote. --- Source: https://himanshuat.com/blogs/fid-was-the-wrong-metric ======================================================================== --- title: "Fitting a saree into 24GB: the context window trick" description: "A saree needs pixel density to hold its pleats, and a full-frame high-resolution pass doesn't fit on a 24GB card. The fix was to stop treating the frame as the unit of inference and hand the model only the region that matters." date: "September 02, 2025" url: "https://himanshuat.com/blogs/fitting-a-saree-into-24gb" --- # Fitting a saree into 24GB: the context window trick At low resolution, a saree pallu flattens against the skin instead of draping over the shoulder, and waist pleats blur into a solid smear. Resolving fine pleats requires high pixel density on the garment. Running the entire full-body frame at that resolution exceeds the VRAM limits of a standard 24GB card. > The unit of inference is the garment, not the frame. Crop to the region that has to be right, and a 24GB card runs the pipeline that a 48GB card was buying you. ## Latent token scaling and frame waste In a full-body e-commerce catalog photo, a model stands in the middle of a tall frame. The garment occupies roughly a third of the canvas. The remaining pixels consist of neutral backdrops, floor surfaces, and empty headroom: regions that a diffusion model still has to encode, denoise across every step, and decode through the VAE. At native resolution, the model distributes compute evenly across the frame. Upscaling the full image to resolve pleat details means allocating most of that compute to background studio sweeps. Memory consumption in diffusion transformers scales with the count of latent tokens, which grows proportionally with surface area. Doubling linear resolution quadruples the latent token footprint. That difference pushes memory requirements from a comfortable 24GB consumer GPU (such as an RTX 4090) up into 48GB enterprise hardware (such as an NVIDIA L40S) primarily to process empty backdrop pixels. We prioritized 24GB compatibility because it represents the standard hardware tier across our self-hosted inference clusters. --- ## Why naive tiling breaks continuous pleats A common approach to high-resolution diffusion is spatial tiling: splitting the image into overlapping sub-blocks, denoising each independently, and blending the seams. While this works for landscapes, it destroys structured garments. A saree pleat is a continuous physical structure. It originates at the waist and falls to the floor, maintaining consistent angle, shadow falloff, and spacing dictated by gravity. When a pleat is split across two independent tiles, each tile is denoised without visibility into the adjacent latent field. The model's fabric prior generates plausible folds in both tiles, but with no phase alignment. When stitched back together, two disconnected sets of pleats collide at the boundary. Alpha feathering across tile boundaries softens the seam but blends out-of-phase geometric lines into a blurry patch that reads as defective tailoring. Tiling fails whenever pixel values depend on structures extending beyond the tile window. Because continuous draping requires long-range structural coherence, we crop directly around the garment region with surrounding context, run inference at native resolution, and composite the result back into the full frame. ```mermaid flowchart LR I[source frame] --> W[roi window] M[mask] --> W W --> U[upscale] U --> P[inpaint] P --> D[downscale] D --> C{composite} I --> C M --> C ``` --- ## Step 1: Cropping and padding the mask We compute the bounding box of the garment mask and expand it with proportional padding. Sufficient surrounding context is necessary so the model can align the garment against body anatomy, such as shoulders and hips. We use `rho = 0.2`. ```python # pipeline/context_window.py """Crop, upscale, inpaint and composite: the context window mechanism. The unit of inference is the garment region, not the frame. The model processes a high-density crop while the delivered asset maintains full source resolution. """ from __future__ import annotations from dataclasses import dataclass import numpy as np NATIVE_RESOLUTIONS = ((1024, 1024), (1344, 1344)) ROI_PADDING = 0.2 @dataclass(frozen=True) class Window: """A crop rectangle in the coordinate space of the source image.""" x0: int y0: int x1: int y1: int @property def size(self) -> tuple[int, int]: return self.x1 - self.x0, self.y1 - self.y0 def roi_from_mask(mask: np.ndarray, rho: float = ROI_PADDING) -> Window: """Bounding box of the mask, padded by rho and clamped to the frame. rho is a fraction of the box dimension, applied on every side, so a tall narrow mask gets proportionally more horizontal context than a fixed pixel pad would provide. """ ys, xs = np.nonzero(mask) if ys.size == 0: raise ValueError("empty mask: nothing to inpaint") y0, y1 = int(ys.min()), int(ys.max()) x0, x1 = int(xs.min()), int(xs.max()) pad_y = int((y1 - y0) * rho) pad_x = int((x1 - x0) * rho) h, w = mask.shape[:2] window = Window( x0=max(0, x0 - pad_x), y0=max(0, y0 - pad_y), x1=min(w, x1 + pad_x), y1=min(h, y1 + pad_y), ) print(f"mask box: {(x0, y0, x1, y1)}") print(f"window: {(window.x0, window.y0, window.x1, window.y1)} size={window.size}") return window def pick_native(window: Window) -> tuple[int, int]: """Choose the inference resolution whose aspect is closest to the crop.""" crop_w, crop_h = window.size crop_aspect = crop_w / crop_h return min( NATIVE_RESOLUTIONS, key=lambda r: abs((r[0] / r[1]) - crop_aspect), ) ``` Padding by a fraction of the bounding box rather than a fixed pixel count accommodates vertical silhouettes. Saree masks are narrow and tall; fixed pixel padding starves the sides of torso context, which the diffusion model requires to anchor the drape against the shoulders. --- ## Step 2: Upscaling the crop to native resolution The cropped region is resized to match standard model dimensions (1024x1024 or 1344x1344), chosen via `pick_native` by aspect ratio proximity. The garment now fills the majority of the latent space rather than a third of the canvas, giving delicate pleat lines sufficient latent tokens to avoid collapsing during denoising. --- ## Step 3: Inpainting only the garment window The inpainting model runs directly on the isolated crop using the specified mask, conditioning, and adapter weights. ```python # pipeline/context_window.py (continued) INFERENCE = { "primary_steps": 30, "refinement_steps": 10, "sampler": "euler_ancestral", "scheduler": "beta", "vae_dtype": "fp16", "mask_dilation_px": 12, # bleeding zone, tuned per garment between 5 and 15 } def run_window( image: np.ndarray, mask: np.ndarray, pipe, conditioning, ) -> np.ndarray: """Inpaint the garment region at native resolution and return the crop.""" window = roi_from_mask(mask) target = pick_native(window) crop = resize(image[window.y0:window.y1, window.x0:window.x1], target) crop_mask = resize(mask[window.y0:window.y1, window.x0:window.x1], target) crop_mask = dilate(crop_mask, radius=INFERENCE["mask_dilation_px"]) generated = pipe( image=crop, mask_image=crop_mask, num_inference_steps=INFERENCE["primary_steps"], **conditioning, ).images[0] generated = pipe( image=generated, mask_image=crop_mask, num_inference_steps=INFERENCE["refinement_steps"], strength=0.25, **conditioning, ).images[0] print(f"inference at {target}, window {window.size}") return np.asarray(generated) ``` We execute 30 primary diffusion steps followed by 10 refinement steps at 0.25 denoise strength. Dilating the mask by 5 to 15 pixels creates a boundary transition zone where the model sees adjacent skin tones, generating a natural boundary rather than a hard composite line. ### Profiling VRAM allocation per stage Tracking `torch.cuda.memory_allocated()` across each stage on an RTX 4090 (24,576 MB total) reveals where the memory savings occur: | Pipeline Stage | Full-Frame Native (2048x3072) | ROI Window Crop (1024x1024) | VRAM Delta | |---|---|---|---| | Model Weights Resident (FP16 Flux DiT) | 11.90 GB | 11.90 GB | 0 GB | | Latent Spatial Grid Tensor ($B \times C \times H/8 \times W/8$) | 4.72 GB | 0.52 GB | -4.20 GB | | Cross-Attention Activation Buffers | 7.20 GB | 1.48 GB | -5.72 GB | | VAE Decoder Allocation | 4.80 GB (OOM spike) | 1.10 GB | -3.70 GB | | **Peak Allocated VRAM** | **28.62 GB (CUDA OOM)** | **15.00 GB (Safe headroom)** | **-13.62 GB** | | Total Inference Latency (30+10 steps) | Failed | 11.84s | n/a | ```text [INFO] mask box detected: (y0=312, y1=1840, x0=420, x1=890) size=(470, 1528) [INFO] padding rho=0.2 applied: window=(y0=0, y1=2145, x0=326, x1=984) [INFO] allocated before inpaint: 12,240.50 MB | max cached: 14,880.00 MB [INFO] running diffusion at (1024, 1024), 30 primary + 10 refinement steps [INFO] pass completed in 11.84s | peak memory: 15,360.25 MB ``` --- ## Step 4: Compositing across the mask boundary The generated crop is downscaled back to its source bounding box dimensions and composited into the original high-resolution frame. The surrounding background remains untouched. Compositing generated pixels into a photographic plate introduces two specific boundary challenges: | Boundary Issue | Visual Manifestation | Resolution | |---|---|---| | Window edge discontinuity | Faint rectangular boundary outline | Feather alpha over the dilated mask boundary rather than the bounding box rectangle | | Resampling noise mismatch | Slight difference in grain between downscaled generation and original plate | Match film grain or position padding boundaries over smooth skin and flat backdrops | Downscaling generated latents acts as a low-pass filter, leaving the output slightly smoother than the surrounding photograph. Alpha blending over the dilated segmentation mask prevents rectangular boundary artifacts from appearing. --- ## Failure modes and edge cases 1. **Unchecked boundary seams:** Visual inspection by raters catches boundary artifacts late in the pipeline. Automated edge-variance checks should validate transitions before assets leave inference workers. 2. **Low-resolution reference photography:** The context window preserves existing reference detail; it cannot synthesize clarity from blurry input photos. 2K or 4K source assets produce significantly crisper outputs. 3. **Multi-window garments:** Garments with sprawling dimensions that exceed a single aspect crop require multi-window coordination, which introduces cross-window blending complexity. --- ## When this is the wrong choice - **The garment already fills the frame.** On a tight portrait or torso crop there is almost no backdrop to skip. You get no memory back and you have added a compositing boundary to defend. - **You have 48GB cards.** On an L40S or an A100, the full frame fits and never OOMs. Every seam problem in Step 4 exists only because we chose to crop. - **The garment has no long-range structure.** Plain t-shirts and shorts have no pleat phase to keep aligned across a boundary, so an ordinary tiled upscaler works and is less code. --- Source: https://himanshuat.com/blogs/fitting-a-saree-into-24gb ======================================================================== --- title: "Two streams: why one diffusion pass couldn't do virtual try-on" description: "Masking a torso and inpainting a garment demos well in an afternoon. Getting back the exact garment, with its weave and its print placement intact, is a different problem. One conditioning stream could not hold both the body and the reference, so Flux-VTON+ runs two." date: "March 18, 2025" url: "https://himanshuat.com/blogs/flux-fill-and-redux-two-streams" --- # Two streams: why one diffusion pass couldn't do virtual try-on Masking a torso and prompting for a red saree in Flux Fill produces a plausible red saree. However, it will not reproduce the specific garment shipped by a brand. For e-commerce catalog production, that discrepancy invalidates the asset. > Text prompts cannot carry weave density or exact pattern alignment. Describing complex embroidery in text still leads the model toward whatever generic fabric priors dominated its pre-training data. One conditioning stream cannot simultaneously anchor body anatomy and enforce reference garment fidelity, which led us to design the dual-stream architecture in Flux-VTON+. ## Division of responsibilities across conditioning streams A conditioning stream supplies guidance to the diffusion sampler. Flux Fill takes an image, an inpainting mask, and a text prompt, filling the masked area with content consistent with surrounding pixels. Given a model's torso and the prompt "red saree", it generates a generic saree. It lacks any representation of the specific gold zari border, thread count, or motif repeat. Flux Redux provides the second conditioning stream. It encodes the reference garment directly, feeding high-dimensional structural tokens into the transformer's cross-attention layers. The division of labor is separated by function: | Stream | Domain | Responsibility | Limitation | |---|---|---|---| | Flux Fill | Spatial positioning | Body contours, pose alignment, lighting on skin, garment boundaries | Cannot infer garment identity | | Flux Redux | Garment structure | Weave texture, print repeats, structural reference details | Cannot determine body placement | Neither stream can fulfill the other's role. --- ## Step 1: Identifying style drift and spatial distortion Single-stream pipelines fail in two distinct ways: **Style drift** happens under inpainting-only conditioning. The output matches the reference in base color and rough silhouette but loses structural detail. A cable-knit sweater flattens into a plain red top, and a hand-blocked motif degrades into a generic floral print. Low-frequency color persists, but high-frequency fabric structure is lost. **Spatial distortion and amputation** happens when you condition heavily on reference structure without enough spatial anchoring. The model renders an accurate garment texture that disregards the subject's physical anatomy. Drapes ignore shoulder positions, hem lengths shift arbitrarily, and hands resting on hips get painted over with fabric. **The tell:** if the color survived and the structure didn't, the reference stream is too weak. If the texture is right but it sits in the wrong place on the body, the mask is the problem. --- ## Step 2: Segmenting with SAM2 and building the transition zone Because inpainting relies on precise spatial boundaries, faulty masks degrade both conditioning streams. We use Segment Anything 2 (SAM2) with hierarchical prompts, followed by morphological dilation to create a transition zone. ```python # mask_garment.py """Produce the inpainting mask for a source image. SAM2 provides a tight garment boundary. Morphological dilation creates a transition zone that the sampler repaints, allowing new fabric to meet skin without a visible seam. """ import cv2 import numpy as np import torch from sam2.build_sam import build_sam2 from sam2.sam2_image_predictor import SAM2ImagePredictor GARMENT_PROMPTS = ["shirt", "tshirt", "top", "saree", "dress"] # Transition zone radius in pixels at inference crop resolution. # 5 for tight-fitting tops, up to 15 for flowing drapes. DILATION_PX = 9 # Minimum mask fraction to filter out false segmentation hits. MIN_MASK_FRACTION = 0.04 def build_predictor(checkpoint: str, config: str) -> SAM2ImagePredictor: """Load SAM2 once per worker; the encoder pass is compute-heavy.""" model = build_sam2(config, checkpoint, device="cuda") return SAM2ImagePredictor(model) def segment_garment(predictor, image_bgr: np.ndarray, prompts=GARMENT_PROMPTS): """Run hierarchical prompts and select the highest-scoring mask.""" image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB) predictor.set_image(image_rgb) best_mask, best_score, best_prompt = None, -1.0, None for prompt in prompts: masks, scores, _ = predictor.predict(text=prompt, multimask_output=True) top = int(np.argmax(scores)) if scores[top] > best_score: best_mask, best_score, best_prompt = masks[top], float(scores[top]), prompt return best_mask.astype(np.uint8), best_score, best_prompt def dilate(mask: np.ndarray, radius_px: int = DILATION_PX) -> np.ndarray: """Expand mask by radius_px using an elliptical structuring element.""" k = 2 * radius_px + 1 element = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (k, k)) return cv2.dilate(mask, element, iterations=1) def prepare_mask(predictor, image_bgr, radius_px=DILATION_PX): raw, score, prompt = segment_garment(predictor, image_bgr) coverage = float(raw.mean()) if coverage < MIN_MASK_FRACTION: raise ValueError(f"Mask too small for '{prompt}': coverage={coverage:.3f}") final = dilate(raw, radius_px) grown = float(final.mean()) - coverage print(f"prompt={prompt} sam2_score={score:.3f} coverage={coverage:.3f} " f"radius={radius_px}px grown_by={grown:.3f}") return final ``` Dilation radius directly affects edge quality. If the transition zone is too narrow, the sampler cannot observe surrounding skin pixels, preventing natural lighting integration and creating a pasted-on look. If the zone is too wide, the mask erases anatomical references like necklines and shoulders, encouraging the model to synthesize distorted anatomy. A range of 5 to 15 pixels accommodates most garments, tuned per category based on drape flexibility. --- ## Step 3: Extracting structural tokens with Flux Redux Standard CLIP vision encoders compress reference imagery into high-level semantic embeddings, effectively reducing an image to a text-like caption. Flux Redux preserves intermediate spatial token representations, allowing micro-textures, print repeats, and logo typography to pass through the bottleneck. ```python # reference_stream.py """Encode the reference garment into structural conditioning tokens. C_style = ReduxEncoder(SigCLIP(I_ref)) Garment backgrounds are removed before encoding to prevent studio sweeps or mannequin textures from polluting cross-attention tokens. """ from dataclasses import dataclass import torch from PIL import Image @dataclass class ReferenceStreamConfig: redux_strength: float = 0.85 # scales C_style before cross-attention cutout_feather_px: int = 3 # softens RMBG alpha edge square_pad: bool = True # pads rather than crops to prevent pattern distortion class ReferenceStream: def __init__(self, siglip, redux, rmbg, config: ReferenceStreamConfig): self.siglip = siglip self.redux = redux self.rmbg = rmbg self.config = config def cutout(self, ref: Image.Image) -> Image.Image: """Isolate garment on transparent background using RMBG.""" alpha = self.rmbg(ref) garment = ref.convert("RGBA") garment.putalpha(alpha.filter_edge(self.config.cutout_feather_px)) if self.config.square_pad: garment = pad_to_square(garment) return garment @torch.inference_mode() def encode(self, ref: Image.Image) -> torch.Tensor: """Return C_style tokens for injection into cross-attention.""" garment = self.cutout(ref) pixel_values = self.siglip.preprocess(garment).to("cuda", torch.bfloat16) # SigCLIP extracts visual features; Redux maps them into transformer tokens features = self.siglip(pixel_values) c_style = self.redux(features) c_style = c_style * self.config.redux_strength print(f"c_style tokens={tuple(c_style.shape)} strength={self.config.redux_strength}") return c_style ``` `redux_strength` balances reference fidelity against spatial flexibility. Increasing strength improves texture accuracy until it begins overriding body pose, imprinting flat-lay geometry onto standing subjects. Decreasing strength reintroduces style drift. Background isolation with RMBG is mandatory: background studio props or hanger artifacts in reference photos otherwise bleed into cross-attention conditioning. --- ## Step 4: Two-pass sampling with refinement Both conditioning paths feed into a unified sampling pipeline, followed by a localized low-denoise refinement pass over the garment mask. ```mermaid flowchart LR R[reference] --> C[RMBG cutout] C --> S[SigCLIP + Redux] I[source image] --> M[SAM2 mask] M --> D[dilate] S --> F[Flux Fill sampler] D --> F F --> P{soft edge?} P -->|refine| F P -->|done| V[VAE decode] ``` ```python # run_vton.py """Dual-stream try-on pass: Fill conditioning on the masked body, Redux conditioning on the reference garment, unified sampler with refinement. """ SAMPLER = { "sampler_name": "euler_ancestral", "scheduler": "beta", "cfg": 3.5, } PRIMARY = {**SAMPLER, "steps": 30, "denoise": 1.0} REFINE = {**SAMPLER, "steps": 10, "denoise": 0.28} # FP16 VAE decoding prevents color blotching on large solid-color fabrics VAE_DTYPE = "fp16" def try_on(pipe, source, mask, reference, ref_stream, prompt: str): """Execute dual-stream inpainting and return composited image.""" c_style = ref_stream.encode(reference) latents = pipe.fill( image=source, mask_image=mask, prompt=prompt, style_conditioning=c_style, **PRIMARY, ) # Low-denoise refinement pass settles boundary transitions and fine threads latents = pipe.fill( image=latents, mask_image=mask, prompt=prompt, style_conditioning=c_style, **REFINE, ) image = pipe.decode(latents, vae_dtype=VAE_DTYPE) print(f"primary_steps={PRIMARY['steps']} refine_steps={REFINE['steps']} " f"sampler={SAMPLER['sampler_name']}/{SAMPLER['scheduler']}") return image ``` We select Euler Ancestral with a Beta scheduler because ancestral noise injection preserves the stochastic micro-roughness of natural textiles, whereas deterministic samplers tend to smooth out subtle weaves. Thirty primary steps combined with ten refinement steps consistently settled boundary artifacts. --- ## Step 5: Disentangling architectural gains from baselines In our benchmark comparisons against ControlNet and T2I-Adapter baselines on SDXL, spatial conditioning alone proved insufficient for texture synthesis. While pose and depth control maintain body geometry, they do not resolve fabric texture transfer. On our 500-image evaluation dataset, SDXL with ControlNet achieved an SSIM of 0.45 on Global Traditional garments, compared to 0.85 for the dual-stream Flux-VTON+ pipeline. For Western Casual garments, the baseline gap was considerably narrower (0.82 vs 0.94), confirming that standard diffusion priors already handle simple fitted apparel. --- ## Failure modes and edge cases 1. **Sheer and semi-transparent fabrics:** Chiffon and lace require alpha blending of underlying skin tones rather than solid inpainting. Dual-stream diffusion currently renders sheer fabrics mostly opaque. 2. **Dynamic athletic poses:** Severe articulation (such as yoga or dance) stresses both SAM2 segmentation and spatial conditioning, resulting in local geometry warping. 3. **Hyperparameter coupling:** Dilation radius and Redux conditioning strength interact. Expanding the dilation zone gives the sampler more freedom, increasing the likelihood that strong Redux conditioning distorts body anatomy. --- ## When this is the wrong choice - **There is no specific garment to reproduce.** Mood boards and concept apparel have no manufacturing reference to be faithful to. "Red saree" as a prompt is the whole requirement, and single-stream Flux Fill answers it. - **The fabric is flat and solid.** A plain cotton has no weave, no print repeat, and no motif alignment. Redux tokens are carrying nothing, and you pay for the encode anyway. - **The reference photo will not segment cleanly.** RMBG has to isolate the garment before encoding. If the reference is a hanger shot against a busy background, the second stream feeds studio props into cross-attention and makes the output worse than one stream would have. --- Source: https://himanshuat.com/blogs/flux-fill-and-redux-two-streams ======================================================================== --- title: "From a ComfyUI graph to an API a brand's team can call" description: "Our try-on pipeline lived as a ComfyUI graph with implicit state, hand-picked seeds, and custom nodes tracking whatever was on main that week. Turning it into something another company's engineers could call meant writing down which code ran, which seed, when the work happens, and what we refuse." date: "September 09, 2025" url: "https://himanshuat.com/blogs/from-comfyui-graph-to-an-api" --- # From a ComfyUI graph to an API a brand's team can call During one integration week, our endpoint received three distinct inputs: a 4K studio flat-lay of a lehenga, a smartphone photo of a kurta hanging on a door, and a PNG with an alpha channel. All three came from engineers who reasonably assumed a virtual try-on API accepts any image of clothing. The alpha channel crashed our PIL preprocessor, resulting in a terminal traceback on my machine and a 504 gateway timeout for the caller. Running the workflow interactively in ComfyUI lets you intervene. You watch the browser tab, drop the denoise when fabric goes plastic, and re-seed until segmentation stops clipping fingers. Once beta API keys went out to design partners like GettoIndia and EcomBuddha, nobody was watching the tab. > A ComfyUI graph is not an API with the wrapper missing. What makes it callable by someone else is the part the graph never had to write down: which code ran, which seed, when the work happens, and what you refuse. ## Operational gaps between UI graphs and backend services Our initial API wrapper took a single day to build. ComfyUI natively exposes an HTTP endpoint accepting workflow graphs in JSON format. I built a lightweight Python service that loaded our saved workflow, injected incoming model and garment URLs into the `LoadImage` nodes, dispatched the execution, and polled for output images. While sufficient for internal demos, this setup failed across four areas when exposed to external engineering teams: | Failure Vector | Underlying System State | Impact on API Client | |---|---|---| | Floating dependencies | Custom nodes (SAM2, RMBG, LoRA merging) tracked `main` git branches | Unversioned, non-reproducible output across deployments | | Resident GPU state | ComfyUI process cached in-memory LoRA merges from earlier calls | Changing weights on disk without restarting workers served stale cached layers | | Unstructured exceptions | VRAM allocations, aspect-ratio mismatches, and empty masks threw Python tracebacks | HTTP timeouts without actionable error messages | | Missing input validation | No formal constraints on color spaces, dimensions, or apparel categories | Silent generation failures on unsupported garments | To transition from an internal research tool to an enterprise service, we defined four operational guarantees: explicit code manifests, deterministic seeding, asynchronous job processing, and rigid input validation. --- ## Step 1: Versioning execution manifests rather than UI graphs We replaced loose workflow JSON files with comprehensive manifests that track model weight hashes, exact git commits for all node packages, and fixed merge hyperparameters. ```json // manifests/vton-2025-09-a.json { "manifest_id": "vton-2025-09-a", "created": "2025-09-02", "graph": { "file": "graphs/flux_vton_plus.api.json", "sha256": "9f2c1e0b7a4d3f88c5b6e2a1d0c9f4b3e8a7d6c5b4a39281706f5e4d3c2b1a09" }, "nodes": [ { "repo": "comfyui-sam2", "commit": "4b1e9c2" }, { "repo": "comfyui-rmbg", "commit": "a07d331" }, { "repo": "comfyui-essentials", "commit": "c19f4ab" } ], "models": [ { "role": "base", "file": "flux_fill.safetensors", "sha256": "1a2b3c4d" }, { "role": "redux", "file": "flux_redux.safetensors", "sha256": "5e6f7a8b" }, { "role": "vae", "file": "ae.safetensors", "sha256": "9c0d1e2f", "dtype": "fp16" }, { "role": "lora_drape", "file": "drape_physics_r32.safetensors", "sha256": "3a4b5c6d" }, { "role": "lora_occlusion", "file": "occlusion_depth_r32.safetensors", "sha256": "7e8f9a0b" } ], "merge": { "node": "model_merge_lora", "lambda_drape": 0.6, "lambda_occlusion": 0.4 }, "sampling": { "sampler": "euler_ancestral", "scheduler": "beta", "steps_primary": 30, "steps_refine": 10 }, "context_window": { "roi_padding": 0.2, "mask_dilation_px": 9, "inference_resolution": [1344, 1344] } } ``` Manifests eliminate ambiguity around adapter ratios (such as 0.6 drape and 0.4 occlusion). Every API request pins a specific manifest ID, and responses echo the exact manifest executed. Old manifests remain pinned in production as long as their corresponding weights reside on disk, preventing silent breaking changes for live client catalogs. --- ## Step 2: Deriving deterministic seeds from request payloads Interactive diffusion relies on randomized seeds to generate candidate varieties. In a production API, randomized execution prevents caching, invalidates regression test suites, and makes customer bugs unreproducible. ```python # api/seeding.py import hashlib import json from dataclasses import dataclass @dataclass class TryOnRequest: """A single try-on job received from an API caller.""" request_id: str model_image_key: str garment_image_key: str category: str manifest_id: str seed: int | None = None def derive_seed(req: TryOnRequest) -> int: """Return a deterministic seed for the request. If the caller provides an explicit seed, we use it directly to support exact replays. Otherwise, we hash the request payload into a uint32. Worker retries following transient hardware failures land on the identical seed rather than altering fabric drape. """ if req.seed is not None: return req.seed % (2 ** 32) payload = json.dumps( { "model": req.model_image_key, "garment": req.garment_image_key, "category": req.category, "manifest": req.manifest_id, }, sort_keys=True, ).encode() return int.from_bytes(hashlib.sha256(payload).digest()[:4], "big") def apply_seed(graph: dict, seed: int) -> dict: """Inject the computed seed into all sampler nodes and lock randomisation.""" for node in graph.values(): if node.get("class_type") in {"KSampler", "KSamplerAdvanced"}: node["inputs"]["seed"] = seed node["inputs"]["noise_seed"] = seed node["inputs"]["control_after_generate"] = "fixed" return graph ``` Identical seeds run on matching GPU architectures with identical drivers yield visually consistent renders. Responses always return the derived seed so brand creative directors can store approved seeds across SKU batches. --- ## Step 3: Asynchronous job queues and worker leasing A full-body try-on pass takes approximately 12 seconds on an RTX 4090. Holding synchronous HTTP connections open for that duration creates connection pool exhaustion and exposes clients to gateway timeouts. We designed the service around asynchronous job polling and webhooks using Redis for queue management. ```mermaid flowchart LR A[submit] --> B{under limit} B -->|no| R[rejected] B -->|yes| Q[202 queued] Q --> W[worker lease] W -->|crash| Q W --> D[result] D --> P[poll or webhook] ``` ```python # api/queue.py import json import time import uuid import redis r = redis.Redis(decode_responses=True) QUEUE_KEY = "vton:pending" LEASE_TTL_SECONDS = 180 MAX_INFLIGHT_PER_KEY = 4 def submit(api_key: str, payload: dict) -> dict: """Validate concurrency and register a queued job.""" inflight = r.scard(f"vton:inflight:{api_key}") if inflight >= MAX_INFLIGHT_PER_KEY: return {"status": "rejected", "reason": "concurrency_limit"} job_id = f"job_{uuid.uuid4().hex[:12]}" record = { "job_id": job_id, "api_key": api_key, "status": "queued", "manifest_id": payload["manifest_id"], "submitted_at": time.time(), "payload": payload, } r.hset(f"vton:job:{job_id}", mapping={"data": json.dumps(record)}) r.sadd(f"vton:inflight:{api_key}", job_id) r.lpush(QUEUE_KEY, job_id) return {"job_id": job_id, "status": "queued", "queue_depth": r.llen(QUEUE_KEY)} def claim(worker_id: str) -> dict | None: """Lease a job from the pending queue. If a worker crashes mid-inference, the lease expires and an automated sweeper requeues the job. Deterministic seeding ensures that the retry generates the expected image. """ job_id = r.brpoplpush(QUEUE_KEY, f"vton:leased:{worker_id}", timeout=30) if job_id is None: return None r.setex(f"vton:lease:{job_id}", LEASE_TTL_SECONDS, worker_id) raw = r.hget(f"vton:job:{job_id}", "data") record = json.loads(raw) record["status"] = "running" r.hset(f"vton:job:{job_id}", mapping={"data": json.dumps(record)}) return record ``` Our architecture uses Next.js for client interfaces, NestJS for REST APIs and PostgreSQL metadata management, Redis for leasing queues, S3 for object storage, and headless Python workers dedicated strictly to consuming jobs from the queue. --- ## Step 4: Explicit API input contracts and rejection policies Rather than attempting to render unsupported edge cases and producing broken outputs, the API validates inputs up front and returns descriptive HTTP 422 errors. ```yaml # api/surface.yaml categories: supported: - dresses_one_pieces - tops - bottoms planned: - intimates_swimwear - coords_sets input_requirements: model_image: color_space: rgb # alpha channels rejected during pre-validation min_short_side: 1024 max_faces: 1 garment_image: background: any # RMBG segments background prior to conditioning on_body_reference: preferred rejections: category_unsupported: http: 422 hint: "This category has no specialised model yet. See categories.planned." no_garment_found: http: 422 hint: "Segmentation returned an empty mask for the requested category." sheer_fabric_unsupported: http: 422 hint: "Lace and chiffon currently render opaque. Not accepted." pose_out_of_distribution: http: 422 hint: "Non-standing, non-seated poses are outside the trained range." ``` Returning a clear 422 status code with diagnostic hints immediately informs client developers of input issues, preventing billing disputes over corrupted renders. --- ## Failure modes and edge cases 1. **Missing idempotency keys:** Network timeouts during job submission caused clients to retry requests, occasionally generating duplicate GPU tasks. Adding request idempotency keys resolved duplicate submissions. 2. **Aggressive webhook retry loops:** Webhook delivery engines must use exponential backoff with circuit breakers to avoid overwhelming client staging servers during outages. 3. **Unbounded manifest retention:** Keeping every experimental manifest deployable indefinitely leads to disk bloat on inference workers. Deprecation schedules must accompany version releases. --- ## When this is the wrong choice - **The graph is still changing shape.** While you are testing node architectures, rewiring a canvas beats updating a manifest schema, a seed derivation, and a rejection table for a topology you will throw away. - **One operator reviews every output.** An internal studio where an artist looks at each render can tune parameters by hand. Determinism buys you reproducibility for bugs nobody is going to file. - **The pass finishes in under half a second.** A synchronous HTTP endpoint is fine at that latency. Leasing queues, webhook retries, and idempotency keys exist because a full-body try-on pass takes about 12 seconds on a 4090. - **You control both sides of the call.** Manifests, 422 hints, and input contracts are for engineers who cannot ask you what the endpoint accepts. --- Source: https://himanshuat.com/blogs/from-comfyui-graph-to-an-api ======================================================================== --- title: "Graph Agents and When the Graph Earns Its Keep" description: "A graph framework introduces an additional control flow abstraction. Here is when cyclic state machines, checkpoint durability, and human approval justify graph infrastructure over plain loops." date: "May 12, 2026" url: "https://himanshuat.com/blogs/graph-agents-when-the-graph-earns-its-keep" --- # Graph Agents and When the Graph Earns Its Keep A node configured with a static `add_edge` also returned `Command(goto=...)`. Both fired during execution, the downstream node ran twice, and the paid API tool calls inside it were billed twice. A graph framework is a control flow layer bolted on top of Python, and it charges rent. > The graph earns its keep when a workflow needs cycles, durable checkpoints, and a human approval gate at the same time. With one of the three, a while loop and a queue are less to maintain. ### Chapter 0: Super-steps and Pregel execution Most diagrams draw [LangGraph](https://docs.langchain.com/oss/python/langgraph/graph-api) workflows as directed acyclic graphs. The runtime underneath is [Pregel](https://research.google/pubs/pregel-a-system-for-large-scale-graph-processing/) message passing — Malewicz and co-authors' 2010 SIGMOD paper on Google's large-scale graph processing system, where computation advances in synchronized super-steps — and the difference shows up the first time you debug a resume. The API details below track LangGraph 1.2, the current release line. Execution advances through discrete **super-steps**: 1. A node activates upon receiving inbound state or messages. 2. The node runs its function, writes state updates, and yields control. 3. The runtime evaluates transitions and dispatches state to downstream nodes. 4. Execution terminates when no nodes are active and no messages remain in transit. Nodes activated concurrently in the same super-step execute in parallel. Sequential transitions span across distinct super-steps. Three behaviors follow from that: - Concurrent writes to the same state key conflict when executed within the same super-step. - On process resumption from a checkpoint, the entire node re-executes because the super-step is the atomic unit of recovery. - `recursion_limit` tracks completed super-steps rather than individual LLM API calls. --- ### Step 1: Design the state schema before node logic Define the state schema using `TypedDict`. Field annotations dictate how data accumulates across super-steps. ```python # graph.py import operator from typing import Annotated, TypedDict from langchain_core.messages import AnyMessage from langgraph.graph import END, START, StateGraph from langgraph.graph.message import add_messages class ResearchState(TypedDict): """State persisted across research graph super-steps.""" messages: Annotated[list[AnyMessage], add_messages] findings: Annotated[list[str], operator.add] question: str verdict: str class ResearchInput(TypedDict): """Input contract accepted by the graph.""" question: str class ResearchOutput(TypedDict): """Output schema returned to external callers.""" verdict: str findings: list[str] def plan(state: ResearchState) -> dict: """Generates search plan from input question.""" return {"messages": [("assistant", f"Plan for: {state['question']}")]} def search(state: ResearchState) -> dict: """Retrieves document evidence.""" return {"findings": [f"Evidence for {state['question']}"]} def critique(state: ResearchState) -> dict: """Evaluates sufficiency of findings.""" is_sufficient = len(state.get("findings", [])) >= 2 return {"verdict": "sufficient" if is_sufficient else "insufficient"} def route(state: ResearchState) -> str: """Evaluates state to loop back or terminate.""" return END if state["verdict"] == "sufficient" else "search" builder = StateGraph( ResearchState, input_schema=ResearchInput, output_schema=ResearchOutput, ) builder.add_node("plan", plan) builder.add_node("search", search) builder.add_node("critique", critique) builder.add_edge(START, "plan") builder.add_edge("plan", "search") builder.add_edge("search", "critique") builder.add_conditional_edges("critique", route, ["search", END]) graph = builder.compile() ``` The compiled graph enforces a cycle: ```mermaid flowchart LR ST([START]) --> P[plan] P --> S[search] S --> C{critique} C -->|insufficient| S C -->|sufficient| E([END]) ``` `input_schema` and `output_schema` restrict public interfaces while allowing internal graph state to carry execution scratchpads. Channel reducers manage mutation semantics: `add_messages` merges messages by ID, and `operator.add` concatenates lists. Keys without reducers use last-write-wins overwrite semantics. --- ### Step 2: Define reducer conflict policies explicitly When multiple nodes activate concurrently in the same super-step, writing to unreduced keys causes runtime exceptions: ```python # fanout_conflict.py from typing import TypedDict from langgraph.graph import START, StateGraph class State(TypedDict): topic: str notes: list[str] def west(state: State) -> dict: return {"notes": ["west: p99 latency normal"]} def east(state: State) -> dict: return {"notes": ["east: p99 latency elevated"]} builder = StateGraph(State) builder.add_node("west", west) builder.add_node("east", east) builder.add_edge(START, "west") builder.add_edge(START, "east") graph = builder.compile() # graph.invoke({"topic": "p99", "notes": []}) # Raises: langgraph.errors.InvalidUpdateError (Can receive only one value per step) ``` Adding `Annotated[list[str], operator.add]` resolves the error by defining an append policy. However, `operator.add` appends updates in whatever order concurrent nodes complete. When deterministic ordering matters, define a custom reducer function that sorts incoming values by node ID or timestamp before writing to state. --- ### Step 3: Dynamic fan-out using Send When the number of parallel tasks is determined at runtime, use the [`Send`](https://reference.langchain.com/python/langgraph/types/Send/) primitive instead of fixed static edges. LangGraph documents it as the way to invoke a node with a custom state that need not match the main graph's schema. ```python # fanout_send.py import operator from typing import Annotated, TypedDict from langgraph.graph import END, START, StateGraph from langgraph.types import Send class MapState(TypedDict): documents: list[str] summaries: Annotated[list[str], operator.add] report: str class WorkerState(TypedDict): document: str def dispatch(state: MapState) -> list[Send]: """Spawns parallel worker nodes for each document in the list.""" return [Send("summarise", {"document": doc}) for doc in state["documents"]] def summarise(state: WorkerState) -> dict: return {"summaries": [f"Summary: {state['document'][:30]}"]} def reduce_report(state: MapState) -> dict: return {"report": "\n".join(state["summaries"])} builder = StateGraph(MapState) builder.add_node("summarise", summarise) builder.add_node("reduce_report", reduce_report) builder.add_conditional_edges(START, dispatch, ["summarise"]) builder.add_edge("summarise", "reduce_report") builder.add_edge("reduce_report", END) graph = builder.compile() ``` Each `Send` instance delivers an isolated payload to its target node: ```mermaid flowchart LR D[dispatch] --> S1[summarise doc 1] D --> S2[summarise doc 2] D --> S3[summarise doc 3] S1 --> R[reduce_report] S2 --> R S3 --> R ``` --- ### Step 4: Typed routing with Command and Literal targets The [`Command`](https://reference.langchain.com/python/langgraph/types/Command/) object combines state updates and dynamic edge transitions into a single return type: ```python # supervisor.py from typing import Literal, TypedDict from langgraph.graph import END, START, StateGraph from langgraph.types import Command class TeamState(TypedDict): task: str draft: str review_notes: str def supervisor(state: TeamState) -> Command[Literal["writer", "reviewer", "__end__"]]: """Selects the next node and updates task metadata.""" if not state["draft"]: return Command(update={"task": state["task"].strip()}, goto="writer") if not state["review_notes"]: return Command(goto="reviewer") return Command(goto=END) def writer(state: TeamState) -> Command[Literal["supervisor"]]: return Command(update={"draft": "Draft body"}, goto="supervisor") def reviewer(state: TeamState) -> Command[Literal["supervisor"]]: return Command(update={"review_notes": "Validation notes"}, goto="supervisor") builder = StateGraph(TeamState) builder.add_node("supervisor", supervisor) builder.add_node("writer", writer) builder.add_node("reviewer", reviewer) builder.add_edge(START, "supervisor") graph = builder.compile() ``` Using `Literal` in return type annotations allows static analyzers and graph visualization tools to trace potential paths. Avoid combining static `builder.add_edge("node_a", "node_b")` with a dynamic `Command(goto="node_b")` on the same node, as both transition paths will trigger duplicate executions. --- ### Step 5: Configure checkpoint durability and recursion budgets Persist execution state to handle node failures and support human approval interruptions: ```python # run.py from langgraph.checkpoint.sqlite import SqliteSaver from langgraph.errors import GraphRecursionError from graph import builder def execute_thread(question: str, thread_id: str) -> dict: with SqliteSaver.from_conn_string("checkpoints.db") as saver: graph = builder.compile(checkpointer=saver) config = { "configurable": {"thread_id": thread_id}, "recursion_limit": 25, } try: return graph.invoke({"question": question}, config, durability="sync") except GraphRecursionError: snapshot = graph.get_state(config) return { "verdict": "budget_exhausted", "findings": snapshot.values.get("findings", []), } ``` Key operational guidelines: - Set [`durability="sync"`](https://reference.langchain.com/python/langgraph/types/Durability) to guarantee that state is written to storage before transitioning to the next super-step. LangGraph's reference describes the `sync` mode as persisting changes "synchronously before the next step starts"; the alternatives are `async` and `exit`. - Lower `recursion_limit` from the default 1000 to a realistic bound (such as 25) to catch cyclical loops early. The [graph API docs](https://docs.langchain.com/oss/python/langgraph/graph-api) put that default at 1000 super-steps starting in LangGraph 1.0.6. - Keep state payloads compact. Store large document corpora or raw vector payloads in external object stores, keeping only record IDs and summaries in graph state. --- ### Step 6: Inspect state history rather than stack traces Because super-steps discard local execution frames upon completion, debug graph behavior by inspecting checkpoint history: ```python config = {"configurable": {"thread_id": "session-104"}} for state_snapshot in graph.get_state_history(config): print(f"Step: {state_snapshot.metadata.get('step')}") print(f"Node: {state_snapshot.metadata.get('source')}") print(f"State Values: {state_snapshot.values}\n---") ``` Tracing state mutations across snapshots isolates the exact super-step where state diverged from expected values. --- ### When this is the wrong choice - **The workflow is a straight line.** No cycle, no retry edge, nothing that routes backwards. A sequence of function calls does the same work, and you can read the control flow top to bottom instead of reconstructing it from `add_conditional_edges` calls. - **You never pause mid-run.** Durable checkpointing is what pays for the Pregel model. If a failed run is simply retried from the start, `SqliteSaver`, thread IDs, and `durability="sync"` are bookkeeping you maintain for a guarantee you do not use. - **State is large.** The super-step is the atomic unit of recovery, so the whole node re-executes on resume and the whole state is written between steps. Carry big payloads in graph state and you pay that write on every transition. --- ### Tradeoff summary | Architecture | Control flow mechanism | Primary tradeoff | |---|---|---| | Plain Python Loop | While loops with functions | Low overhead; requires custom checkpointing logic for pauses | | LangGraph | Pregel super-steps on StateGraph | Built-in durable checkpoints and pause/resume; requires state schema discipline | | [Pydantic Graph](https://pydantic.dev/docs/ai/graph/graph/) | Class-based nodes with typed transitions | Compile-time type safety; limited dynamic graph introspection | --- Source: https://himanshuat.com/blogs/graph-agents-when-the-graph-earns-its-keep ======================================================================== --- title: "LLM as judge, what the numbers actually say" description: "I let an LLM judge gate whether changes shipped because public benchmarks cited 80% human agreement. Six weeks in, an apparent performance improvement turned out to be judge variance across reruns. This is what that 80% figure measures and why chance-corrected agreement deflates headline metrics." date: "June 23, 2026" url: "https://himanshuat.com/blogs/llm-as-judge-what-the-numbers-actually-say" --- # LLM as judge, what the numbers actually say An automated LLM judge decided whether our production prompts shipped. I had implemented it because industry write-ups reported that LLM evaluators agreed with human annotators roughly 80% to 85% of the time. Six weeks later, a reported accuracy improvement turned out to be the evaluator flipping verdicts on a rerun. > A judge that agrees with human raters 85% of the time is not 85% accurate. It is 85% concordant with a group of humans that only agree with each other roughly 80% of the time on subjective criteria. Read the primary papers and the headline agreement percentage turns out to be hiding the statistical adjustment that matters. ### Chapter 0: What agreement metrics actually measure The 85% agreement figure originates from the MT-Bench and Chatbot Arena benchmark (Zheng et al., NeurIPS 2023). The dataset spans 80 multi-turn queries across 8 categories with 3,000 expert votes and 30,000 crowdsourced pairwise judgments. Excluding ties, GPT-4 pairwise judgments agreed with human experts 85% of the time. The accompanying baseline metric is frequently omitted: **human-to-human agreement was 81% to 82%.** The judge matches the concordance rate between two independent human annotators. It does not measure ground-truth correctness, because open-ended generation has no deterministic label to be correct against. Two domain experts disagree on roughly one subjective rating in five. The judge models the human preference distribution, and its ceiling is the clarity of your rubric. --- ### Step 1: Apply chance correction via Cohen's Kappa Raw percentage agreement inflates apparent reliability when the underlying label distribution is imbalanced. In large-scale empirical studies evaluating LLM judges across multiple providers and benchmarks (such as arXiv 2606.19544), **kappa deflation is consistent across models.** Between raw percentage agreement and Cohen's kappa on MT-Bench, agreement scores drop by 33 to 41 percentage points. Raw agreement credits instances where both the evaluator and the human select the winning candidate simply because one option dominates the dataset. Cohen's kappa ($\kappa$) subtracts the probability of agreement occurring by chance: $$ \kappa = \frac{P_o - P_e}{1 - P_e} $$ where $P_o$ is observed agreement and $P_e$ is expected chance agreement. | Cohen's Kappa Range | Landis and Koch Interpretation | |---|---| | 0.21 to 0.40 | Fair agreement | | 0.41 to 0.60 | Moderate agreement | | 0.61 to 0.80 | Substantial agreement | | 0.81 to 1.00 | Near-perfect agreement | Krippendorff's alpha ($\alpha$) extends this formulation to multi-annotator panels and missing data, setting standard statistical reliability thresholds at $\alpha \ge 0.800$. **The diagnostic check:** if your evaluation pipeline reports only raw percentage agreement, you are measuring label class imbalance rather than true scoring reliability. --- ### Step 2: Report metrics per evaluation criterion Aggregating scores across different evaluation criteria conceals severe per-task variance. In clinical and technical evaluation studies, aggregate kappa scores of 0.74 frequently mask individual sub-criteria spanning from 0.34 (barely above chance) to 0.97 (high consistency). ```python # agreement.py from collections import Counter def cohens_kappa(a: list[str], b: list[str]) -> float: """Chance-corrected agreement between two raters over identical items.""" if len(a) != len(b) or not a: raise ValueError("Rater vectors must be non-empty and of equal length.") n = len(a) observed = sum(1 for x, y in zip(a, b) if x == y) / n ca, cb = Counter(a), Counter(b) expected = sum((ca[k] / n) * (cb[k] / n) for k in set(a) | set(b)) if expected == 1.0: return 1.0 return (observed - expected) / (1 - expected) def per_criterion_report(labels: dict[str, tuple[list[str], list[str]]]) -> dict: """Computes kappa and raw agreement per criterion separately.""" out = {} for criterion, (human, judge) in labels.items(): raw = sum(1 for x, y in zip(human, judge) if x == y) / len(human) kappa = cohens_kappa(human, judge) out[criterion] = { "raw_agreement": round(raw, 3), "kappa": round(kappa, 3), "deflation_gap": round((raw - kappa) * 100, 1), } return out ``` Disaggregating tells you which criterion the judge is unreliable on, which is the one to rewrite. **The diagnostic check:** if you report one number for the whole rubric, you cannot tell a judge that is wrong everywhere from a judge that is wrong on one criterion. --- ### Step 3: Evaluate pairwise candidates in both presentation orders Position bias occurs when the evaluator systematically prefers the first or second candidate passage regardless of content. Empirical measurements of consistency when swapping candidate positions ($A/B$ vs. $B/A$): | Evaluator Model | Default Prompt Consistency | Anonymized Prompt Consistency | |---|---|---| | GPT-4 | 65.0% | 66.2% | | GPT-3.5 | 46.2% | 51.2% | | Claude-v1 | 23.8% | 56.2% | A consistency score of 50% indicates that reversing presentation order flips the verdict half the time. ```python # position_bias.py from dataclasses import dataclass from itertools import product @dataclass class PairJudgment: task_id: str order: str # "ab" or "ba" winner: str # "a", "b", or "tie" def calculate_consistency(judgments: list[PairJudgment]) -> float: """Calculates fraction of tasks where swapping presentation order did not alter winner.""" by_task: dict[str, dict[str, str]] = {} for j in judgments: by_task.setdefault(j.task_id, {})[j.order] = j.winner complete = [v for v in by_task.values() if "ab" in v and "ba" in v] if not complete: raise ValueError("No tasks evaluated in both presentation orders.") agree = sum(1 for v in complete if v["ab"] == v["ba"]) return agree / len(complete) def generate_swapped_pairs(tasks: list[str], answers: dict[str, tuple[str, str]]): """Emits every candidate pair in both presentation orders.""" for task_id, order in product(tasks, ("ab", "ba")): a, b = answers[task_id] yield task_id, order, (a, b) if order == "ab" else (b, a) ``` **The diagnostic check:** evaluate every test pair in both orders. If the verdicts disagree, mark the comparison as unresolved rather than taking an arbitrary single-order result. --- ### Step 4: Supply explicit reference answers Putting a verified reference answer in the judge prompt cuts grading error rates: | Judge Configuration | Math / Reasoning Failure Rate | |---|---| | Default zero-shot prompt | 70% | | Chain-of-thought reasoning | 30% | | Reference-guided evaluation | 15% | Grounding the evaluation against a known golden response shifts the task from unconstrained open-ended generation to direct semantic verification. --- ### Step 5: Embed adversarial canary probes Research on reward model vulnerabilities demonstrates that content-free text strings (such as a single colon, a period, or standard boilerplate prefixes like `"Thought process:"`) can elicit false-positive passing scores from uncalibrated evaluators. Include content-free test cases in evaluation suites to catch prompt degradation: ```python # canaries.py CANARY_PROBES = [ ":", ".", "Thought process:", "Let's solve this problem step by step.", ] def audit_canary_responses(judge_fn, test_tasks: list[dict]) -> dict: """Verifies whether the evaluator awards passing scores to empty or meaningless inputs.""" false_positives = [] for task in test_tasks: for probe in CANARY_PROBES: verdict = judge_fn(question=task["question"], candidate_answer=probe) if verdict.get("passed"): false_positives.append({"task_id": task["id"], "probe": probe}) return { "total_probes": len(test_tasks) * len(CANARY_PROBES), "false_positive_count": len(false_positives), "suite_valid": len(false_positives) == 0, } ``` If the evaluator passes any empty canary probe, the scoring threshold is invalid and should not gate deployment pipelines. --- ### Step 6: Multi-model panels and cost routing Using a panel of diverse smaller models (PoLL architecture) from **disjoint model families** (such as mixing Claude-3.5-Haiku, GPT-4o-mini, and Llama-3.1-8B) prevents shared family biases while reducing evaluation costs relative to a single frontier model. ```mermaid flowchart TD A[Candidate Output] --> B{Deterministic Checks} B -->|Linter / Schema Passes| C[Deterministic Verdict] B -->|Requires Subjective Eval| D[Judge: Order AB] B -->|Requires Subjective Eval| E[Judge: Order BA] D --> F{Verdicts Match?} E --> F F -->|Yes| G[Record Verified Verdict] F -->|No / Flipped| H[Flag as Unresolved] ``` Run the deterministic verifications first: JSON schema validators, unit tests, regex matchers. Send the judge only the properties that no compiler can check. --- ### When this is the wrong choice - **The property has a deterministic check.** Schema validity, a passing test, an exact-match answer. A validator answers those exactly, for free, every time. A judge answers them at a cost, with the position bias and rerun variance described above. - **The rubric is still moving.** Kappa measures agreement against a fixed rubric. Score against a rubric you rewrote last week and the change you observe is rubric drift, not a model improvement, which is the same mistake as reading a rerun flip as progress. - **You can only afford one presentation order.** Order-swapped evaluation doubles judge calls. At the consistency rates in the table above, single-order verdicts on close pairs are near noise, so a smaller human-reviewed sample buys more information than a larger automated one. --- Source: https://himanshuat.com/blogs/llm-as-judge-what-the-numbers-actually-say ======================================================================== --- title: "Loop Engineering" description: "Agents fail in production not because models lack reasoning capacity, but because unmonitored variance causes identical tasks to succeed once and fail on subsequent runs. Here is how to engineer verification ladders, context budgets, and deterministic checklists." date: "June 02, 2026" url: "https://himanshuat.com/blogs/loop-engineering" --- # Loop Engineering An agent succeeds during a local test demonstration. A week later, it fails on the exact same task in a staging environment. Nothing about the prompt changed. > Prompt tuning optimizes the mean. What kills you in production is the variance, and variance is a property of the execution loop, not the prompt. ### Chapter 0: Anatomy of an agent loop An agent loop runs four stages: **gather context, take action, verify work, repeat.** Most homegrown implementations skip verification. Without an evaluation gate the loop keeps taking actions and never notices that an intermediate output has drifted off its constraints. --- ### Step 1: Measuring pass^k instead of pass@1 Shunyu Yao and co-authors' [τ-bench](https://arxiv.org/abs/2406.12045), published in June 2024, measures tool-calling reliability using `pass^k`: the percentage of tasks where **all k** independent attempts succeed consecutively. Their headline result is the gap between the two metrics. State-of-the-art function-calling agents (gpt-4o at the time of writing) succeed on under 50% of tasks on a single attempt, and `pass^8` falls below 25% in the retail domain. Measuring single-run success gives you an optimistic number. Running the same task k times and reporting `pass^k` is what exposes how stable the loop actually is. **The tell:** if you have never run the same task twice under the same conditions, you do not know your failure rate. You know your best case. --- ### Step 2: The verification ladder: deterministic checks first Verification should follow a tiered cost hierarchy: 1. **Deterministic syntax and type checks:** fast, local linters (such as `tsc`, `ruff`, or AST parsers). 2. **Programmatic assertions:** test suites and schema validators. 3. **Visual evaluation:** rendering screenshots to verify UI state. 4. **Model-based grading:** LLM-as-a-judge for semantic nuances that code cannot test. ```python # loop.py from dataclasses import dataclass, field from typing import Callable, Literal, List, Tuple Verdict = Literal["pass", "fail", "unknown"] @dataclass class Check: """A single verification check with associated execution cost.""" name: str cost: int run: Callable[[str], Verdict] @dataclass class VerificationLadder: """Executes checks in ascending cost order, halting on first failure.""" checks: List[Check] = field(default_factory=list) def verify(self, artifact: str) -> Tuple[Verdict, List[str]]: executed_checks: List[str] = [] for check in sorted(self.checks, key=lambda c: c.cost): executed_checks.append(check.name) verdict = check.run(artifact) if verdict == "fail": return "fail", executed_checks return "pass", executed_checks ``` Ordering checks by cost means broken syntax fails on the cheapest rung, before you pay a model judge to read it. **The tell:** if your linter and your judge run on the same artifact in the same turn, the ladder is not sorted. --- ### Step 3: Managing context budgets and compaction Chroma's [context rot report](https://www.trychroma.com/research/context-rot) from July 2025 evaluated 18 models — GPT-4.1, Claude 4, Gemini 2.5, Qwen3 — and found that models do not use their context uniformly: performance grows increasingly unreliable as input length grows, on tasks as simple as retrieval. Models degrade when the window fills with unfocused tokens. Stale tool output competes with your instructions for attention. Manage context through active compaction: - Blank or truncate stale tool observations once downstream actions consume their outputs. - Summarize multi-turn execution histories into concise decision logs when approaching window limits. - Delegate verbose exploratory tasks to sub-agents that return concise summary payloads to the parent orchestrator. ```python # compaction.py from dataclasses import dataclass from typing import List, Dict, Any @dataclass class Transcript: messages: List[Dict[str, Any]] files_touched: List[str] def clear_stale_tool_outputs(messages: List[Dict[str, Any]], retain_latest: int = 2) -> List[Dict[str, Any]]: """Replaces older verbose tool outputs with truncated placeholders.""" seen_tool_messages = 0 updated_messages = [] for msg in reversed(messages): if msg.get("role") == "tool": seen_tool_messages += 1 if seen_tool_messages > retain_latest: msg = {**msg, "content": "[output cleared from working context]"} updated_messages.append(msg) return list(reversed(updated_messages)) ``` --- ### Step 4: Enforce execution bounds and token caps An unbounded loop will cycle forever on an API error it does not know how to interpret. Enforce explicit turn and token budgets: ```python # budget.py from dataclasses import dataclass class BudgetExhaustedError(RuntimeError): """Raised when turn or token limits are exceeded.""" pass @dataclass class ExecutionBudget: max_turns: int max_tokens: int turns_used: int = 0 tokens_used: int = 0 @property def remaining_turns(self) -> int: return max(0, self.max_turns - self.turns_used) def record_step(self, token_count: int) -> None: self.turns_used += 1 self.tokens_used += token_count if self.turns_used >= self.max_turns: raise BudgetExhaustedError(f"Turn limit ({self.max_turns}) exceeded.") if self.tokens_used >= self.max_tokens: raise BudgetExhaustedError(f"Token budget ({self.max_tokens}) exceeded.") def should_terminate(self) -> bool: return self.remaining_turns <= 1 ``` --- ### Step 5: Anchor progress to external state checklists Long-running workflows spanning multiple steps require externalized state tracking. Models can declare tasks complete prematurely if progress is tracked only within ephemeral prompt contexts. Maintain an external checklist in structured JSON to track task completion: ```python # checklist.py import json from pathlib import Path from typing import Any, Dict, List, Optional CHECKLIST_PATH = Path("execution_checklist.json") def load_checklist() -> List[Dict[str, Any]]: return json.loads(CHECKLIST_PATH.read_text()) def get_next_pending_task(checklist: List[Dict[str, Any]]) -> Optional[Dict[str, Any]]: for item in checklist: if not item.get("passed"): return item return None def record_task_verification(task_id: str, test_evidence: str) -> None: if not test_evidence.strip(): raise ValueError(f"Task {task_id} requires concrete test execution evidence.") checklist = load_checklist() for item in checklist: if item["id"] == task_id: item["passed"] = True item["evidence"] = test_evidence break CHECKLIST_PATH.write_text(json.dumps(checklist, indent=2)) ``` Requiring explicit execution output (such as passing test runners or status codes) prevents models from assuming completion without verification. **The tell:** if a task can be marked done without a command output pasted next to it, the checklist is decoration. --- ### When this is the wrong choice - **The task runs once and a person reads the result.** Everything above buys reliability across repeated runs. If nobody reruns the task, `pass^k` and `pass@1` are the same number, and the loop machinery costs turns for nothing. - **You have no cheap deterministic check to put on the bottom rung.** The ladder works because `tsc` or `ruff` costs almost nothing to run every turn. Where the output is prose with no compiler behind it, every rung is a model call and verification costs about as much as generation. - **The whole job fits in one context window.** Compaction and external checklists exist to survive window pressure over a long horizon. Below that, clearing stale tool output deletes information the model still needs, and the JSON checklist is a second source of truth to keep in sync. --- ### Engineering checklist for reliable loops 1. **Benchmark pass^k:** measure stability across repeated runs rather than optimizing single-turn demos. 2. **Implement tiered verification:** evaluate cheap, deterministic type and syntax checks before calling model judges. 3. **Prune context aggressively:** clear stale tool observation payloads to prevent context degradation. 4. **Enforce hard execution bounds:** configure explicit turn and token ceilings. 5. **Externalize task checklists:** track state transitions and verification evidence outside ephemeral model context. --- Source: https://himanshuat.com/blogs/loop-engineering ======================================================================== --- title: "LoRA finetuning: the hyperparameters settled in a week, the data never did" description: "An early draping adapter kept rendering identical flat planes across shoulder folds, regardless of prompt. This is memorization rather than learned geometric priors, traced back to pinned studio photos. Hyperparameters stabilize quickly in Low-Rank Adaptation; dataset curation and caption isolation determine generalization." date: "April 08, 2025" url: "https://himanshuat.com/blogs/lora-finetuning-what-moved-the-needle" --- # LoRA finetuning: the hyperparameters settled in a week, the data never did An early draping adapter kept rendering the same flat plane across the back of the shoulder, under prompts that shared no vocabulary at all. That artifact is the signature of dataset memorization, not a learned spatial prior. It traced back to a cluster of studio photographs where stylists had pinned the fabric tight behind the subject. In Low-Rank Adaptation (LoRA) fine-tuning, the hyperparameters settle in a handful of sweeps. Dataset quality, caption token isolation, and aspect ratio bucketing decide whether the adapter generalizes. > Rank and alpha define capacity and update magnitude. The dataset distribution defines the upper bound on generation quality. ### Chapter 0: Mechanics of alpha over rank scaling [LoRA](https://arxiv.org/abs/2106.09685), introduced by Edward Hu and co-authors in 2021, freezes base model weights $W_0$ and injects trainable rank decomposition matrices: $$ W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} (B \cdot A) $$ where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ with rank $r \ll \min(d, k)$. Rank ($r$) sets parameter capacity. The scaling factor $\frac{\alpha}{r}$ regulates how strongly adapter updates modify the frozen base representations. Section 4.1 of the paper puts it plainly: "We then scale $\Delta W x$ by $\alpha/r$, where $\alpha$ is a constant in $r$." --- ### Step 1: Fix the scale ratio below 1.0 to prevent style collapse Setting $r = 32$ and $\alpha = 16$ yields a scaling factor of: $$ \text{scale} = \frac{\alpha}{r} = \frac{16}{32} = 0.5 $$ | Alpha ($\alpha$) | Scale Factor ($\frac{\alpha}{r}$) | Observed Empirical Behavior | |---|---|---| | 16 | 0.5 | Target physical priors learned; base model style and photorealism preserved | | 32 | 1.0 | Stronger drape effect; noticeable background and contrast overfitting | | 64 | 2.0 | Severe style collapse; output mimics training set color grading and lighting | A 0.5 ratio damps adapter updates relative to base attention activations, so the network picks up the fold geometry without overwriting what it already knows. Use a base learning rate of `1e-4` with cosine decay. The cosine schedule lowers step sizes in later epochs, preventing the optimizer from memorizing spurious high-frequency noise from small image sets. **The diagnostic check:** if increasing $\alpha$ amplifies the target concept while simultaneously degrading lighting diversity across unrelated prompts, the adapter update scale is too high. --- ### Step 2: Freeze text encoders and isolate attention projections Train attention projection layers in the denoiser (Transformer blocks or UNet) while keeping text encoders frozen. If the base foundation model already understands the semantic concept (for example, the noun "saree" or "tuxedo"), fine-tuning text encoders introduces shortcut paths: the optimizer rebinds text embeddings directly to training image artifacts rather than learning spatial cross-attention relationships. Confining updates to the Query, Key, Value, and Output projections forces the adapter to learn relationships between spatial image patches instead. **The diagnostic check:** if the adapter only produces the target concept when the trigger token appears verbatim, and ignores paraphrases the base model understands, the text encoder learned a shortcut. --- ### Step 3: Configure aspect ratio bucketing and gradient accumulation Small batch sizes (such as `train_batch_size = 4`) paired with `gradient_accumulation_steps = 4` preserve bucket diversity. [Aspect ratio bucketing](https://github.com/NovelAI/novelai-aspect-ratio-bucketing), released by NovelAI under MIT after they found square-crop training was generating "humans... without feet or heads", groups non-square images (such as tall full-body portraits or wide flat-lays) into discrete resolution bins. Large physical batches from a single bucket skew gradients toward specific aspect ratios. Accumulating gradients across multiple small batches averages updates across varied compositions. The config below is [`kohya-ss/sd-scripts`](https://github.com/kohya-ss/sd-scripts) TOML, the training scripts most FLUX and SDXL LoRA work runs on. ```toml # configs/drape_lora.toml # Low-Rank Adaptation configuration for diffusion models [model] pretrained_model_name_or_path = "/models/flux/base" vae = "/models/flux/vae" mixed_precision = "bf16" save_precision = "fp16" [network] network_module = "networks.lora_flux" network_dim = 32 # Rank (r) network_alpha = 16 # Alpha (a) -> scale = 0.5 network_train_unet_only = true # Keep text encoder frozen [optimizer] optimizer_type = "AdamW8bit" learning_rate = 1e-4 lr_scheduler = "cosine" lr_warmup_steps = 100 max_grad_norm = 1.0 [training] train_batch_size = 4 gradient_accumulation_steps = 4 gradient_checkpointing = true cache_latents = true cache_latents_to_disk = true seed = 42 max_train_epochs = 12 save_every_n_epochs = 1 [dataset] resolution = "1024,1024" enable_bucket = true bucket_reso_steps = 64 min_bucket_reso = 768 max_bucket_reso = 1536 caption_extension = ".txt" shuffle_caption = false keep_tokens = 1 # Trigger token fixed at position 0 ``` Using 8-bit AdamW ([`AdamW8bit`](https://huggingface.co/docs/bitsandbytes/main/en/optimizers), from Tim Dettmers and co-authors' [8-bit Optimizers via Block-wise Quantization](https://arxiv.org/abs/2110.02861)) reduces optimizer memory footprint from 8 bytes to 2 bytes per parameter, freeing VRAM for higher training resolutions (1024x1024+). --- ### Step 4: Technical filtering and automated image validation Mechanical filters reject low-quality inputs prior to training: ```python # data/filter_corpus.py from dataclasses import dataclass from pathlib import Path import cv2 import numpy as np @dataclass(frozen=True) class FilterConfig: min_short_side: int = 1024 min_sharpness: float = 120.0 # Variance of Laplacian min_aspect: float = 0.5 # Portrait limit max_aspect: float = 1.6 # Landscape limit max_clipped_fraction: float = 0.04 # Blown highlights threshold def calculate_sharpness(gray: np.ndarray) -> float: return float(cv2.Laplacian(gray, cv2.CV_64F).var()) def calculate_clipped_fraction(gray: np.ndarray) -> float: return float((gray >= 250).mean()) def inspect_image(path: Path, cfg: FilterConfig): image = cv2.imread(str(path)) if image is None: return False, "unreadable" h, w = image.shape[:2] gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) aspect = w / h if min(h, w) < cfg.min_short_side: return False, f"resolution {w}x{h}" if not (cfg.min_aspect <= aspect <= cfg.max_aspect): return False, f"aspect {aspect:.2f}" if calculate_sharpness(gray) < cfg.min_sharpness: return False, "soft_focus" if calculate_clipped_fraction(gray) > cfg.max_clipped_fraction: return False, "clipped_highlights" return True, None ``` ```mermaid flowchart TD A[Repeated Artifact in Generation] --> B{Style or Geometry?} B -->|Style / Color Bleed| C[Scale Alpha / Rank Too High] B -->|Fixed Geometric Distortion| D[Isolate Bad Sub-cluster in Dataset] C --> E[Lower Alpha to 0.5 Ratio] D --> F[Prune / Recaption Dataset] F --> G[Retrain Checkpoint] E --> G ``` --- ### Step 5: Caption token isolation Captions must explicitly describe background, lighting, and camera perspective so the model does not entangle those traits with the trigger token. If every training image sits on a solid grey studio background and no caption says so, the trigger token absorbs the grey studio into its concept. Naming the non-target elements is what decouples them from the adapter weights. **The diagnostic check:** generate with the trigger token and a prompt that specifies a different background. If the grey studio comes back anyway, the captions did not isolate it. --- ### Step 6: Evaluation across fixed seed grids and out-of-domain prompts Select optimal training checkpoints using a standardized validation suite: 1. **Fixed seed grids:** Generate samples for each saved epoch using identical prompt sets and constant seeds. 2. **Partial adapter weights:** Evaluate checkpoints at merged strengths ($0.5$ to $0.7$) to verify interoperability with secondary adapters. 3. **Out-of-domain regression checks:** Generate non-target subjects (such as basic denim or leather jackets) to detect whether the adapter induces catastrophic forgetting on general attire. --- ### When this is the wrong choice - **The base model already renders the concept.** Step 2 turns on exactly this. If the base understands the noun, an adapter mostly rebinds the token to your training artifacts, and prompt work costs nothing and is reversible. - **Most of your images fail the mechanical filters.** The filters in Step 4 reject soft focus, blown highlights, and off-aspect frames. If the set thins out badly under them, no rank or alpha setting recovers the missing distribution. You get the flat shoulder plane at the top of this post. - **The defect is a caption problem, not a capacity problem.** Step 5's failure mode is entanglement. When the trigger token has absorbed the background or the lighting, re-captioning the set you already have fixes it, and retraining at a different rank does not. --- Source: https://himanshuat.com/blogs/lora-finetuning-what-moved-the-needle ======================================================================== --- title: "Modern Neovim from Scratch: Lua Modular Architecture, Lazy.nvim, LSP, and Treesitter" description: "A practical guide to configuring a fast, modular Neovim development environment using Lua, Lazy.nvim, native LSP with Mason, nvim-cmp, and Treesitter." date: "September 12, 2023" url: "https://himanshuat.com/blogs/my-nvim-setup" --- # Modern Neovim from Scratch: Lua Modular Architecture, Lazy.nvim, LSP, and Treesitter Vim configurations historically relied on monolithic `init.vim` files written in Vimscript, external node processes like `coc.nvim`, and plugin managers like `vim-plug`. Neovim v0.9+ replaces all three with a [native Lua runtime](https://neovim.io/doc/user/lua/), an [embedded Language Server Protocol (LSP) client](https://neovim.io/doc/user/lsp/), and syntax parsing through [Tree-sitter](https://neovim.io/doc/user/treesitter/). All three are in core: `:help lua` says "the Lua 5.1 script engine is builtin and always available," and `:help lsp` says "Nvim provides an LSP client." The config below is split across five files so that a broken plugin is a broken file, not a broken editor. ``` ~/.config/nvim/ ├── init.lua └── lua/ ├── config/ │ ├── options.lua # Editor defaults and UI behaviors │ ├── keymaps.lua # Keybindings and ergonomics │ └── lazy.lua # Lazy.nvim bootstrapper └── plugins/ ├── lsp.lua # Mason, nvim-lspconfig, and nvim-cmp ├── treesitter.lua # AST syntax parsing and highlighting ├── telescope.lua # Fuzzy finding and ripgrep search └── colorscheme.lua ``` ## Step 1: Set the entry point and editor defaults `~/.config/nvim/init.lua` is the entry point. It does nothing except require the modules under `lua/`. ```lua -- ~/.config/nvim/init.lua require("config.options") require("config.keymaps") require("config.lazy") ``` ### Editor options (`lua/config/options.lua`) Define standard buffer and window behavior using `vim.opt`: ```lua -- ~/.config/nvim/lua/config/options.lua local opt = vim.opt -- Line numbers opt.number = true opt.relativenumber = true -- Tabs and Indentation opt.tabstop = 2 opt.shiftwidth = 2 opt.expandtab = true opt.autoindent = true opt.smartindent = true -- Search behavior opt.ignorecase = true opt.smartcase = true -- Visuals and Splits opt.termguicolors = true opt.signcolumn = "yes" opt.scrolloff = 8 opt.splitright = true opt.splitbelow = true opt.cursorline = true -- System integration opt.clipboard = "unnamedplus" opt.undofile = true -- Persistent undo history across editor restarts opt.updatetime = 250 opt.timeoutlen = 300 ``` ### Ergonomic keymaps (`lua/config/keymaps.lua`) Set the spacebar as the leader key and map window, buffer, and navigation shortcuts: ```lua -- ~/.config/nvim/lua/config/keymaps.lua vim.g.mapleader = " " vim.g.maplocalleader = " " local map = vim.keymap.set -- Clear search highlights on pressing Escape in normal mode map("n", "", "nohlsearch") -- Window navigation using Ctrl + hjkl map("n", "", "h", { desc = "Move to left window" }) map("n", "", "j", { desc = "Move to lower window" }) map("n", "", "k", { desc = "Move to upper window" }) map("n", "", "l", { desc = "Move to right window" }) -- Keep cursor centered during half-page scrolling and search jumps map("n", "", "zz") map("n", "", "zz") map("n", "n", "nzzzv") map("n", "N", "Nzzzv") -- Move selected text blocks up and down in visual mode map("v", "J", ":m '>+1gv=gv") map("v", "K", ":m '<-2gv=gv") -- Buffer management map("n", "bd", "bdelete", { desc = "Delete buffer" }) map("n", "[b", "bprevious", { desc = "Previous buffer" }) map("n", "]b", "bnext", { desc = "Next buffer" }) ``` ## Step 2: Bootstrap package management with Lazy.nvim [`lazy.nvim`](https://github.com/folke/lazy.nvim) provides fast plugin loading with automatic lazy-loading on keymaps, filetypes, or events. Create `lua/config/lazy.lua` to clone and initialize the package manager automatically: ```lua -- ~/.config/nvim/lua/config/lazy.lua local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim" if not vim.loop.fs_stat(lazypath) then vim.fn.system({ "git", "clone", "--filter=blob:none", "https://github.com/folke/lazy.nvim.git", "--branch=stable", lazypath, }) end vim.opt.rtp:prepend(lazypath) require("lazy").setup({ spec = { { import = "plugins" }, }, defaults = { lazy = true, }, install = { colorscheme = { "tokyonight", "habamax" }, }, }) ``` ## Step 3: Replace regex highlighting with Tree-sitter Traditional Vim syntax highlighting relies on regular expressions, which can be slow and fail to parse complex nested language scopes. [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) generates a concrete syntax tree (AST) directly in memory, updating incrementally on every keystroke. [`nvim-treesitter`](https://github.com/nvim-treesitter/nvim-treesitter) installs and updates the parsers: ```lua -- ~/.config/nvim/lua/plugins/treesitter.lua return { "nvim-treesitter/nvim-treesitter", build = ":TSUpdate", event = { "BufReadPost", "BufNewFile" }, config = function() require("nvim-treesitter.configs").setup({ ensure_installed = { "c", "cpp", "lua", "vim", "vimdoc", "python", "rust", "go", "typescript", "tsx", "javascript", "html", "css", "json", }, auto_install = true, highlight = { enable = true, additional_vim_regex_highlighting = false, }, indent = { enable = true, }, }) end, } ``` ## Step 4: Wire fuzzy finding to ripgrep with Telescope [Telescope](https://github.com/nvim-telescope/telescope.nvim) uses `ripgrep` and `fd` to scan files, git status, and code symbols: ```lua -- ~/.config/nvim/lua/plugins/telescope.lua return { "nvim-telescope/telescope.nvim", tag = "0.1.6", dependencies = { "nvim-lua/plenary.nvim", { "nvim-telescope/telescope-fzf-native.nvim", build = "make", cond = function() return vim.fn.executable("make") == 1 end, }, }, cmd = "Telescope", keys = { { "ff", "Telescope find_files", desc = "Find Files" }, { "fg", "Telescope live_grep", desc = "Live Grep" }, { "fb", "Telescope buffers", desc = "Find Buffers" }, { "fh", "Telescope help_tags", desc = "Help Tags" }, }, config = function() local telescope = require("telescope") telescope.setup({ defaults = { path_display = { "truncate" }, file_ignore_patterns = { "node_modules", ".git/" }, }, }) pcall(telescope.load_extension, "fzf") end, } ``` ## Step 5: Attach native LSP and autocompletion Language Server Protocol integration requires three coordinating layers: 1. **[`mason.nvim`](https://github.com/mason-org/mason.nvim):** Manages language server binaries (`gopls`, `pyright`, `ts_ls`, `rust-analyzer`). 2. **[`nvim-lspconfig`](https://github.com/neovim/nvim-lspconfig):** Configures client options and hooks keymaps on server attachment (`on_attach`). 3. **[`nvim-cmp`](https://github.com/hrsh7th/nvim-cmp):** Renders autocompletion popups, snippet expansions (through [`LuaSnip`](https://github.com/L3MON4D3/LuaSnip)), and signature help. ```lua -- ~/.config/nvim/lua/plugins/lsp.lua return { -- LSP Configuration { "neovim/nvim-lspconfig", event = { "BufReadPre", "BufNewFile" }, dependencies = { "williamboman/mason.nvim", "williamboman/mason-lspconfig.nvim", "hrsh7th/cmp-nvim-lsp", }, config = function() local lspconfig = require("lspconfig") local capabilities = require("cmp_nvim_lsp").default_capabilities() -- Keybindings applied when LSP attaches to buffer local on_attach = function(_, bufnr) local nmap = function(keys, func, desc) vim.keymap.set("n", keys, func, { buffer = bufnr, desc = "LSP: " .. desc }) end nmap("gd", vim.lsp.buf.definition, "Goto Definition") nmap("gr", vim.lsp.buf.references, "Goto References") nmap("gi", vim.lsp.buf.implementation, "Goto Implementation") nmap("K", vim.lsp.buf.hover, "Hover Documentation") nmap("rn", vim.lsp.buf.rename, "Rename Symbol") nmap("ca", vim.lsp.buf.code_action, "Code Action") nmap("d", vim.diagnostic.open_float, "Show Diagnostic") nmap("[d", vim.diagnostic.goto_prev, "Previous Diagnostic") nmap("]d", vim.diagnostic.goto_next, "Next Diagnostic") end require("mason").setup() require("mason-lspconfig").setup({ ensure_installed = { "lua_ls", "pyright", "ts_ls", "gopls", "rust_analyzer", }, handlers = { function(server_name) lspconfig[server_name].setup({ capabilities = capabilities, on_attach = on_attach, }) end, ["lua_ls"] = function() lspconfig.lua_ls.setup({ capabilities = capabilities, on_attach = on_attach, settings = { Lua = { diagnostics = { globals = { "vim" }, }, }, }, }) end, }, }) end, }, -- Autocompletion Engine { "hrsh7th/nvim-cmp", event = "InsertEnter", dependencies = { "hrsh7th/cmp-nvim-lsp", "hrsh7th/cmp-buffer", "hrsh7th/cmp-path", "L3MON4D3/LuaSnip", "saadparwaiz1/cmp_luasnip", }, config = function() local cmp = require("cmp") local luasnip = require("luasnip") cmp.setup({ snippet = { expand = function(args) luasnip.lsp_expand(args.body) end, }, mapping = cmp.mapping.preset.insert({ [""] = cmp.mapping.scroll_docs(-4), [""] = cmp.mapping.scroll_docs(4), [""] = cmp.mapping.complete(), [""] = cmp.mapping.confirm({ select = true }), [""] = cmp.mapping(function(fallback) if cmp.visible() then cmp.select_next_item() elseif luasnip.expand_or_jumpable() then luasnip.expand_or_jump() else fallback() end end, { "i", "s" }), [""] = cmp.mapping(function(fallback) if cmp.visible() then cmp.select_prev_item() elseif luasnip.jumpable(-1) then luasnip.jump(-1) else fallback() end end, { "i", "s" }), }), sources = cmp.config.sources({ { name = "nvim_lsp" }, { name = "luasnip" }, { name = "buffer" }, { name = "path" }, }), }) end, }, } ``` ## Step 6: Pick a colorscheme Install [Tokyo Night](https://github.com/folke/tokyonight.nvim) for a high-contrast terminal interface: ```lua -- ~/.config/nvim/lua/plugins/colorscheme.lua return { { "folke/tokyonight.nvim", lazy = false, priority = 1000, config = function() vim.cmd.colorscheme("tokyonight-night") end, }, } ``` ## Verifying startup performance Profile your startup time using Neovim's built-in profiler flag: ```bash nvim --startuptime /tmp/nvim_startup.log tail -n 10 /tmp/nvim_startup.log ``` With lazy-loading across syntax, LSP, and search, cold start stays under 30 milliseconds. Everything except the colorscheme is deferred until a buffer, a keymap, or an insert-mode event asks for it. ## When this is the wrong choice - **You edit files on machines you don't control.** This config assumes `git`, `make`, `ripgrep`, and `fd` are present and that Mason can reach the network to download server binaries. On a locked-down jump host, none of that holds, and stock vim with no plugins is the thing that always starts. - **You already have a config you know by heart.** The payoff here is startup time and a file layout you can bisect. Neither is worth trading away the muscle memory of a setup that already works. - **Your language has no Tree-sitter grammar and no language server.** Steps 3 and 5 are most of the value. Without them you have installed a plugin manager, a fuzzy finder, and a colorscheme. --- Source: https://himanshuat.com/blogs/my-nvim-setup ======================================================================== --- title: "One LoRA per failure mode" description: "Put a hand on a hip and our try-on pipeline painted the garment straight over the fingers. The obvious fix, one bigger dataset covering every hard case at once, quietly made the draping worse. Splitting the finetune into two adapters, one per named failure mode, is what actually worked." date: "May 20, 2025" url: "https://himanshuat.com/blogs/one-lora-per-failure-mode" --- # One LoRA per failure mode When a model in the input photo rested a hand on her hip, our early try-on pipeline painted garment fabric directly over her fingers. The hand disappeared into the cloth. We called this defect the "Amputated Hand." I initially tried to fix it by pooling all failure cases (sarees, kimonos, jewelry occlusions, and hand poses) into a single dataset for joint fine-tuning. > Pooling disparate failure modes into one training set creates conflicting optimization targets within a fixed adapter rank budget, yielding averaged behavior that fixes neither problem. ### Capacity limits of single low-rank adapters A single LoRA trained on all difficult edge cases failed because the two primary defects demand opposing model behaviors: **Fabric draping is a generative task:** A saree pallu must fall across the shoulder with natural volume, and pleats must gather at the waist casting directional shadows. The adapter must push the diffusion model to synthesize new geometric folds within the masked region. **Hand occlusion is a suppression task:** When fingers rest over a torso, the model must suppress generation across those specific pixels, treating foreground anatomy as immutable. | Characteristic | Draping Physics (Expert A) | Occlusion and Depth (Expert B) | |---|---|---| | Objective | Generative structure synthesis | Pixel value preservation and suppression | | Mask interaction | Generate novel folds inside mask | Preserve marked foreground regions | | Optimization signal | High-frequency textile folds | Boundary fidelity to foreground anatomy | At rank 32, a LoRA possesses a constrained subspace of parameter updates. Forcing it to simultaneously synthesize complex folds and suppress generation over foreground pixels spends that capacity on a compromised average. Adding occlusion samples restored hands but flattened pleats. Rebalancing toward draping samples restored pleat depth but caused fabric to overpaint fingers again. Evaluating each iteration required a complete training run with no layer-wise attribution. --- ### Step 1: Triage the failure logs before expanding the dataset Rather than scraping more uncategorized data, we classified rejected renders by failure type. Reviewers tagged rejected outputs with concise notes ("hand clipped", "pleats flat", "kimono looks like bathrobe"). We parsed these logs to map recurring terms into concrete failure categories: ```python # triage_failures.py """Bucket rejected try-on renders into named failure modes. Reviewers tag rejected renders with short labels. This script maps labels to specific failure categories to focus dataset collection. """ import json from collections import Counter from pathlib import Path MODE_PATTERNS = { "drape": ["pleat", "fold", "flat texture", "pallu", "no volume", "plastered"], "occlusion": ["hand", "finger", "arm", "jewel", "necklace", "painted over"], "style_drift": ["wrong texture", "lost weave", "flat colour", "not the fabric"], "semantic": ["bathrobe", "wrong garment", "looks like a dress"], } def classify(tags: list[str]) -> set[str]: """Return every failure mode matching a render's tags. A render displaying both flat pleats and occluded hands counts toward both categories to prevent hiding secondary failure modes. """ blob = " ".join(tags).lower() modes = {mode for mode, pats in MODE_PATTERNS.items() if any(p in blob for p in pats)} return modes or {"unclassified"} def summarise(reject_log: Path) -> Counter: counts = Counter() for line in reject_log.read_text().splitlines(): record = json.loads(line) for mode in classify(record["tags"]): counts[mode] += 1 return counts if __name__ == "__main__": counts = summarise(Path("data/rejects.jsonl")) total = sum(counts.values()) for mode, n in counts.most_common(): print(f"{mode:>14} {n:>5} {n / total:6.1%}") print(f"{'total defects':>14} {total:>5}") ``` Two defect modes dominated the logs: drape collapse and foreground occlusion. **The tell:** if the reject tags do not cluster into a handful of named buckets, you have a data quality problem, not a set of failure modes to split on. --- ### Step 2: Test each failure mode against four criteria Before allocating compute to train an isolated adapter, a candidate failure mode must satisfy four conditions: 1. **Formulated as an actionable behavior:** "Treat foreground hands as immutable" can be supervised in a dataset; "hands look broken" cannot. 2. **Isolatable in training data:** The target behavior must represent the primary variable across training pairs. 3. **Presents conflicting gradients with existing adapters:** Separate adapters are only needed when joint training causes parameter interference. 4. **Represents a base weight gap rather than a conditioning issue:** Style drift was resolved by adding Flux Redux image conditioning without training new adapter weights. ```mermaid flowchart LR R[rejects] --> T{evaluate criteria} T -->|drape physics| D[drape set] T -->|foreground overlap| O[occlusion set] T -->|conditioning missing| X[Flux Redux] T -->|base semantic prior| N[out of scope] D --> A[expert A] O --> B[expert B] A --> M[fuse and serve] B --> M ``` Draping physics and occlusion boundaries satisfied all four criteria. --- ### Step 3: Weigh single-adapter simplicity against specialists Maintaining a single unified adapter offers operational advantages: one training schedule, one artifact to version, and no merge step. However, when a single rank-32 adapter cannot simultaneously learn orthogonal generative and suppressive behaviors, splitting the task into specialized models becomes necessary. --- ### Step 4: Train the drape specialist with aspect bucketing Expert A (draping physics) was trained on 5,000 curated images of sarees, hanfus, kimonos, and structured couture. These garments do not conform to body contours like t-shirts; their folds are dictated by wrapping technique and gravity. ```bash # train_drape_lora.sh # Expert A: draping physics. Single NVIDIA L40S (48GB). # Rank 32 and alpha 16 match Expert B for predictable weight composition. accelerate launch --num_cpu_threads_per_process 8 \ sdxl_train_network.py \ --pretrained_model_name_or_path "$FLUX_BASE" \ --train_data_dir "./datasets/drape_5k" \ --output_dir "./out/lora_drape" \ --output_name "expert_a_drape" \ --resolution 1024,1024 \ --network_module networks.lora \ --network_dim 32 \ --network_alpha 16 \ --learning_rate 1e-4 \ --lr_scheduler cosine \ --train_batch_size 4 \ --gradient_accumulation_steps 4 \ --gradient_checkpointing \ --optimizer_type AdamW8bit \ --mixed_precision bf16 \ --save_every_n_epochs 1 \ --caption_extension .txt \ --keep_tokens 1 \ --enable_bucket \ --min_bucket_reso 768 \ --max_bucket_reso 1536 ``` Aspect ratio bucketing is essential for full-length garments: cropping vertical saree shots to square frames removes the lower pleat cascades that the adapter is meant to learn. --- ### Step 5: Supervise occlusion boundaries explicitly Expert B (occlusion and depth) was trained on 3,000 images showing hands on hips, crossed arms, and jewelry over fabric. We prepared masks by segmenting both the garment and the occluding foreground object with SAM2, then subtracting the foreground mask from the garment mask. The adapter learns to leave foreground elements untouched because the supervision explicitly holds them out. ```python # prep_occlusion_set.py """Build Expert B training set: garment masks with foreground held out. For each image, SAM2 segments the garment and the occluding limb or accessory. Subtracting the foreground mask ensures the model learns to preserve foreground structures rather than repainting over them. """ from pathlib import Path import numpy as np from PIL import Image from sam2.build_sam import build_sam2 from sam2.sam2_image_predictor import SAM2ImagePredictor GARMENT_PROMPTS = ["shirt", "tshirt", "top", "saree", "dress"] FOREGROUND_PROMPTS = ["hand", "fingers", "forearm", "necklace", "bracelet"] DILATE_PX = 9 def dilate(mask: np.ndarray, radius: int) -> np.ndarray: """Expand boolean mask using a square structuring element.""" from scipy.ndimage import binary_dilation kernel = np.ones((radius * 2 + 1, radius * 2 + 1), dtype=bool) return binary_dilation(mask, structure=kernel) def build_masks(predictor: SAM2ImagePredictor, image: Image.Image): predictor.set_image(np.array(image)) garment, _, _ = predictor.predict(text=GARMENT_PROMPTS, multimask_output=False) foreground, _, _ = predictor.predict(text=FOREGROUND_PROMPTS, multimask_output=False) garment = dilate(garment[0].astype(bool), DILATE_PX) foreground = dilate(foreground[0].astype(bool), DILATE_PX * 2) return garment & ~foreground, foreground def main(src: Path, dst: Path): predictor = SAM2ImagePredictor(build_sam2("sam2_hiera_l.yaml", "sam2_hiera_large.pt")) dst.mkdir(parents=True, exist_ok=True) kept = skipped = 0 for path in sorted(src.glob("*.jpg")): image = Image.open(path).convert("RGB") target, foreground = build_masks(predictor, image) if foreground.mean() < 0.005: skipped += 1 continue Image.fromarray((target * 255).astype(np.uint8)).save(dst / f"{path.stem}_mask.png") image.save(dst / path.name) (dst / f"{path.stem}.txt").write_text("hand resting over garment, hand in front of fabric") kept += 1 print(f"Kept {kept} / skipped {skipped} (occluder too small)") if __name__ == "__main__": main(Path("raw/occlusion"), Path("datasets/occlusion_3k")) ``` Both adapters share identical training hyperparameter shapes (rank 32, alpha 16, cosine learning rate schedule), making them directly comparable when merged. --- ### Failure modes and edge cases 1. **Inter-adapter interference upon merging:** While each adapter succeeds independently, direct linear addition into base weights can introduce interference across overlapping attention layers. 2. **Sheer fabric transparency:** Neither drape nor occlusion adapters model alpha transparency in sheer lace or chiffon. 3. **Dataset distribution bias:** Both datasets primarily cover standing and seated catalog poses; unconventional poses continue to exhibit geometry warping. --- ### When this is the wrong choice - **The tasks do not conflict.** If adding the new training data leaves existing behavior intact, criterion 3 fails and there is nothing to split. Keep one adapter and skip the merge step entirely. - **The failure is a conditioning gap.** Missing reference texture or a pattern repeat is fixed by image conditioning, which is how style drift got resolved here. Training new weights for it spends compute on a problem that has no weight component. - **Serving cannot load more than one checkpoint.** If the infrastructure supports neither multi-adapter builds nor offline weight fusion, a single compromised adapter is the only thing you can actually ship, and the compromise is real: hands or pleats, not both. --- Source: https://himanshuat.com/blogs/one-lora-per-failure-mode ======================================================================== --- title: "Production Agent Architecture: Memory, Approval Gates, and Durable Checkpointing" description: "What actually breaks when you put a LangGraph agent in production: memory interfaces for vector stores, human-in-the-loop approval routing for critical operations, TypeScript orchestrators with transactional state persistence, and the deployment considerations that don't appear in tutorials." date: "August 05, 2025" url: "https://himanshuat.com/blogs/production-grade-agent-architecture-memory-human-in-the-loop" --- # Production Agent Architecture: Memory, Approval Gates, and Durable Checkpointing Tutorial agents run to completion. A production agent reaches a critical operation (deploy to prod, wire a transfer, delete customer data) and has to stop, hold, and wait for a person. > The hard part of a production agent is not the reasoning. It is stopping mid-run, surviving a process restart, and resuming without replaying the tool calls that already succeeded. Three pieces carry that: a typed memory interface that keeps agent code away from vector store drivers, approval routing declared in the state schema, and an orchestrator that persists state on every transition. --- ## Chapter 0: What "production-ready" actually means here An agent becomes production-ready when it can: 1. Retrieve episodic context from a vector store without coupling business logic to embedding provider APIs 2. Halt at critical operations and wait for a human decision, holding that state across process restarts 3. Resume from exactly the last saved checkpoint without re-executing completed steps 4. Expose an audit trail of every approval decision and state transition None of these are available in the [default LangGraph JS quickstart](https://docs.langchain.com/oss/javascript/langgraph/quickstart), which builds a calculator agent with three arithmetic tools and conditional tool-call routing, and stops there. --- ## Step 1: Put the vector store behind a typed interface Agents should not call embedding APIs directly. Wrapping the vector store behind a typed interface makes the embedding provider swappable without touching agent logic. ```typescript // interfaces/memory.ts export interface MemoryRecord { id: string; content: string; timestamp: number; metadata?: Record; embedding?: number[]; } export interface IVectorStoreClient { /** Embeds content and stores memory records. */ add(records: Omit[]): Promise; /** Cosine similarity search over stored records. */ search(query: string, topK: number): Promise; } ``` ### Qdrant implementation This uses [`@qdrant/js-client-rest`](https://github.com/qdrant/qdrant-js), the REST client from Qdrant's official JavaScript SDK. ```typescript // services/vectorStoreClient.ts import { v4 as uuidv4 } from 'uuid'; import { IVectorStoreClient, MemoryRecord } from '../interfaces/memory'; import { EmbeddingService } from './embeddingService'; import { QdrantClient } from '@qdrant/js-client-rest'; export class QdrantVectorStoreClient implements IVectorStoreClient { private readonly embeddingService: EmbeddingService; private readonly qdrantClient: QdrantClient; private readonly collectionName: string; constructor(embeddingService: EmbeddingService, qdrantHost: string, collectionName: string = "agent_memories") { this.embeddingService = embeddingService; this.qdrantClient = new QdrantClient({ host: qdrantHost }); this.collectionName = collectionName; } public async initCollection(): Promise { const { collections } = await this.qdrantClient.getCollections(); const exists = collections.some(c => c.name === this.collectionName); if (!exists) { await this.qdrantClient.createCollection(this.collectionName, { vectors: { size: 1536, distance: 'Cosine' }, }); } } public async add(records: Omit[]): Promise { if (!records.length) return; const embeddings = await this.embeddingService.embedBatch(records.map(r => r.content)); const points = records.map((record, i) => ({ id: uuidv4(), vector: embeddings[i], payload: { content: record.content, timestamp: record.timestamp, metadata: record.metadata }, })); await this.qdrantClient.upsert(this.collectionName, { wait: true, points }); } public async search(query: string, topK: number): Promise { if (!query) return []; const [queryEmbedding] = await this.embeddingService.embedBatch([query]); const results = await this.qdrantClient.search(this.collectionName, { vector: queryEmbedding, limit: topK, with_payload: true, }); return results.map(r => ({ id: r.id.toString(), content: (r.payload as any).content, timestamp: (r.payload as any).timestamp, metadata: (r.payload as any).metadata, })); } } ``` The `EmbeddingService` is a thin wrapper over the OpenAI embeddings API. Swap it for a local [`bge-small-en-v1.5`](https://huggingface.co/BAAI/bge-small-en-v1.5) instance and the agent code does not change. **The tell:** if replacing the embedding provider means editing a file with the word `agent` in its name, the interface is not doing its job. --- ## Step 2: Declare approval gates in the state schema Tools declare whether they require authorization via an `isCritical` flag: ```typescript // types/tools.ts export interface Tool { name: string; description: string; func: (args: any) => Promise; isCritical: boolean; } export const tools: Tool[] = [ { name: "read_database", description: "Queries read replica analytics tables.", func: async (args: { query: string }) => `Data for: ${args.query}`, isCritical: false, }, { name: "deploy_code", description: "Deploys code to production cluster.", func: async (args: { project_id: string; version: string }) => `Deployed ${args.project_id} v${args.version}`, isCritical: true, } ]; ``` The approval state lives directly in the agent state schema, so checkpointers persist pending decisions when the process suspends. This is the same contract [LangGraph's own `interrupt` documentation](https://docs.langchain.com/oss/javascript/langgraph/interrupts) describes — it notes that persistence is what makes a pause survivable, and tells you to use a durable checkpointer in production rather than an in-memory one: ```typescript // types/agent.ts export interface AgentState { input: string; scratchpad: string[]; tool_calls: { tool_name: string; args: Record }[]; final_answer?: string; awaitingHumanApproval: boolean; proposedAction?: { type: 'tool_call' | 'final_answer'; details: any; }; humanFeedback?: 'approve' | 'reject' | 'modify'; modificationDetails?: string; } ``` The execution graph routes critical actions to the approval gate before executing them: ```mermaid flowchart TD T[agent_think] --> G{Evaluate action} G -->|isCritical = true| P[await_human_approval] G -->|isCritical = false| X[execute_tools] G -->|Final answer| E[END] P --> F[Receive human feedback] F --> T X --> T ``` **The tell:** if the pending decision lives anywhere outside the state object, the checkpointer will not restore the pause, and a restart drops the agent back into the tool call it was waiting on. --- ## Step 3: Persist state on every node transition The orchestrator persists state before and after every node transition. A process restart loads the last checkpoint and continues from exactly that position. ```typescript // orchestrator.ts export class AgentOrchestrator { private store: Map; constructor( private workflowDefinition: WorkflowDefinition, private nodeFunctions: Record, store?: Map ) { this.store = store ?? new Map(); } public async saveState(state: WorkflowState): Promise { // Deep clone to prevent reference mutations corrupting stored state this.store.set(state.workflowId, JSON.parse(JSON.stringify(state))); } public async loadState(workflowId: string): Promise { const item = this.store.get(workflowId); return item ? JSON.parse(JSON.stringify(item)) : undefined; } public async execute(workflowId: string): Promise { let state = await this.loadState(workflowId); if (!state) throw new Error(`Workflow ${workflowId} not found.`); while ( state.currentState !== 'complete' && state.currentState !== 'failed' && !state.interruptionDetails ) { const stepConfig = this.workflowDefinition[state.currentState]; if (!stepConfig) throw new Error(`Unknown state: '${state.currentState}'`); const nodeFn = this.nodeFunctions[stepConfig.node]; if (!nodeFn) throw new Error(`Missing node function: '${stepConfig.node}'`); try { state = await nodeFn(state); if (state.interruptionDetails) { console.log(`[${state.workflowId}] Suspended at: ${state.lastExecutedNode}`); await this.saveState(state); return state; } state.currentState = typeof stepConfig.nextState === 'function' ? stepConfig.nextState(state) : stepConfig.nextState || 'complete'; await this.saveState(state); } catch (error: any) { console.error(`[${workflowId}] Node failed:`, error.message); state.currentState = 'failed'; state.data.error = error.message; await this.saveState(state); return state; } } return state; } public async resumeWithHumanInput( workflowId: string, humanInput: { decision: 'approved' | 'rejected'; comment?: string } ): Promise { let state = await this.loadState(workflowId); if (!state) throw new Error(`Workflow ${workflowId} not found.`); if (!state.interruptionDetails) throw new Error(`Workflow ${workflowId} is not paused.`); state.data.humanDecision = humanInput.decision; state.data.humanComment = humanInput.comment; state.interruptionDetails = undefined; if (humanInput.decision === 'approved') { state.data.humanApproved = true; state.currentState = 'propose_action'; } else { state.data.humanApproved = false; state.currentState = 'failed'; state.data.error = 'Operation rejected by human reviewer.'; } await this.saveState(state); return this.execute(workflowId); } } ``` **The tell:** kill the process mid-run and start it again. If the agent repeats a tool call it already made, the save is landing on the wrong side of the node boundary. --- ## What breaks it in production **Transactional persistence.** The in-memory `Map` above is development-only. In production, replace it with Redis (for low-latency state reads) or Postgres (for durable ACID transactions). Add optimistic locking to prevent concurrent writes from two resume calls on the same workflow. **Audit logging.** Record operator user ID, timestamp, decision, and justification comment alongside every approval state transition. This is non-negotiable for any operation that affects financial records, infrastructure, or customer data. **Idempotent node execution.** Nodes that make external API calls must handle retries gracefully. If a process crash occurs after `deploy_code` executes but before the state is saved, the orchestrator will re-invoke the node on resume. Either use idempotency keys or check whether the action already completed before executing. **Rejection paths need explicit states.** A rejected approval should not silently set `currentState = 'failed'` and stop. In financial or infrastructure workflows, rejected operations often need a compensation step: reversing a partial action, notifying a queue, or escalating to a higher approval tier. --- ## When this is the wrong choice - **Low-stakes reversible operations.** If the action can be undone in under 30 seconds, the approval latency usually costs more than the mistake. - **High-throughput pipelines.** Approval gates that block on human response time don't work at scale. Pre-approve categories of operations and use post-hoc audit review instead. - **When the gate is always approved.** An approval gate that gets rubber-stamped 99% of the time provides no safety value and adds latency. Profile your approval decisions before building the interrupt infrastructure. --- Source: https://himanshuat.com/blogs/production-grade-agent-architecture-memory-human-in-the-loop ======================================================================== --- title: "Production-Grade RAG: A Blueprint for Scalable, Real-Time Architecture" description: "A RAG system fails quietly when it answers from documents that were modified or deleted upstream. Real-time accuracy requires event-driven change data capture, explicit vector index configuration, tiered semantic caching, and telemetry that monitors silent retrieval drift." date: "November 27, 2025" url: "https://himanshuat.com/blogs/production-grade-rag-architecture-blueprint" --- # Production-Grade RAG: A Blueprint for Scalable, Real-Time Architecture The failure mode that degrades production RAG systems is rarely a hard process crash. It is a confident answer synthesized from a document that was updated or deleted upstream, with no HTTP 500 in the trace. Prototypes built with simple document loaders succeed on static corpora. Handling high concurrency, low latency, and real-time freshness in production requires decoupling ingestion, storage, and retrieval services. > The prototype problem is retrieval quality. The production problem is data freshness, and freshness is governed by the ingestion pipeline rather than the generator. Decoupling the pipeline lets each service scale, fail, and recover on its own. ### Chapter 0: Core service boundaries A production RAG deployment is four services: **ingestion**, the **vector database**, the **retrieval gateway**, and the **telemetry plane**. Keeping them separate stops an ingestion spike from starving user-facing queries, and it means a latency regression has one owner instead of four suspects. --- ### Step 1: Push updates via event-driven Change Data Capture Batch re-indexing fails as corpora grow. Periodic re-indexing introduces long staleness windows and consumes excessive GPU embedding hours. Change Data Capture (CDC) via Kafka or AWS SQS streams document mutations into workers in real time: ```mermaid graph TD A["Source Systems (Databases, CMS, APIs)"] --> B("Change Data Capture / Webhooks") B --> C("Message Queue (e.g. Kafka)") C --> D["Ingestion Service (Workers)"] D -- Embeddings --> E["Embedding Service (API / Local Triton)"] D -- Upsert / Delete --> F[Vector Database] D -- Metadata / Content --> G["Metadata Store (PostgreSQL / S3)"] ``` Key ingestion requirements: 1. **Event-driven mutations:** Source systems emit discrete `upsert` and `delete` events. 2. **Buffer isolation:** The queue buffers traffic surges during bulk uploads, isolating vector index nodes from write thrashing. 3. **Idempotent workers:** Upsert operations use deterministic hash keys so reprocessing messages produces zero duplicate records. ```typescript // src/ingestion-worker.ts import { Kafka } from 'kafkajs'; import { EmbeddingService } from './embeddingService'; import { VectorDbClient } from './vectorDbClient'; interface DocumentChangeEvent { id: string; content: string; metadata: Record; type: 'upsert' | 'delete'; timestamp: string; } const kafka = new Kafka({ brokers: [process.env.KAFKA_BROKERS || 'localhost:9092'] }); const consumer = kafka.consumer({ groupId: 'rag-ingestion-group' }); const embeddingService = new EmbeddingService(); const vectorDbClient = new VectorDbClient(); async function startIngestionWorker() { await consumer.connect(); await consumer.subscribe({ topic: 'document-changes', fromBeginning: false }); await consumer.run({ eachMessage: async ({ topic, partition, message, heartbeat }) => { if (!message.value) return; try { const event: DocumentChangeEvent = JSON.parse(message.value.toString()); if (event.type === 'upsert') { const embedding = await embeddingService.getEmbedding(event.content); await vectorDbClient.upsert({ id: event.id, vector: embedding, metadata: { ...event.metadata, ingestion_timestamp: event.timestamp }, content: event.content }); } else if (event.type === 'delete') { await vectorDbClient.delete(event.id); } await heartbeat(); } catch (error) { console.error(`Ingestion error at offset ${message.offset}:`, error); } }, }); } startIngestionWorker().catch(err => { console.error("Ingestion worker startup failed:", err); process.exit(1); }); ``` The `await heartbeat()` call is doing real work. [KafkaJS's consumer documentation](https://kafka.js.org/docs/consuming) warns that the `eachMessage` handler "should not block for longer than the configured session timeout or else the consumer will be removed from the group," and tells you to either raise the session timeout or call `heartbeat` periodically from inside the handler. Handle the `delete` events explicitly. Without them, a document removed upstream keeps surfacing in top-$k$ searches and keeps getting cited. **The tell:** delete a document at the source, then query for something only that document says. If the answer still comes back confident, your ingestion path has no delete. --- ### Step 2: Configure the vector database for partitioned metadata filtering Qdrant, OpenSearch, Milvus, and pgvector all ship defaults that are wrong at scale. Tune the index explicitly. The `M`, `ef_construction`, and `ef_search` knobs below are not vendor inventions — they come from [Malkov and Yashunin's 2016 HNSW paper](https://arxiv.org/abs/1603.09320), which defines the multi-layer proximity graph these databases build their indexes on. The quantization figure is Qdrant's own: its [quantization guide](https://qdrant.tech/documentation/guides/quantization/) puts scalar quantization at a 4x memory reduction going from float32 to uint8, with an error "usually less than 1%." Primary scaling parameters: | Parameter | Function | Production trade-off | |---|---|---| | Sharding | Horizontal distribution across nodes | Increases write capacity and index memory ceiling | | Replication factor | High availability read replicas | Increases concurrent QPS; requires more RAM | | HNSW `M` & `ef_construction` | Graph connectivity during build | Higher values improve recall at the cost of index build latency and memory | | HNSW `ef_search` | Search beam width | Higher values increase Hit@k at the cost of query latency | | Vector quantization (Scalar / Product) | Compression (FP32 to INT8/INT4) | Reduces memory footprint by up to 4x with minimal recall loss | The client below is [`@qdrant/js-client-rest`](https://github.com/qdrant/qdrant-js), the REST package from Qdrant's official JavaScript SDK, which exposes its schema types through the `Schemas` export. ```typescript // src/vectorDbClient.ts import { QdrantClient, Schemas } from '@qdrant/js-client-rest'; export class VectorDbClient { private client: QdrantClient; private collectionName: string; constructor( host: string = process.env.QDRANT_HOST || 'localhost', port: number = parseInt(process.env.QDRANT_PORT || '6333', 10), collectionName: string = process.env.QDRANT_COLLECTION || 'rag_documents' ) { this.client = new QdrantClient({ host, port }); this.collectionName = collectionName; } async upsert(document: { id: string; vector: number[]; metadata: Record; content?: string }) { const point: Schemas['PointStruct'] = { id: document.id, vector: document.vector, payload: { ...document.metadata, content: document.content || null } }; await this.client.upsert(this.collectionName, { wait: true, points: [point], }); } async query( queryVector: number[], limit: number = 5, filters?: Schemas['Filter'] ): Promise }>> { const searchResult = await this.client.search(this.collectionName, { vector: queryVector, limit: limit, filter: filters, with_payload: true, with_vectors: false, }); return searchResult.map(hit => ({ id: hit.id, score: hit.score, payload: hit.payload || {}, })); } async delete(id: string) { await this.client.delete(this.collectionName, { pointsSelector: { points: [id] } }); } } ``` Setting `with_vectors: false` during retrieval minimizes network serialization overhead by returning only text payloads and scores. --- ### Step 3: Implement tiered caching with granular TTLs Caching cuts LLM cost and p50/p95 latency. The catch is that the three artifacts in the retrieval flow go stale at different rates, so they cannot share a TTL: | Cache layer | Key composition | Recommended TTL | Invalidation rationale | |---|---|---|---| | Query embedding | SHA-256 of normalized query text | 24 hours | Deterministic vector representation | | Candidate documents | Query hash + metadata filter hash + top-$k$ | 300 seconds | Aligns with ingestion pipeline update cadence | | Generated LLM answer | Query hash + sorted retrieved chunk IDs + filter hash | 120 seconds | Short window to catch concurrent duplicate queries | ```typescript // src/retrievalService.ts import { EmbeddingService } from './embeddingService'; import { VectorDbClient } from './vectorDbClient'; import { LLMClient } from './llmClient'; import { RedisClient } from './redisClient'; import crypto from 'crypto'; export class RetrievalService { private embeddingService: EmbeddingService; private vectorDbClient: VectorDbClient; private llmClient: LLMClient; private redis: RedisClient; constructor() { this.embeddingService = new EmbeddingService(); this.vectorDbClient = new VectorDbClient(); this.llmClient = new LLMClient(); this.redis = new RedisClient(); } private generateCacheKey(input: string): string { return crypto.createHash('sha256').update(input).digest('hex'); } private async getCached(key: string, fetchFn: () => Promise, ttlSeconds: number): Promise { const cached = await this.redis.get(key); if (cached) { return JSON.parse(cached) as T; } const result = await fetchFn(); await this.redis.set(key, JSON.stringify(result), ttlSeconds); return result; } async retrieveAndGenerate( userQuery: string, filters?: Record, numDocuments: number = 5 ): Promise { // 1. Cached Query Embedding const embeddingCacheKey = this.generateCacheKey(`embedding:${userQuery}`); const queryEmbedding = await this.getCached( embeddingCacheKey, () => this.embeddingService.getEmbedding(userQuery), 86400 // 24 hours ); // 2. Cached Vector Retrieval const filterString = filters ? JSON.stringify(filters) : ''; const retrievalCacheKey = this.generateCacheKey(`retrieval:${userQuery}:${filterString}:${numDocuments}`); const retrievedDocs = await this.getCached( retrievalCacheKey, async () => { const hits = await this.vectorDbClient.query(queryEmbedding, numDocuments, filters); return hits.map(hit => ({ content: hit.payload.content as string, metadata: hit.payload as Record })); }, 300 // 5 minutes ); if (retrievedDocs.length === 0) { return await this.llmClient.generateResponse([{ role: 'user', content: userQuery }]); } // 3. Prompt Assembly & Cached Generation const context = retrievedDocs.map(doc => doc.content).join('\n\n---\n\n'); const promptMessages = [ { role: 'system' as const, content: 'You are an accurate technical assistant. Use only the provided context.' }, { role: 'user' as const, content: `Context:\n${context}\n\nQuestion: ${userQuery}` } ]; const sortedDocIds = retrievedDocs.map(d => d.metadata.id || '').sort().join(','); const responseCacheKey = this.generateCacheKey(`response:${userQuery}:${sortedDocIds}:${filterString}`); return await this.getCached( responseCacheKey, () => this.llmClient.generateResponse(promptMessages), 120 // 2 minutes ); } } ``` --- ### Step 4: Monitor telemetry for silent retrieval degradation CPU load and HTTP 200 counts stay green through every failure this post describes. Instrument the pipeline stages instead: | Subsystem | Telemetry signals | Anomaly indicator | |---|---|---| | Ingestion Queue | Consumer lag, ingestion worker latency, DLQ counts | Queue backlog growing steadily | | Vector Index | p99 query latency, cosine score distributions | Score distribution shifting downward over time | | Cache Tier | Hit rates per tier (Embedding, Retrieval, Generation) | Unexplained drop in retrieval cache hit rate | | LLM Gateway | Generation latency, input/output token counts | Token usage spike or unexpected refusal increase | ```typescript // src/telemetry.ts type MetricType = 'counter' | 'gauge' | 'histogram'; export function logMetric(name: string, value: number, type: MetricType, labels?: Record) { const labelStr = labels ? Object.entries(labels).map(([k, v]) => `${k}="${v}"`).join(',') : ''; console.log(`METRIC: ${name}{${labelStr}} ${value} (type: ${type}, timestamp: ${Date.now()})`); } ``` --- ### When this is the wrong choice - **The corpus does not change.** Everything above is built around freshness. Against a fixed set of documents, a batch job and one index give you identical answers without Kafka, a metadata store, or consumer lag to watch. - **One service, low QPS.** Sharding, replication, and three cache tiers exist to keep ingestion from contending with queries. Below the point where they contend, you have replaced one moving part with six, and each one can be the thing that is down. - **Answers must reflect the last edit immediately.** The candidate-document cache holds for 300 seconds and the answer cache for 120. In a workflow where a correction has to be visible in the next answer, the cache tier is the thing producing the silent staleness this post opens with, and you should turn it off rather than tune it. --- ### Key drift indicators - **Similarity score percentiles:** Track the p50 and p95 cosine similarity scores of top-$k$ retrieved chunks. A persistent decline indicates semantic drift or missing domain documentation. - **Null retrieval frequency:** Track the proportion of queries that return zero passages above your minimum relevance threshold. - **Source collection skew:** Monitor the categorical distribution of retrieved documents to detect when search results over-index on stale collections. --- Source: https://himanshuat.com/blogs/production-grade-rag-architecture-blueprint ======================================================================== --- title: "RAG Foundations: From First Query to a Retriever You Can Actually Trust" description: "Building a RAG system is straightforward. Building one whose retrieval you can measure, debug, and improve is not. This post covers chunking strategies with real Hit Rate and MRR comparisons, embedding model selection against concrete constraints, and the diagnostic checks that matter before you touch the generation side." date: "June 12, 2025" url: "https://himanshuat.com/blogs/rag-foundations" --- # RAG Foundations: From First Query to a Retriever You Can Actually Trust The first RAG system I shipped took about a weekend. The retriever returned plausible-looking chunks, the generator produced fluent answers, and manual spot-checks looked fine. Then I ran it on 200 representative queries with ground-truth answers. Hit Rate at k=5 was 51%. Nearly half the queries never retrieved the document that contained the answer. The generator had been producing confident, well-written hallucinations. > Fluent output is not evidence of correct retrieval. Measure retrieval separately before you evaluate generation quality. This post covers the decisions that actually moved retrieval accuracy: chunk boundary strategy, embedding model selection against real constraints, and the two metrics that replace guesswork. --- ### Chapter 0: What retrieval failure looks like Standard RAG is sequential: embed the query, fetch top-k chunks, construct a prompt, generate. The assumption is that top-k retrieval achieved adequate recall. When it doesn't, the failure is invisible. The model synthesizes a paragraph using whatever was in the context window. If the relevant passage wasn't retrieved, the answer either hallucinates or hedges politely. Both look reasonable to a reviewer who doesn't know the ground truth. Before touching chunk size or embedding models, run 50 to 100 representative queries against your index and check whether the correct document appeared in the results at all. If Hit Rate at k=5 is below 70%, fix retrieval before everything else. --- ### Step 1: Fixed-size chunking and where it breaks Fixed-size chunking splits text every N tokens, with an optional overlap window: ```typescript interface TextChunk { content: string; startIndex: number; endIndex: number; metadata: { sourceId: string; chunkIndex: number; }; } interface ChunkingOptions { chunkSize: number; overlapSize: number; } function fixedSizeChunker(text: string, options: ChunkingOptions): TextChunk[] { const { chunkSize, overlapSize } = options; const chunks: TextChunk[] = []; if (chunkSize <= 0 || overlapSize < 0 || overlapSize >= chunkSize) { throw new Error(`Invalid options: chunkSize=${chunkSize}, overlapSize=${overlapSize}`); } let currentPos = 0; let chunkIndex = 0; while (currentPos < text.length) { const endPos = Math.min(currentPos + chunkSize, text.length); const chunkContent = text.slice(currentPos, endPos); if (chunkContent.trim().length > 0) { chunks.push({ content: chunkContent, startIndex: currentPos, endIndex: endPos, metadata: { sourceId: 'doc-001', chunkIndex: chunkIndex++ }, }); } if (endPos === text.length) break; currentPos = endPos - overlapSize; } return chunks; } ``` Fixed-size chunking works well on uniform prose. It fails on technical documents where key facts sit at section boundaries. The 512-token window cuts a sentence in half. Half lands in one chunk, half in the next, and neither chunk contains a complete retrievable fact. The failure mode to watch: overlap progress stalls. When a natural break snaps `endPos` back close to `startPos`, the next `currentPos = endPos - overlapSize` can move forward by only a few tokens. Text between the snap point and the forced forward index gets silently dropped. --- ### Step 2: Semantic chunking on embedding similarity drops Semantic chunking embeds sentences and cuts on cosine similarity drops, where meaning shifts rather than where the token count hits a threshold: ```typescript function cosineSimilarity(vec1: number[], vec2: number[]): number { const dot = vec1.reduce((sum, val, i) => sum + val * vec2[i], 0); const mag1 = Math.sqrt(vec1.reduce((sum, val) => sum + val * val, 0)); const mag2 = Math.sqrt(vec2.reduce((sum, val) => sum + val * val, 0)); return mag1 > 0 && mag2 > 0 ? dot / (mag1 * mag2) : 0; } async function semanticChunker( text: string, embeddingFunction: (text: string) => Promise, options: { similarityThreshold: number; minChunkSize: number; maxChunkSize: number; } ): Promise { const sentences = text .split(/(?<=[.!?])\s+(?=[A-Z])/) .map(s => s.trim()) .filter(s => s.length > 0); if (sentences.length === 0) return []; const embeddings = await Promise.all(sentences.map(embeddingFunction)); const chunks: TextChunk[] = []; let currentGroup: string[] = [sentences[0]]; let startIndex = 0; for (let i = 1; i < sentences.length; i++) { const similarity = cosineSimilarity(embeddings[i - 1], embeddings[i]); const currentGroupText = currentGroup.join(' '); const potentialSize = currentGroupText.length + sentences[i].length; const shouldSplit = (similarity < options.similarityThreshold && currentGroupText.length >= options.minChunkSize) || potentialSize > options.maxChunkSize; if (shouldSplit) { const chunkContent = currentGroup.join(' '); chunks.push({ content: chunkContent, startIndex, endIndex: startIndex + chunkContent.length, metadata: { sourceId: 'doc-semantic', chunkIndex: chunks.length }, }); startIndex += chunkContent.length + 1; currentGroup = [sentences[i]]; } else { currentGroup.push(sentences[i]); } } if (currentGroup.length > 0) { const chunkContent = currentGroup.join(' '); chunks.push({ content: chunkContent, startIndex, endIndex: startIndex + chunkContent.length, metadata: { sourceId: 'doc-semantic', chunkIndex: chunks.length }, }); } return chunks; } ``` The similarity threshold doesn't generalize across domains. A cutoff of 0.7 works for prose but over-segments dense technical manuals and financial tables. Profile the similarity distribution on your actual corpus before setting a hard number. --- ### Step 3: Embedding model selection against real constraints The embedding model choice should be driven by a measurable constraint, not benchmarks on someone else's dataset. | Model | Source | When to use | |---|---|---| | `bge-small-en-v1.5` | BAAI, open weights | Memory-constrained or on-device; runs on CPU | | `E5-large-v2` | Microsoft, open weights | Highest open-weights accuracy on long technical context | | `all-MiniLM-L6-v2` | Open weights | Fast CPU inference, moderate fidelity | | `text-embedding-3-small` | OpenAI API | Configurable dimension truncation via Matryoshka; lowest API cost | | `text-embedding-3-large` | OpenAI API | Highest API-based accuracy; use when latency budget allows | For air-gapped environments or p99 latency SLAs under 50ms, `bge-small-en-v1.5` via ONNX Runtime was the only viable option on the projects I ran. For everything else, `text-embedding-3-small` with truncated dimensions (768 instead of 1536) cut storage cost in half with less than 2% Hit Rate degradation on my domain test set. If you can't state the specific constraint that dictated your choice (latency, RAM footprint, cost per million tokens, or measured Hit Rate on your corpus), you picked arbitrarily. --- ### Step 4: Measure retrieval with Hit Rate and MRR Two metrics replace guesswork. Build a validation set of 50 to 200 representative queries with ground-truth chunk labels before running any experiments. **Hit Rate at k.** What fraction of queries retrieved at least one correct chunk in the top k results: $$ HitRate@k = \frac{1}{|Q|} \sum_{q=1}^{|Q|} \mathbb{I}(\text{rank}_q \le k) $$ **Mean Reciprocal Rank at k.** How high the first correct chunk ranks on average. Penalizes pipelines that retrieve the right chunk but bury it at position 4 or 5: $$ MRR@k = \frac{1}{|Q|} \sum_{q=1}^{|Q|} \frac{1}{\text{rank}_q} $$ ```typescript async function runEvaluation( retriever: { retrieve: (q: string, k: number) => Promise }, queries: string[], groundTruth: Map>, k: number = 5 ): Promise<{ hitRate: number; mrr: number }> { let totalHit = 0; let totalReciprocalRank = 0; for (const query of queries) { const retrieved = await retriever.retrieve(query, k); const relevant = groundTruth.get(query) || new Set(); let rankOfFirst = 0; for (let i = 0; i < retrieved.length; i++) { const isRelevant = Array.from(relevant).some( gt => retrieved[i].content.includes(gt) || gt.includes(retrieved[i].content) ); if (isRelevant) { totalHit++; rankOfFirst = i + 1; break; } } if (rankOfFirst > 0) totalReciprocalRank += 1 / rankOfFirst; } return { hitRate: totalHit / queries.length, mrr: totalReciprocalRank / queries.length, }; } ``` Hold the test queries and embedding model constant, swap only the chunker, and the delta in Hit Rate and MRR is the actual effect of your chunking decision. Without this, you're guessing. On my own domain corpus (legal contracts, ~4,000 documents): | Chunker | Chunk Size | Hit Rate@5 | MRR@5 | |---|---|---|---| | Fixed-size | 512 tokens | 0.51 | 0.38 | | Fixed-size + overlap | 512 / 64 overlap | 0.61 | 0.44 | | Semantic (threshold 0.75) | Variable | 0.74 | 0.57 | | Semantic + reranker | Variable | 0.81 | 0.67 | The reranker (a cross-encoder run over the top-20 candidates before generation) added the biggest single jump. It's also the most expensive. Profile your p99 latency budget before you add it. --- ### Step 5: Architectural extensions worth evaluating in order Once your baseline Hit Rate is above 70% and you have a working eval loop, these additions are worth testing in order of typical impact: **Hybrid retrieval (BM25 + dense vector, fused via RRF):** Sparse lexical matching handles exact keyword queries that dense embeddings miss. Reciprocal Rank Fusion merges the two ranked lists without requiring calibrated score thresholds. **Hierarchical indexing:** Embed at two granularities: section summaries and sentence-level chunks. Retrieve candidate sections first, then pass child chunks to the context window. Reduces irrelevant context without sacrificing recall. **Query rewriting:** Use an LLM to generate a hypothetical ideal document (HyDE) or decompose multi-hop questions before searching. Adds latency; only worth it if your queries are consistently ambiguous or multi-part. **Context reranking:** A cross-encoder (like `bge-reranker-large`) re-scores the top-20 candidates with full token-level attention. The largest accuracy gain, but ~150ms additional latency per query. --- ### Common failure modes **Regex sentence splitters fail on technical text.** Simple period-based splitting breaks on version numbers (`v2.1.0`), decimal values, code snippets, and abbreviations. Every downstream metric inherits that segmentation error silently. **String containment inflates eval scores.** Evaluating chunk relevance with `includes()` can match boilerplate phrases shared across many chunks. Use token overlap (ROUGE-L) or exact identifier matching instead. **Embedding model version drift.** Re-embedding documents with a newer model version without schema versioning corrupts vector space comparisons. Store the exact model identifier per row. --- ## When this is the wrong choice - **Your documents are already segmented.** FAQ databases, support tickets, API documentation, and code docstrings arrive with natural boundaries. Semantic chunking pays embedding cost at ingestion time to rediscover a split the source already gave you. - **The corpus is single-topic throughout.** Similarity rarely drops below any threshold, so the semantic chunker emits roughly what fixed-size would have emitted, for more money and one more tunable. - **You have no eval set yet.** Swapping chunkers without Hit Rate and MRR gives you no signal about which one won. Build the 50 to 200 query validation set first; every comparison in this post is worthless without it. - **You have a hard latency budget.** The cross-encoder reranker was the largest single accuracy jump in my table and it adds roughly 150ms per query. If that doesn't fit under your p99, take the Hit Rate loss and skip it. --- Source: https://himanshuat.com/blogs/rag-foundations ======================================================================== --- title: "Part 3: Scaling LangGraph: State Persistence, Checkpointing, and Parallelism" description: "Long-running multi-agent workflows require durable state persistence and concurrency management. Here is how to configure SQLite and Redis checkpointers alongside Pregel super-step parallel branching in LangGraph." date: "May 03, 2024" url: "https://himanshuat.com/blogs/scaling-langgraph-persistence-and-parallelism" --- # Part 3: Scaling LangGraph: State Persistence, Checkpointing, and Parallelism An agent process throws midway through execution. Every message it accumulated and every intermediate reasoning step it produced lived in memory, so all of it goes with the process. > A graph without a checkpointer is not resumable. One exception discards every super-step that already succeeded, and a retry pays for all of them again. A stateless workflow that restarts looks like it recovered. It started over. Anything you run in production needs checkpoint persistence, and once you have that, you can also run branches concurrently. ### Chapter 0: Checkpointing mechanisms in graph runtimes A **checkpointer** serializes the entire state of a graph to external storage (such as SQLite, Redis, or PostgreSQL) after each super-step. Stored data includes accumulated message lists, custom state variables, node execution markers, and routing metadata. To resume execution after an interruption or process restart, the graph loads the state snapshot associated with a `thread_id` and continues execution from the last completed node. --- ### Step 1: Local persistence with SqliteSaver For single-instance deployments or local testing, `SqliteSaver` serializes state transitions directly into SQLite tables. The `thread_id` is the primary lookup key for state snapshots: ```python import os from typing import TypedDict, List from langgraph.graph import StateGraph, START from langgraph.checkpoint.sqlite import SqliteSaver class GraphState(TypedDict): messages: List[str] turn: int def chat_node(state: GraphState) -> dict: current_turn = state.get("turn", 0) new_message = f"Agent turn {current_turn}: Step complete." return { "messages": state.get("messages", []) + [new_message], "turn": current_turn + 1 } memory = SqliteSaver.from_conn_string(":memory:") workflow = StateGraph(GraphState) workflow.add_node("chat", chat_node) workflow.set_entry_point("chat") workflow.add_edge("chat", "chat") app = workflow.compile(checkpointer=memory) thread_id = "session_001" initial_state = {"messages": ["User: Run initial batch."], "turn": 0} print(f"Executing thread '{thread_id}':") for i in range(2): output = app.invoke( input=initial_state if i == 0 else {}, config={"configurable": {"thread_id": thread_id}} ) print(output['messages'][-1]) print("\nResuming from saved checkpoint:") output_resumed = app.invoke( input={}, config={"configurable": {"thread_id": thread_id}} ) print("Resumed output:", output_resumed['messages'][-1]) print("Total history length:", len(output_resumed['messages'])) ``` Invoking the compiled graph with an empty input dictionary and an existing `thread_id` automatically retrieves the latest checkpoint from SQLite and executes the next step. --- ### Step 2: Distributed persistence with RedisSaver In distributed environments where multiple worker nodes process tasks concurrently, single-file SQLite databases create file-lock contention. `RedisSaver` moves state snapshots into a networked in-memory key-value store, so any available worker can pick up a suspended thread: ```python import os import redis from typing import TypedDict, List from langgraph.graph import StateGraph from langgraph.checkpoint.redis import RedisSaver class GraphState(TypedDict): messages: List[str] turn: int def chat_node(state: GraphState) -> dict: current_turn = state.get("turn", 0) new_message = f"Turn {current_turn} processed via Redis." return { "messages": state.get("messages", []) + [new_message], "turn": current_turn + 1 } # Connect to Redis redis_url = os.getenv("REDIS_URL", "redis://localhost:6379/0") memory = RedisSaver.from_conn_info(host="localhost", port=6379, db=0) workflow = StateGraph(GraphState) workflow.add_node("chat", chat_node) workflow.set_entry_point("chat") workflow.add_edge("chat", "chat") app_redis = workflow.compile(checkpointer=memory) thread_id_redis = "distributed_thread_42" initial_state = {"messages": ["Init distributed task."], "turn": 0} output = app_redis.invoke( input=initial_state, config={"configurable": {"thread_id": thread_id_redis}} ) print("First turn output:", output['messages'][-1]) ``` | Checkpointer | Storage engine | Concurrency model | Best fit | |---|---|---|---| | `SqliteSaver` | Local SQLite file / memory | Single-process only | Local development and tests | | `RedisSaver` | Redis cluster | Multi-process networked workers | Low-latency distributed agents | | `PostgresSaver` | PostgreSQL tables | Transactional ACID storage | High-durability enterprise workflows | --- ### Step 3: Branch parallelism via Pregel super-steps LangGraph executes graphs using the Pregel computational model. Computations progress through synchronized super-steps: 1. In each super-step, all nodes whose inbound edge dependencies are met execute concurrently. 2. Nodes emit state updates and pass messages to downstream targets. 3. The super-step completes when all active nodes finish their execution cycle. To execute tasks in parallel, define multiple outgoing edges from a single parent node without adding edges between the sibling workers: ```mermaid flowchart LR P[plan_topics] --> A[research_ethics] P --> B[research_quantum] P --> C[research_neuroscience] A --> W[compile_report] B --> W C --> W W --> E[END] ``` ```python from typing import List, Literal, TypedDict from langgraph.graph import StateGraph, END import time class MultiAgentState(TypedDict): research_topics: List[str] report_sections: List[str] final_report: str current_step: Literal["plan", "research", "write", "finish"] def plan_topics(state: MultiAgentState) -> dict: topics = ["AI Ethics", "Quantum Computing", "Neuroscience in AI"] print("Planner assigned topics:", topics) return {"research_topics": topics, "current_step": "research"} def research_ai_ethics(state: MultiAgentState) -> dict: time.sleep(0.1) section = "AI Ethics: Evaluated algorithmic transparency and alignment." return {"report_sections": state.get("report_sections", []) + [section]} def research_quantum_computing(state: MultiAgentState) -> dict: time.sleep(0.08) section = "Quantum Computing: Analyzed qubit coherence algorithms." return {"report_sections": state.get("report_sections", []) + [section]} def research_neuroscience_ai(state: MultiAgentState) -> dict: time.sleep(0.12) section = "Neuroscience in AI: Modeled sparse activation graphs." return {"report_sections": state.get("report_sections", []) + [section]} def compile_report(state: MultiAgentState) -> dict: sections = "\n".join(state.get("report_sections", [])) report = f"Consolidated Findings:\n{sections}" return {"final_report": report, "current_step": "finish"} workflow_parallel = StateGraph(MultiAgentState) workflow_parallel.add_node("plan", plan_topics) workflow_parallel.add_node("research_ethics", research_ai_ethics) workflow_parallel.add_node("research_quantum", research_quantum_computing) workflow_parallel.add_node("research_neuroscience", research_neuroscience_ai) workflow_parallel.add_node("compile_report", compile_report) workflow_parallel.set_entry_point("plan") # Fan-out to independent research nodes workflow_parallel.add_edge("plan", "research_ethics") workflow_parallel.add_edge("plan", "research_quantum") workflow_parallel.add_edge("plan", "research_neuroscience") # Fan-in synchronization barrier: compile_report waits for all three nodes workflow_parallel.add_edge("research_ethics", "compile_report") workflow_parallel.add_edge("research_quantum", "compile_report") workflow_parallel.add_edge("research_neuroscience", "compile_report") workflow_parallel.add_edge("compile_report", END) app_parallel = workflow_parallel.compile() ``` The runtime executes `research_ethics`, `research_quantum`, and `research_neuroscience` in parallel within the same super-step, waiting for all three to complete before triggering `compile_report`. --- ### Step 4: Node-level async concurrency Pregel coordinates high-level graph topology. However, I/O-bound tasks within individual nodes (such as querying multiple APIs or embedding text batches) should use native asynchronous Python concurrency: ```python import asyncio import httpx async def fetch_api_data(endpoints: List[str]) -> List[dict]: """Fetches multiple external endpoints concurrently using httpx.""" async with httpx.AsyncClient() as client: tasks = [client.get(url, timeout=10.0) for url in endpoints] responses = await asyncio.gather(*tasks, return_exceptions=True) valid_results = [] for r in responses: if isinstance(r, httpx.Response) and r.status_code == 200: valid_results.append(r.json()) return valid_results ``` Using `asyncio.gather` inside node definitions prevents network round trips from blocking the overall Pregel event loop. --- ## When this is the wrong choice - **The whole run is one request and cheap to repeat.** A checkpointer writes the full graph state to storage after every super-step. If re-running the workflow costs one model call, those writes buy you nothing and add a storage dependency to a process that had none. - **The concurrency you need lives inside a single node.** Splitting three API calls into three graph nodes gets you the same wall-clock time as `asyncio.gather` in one node, plus three checkpoint writes and a fan-in barrier. Fan out in the graph only when the branches are separately resumable. - **The branches depend on each other.** A super-step runs nodes whose inbound edges are satisfied. If the quantum research needs the ethics section first, the barrier serializes them anyway, and you have written a parallel-looking graph that executes sequentially. - **You are on one process and staying there.** `SqliteSaver` covers this. Redis and PostgreSQL exist for multiple workers contending on the same threads; adopting them earlier adds an operational service to keep alive for no concurrency you actually have. --- ### Architecture rules for scaled graphs 1. **State payload optimization:** avoid storing raw binary files or extensive document corpora in checkpoint stores. Store pointers (such as S3 URIs or database primary keys) to prevent storage bloat. 2. **Deterministic reducer policies:** when multiple parallel nodes write to the same list field within a super-step, use explicit reducer functions to prevent race conditions. 3. **Appropriate store selection:** use SQLite for local unit testing, Redis for low-latency session caching, and PostgreSQL for long-term auditable checkpoints. --- Source: https://himanshuat.com/blogs/scaling-langgraph-persistence-and-parallelism ======================================================================== --- title: "The Self-Correcting RAG: Implementing Agentic and Recursive Retrieval Loops" description: "Static RAG retrieves once and assumes top-k results contain complete context. Recursive retrieval loops inspect accumulated context, identify missing facts, synthesize targeted follow-up queries, and terminate on explicit criteria or iteration bounds." date: "November 18, 2025" url: "https://himanshuat.com/blogs/self-correcting-rag-agentic-recursive-retrieval" --- # The Self-Correcting RAG: Implementing Agentic and Recursive Retrieval Loops A single-shot search returns five candidate chunks. The model synthesizes a fluent paragraph, but the output fails in the specific area where those five passages were silent. Standard RAG architectures execute sequentially: embed query, retrieve top-$k$, construct prompt, generate answer. This assumes the initial search step achieved high recall and precision. > Single-shot pipelines cannot detect when initial retrieval is incomplete. Reasoning over retrieved context requires evaluating whether current information satisfies query constraints before generating. Multi-hop queries and complex technical questions require iterative investigation: identify missing evidence, formulate targeted secondary queries, and aggregate evidence across retrieval steps. This is a published pattern, not a new one. [Self-RAG](https://arxiv.org/abs/2310.11511) (Asai et al., 2023) trains a language model to retrieve passages on demand and reflect on those passages and its own generations using special reflection tokens. [CRAG](https://arxiv.org/abs/2401.15884) (Yan et al., 2024) puts a lightweight retrieval evaluator in front of generation, returning a confidence degree that decides which knowledge action fires next. [FLARE](https://arxiv.org/abs/2305.06983) (Jiang et al., 2023) actively decides when and what to retrieve across the course of generation rather than once up front. What follows is the same control structure at the prompt level, with no training step. ### Chapter 0: Anatomy of an iterative retrieval loop An agentic retrieval loop connects retrieval and generation into a cyclical state machine. Embed the user query, run an initial vector search, then hand the accumulated context back to an evaluator along with the original query. The evaluator answers one question: does this context already support a complete answer? If not, it names the gap and writes a follow-up query. Synthesis runs once the evaluator says COMPLETE or the iteration cap trips. ```mermaid flowchart TD Q[user query] --> E[embed + search] E --> C[accumulate context] C --> J{context sufficient?} J -->|NEEDS_MORE_INFO| F[write follow-up query] F --> E J -->|COMPLETE| S[synthesize response] ``` --- ### Step 1: Wrap vector retrieval into an invokable primitive The retriever has to be callable from inside the loop, not wired into a one-shot pipeline. ```typescript interface VectorDB { search: (embedding: number[], k: number) => Promise; } interface Document { id: string; content: string; metadata?: Record; embedding?: number[]; } const getEmbedding = async (text: string): Promise => { const response = await fetch('https://api.openai.com/v1/embeddings', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`, }, body: JSON.stringify({ input: text, model: 'text-embedding-3-small', }), }); if (!response.ok) { throw new Error(`Embedding API error: ${response.statusText}`); } const data = await response.json(); return data.data[0].embedding; }; ``` Parameterized $k$ values allow broad candidate exploration on the initial pass ($k=5$) and targeted precision retrieval on follow-ups ($k=2$ or $3$). --- ### Step 2: Enforce structured schema validation on evaluation calls Loop control needs a deterministic output from the model, not freeform text. Pin the response format to JSON: ```typescript type LLMCallOptions = { model: string; temperature?: number; max_tokens?: number; }; const callLLM = async ( messages: Array<{ role: 'system' | 'user' | 'assistant', content: string }>, options: LLMCallOptions ): Promise => { const response = await fetch('https://api.openai.com/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`, }, body: JSON.stringify({ model: options.model, messages: messages, temperature: options.temperature ?? 0.0, max_tokens: options.max_tokens, response_format: { type: "json_object" } }), }); if (!response.ok) { throw new Error(`LLM API error: ${response.statusText}`); } const data = await response.json(); return data.choices[0].message.content; }; ``` `response_format: { type: "json_object" }` is JSON mode, and OpenAI's [structured outputs guide](https://developers.openai.com/api/docs/guides/structured-outputs) draws the line explicitly: "while both ensure valid JSON is produced, only Structured Outputs ensure schema adherence." So this call buys parseable JSON, not a guarantee that the fields the loop reads are present. If you want the schema enforced, that is `json_schema`, not `json_object`. --- ### Step 3: Extract structured critique and follow-up queries The evaluation prompt requires three fields: termination status, a critique explaining missing context, and a specific search query designed to resolve the gap. Tracking previous search queries prevents the model from repeating redundant lookups. ```typescript interface AgentDecision { status: 'COMPLETE' | 'NEEDS_MORE_INFO'; followUpQuery?: string; critique?: string; } const getAgentDecisionPrompt = ( originalQuery: string, currentContext: string[], accumulatedSearchHistory: string[], maxTokens: number ): Array<{ role: 'system' | 'user', content: string }> => [ { role: 'system', content: `You are an evaluation agent in a retrieval pipeline. Evaluate the supplied context against the user query. Determine if current context contains sufficient information to produce a complete, factual answer. If sufficient, return status: 'COMPLETE'. If insufficient, return status: 'NEEDS_MORE_INFO', provide a concise 'critique' of missing data, and formulate a targeted 'followUpQuery'. Output must be a valid JSON object matching the schema: {"status": "COMPLETE" | "NEEDS_MORE_INFO", "critique": string, "followUpQuery": string}.` }, { role: 'user', content: `Original Query: "${originalQuery}" --- Accumulated Context: ${currentContext.length > 0 ? currentContext.map((doc, i) => `Document ${i + 1}:\n${doc}`).join('\n---\n') : 'No context yet.'} --- Search History: ${accumulatedSearchHistory.length > 0 ? accumulatedSearchHistory.join('\n') : 'No previous searches.'} Evaluate context completeness:` } ]; const evaluateContext = async ( originalQuery: string, currentContext: string[], accumulatedSearchHistory: string[], llmOptions: LLMCallOptions, maxTokensForFinalAnswer: number ): Promise => { const prompt = getAgentDecisionPrompt(originalQuery, currentContext, accumulatedSearchHistory, maxTokensForFinalAnswer); const responseJson = await callLLM(prompt, llmOptions); try { return JSON.parse(responseJson) as AgentDecision; } catch (error) { console.error("Failed to parse agent decision JSON:", responseJson, error); return { status: 'NEEDS_MORE_INFO', followUpQuery: `Targeted search for "${originalQuery}" specifics`, critique: "JSON parse fallback triggered." }; } }; ``` --- ### Step 4: Bound loop execution with strict termination limits Unbounded recursive loops risk infinite recursion and runaway API costs. Enforce hard limits on iteration counts, token accumulation, and search width: ```typescript const MAX_RAG_ITERATIONS = 3; const FINAL_ANSWER_MAX_TOKENS = 1500; interface RecursiveRAGResult { answer: string; iterations: number; finalContext: string[]; searchHistory: string[]; } const runRecursiveRAG = async ( userQuery: string, vectorDB: VectorDB, llmOptions: LLMCallOptions, k_initial: number = 5, k_followup: number = 3 ): Promise => { let accumulatedContext: string[] = []; let searchHistory: string[] = []; let currentIteration = 0; let currentSearchQuery = userQuery; while (currentIteration < MAX_RAG_ITERATIONS) { currentIteration++; searchHistory.push(currentSearchQuery); const embedding = await getEmbedding(currentSearchQuery); const docs = await vectorDB.search(embedding, currentIteration === 1 ? k_initial : k_followup); const newContexts = docs.map(d => d.content); accumulatedContext = Array.from(new Set([...accumulatedContext, ...newContexts])); const decision = await evaluateContext( userQuery, accumulatedContext, searchHistory, llmOptions, FINAL_ANSWER_MAX_TOKENS ); if (decision.status === 'COMPLETE') { break; } else { currentSearchQuery = decision.followUpQuery || userQuery; } } const finalSynthesisPrompt = [ { role: 'system' as const, content: `Synthesize a concise, accurate answer based strictly on the provided context. If facts are incomplete, state the omission explicitly.` }, { role: 'user' as const, content: `Original Query: "${userQuery}"\n\nContext:\n${accumulatedContext.join('\n---\n')}` } ]; const finalAnswer = await callLLM(finalSynthesisPrompt, { ...llmOptions, max_tokens: FINAL_ANSWER_MAX_TOKENS }); return { answer: finalAnswer, iterations: currentIteration, finalContext: accumulatedContext, searchHistory: searchHistory }; }; ``` --- ### Step 5: Route evaluation and synthesis to appropriate model tiers The evaluation step performs a narrow classification task across iterations, while the synthesis step generates the final response. Using a fast, lightweight model (such as `gpt-4o-mini` or `claude-3-haiku`) for context evaluation reduces per-iteration latency. Reserve larger parameter models for the final synthesis pass. --- ### Key production failure modes - **Repetitive search queries:** If the model fails to incorporate previous search history, follow-up iterations retrieve identical chunks. Include search logs in evaluation prompts. - **Context dilution:** As documents accumulate across iterations, irrelevant text can overwhelm LLM attention. Apply context compression, deduplication, or [cross-encoder reranking](https://sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html) prior to final synthesis — the Sentence Transformers retrieve-and-rerank docs describe the two-stage shape, where a bi-encoder pulls candidates and a cross-encoder scores each query-document pair directly because it "performs attention across the query and the document." - **Latency overhead:** Each iteration adds sequential network calls (embedding generation, vector query, LLM evaluation). Measure end-to-end latency and verify that accuracy gains justify the multi-second execution cost. --- ## When this is the wrong choice - **The question is single-hop.** If the answer sits in one document, the first search finds it and the evaluator returns `COMPLETE` on iteration one. You paid an extra embedding call, a vector query, and an LLM classification to be told the pipeline was already done. - **You have a tight latency budget.** With `MAX_RAG_ITERATIONS = 3`, the worst case is three embedding calls, three vector queries, three evaluator calls, and a synthesis call, all sequential. That ceiling is the number to hold against your p99, not the average. - **Retrieval is broken rather than incomplete.** The loop assumes a follow-up query will surface what the first one missed. If the index is badly chunked or the embedding model is wrong for the corpus, every iteration searches the same broken space and the evaluator just spends money confirming it. --- Source: https://himanshuat.com/blogs/self-correcting-rag-agentic-recursive-retrieval ======================================================================== --- title: "Serverless GPUs for bursty training" description: "Our fine-tuning load arrived in bursts: intensive compute sprints followed by quiet intervals. Rented static GPU instances handle this pattern poorly. An architectural breakdown of splitting workloads between dedicated EC2 nodes and serverless Modal GPU workers." date: "June 17, 2025" url: "https://himanshuat.com/blogs/serverless-gpus-for-bursty-training" --- # Serverless GPUs for bursty training For long stretches, our GPU instance sat idle. Then client data arrived, we needed three LoRA variants trained concurrently by the weekend, and a single reserved node created an immediate bottleneck. The cost was only part of the problem. On small engineering teams without dedicated MLOps staff, maintaining a long-lived GPU server turns into a configuration liability. > A server you never rebuild is a machine whose configuration state is unknown. That debt appears when an experimental run fails for reasons that cannot be reproduced. ### Chapter 0: Defining burst workloads A burst workload has two defining characteristics: it is separated by long idle windows, and its primary efficiency metric is total wall-clock time to completion. Interactive research (debugging ComfyUI nodes, testing diffusion mask dilations on test images) requires an always-warm instance with zero cold-start overhead. Burst jobs (training four LoRA adapters in parallel across parameter sweeps) require elastic fan-out. Sizing a static instance for burst peaks leaves hardware idle most of the month; sizing it for interactive work forces training experiments into sequential queues. --- ### Step 1: Decouple interactive R&D from elastic batch jobs Averaging interactive and batch requirements yields an infrastructure setup that fails both constraints: | Dimension | Interactive R&D | Elastic Burst Training | |---|---|---| | Workload profile | Extended hours at low GPU utilization | Days of idle state followed by high-concurrency spikes | | Key metric | Time to first token or frame | Wall-clock time to complete batch | | Cold-start tolerance | Zero tolerance (requires live shell) | Tolerates 30 to 60 second initialization | | Target platform | Dedicated Amazon EC2 node | Serverless GPU workers (Modal) | The hybrid architecture uses dedicated EC2 instances for interactive experimentation, Modal for parallelized training sweeps, and Amazon S3 as the centralized artifact and dataset store. ```mermaid flowchart LR W[Workload Type] --> Q{Interactive or Batch?} Q -->|Interactive| E[Dedicated EC2 Instance] Q -->|Batch Burst| M[Modal Serverless Workers] E --> S[(Amazon S3)] M --> S ``` The dedicated instance maintains an active development environment. Serverless workers handle parallelized hyperparameter sweeps. A single static node serializes research: testing four rank-32 adapters at different learning rates requires running them back-to-back. Serverless fan-out executes all four runs concurrently. --- ### Step 2: Declare infrastructure environments in code Define the container in Python next to the function it runs, and every training run executes against the same pinned specification: ```python # modal_app.py import modal app = modal.App("aurax-burst") CUDA_TAG = "12.4.1-devel-ubuntu22.04" image = ( modal.Image.from_registry(f"nvidia/cuda:{CUDA_TAG}", add_python="3.11") .apt_install("git", "libgl1", "libglib2.0-0") .pip_install( "torch==2.4.0", "diffusers==0.30.0", "transformers==4.44.0", "accelerate==0.33.0", "safetensors==0.4.4", "bitsandbytes==0.43.3", "peft==0.12.0", "boto3==1.34.162", "pillow==10.4.0", ) .run_commands( "git clone https://github.com/kohya-ss/sd-scripts /opt/sd-scripts", "cd /opt/sd-scripts && git checkout 5b6c2e2 && pip install -r requirements.txt", ) .env({"HF_HOME": "/weights/hf", "PYTHONUNBUFFERED": "1"}) ) weights = modal.Volume.from_name("aurax-weights", create_if_missing=True) outputs = modal.Volume.from_name("aurax-outputs", create_if_missing=True) secrets = [modal.Secret.from_name("aws-s3"), modal.Secret.from_name("hf-token")] ``` Pin the third-party training repo to an exact commit hash. Otherwise an upstream change lands in the middle of a sweep and the run you want to reproduce no longer exists. --- ### Step 3: Decouple model checkpoints from container layers Separating code dependencies from large binary weights prevents unnecessary image rebuilds: | Artifact Type | Update Frequency | Storage Target | |---|---|---| | System binaries, pip packages, scripts | Code updates | Container image layers | | Training datasets, logs, outputs | Per-run schedule | Network volume mount | | Base foundation weights | Infrequent updates | Versioned network volume mount | Baking 20GB+ foundation model weights into container image layers forces multi-gigabyte uploads on minor dependency changes. Storing weights in network volumes enables sub-second container startups with cached filesystem layers. ```python # modal_app.py (continued) @app.function( image=image, volumes={"/weights": weights}, secrets=secrets, timeout=3600, ) def sync_weights(keys: list[str]) -> None: """Synchronizes base checkpoints and adapters from S3 to shared volume.""" import os from pathlib import Path import boto3 s3 = boto3.client("s3") bucket = os.environ["AURAX_WEIGHTS_BUCKET"] for key in keys: dest = Path("/weights") / key if dest.exists(): print(f"Skipping {key} (already present)") continue dest.parent.mkdir(parents=True, exist_ok=True) print(f"Downloading {key} from S3...") s3.download_file(bucket, key, str(dest)) weights.commit() print("Volume synchronized and committed.") ``` --- ### Step 4: Fan out parameter sweeps across independent GPU containers With container environments and weight storage decoupled, training sweeps execute in parallel across independent L40S/A100 instances: ```python # train_burst.py import modal from modal_app import app, image, outputs, secrets, weights @app.function( image=image, gpu="L40S", volumes={"/weights": weights, "/out": outputs}, secrets=secrets, timeout=21600, retries=modal.Retries(max_retries=1, backoff_coefficient=1.0), ) def train_lora(run: dict) -> str: """Executes single adapter training run inside an isolated container.""" import subprocess name = run["name"] cmd = [ "accelerate", "launch", "--num_cpu_threads_per_process", "8", "/opt/sd-scripts/sdxl_train_network.py", "--pretrained_model_name_or_path", "/weights/flux/base.safetensors", "--train_data_dir", f"/weights/datasets/{run['dataset']}", "--output_dir", "/out/lora", "--output_name", name, "--resolution", "1024,1024", "--network_module", "networks.lora", "--network_dim", "32", "--network_alpha", "16", "--learning_rate", str(run.get("lr", 1e-4)), "--lr_scheduler", "cosine", "--train_batch_size", "4", "--gradient_accumulation_steps", "4", "--gradient_checkpointing", "--optimizer_type", "AdamW8bit", "--mixed_precision", "bf16", "--save_every_n_epochs", "1", ] subprocess.run(cmd, check=True) outputs.commit() artifact = f"/out/lora/{name}.safetensors" return artifact @app.local_entrypoint() def main(): runs = [ {"name": "drape_lr1e4", "dataset": "drape_5k", "lr": 1e-4}, {"name": "drape_lr5e5", "dataset": "drape_5k", "lr": 5e-5}, {"name": "occ_lr1e4", "dataset": "occlusion_3k", "lr": 1e-4}, {"name": "occ_lr5e5", "dataset": "occlusion_3k", "lr": 5e-5}, ] for artifact in train_lora.map(runs): print("Completed run artifact:", artifact) ``` `train_lora.map(runs)` allocates four containers simultaneously. Total wall-clock time equals the duration of a single run rather than the sum of four sequential jobs. --- ### Step 5: Amortize model loading overhead in batch inference For batch inference, container cold-start overhead must be amortized across multiple requests: ```python # batch_infer.py import modal from modal_app import app, image, outputs, secrets, weights @app.cls( image=image, gpu="L40S", volumes={"/weights": weights, "/out": outputs}, secrets=secrets, scaledown_window=300, # Retain warm container for 5 minutes max_containers=8, ) class TryOnWorker: @modal.enter() def load(self): """Loads weights once during container initialization.""" import torch from diffusers import FluxFillPipeline self.pipe = FluxFillPipeline.from_pretrained( "/weights/flux/fill", torch_dtype=torch.bfloat16 ).to("cuda") self.pipe.load_lora_weights("/weights/lora/aurax_merged.safetensors") @modal.method() def render(self, job: dict) -> str: out = self.pipe( prompt=job["prompt"], image=job["image"], mask_image=job["mask"], num_inference_steps=30, height=1024, width=1024, ).images[0] path = f"/out/renders/{job['id']}.png" out.save(path) return path ``` `@modal.enter()` loads weights into GPU VRAM once per container lifecycle, so every later `render` call pays only for the forward pass. --- ### Operational considerations and failure modes - **Cold-start latency boundaries:** If individual inference jobs execute in under 2 seconds, a 30-second cold-start penalty dominates total latency unless warm pools (`scaledown_window`) are maintained. - **Ephemeral container debugging:** Terminated serverless containers cannot be inspected via SSH. Structure scripts to log memory statistics, loss values, and error traces to S3 or persistent network volumes on failure. --- ## When this is the wrong choice - **The GPU is busy most of the month.** This split trades a lower average price for cold starts, an image build, and a control plane. A node running at steady utilization has no idle hours to reclaim, and reserved capacity is cheaper per GPU-hour than per-second billing. - **The work is interactive.** Debugging ComfyUI nodes or checking mask dilations on test images means a live shell and a fast edit-run loop. Paying 30 to 60 seconds of container init per attempt makes that loop unusable. That workload belongs on the dedicated instance, which is why the architecture keeps one. - **There is only one run.** Fan-out pays because four adapters finish in the wall-clock time of one. A single training job gets no speedup from `train_lora.map`, and you have added an image definition, a weights sync, and a deploy step to a script that would have run on the box you already had. - **You cannot get diagnostics out of a dead container.** If the failure you are chasing needs a live process to inspect, a terminated serverless container gives you nothing back. Reproduce it on the EC2 node first, then move the sweep. --- Source: https://himanshuat.com/blogs/serverless-gpus-for-bursty-training ======================================================================== --- title: "StyleX Architecture: Compile-Time CSS Extraction, Deterministic Resolution, and Atomic Bundling" description: "An architectural breakdown of StyleX, examining how compile-time Babel AST transforms eliminate CSS-in-JS runtime overhead, guarantee deterministic class merges, and generate atomic stylesheets." date: "December 16, 2023" url: "https://himanshuat.com/blogs/stylex-new-frontend-styling-framework" --- # StyleX Architecture: Compile-Time CSS Extraction, Deterministic Resolution, and Atomic Bundling Frontend styling solutions have historically traded runtime performance for developer ergonomics: - **Runtime CSS-in-JS (Emotion, Styled-Components):** Offers scoped styles, dynamic props, and colocation, but incurs runtime JavaScript evaluation, repeated string hashing, dynamic `