The backtest is the experiment, and most of them aren't controlled
Self-taught quant research mostly fails on bias-laden backtests rather than on the math. So I front-loaded the pitfall literature before running anything, and it changed what I built first — infrastructure, not strategies.
The failure mode I was warned about isn't running out of math. It's producing a beautiful equity curve that means nothing, believing it, and finding out eighteen months later.
Most self-taught quant research fails on bias-laden backtests and untargeted reading. Not on stochastic calculus.
A backtest is an experiment. Almost nobody treats it like one, and an uncontrolled experiment returns a number whether or not there's an effect.
So I inverted the usual order. Before the first backtest, I read the literature on how backtests lie. Before the first strategy, I built the machinery that makes a result reproducible.
Chapter 0 — What a backtest actually claims
A backtest claims: had I run this rule over this period, I would have earned this.
That claim is only as good as the counterfactual. Every bias is a way the counterfactual is false while the arithmetic is correct.
The named ones I had to internalise before I could trust anything:
- Lookahead. The rule used information that didn't exist at decision time.
- Survivorship. The universe silently excludes what died.
- Selection. You chose the period, the universe, or the parameters after seeing results.
- P-hacking. You ran many specifications and reported the one that worked.
- Regime change. The relationship existed and then stopped.
None of these produce an error. They all produce a number, which is what makes them dangerous.
Step 1 — Read the pitfall canon before the textbooks
The reading order matters more than the reading list. Pitfall literature first, because it changes how you read everything after it.
The four I front-loaded:
- Bailey & López de Prado (2014), The Deflated Sharpe Ratio. How to discount a Sharpe ratio for the number of trials that produced it.
- Bailey, Borwein, López de Prado & Zhu, Pseudo-Mathematics and Financial Charlatanism. The case that backtest overfitting is the field's default state rather than its edge case.
- Harvey, Liu & Zhu (2016), …and the Cross-Section of Expected Returns. Multiple testing applied to the factor zoo — what happens to published factors when you correct for how many were tried.
- Aronson, Evidence-Based Technical Analysis. Bias enumeration done properly.
The shared argument across all four is one idea: a result's credibility depends on how many results you didn't report. A Sharpe ratio without a trial count is an unlabelled measurement.
Reading these after building a strategy would have been useless. I'd have had a result to defend.
The tell: if you can't say how many specifications you tried before the one you're showing, you can't interpret its Sharpe.
Step 2 — Build reproducibility before you build signal
This is the part I'd have skipped and shouldn't have. It reads like infrastructure procrastination. It's actually the control group.
What has to exist before the first real backtest:
- Experiment tracking — mlflow or wandb, logging every backtest with its config. Not the good ones. Every one. This is what makes the trial count in Step 1 a number instead of a guess.
- Data versioning — dvc, or hashed parquet snapshots. A result you can't re-run against the same bytes isn't a result.
- Deterministic notebooks — seeded RNG, library versions pinned in a lockfile.
- Walk-forward and purged k-fold utilities — mlfinlab implements López de Prado's versions. Standard k-fold leaks across time-series folds; purging is what removes the leak.
The ordering argument is the same one from Step 1, turned into engineering. If tracking is opt-in, you'll log the runs you're proud of, and your trial count will be wrong in the direction that flatters you.
The tell: if your experiment tracker has fewer runs in it than you've actually run, its purpose has already been defeated.
Step 3 — Replicate before you originate
The learning loop isn't reading papers. It's re-implementing them.
The three-pass protocol I use, and the honest cost of each:
| pass | time | what you get | |---|---|---| | 1 | ~5 min | title, abstract, intro, headers, conclusion. Decide if pass 2 is worth it. | | 2 | ~1 hr | figures, tables, methodology. Skip proofs. Note data, universe, sample period, claimed result. | | 3 | ~3 hr | re-derive the math, re-implement the core experiment, sanity-check the result. |
The target that matters: at least five pass-3 papers a year. Pass-1 everything else.
Five is a small number and it still sounds ambitious once you've done one. That gap — between how easy a paper is to read and how hard it is to reproduce — is most of what pass 3 teaches.
The foundational set to replicate against is well-defined: Markowitz (1952), Sharpe (1964), Black & Scholes (1973), Fama & French (1993) and (2015), Carhart (1997), Jegadeesh & Titman (1993), Asness/Moskowitz/Pedersen, Hou/Xue/Zhang, and Gu, Kelly & Xiu (2020).
The tell: if your reproduction matches the paper on the first try, check your data for lookahead before you celebrate.
Step 4 — Pick one niche and let it constrain the reading
Untargeted reading is the second named failure mode, and it's the one that feels productive while it's happening.
The fix is structural: pick a narrow niche first, and let it decide what's worth a pass 3. Without an anchor, every interesting paper is equally interesting, which means the queue grows faster than it drains.
The infrastructure that supports this is unglamorous — an SSRN account following FEN and CMBO feeds, the arXiv q-fin daily for q-fin.ST, q-fin.PR and q-fin.CP, a Zotero queue, and one paper a week with notes.
The tell: if your "papers to read" queue is growing monotonically, you don't have a niche yet.
What breaks it
Tracking that's easy to bypass. The moment logging a run takes an extra step, exploratory runs stop being logged, and exploratory runs are exactly the ones the deflated Sharpe correction needs to count.
Purging that isn't applied consistently. Purged cross-validation on the final model and plain k-fold during exploration means the leak happened during the part where you made the decisions.
Replication that stops at "the number matches." A reproduction can agree with the paper and still be wrong in the same way the paper was. Matching a published result is evidence about your implementation, not about the effect.
Reading the canon and then not applying it. Knowing about selection bias does not remove selection bias. The corrections have to be mechanical — in the tracker, in the CV splitter — or they don't happen under deadline.
When this much rigour is the wrong choice
- You're learning the mechanics of an instrument. Pricing a vanilla option by hand to understand it doesn't need experiment tracking. The machinery is for claims about edge, not for exercises.
- The result will never inform a decision. A study run purely to understand a paper's method can stop at pass 3. Reproducibility infrastructure earns its cost when someone might act on the output.
- You haven't picked a niche. Building walk-forward tooling before you know what you're testing is the same procrastination as untargeted reading, wearing engineering clothes.
- The effect size is enormous and mechanical. Some relationships are accounting identities rather than discoveries. Multiple-testing corrections aren't the binding constraint there.
The rigour scales with how much the claim would cost you if it were false. For most of what I'll produce in the first year, that cost is my own time — which is exactly the budget the failure mode consumes.
The shift
The instinct is to learn the math, then build a strategy, then check it. That order puts verification last, which is where it always gets cut.
Front-loading the pitfall canon inverted it. Now the reading list is chosen by a niche, the trial count is a logged number rather than a memory, and the cross-validation splitter refuses to leak whether or not I remember to ask it to.
- Read how backtests lie before you run one. Afterwards, you have a result to defend.
- Make the corrections mechanical. Knowing about p-hacking doesn't prevent p-hacking; a tracker that logs every run does.
- Count reproductions, not papers. Five pass-3s a year beats a hundred abstracts.
A Sharpe ratio is a measurement. The number of times you measured is part of the measurement, and it's the part that never makes it into the chart.