554 million ticks and the wrong reason mmap loses
Everyone repeats that mmap is bad for large scans. On Apple Silicon that turned out to be true, false, and true again depending on working-set size — and the mechanism everybody names for it is not the mechanism doing the work.
I had a 22 GB tick CSV, 554 million rows, and two ways to read it.
The received wisdom said use read(). Crotty et al. argued at CIDR'22 that mmap is a poor choice for database scans, and the paper has been repeated often enough that it functions as a rule.
Then I measured it, and mmap won by 1.2 to 1.3x. Then I made the file bigger and mmap lost by a factor of 0.78.
The rule isn't wrong. It's unconditional, and the thing it's conditional on is working-set size relative to RAM.
The other half of the received wisdom is the explanation: Apple Silicon uses 16 KiB base pages, so of course the fault behaviour differs from 4 KiB x86. That part turns out to be the wrong mechanism entirely.
Chapter 0 — What the two strategies actually are
Buffered read() copies bytes from the page cache into a buffer you own. You reuse one small buffer forever. The kernel reads ahead in large chunks because it knows you're streaming.
mmap maps the file into your address space and lets the MMU fault pages in on demand. No copy, no buffer management. You touch a byte, the hardware traps, the kernel populates the page.
The pitch for mmap is that it deletes the copy. The case against it is that it converts a sequential I/O problem into a page-fault problem, and page faults are a bad way to express "read the next 64 KiB."
Both claims are true. Which one dominates is an empirical question, and it's the one the literature settled at the level of mechanism rather than measurement.
Step 1 — Hold the parser constant, or measure nothing
The trap in any I/O benchmark is that you end up measuring your parser.
The parser here is a hand-rolled, validating, zero-copy byte scanner, identical across every arm. Isolated from I/O it sustains around 11 GB/s — comfortably faster than the disk.
That number is the whole license for the rest of the experiment. If the parser is never the bottleneck, whatever separates the arms is I/O.
The arms:
| arm | what it tests |
|---|---|
| buffered read() | the baseline the literature recommends |
| mmap | demand paging, no advice |
| mmap + MADV_SEQUENTIAL | can advice close the gap? |
| mmap + MADV_WILLNEED | can eager population close the gap? |
| parallel mmap stream | does concurrency change the shape? |
| csv + serde | a stand-in for what most people actually write |
Each arm runs in a fresh subprocess, so getrusage peak RSS and the minor/major fault counters are clean rather than accumulated. Warmup runs are discarded. The working-set sweep uses newline-aligned byte limits so every arm parses the same rows.
The tell: if your fast arm and your slow arm don't share a parser, you have a parser benchmark with an I/O label on it.
Step 2 — Sweep the working set until the answer changes sign
A single file size gives you a single verdict, which is how a conditional result gets published as an unconditional one.
Sweeping produces a clean crossover:
| working set vs RAM | mmap relative to read() |
|---|---|
| fits | 1.2–1.3x faster |
| exceeds | 0.78x — slower |
The crossover falls at roughly one-third to two-thirds of physical memory. Not at 100%. The strategies diverge well before you exhaust RAM.
A benchmark that reports one number for "mmap vs read" has picked a side of a crossover and called it a finding.
The tell: if you never varied the input size, you don't know which side of the crossover your result is on.
Step 3 — Attribute the slowdown per fault, not per second
Wall-clock tells you mmap lost. It doesn't tell you why, and "mmap is slow for scans" is a restatement, not a cause.
The fault counters do tell you:
- Buffered
read()takes essentially zero disk faults at any size. The kernel reads ahead; nothing traps. - mmap's disk faults climb to 1.34 million, one 16 KiB page at a time.
That's the mechanism. Not copy avoidance, not TLB pressure, not cache behaviour. The mapped arm is paying a trap per page and the buffered arm is paying none.
Why the two strategies diverge once the working set exceeds RAM: the buffered arm never reaches the decision node at all.
The tell: a performance claim you can't attribute to a counter is a story. Find the counter that moves.
Step 4 — Check whether your mechanism survives crossing the OS
This is the step that changed the paper.
The obvious explanation for the fault count is page size: macOS uses 16 KiB pages, Linux uses 4 KiB, so Linux should fault more, not less. Four times more, if page size is what's driving it.
It faults roughly four times less.
Two things on the Linux side account for it. Fault-around populates 64 KiB per fault rather than one page. And anonymous memory gets folded into huge pages.
So the same scan, with pages one quarter the size, takes a quarter of the faults — because the OS decided to populate more per trap.
The control that closes it: x86-64 and AArch64 Linux agree with each other. Same fault-population behaviour across two instruction sets.
The effect is the operating system's fault-population policy. Not the page size, and not the ISA.
That inverts the intuition the field has been carrying. The page-size story predicts the opposite sign of the effect that actually shows up.
The tell: if your explanation predicts a direction, go find a platform where the direction should flip. If it doesn't flip, the explanation is wrong.
Step 5 — Try to rescue the losing arm before you write it off
madvise is the obvious rescue. If the problem is fault frequency, tell the kernel what's coming.
It doesn't work on macOS:
MADV_SEQUENTIALbuys about 15% — and leaves the fault count unchanged. It's making each fault cheaper, not making fewer of them. The mechanism I identified in Step 3 is untouched.MADV_WILLNEEDdoubles major faults and is the slowest arm in the entire experiment. The eager-population hint makes it actively worse.
The 15% is the more interesting of the two, because it's the number that would have let me claim advice "helps" if I'd only looked at wall-clock. The fault counter is what says it's not addressing the cause.
The tell: when a fix improves time but not the counter you blamed, you fixed something else.
What breaks it
Timing under virtualization is not timing. The cross-OS comparison runs under virtualized and emulated Linux. So I take only fault counts from it, which are architectural, and never timing. This constrains the claim: I can say Linux's fault policy populates more per trap, and I cannot say Linux is faster.
One machine, one disk, one schema. All measurements are on a single Apple Silicon machine with one NVMe device and one dataset schema. The crossover ratio is a property of that configuration until a second storage device says otherwise.
The parser being fast is load-bearing. At 11 GB/s the parser is far from the bottleneck. On a slower parser — a csv + serde pipeline, which is what most people actually deploy — the I/O difference compresses, because the bottleneck moves. The result is about I/O strategy for a workload where I/O is the constraint.
Bare-metal Linux confirmation and a second storage device are the outstanding experiments before this is submittable. Targeting arXiv (cs.OS, cs.PF, cs.DB), then a workshop such as DaMoN or HotStorage.
When mmap is the wrong choice
- Your working set exceeds RAM. This is the case the literature was right about. Above the crossover you pay 1.34 million traps for a copy you weren't spending much on.
- You're streaming once and discarding. Demand paging's advantage is reuse. A single sequential pass over data you'll never touch again is exactly the shape buffered reads with readahead were built for.
- Your parser is the bottleneck. If you're at
csv+serdespeeds rather than 11 GB/s, the I/O strategy is not what's costing you, and swapping it is effort spent on the wrong layer. - You need predictable latency. Faults are a distribution, not a constant.
read()taking essentially zero disk faults at any size is a stability property, not just a speed one.
Conversely: below the crossover, on a re-scanned working set, with a parser fast enough to notice, mmap's 1.2–1.3x is real and worth taking.
The shift
I went in expecting to confirm a known result and got a conditional one, with the wrong mechanism attached to it.
The measured claim is narrower and more useful than the rule it replaces. mmap doesn't lose to read(). mmap loses above a working-set threshold that lands between one-third and two-thirds of RAM, and the thing that sets that threshold is how many bytes your OS populates per fault.
- Sweep the input size. A single-size benchmark picks a side of a crossover and calls it a law.
- Attribute to a counter. Wall-clock says who won. Fault counts say why, and "why" is what transfers to your workload.
- Test the mechanism where it should flip. Page size predicted Linux would fault more. It faulted four times less, and that's how the real cause surfaced.
The received wisdom had the right verdict for the wrong reason, which meant it was also right in cases where it shouldn't have been and wrong in cases nobody checked.