554 million ticks and the wrong reason mmap loses

I had a 22 GB tick CSV containing 554 million rows and two architectural strategies to scan it.

Standard database systems literature (such as Crotty et al., CIDR 2022) advises against mmap for large database scans, favoring buffered read().

I profiled sequential scans across a range of dataset sizes and found a clear crossover. mmap beat read() by 1.2x to 1.3x while the data fit comfortably in memory, then fell to 0.78x once the dataset exceeded physical RAM.

Memory-mapped file I/O performance is not unconditional. Its efficiency relative to buffered streaming is governed directly by working-set size relative to available physical memory.

The common explanation cites Apple Silicon's 16 KiB base page size against x86's 4 KiB. The fault counters say page size is not the governing mechanism.

---

Chapter 0: Comparing buffered reads and demand-paged memory maps

Buffered read(): Copies bytes from the operating system page cache into user-space application buffers. The application reuses a fixed memory buffer, and the kernel pre-fetches contiguous pages sequentially via readahead heuristics.

Memory mapping (mmap): Maps file descriptors directly into the virtual address space, letting the Memory Management Unit (MMU) fault pages in on demand. It eliminates user-space buffer copies, but translates sequential I/O into individual hardware page faults.

Skipping the copy helps while the pages are resident. Once the kernel has to go to storage for them, every fault is a blocking trap, and a sequential scan produces a lot of them.

---

Step 1: Isolate parser throughput from I/O overhead

An I/O benchmark that uses a slow CSV parser ends up profiling string conversions rather than kernel storage paths.

To isolate I/O behavior, I benchmarked against a zero-allocation, SIMD-aligned byte scanner capable of sustaining 11 GB/s in memory, well above NVMe line rates.

Each test arm executes inside a fresh child subprocess so getrusage captures clean peak Resident Set Size (RSS) and major/minor page fault counters.

---

Step 2: Characterize the working-set crossover threshold

Testing across varying file sizes identifies the precise memory boundary:

The crossover boundary occurs between one-third and two-thirds of physical RAM, well before physical memory is 100% exhausted.

---

Step 3: Quantify major vs. minor page faults

Hardware counters explain why mmap degrades on out-of-core scans:

• Buffered read(): Incurs near zero major disk page faults across all dataset sizes. The kernel readahead engine continuously fills buffers ahead of application consumption.
• mmap: Incurs 1.34 million major page faults on the 22 GB file, triggering discrete kernel traps for individual 16 KiB pages.

The bottleneck in out-of-core memory-mapped scans is the accumulation of blocking page-fault interrupts rather than user-space copy latency.

---

Step 4: Compare cross-OS kernel fault-around policies

If page size were the governing factor, Linux systems with 4 KiB base pages should incur four times more page faults than macOS with 16 KiB pages.

On identical datasets, Linux incurs roughly four times fewer faults than macOS.

Two Linux kernel mechanisms explain this inversion:

1. Kernel fault-around: When handling a page fault on mapped files, the Linux kernel speculatively populates up to 64 KiB of contiguous page tables around the fault address in a single trap.
2. Transparent Huge Pages (THP): Folds contiguous anonymous pages into 2 MiB blocks.

The governing performance variable is the kernel's fault-around allocation policy, not the CPU architecture's base hardware page size.

---

Step 5: Evaluate madvise prefetch hints

On macOS, the two madvise flags behave differently from what their names imply:

• MADV_SEQUENTIAL: Improves total scan time by approximately 15%, but leaves total page fault counts unchanged. It optimizes page replacement priority without altering single-page fault granularity.
• MADV_WILLNEED: Triggers aggressive page table population that evicts active memory, doubling major page faults and yielding the slowest throughput in the benchmark suite.

---

When this is the wrong choice

• The scan will not fit in memory. Above roughly two thirds of physical RAM, mmap ran at 0.78x of buffered read() and took 1.34 million major faults on the 22 GB file. There is no madvise flag that recovers this; MADV_WILLNEED made it worse. Stream it.
• You are on Linux. Every crossover number here came off Apple Silicon. Linux fault-around populates up to 64 KiB of page tables per trap and takes about four times fewer faults on the same data, so the point where mmap stops winning sits somewhere else. Measure it on your own kernel before you inherit my threshold.
• Your parser is the bottleneck. I ran this against a zero-allocation SIMD scanner sustaining 11 GB/s precisely so the parser would not hide the kernel behavior. If your pipeline spends its time in serde and string conversion, the I/O path is not what is costing you, and swapping it is a rewrite for a difference you will not be able to see.

---

Engineering takeaways

1. In-memory scans benefit from mmap: For datasets that fit within 50% of available system memory, memory mapping provides a 1.2x to 1.3x speedup by eliminating buffer copying.
2. Out-of-core scans require buffered streaming: For datasets exceeding physical memory, buffered read() avoids millions of major fault traps, maintaining consistent NVMe streaming bandwidth.
3. Profile hardware counters directly: Monitor ru_majflt and ru_minflt via getrusage to isolate page fault overhead from raw parser computational cost.