Systems·ongoing·open
It's the Fault Policy, Not the Page Size: Why mmap Loses to read() for Out-of-Core Scans on Apple Silicon
A fault-attributed re-measurement of the CIDR'22 result that mmap underperforms read() for large scans, on 16 KiB-page Apple Silicon, which is absent from that literature. Uses a real 22 GB, 554-million-row financial tick CSV with the parser held constant so the experiment measures the I/O strategy rather than parser cleverness.
abstract
- Memory-mapping a file and letting the kernel handle paging is a common way to scan large datasets, but Crotty et al. (CIDR'22) argued mmap is a poor choice for database scans. That argument was made at the level of mechanism rather than measurement, predates several years of Linux virtual-memory evolution, and was conducted on x86 with 4 KiB pages. Apple Silicon uses 16 KiB base pages and a different memory-management stack, and is essentially unstudied. I re-measure the canonical result on Apple Silicon using a fixed-schema workload: parsing a 22 GB, 554-million-row financial tick CSV.
method
- The parser is a hand-rolled, validating, zero-copy byte scanner held constant across every I/O arm, and sustains around 11 GB/s when isolated from I/O, comfortably faster than the disk. Whatever separates the strategies is therefore I/O rather than compute. Arms cover buffered read(), mmap, mmap with MADV_SEQUENTIAL, mmap with MADV_WILLNEED, and a parallel mmap stream, plus a csv+serde baseline. Each arm runs in a fresh subprocess for clean getrusage peak RSS and minor/major fault counters, with warmup discarded and a working-set sweep via newline-aligned byte limits.
results
- Sweeping the working-set size produces a clean crossover: mmap is 1.2 to 1.3 times faster than buffered read() while the working set fits in RAM, and 0.78 times slower once it exceeds it, with the crossover falling at roughly one-third to two-thirds of physical memory. The slowdown is attributable per fault. Buffered read() takes essentially zero disk faults at any size, whereas mmap's disk faults climb to 1.34 million, one 16 KiB page at a time. The surprise is cross-OS: 4 KiB Linux takes roughly 4 times fewer minor faults than 16 KiB macOS for the same scan, because Linux populates 64 KiB per fault via fault-around and folds anonymous memory into huge pages. x86-64 and AArch64 Linux agree, so the effect is the operating system's fault-population policy, not the page size and not the ISA. madvise cannot recover the gap on macOS: MADV_SEQUENTIAL gives about 15% but leaves the fault count unchanged, and MADV_WILLNEED doubles major faults and is the slowest arm.
status
- In progress. The measurements above are complete on one Apple Silicon machine with one NVMe device and one dataset schema. The cross-OS comparison runs under virtualized and emulated Linux, so I take only fault counts from it, which are architectural, and never timing. Bare-metal Linux confirmation and a second storage device are the outstanding experiments before this is submittable. Targeting arXiv (cs.OS, cs.PF, cs.DB), then a workshop such as DaMoN or HotStorage.
How it works
Why the two I/O strategies diverge once the working set exceeds RAM.