---
title: "554 million ticks and the wrong reason mmap loses"
description: "Industry lore states that memory mapping is inefficient for large sequential database scans. On Apple Silicon, empirical profiling shows mmap is faster, slower, and faster again based on working-set ratios relative to physical RAM, governed by kernel fault-population policies rather than base page size."
date: "August 12, 2026"
url: "https://himanshuat.com/blogs/the-fault-policy-not-the-page-size"
---
# 554 million ticks and the wrong reason mmap loses
I had a 22 GB tick CSV containing 554 million rows and two architectural strategies to scan it.
Standard database systems literature (such as Crotty et al., CIDR 2022) advises against `mmap` for large database scans, favoring buffered `read()`.
I profiled sequential scans across a range of dataset sizes and found a clear crossover. `mmap` beat `read()` by 1.2x to 1.3x while the data fit comfortably in memory, then fell to 0.78x once the dataset exceeded physical RAM.
> Memory-mapped file I/O performance is not unconditional. Its efficiency relative to buffered streaming is governed directly by working-set size relative to available physical memory.
The common explanation cites Apple Silicon's 16 KiB base page size against x86's 4 KiB. The fault counters say page size is not the governing mechanism.
---
### Chapter 0: Comparing buffered reads and demand-paged memory maps
**Buffered `read()`:** Copies bytes from the operating system page cache into user-space application buffers. The application reuses a fixed memory buffer, and the kernel pre-fetches contiguous pages sequentially via readahead heuristics.
**Memory mapping (`mmap`):** Maps file descriptors directly into the virtual address space, letting the Memory Management Unit (MMU) fault pages in on demand. It eliminates user-space buffer copies, but translates sequential I/O into individual hardware page faults.
Skipping the copy helps while the pages are resident. Once the kernel has to go to storage for them, every fault is a blocking trap, and a sequential scan produces a lot of them.
---
### Step 1: Isolate parser throughput from I/O overhead
An I/O benchmark that uses a slow CSV parser ends up profiling string conversions rather than kernel storage paths.
To isolate I/O behavior, I benchmarked against a zero-allocation, SIMD-aligned byte scanner capable of sustaining **11 GB/s** in memory, well above NVMe line rates.
| Experimental Arm | Target Evaluation Mechanism |
|---|---|
| Buffered `read()` | Baseline sequential streaming with kernel readahead |
| `mmap` (default) | Demand paging without explicit kernel advice |
| `mmap` + `MADV_SEQUENTIAL` | Kernel sequential pre-fetch advice |
| `mmap` + `MADV_WILLNEED` | Eager asynchronous page fault pre-population |
| Parallel `mmap` chunking | Multi-threaded partitioned range scanning |
| Baseline `csv` + `serde` | Standard desktop application parser reference |
Each test arm executes inside a **fresh child subprocess** so `getrusage` captures clean peak Resident Set Size (RSS) and major/minor page fault counters.
---
### Step 2: Characterize the working-set crossover threshold
Testing across varying file sizes identifies the precise memory boundary:
| Working-Set Size Relative to RAM | `mmap` Throughput Relative to Buffered `read()` |
|---|---|
| Fits within available RAM (< 35% RAM) | **1.2x to 1.3x faster** (copy elimination dominates) |
| Exceeds physical RAM (> 65% RAM) | **0.78x (slower)** (major page faults dominate) |
The crossover boundary occurs between **one-third and two-thirds of physical RAM**, well before physical memory is 100% exhausted.
```mermaid
flowchart TD
F["22 GB Tick CSV (554M Rows)"] --> R["Buffered read()"]
F --> M["mmap Direct Mapping"]
R --> RB["Small Reused User Buffer;
Kernel Prefetches Large Blocks"]
RB --> RF["~0 Major Faults Across All Working-Set Sizes"]
M --> MF["Page Fault Handlers Triggered per 16 KiB Page"]
MF --> WS{"Working Set vs Physical RAM"}
WS -->|"Working Set < RAM"| MIN["Minor Page Faults (Soft Traps);
1.2x to 1.3x Faster Than read()"]
WS -->|"Working Set > RAM"| MAJ["Major Page Faults (1.34M Disk Traps);
0.78x Slower Than read()"]
RF --> OUT["read() Wins Out-of-Core"]
MAJ --> OUT
MIN --> OUT2["mmap Wins In-Memory"]
```
---
### Step 3: Quantify major vs. minor page faults
Hardware counters explain why `mmap` degrades on out-of-core scans:
- **Buffered `read()`:** Incurs **near zero major disk page faults** across all dataset sizes. The kernel readahead engine continuously fills buffers ahead of application consumption.
- **`mmap`:** Incurs **1.34 million major page faults** on the 22 GB file, triggering discrete kernel traps for individual 16 KiB pages.
The bottleneck in out-of-core memory-mapped scans is the accumulation of blocking page-fault interrupts rather than user-space copy latency.
---
### Step 4: Compare cross-OS kernel fault-around policies
If page size were the governing factor, Linux systems with 4 KiB base pages should incur four times *more* page faults than macOS with 16 KiB pages.
On identical datasets, Linux incurs roughly **four times fewer faults** than macOS.
Two Linux kernel mechanisms explain this inversion:
1. **Kernel fault-around:** When handling a page fault on mapped files, the Linux kernel speculatively populates up to **64 KiB of contiguous page tables** around the fault address in a single trap.
2. **Transparent Huge Pages (THP):** Folds contiguous anonymous pages into 2 MiB blocks.
The governing performance variable is the kernel's **fault-around allocation policy**, not the CPU architecture's base hardware page size.
---
### Step 5: Evaluate `madvise` prefetch hints
On macOS, the two `madvise` flags behave differently from what their names imply:
- **`MADV_SEQUENTIAL`:** Improves total scan time by approximately 15%, but leaves total page fault counts unchanged. It optimizes page replacement priority without altering single-page fault granularity.
- **`MADV_WILLNEED`:** Triggers aggressive page table population that evicts active memory, doubling major page faults and yielding the slowest throughput in the benchmark suite.
---
## When this is the wrong choice
- **The scan will not fit in memory.** Above roughly two thirds of physical RAM, `mmap` ran at 0.78x of buffered `read()` and took 1.34 million major faults on the 22 GB file. There is no `madvise` flag that recovers this; `MADV_WILLNEED` made it worse. Stream it.
- **You are on Linux.** Every crossover number here came off Apple Silicon. Linux fault-around populates up to 64 KiB of page tables per trap and takes about four times fewer faults on the same data, so the point where `mmap` stops winning sits somewhere else. Measure it on your own kernel before you inherit my threshold.
- **Your parser is the bottleneck.** I ran this against a zero-allocation SIMD scanner sustaining 11 GB/s precisely so the parser would not hide the kernel behavior. If your pipeline spends its time in `serde` and string conversion, the I/O path is not what is costing you, and swapping it is a rewrite for a difference you will not be able to see.
---
### Engineering takeaways
1. **In-memory scans benefit from `mmap`:** For datasets that fit within 50% of available system memory, memory mapping provides a 1.2x to 1.3x speedup by eliminating buffer copying.
2. **Out-of-core scans require buffered streaming:** For datasets exceeding physical memory, buffered `read()` avoids millions of major fault traps, maintaining consistent NVMe streaming bandwidth.
3. **Profile hardware counters directly:** Monitor `ru_majflt` and `ru_minflt` via `getrusage` to isolate page fault overhead from raw parser computational cost.
---
Source: https://himanshuat.com/blogs/the-fault-policy-not-the-page-size