Where Linux charges I/O: cache reads, mapped writes and cancelled writeback

The same payload appears in different counters

Each scenario used a fresh 4 MiB file. The values below are per-process counter increases, in MiB, not rates. All 5 repetitions produced the same displayed values. Raw snapshots retain byte precision and syscall counts.

Observed on Linux 6.8.0-146-generic, x86_64; each row names the process whose counters were read. The last two rows belong to a truncator, not its writer.

The cached reader returned and checked every byte. Its rchar increased by 4,194,304, while read_bytes stayed unchanged. It made 64 successful 64 KiB reads plus one EOF read, explaining the recorded 65 read syscalls. Preparation had written and synchronized the file without evicting its cached contents.

The mapping case changed every byte of a shared mapping and verified the resulting file contents. It recorded 4 MiB in write_bytes, but zero in both wchar and syscw. Counting write-family calls alone would miss this modification path. See the separate mapping, visibility and synchronization experiments for the persistence boundary.

Why writes are charged before writeback finishes

Linux distinguishes bytes returned through read/write-family interfaces from storage-related accounting. Kernel documentation places write_bytes accounting at page-dirtying time: the process that changes cached data can be identified then, even if a different execution context writes it back later. A larger counter therefore does not certify that those bytes have completed physical I/O. [1, section 3.3]

This design rationale is easy to miss in current monitoring summaries. The 2007 paper Where is your application stuck? discusses Andrew Morton's per-task I/O accounting as related work. Section 3.5 explains the attribution problem with delayed writes and illustrates why syscall counters miss memory-mapped modifications. Its original demonstration used a different, much larger mapping. Our small experiment checks the mechanism on a later kernel; it is not a numerical replication of that example. [3, pp. 75-77]

The process cancelling a write need not have written anything

In the dirty-file case, worker A wrote 4 MiB and stopped. Worker B then opened that same file and shortened it to zero. A remained alive and stopped throughout B's operation, so the observation does not rely on a process disappearing or its counters being reset.

A retained its 4 MiB charge. B gained 4 MiB in cancelled_write_bytes and zero in write_bytes. Subtracting B's two increases yields -4 MiB. Nothing flowed backwards through a disk: the subtraction combines a write charge with cancellation of somebody else's dirty data.

The control changed one step: A completed fsync() before B truncated. B's cancellation increase was then zero in every repetition. That comparison supports the dirty-writeback interpretation, while neither counter sequence measures physical media traffic.

The kernel documentation explicitly allows this cross-process attribution. The manual describes cancelled_write_bytes as writeback avoided for data already charged, and warns that a process's derived net can be negative. Background writeback can reduce or eliminate the cancellation seen in a rerun; zero cancellation is a valid result. [1] [2]

A parent's counter can rise without a new write

We extended the experiment to follow one child through four states: stopped before writing, stopped after writing and synchronizing, exited but not reaped, and reaped by its parent. The parent performed no file writes. Its /proc/PID/io gained the child's contribution only at the last step.

Changes in wchar, in MiB. Parent columns use the stopped baseline; the own-task column measures the reaping step. Each payload was tested 5 times on Linux 6.8.0-146-generic; every repetition gave the displayed values. The zero-payload child is a no-write control.

The distinction is between observing termination and consuming its status. The parent first used waitid(..., WEXITED | WNOWAIT), which left the exited child waitable; the observer confirmed its zombie state. A subsequent waitpid() reaped it. Waiting for an earlier stopped state had not transferred its counters either. [4]

The upstream Linux v6.8 implementation explains the boundary: wait_task_zombie() takes the WNOWAIT path before adding the child's accounting to the parent's shared signal structure. The ordinary reap path adds both the child's task counters and its accumulated group counters. Separately, do_io_accounting() includes the shared totals for the process view, but uses the individual task counters for the thread view. These are version-pinned source observations, not a claim that every vendor kernel has identical code. [5] [6]

We also sampled /proc/PID/task/PID/io for the single-threaded parent. Its own wchar and write_bytes stayed unchanged at reaping. The process view gained 1 or 4 MiB in both counters, matching the child; the no-write control gained neither. Thus the increase describes inherited accounting, not a second write or a new burst of disk traffic.

Monitoring consequence: retaining a child's completed total and adding it to a parent total that now includes that child double-counts the contribution. For the 4 MiB trial, that mistaken sum is 8 MiB. A parent's rate can also jump when it reaps old work, rather than when the child performed the writes. Track process identity and lifecycle, and choose whether a series represents an individual task or an inclusive process total. Merely switching to the leader's task path would omit other threads in a multithreaded process; sampling tasks alone can also miss short-lived work.

The child-inheritance harness is standalone; its separate JSON result preserves all 15 trials, seven counters, raw snapshots and source hash. Run python3 observe_children.py --repetitions 5 --mib 0 1 4 --output children.json. Signals and anonymous shared memory coordinate the workers, so control messages do not inflate their file-I/O counters.

The observer reads the child at its last stop before _exit(); no payload work remains after that stop. Reading the zombie's I/O file failed with permission denied on this host, so the protocol checks zombie state without requiring that read. The inherited totals matched the pre-exit snapshot for every recorded counter. This tests one parent and one child with default SIGCHLD handling, not orphan adoption, subreapers, concurrent threads or a whole process-tree collector. The original four-scenario data remain unchanged.

Build the metric around the question

For application demand: distinguish returned read/write bytes from memory-mapped access. Our reader and mapping rows show why neither rchar nor wchar alone describes all file activity.

For process attribution: retain write and cancellation counters separately. A derived signed difference may be useful if named accurately, but treating it as an always-increasing counter, interpreting a decrease as a reset, or silently clamping it to zero changes its meaning. This is a monitoring implication of the measured negative difference, not a claim about a particular exporter.

For device work: measure the relevant device layer and align the observation interval. The process table does not establish request completion, latency or controller-internal writes. In particular, write_bytes / wchar is not a NAND write-amplification estimate: the mapped-write row has a zero denominator despite modifying a file. Compare write amplification and the existing IOPS and transfer-size accounting explanation.

For process trees: the manual says this interface includes waited-for children. Summing historical child counters with an ancestor after reaping can count the same contribution twice. The child-reaping experiment above measures that transfer separately from the original cache and truncation scenarios. The interface also has permission checks and documented non-atomic-counter caveats on some systems. [2]

Reproduce the four scenarios

The standalone Python harness needs Linux, Python 3 and GNU stat. It creates a private temporary directory, checks returned bytes and contents, and removes its files. It changes no mount settings, cache policy or system tunables, and never opens an existing data file or block device.

python3 observe.py --repetitions 5 --mib 4 --output my-results.json

A separate parent reads /proc/PID/io while each worker is stopped with SIGSTOP, both before and after the action. The workers do not read their own counters, and no control-pipe payload contaminates their byte totals. A timeout bounds the run. Use --temp-dir to choose an existing disposable test area; the output path must not already exist.

The measured JSON records every before/after snapshot, source SHA-256, timestamp and environment. The published run used Python 3.12.3; stat -f returned ext2/ext3, a magic-number label that does not distinguish ext-family versions. Repetitions check consistency on this host, not a population failure rate.

Cache residency and writeback scheduling remain uncontrolled. The harness therefore validates operations and arithmetic without requiring every host to reproduce the exact cancellation or cache-hit values. It does not test NFS, SMB, direct I/O, cgroup accounting, firmware counters, power loss or performance.

Corpus: process I/O accounting and page cache.

Primary sources

1. Linux kernel documentation: /proc/PID/io accounting, section 3.3
2. Linux man-pages 6.19: proc_pid_io(5)
3. Nagar, Singh, Kashyap, Seetharaman, Sharoff and Banerjee: Where is your application stuck? Linux Symposium 2007, vol. 2, pp. 71-78, section 3.5
4. Linux man-pages: wait(2), waitid WNOWAIT semantics
5. Linux v6.8 kernel/exit.c: wait_task_zombie and child I/O accumulation
6. Linux v6.8 fs/proc/base.c: process and task I/O accounting views