Parquet checksums: a successful read is not a file integrity check

Published 17 October 2026 / Measured API behavior

A Parquet file can return a wrong value without an error even when it contains a checksum for that page. In our PyArrow 21.0.0 experiment, enabling verification detects the changed bit. But it neither requires a checksum to exist nor validates data excluded from the read.

Three independent requirements

For a page checksum to detect a change, the writer must have stored it, the reader must check it, and the operation must actually validate the affected page. Treating these as one feature hides distinct failure modes.

Parquet separates row groups into column chunks and pages, with metadata describing their structure. Selecting one column is useful precisely because the application need not decode all other columns. A readable footer and correct declared row count therefore do not establish payload integrity. [metadata]

The format supports a page CRC32 over serialized page bytes, excluding the page header. This is a page-body check, not a certificate for the entire file, footer or application schema. [page CRC]

In the version tested here, write_page_checksum=False and page_checksum_verification=False are independent defaults. Turning on the reader setting does not retroactively supply evidence the writer omitted. [reader API] [writer API]

Row groups are not pages

A row group partitions rows logically. Each column has a separate column chunk for that group, containing its pages. In this flat two-column fixture, two row groups therefore produce four column chunks. Selecting the second group and selecting only id exclude different portions of the same file. [format concepts]

The footer tells the reader where chunks are located. Page-body CRCs do not validate that footer or the page headers. This follows from the documented format, not from a metadata-corruption experiment: our injected fault changes a payload value only. A full logical scan, a trusted artifact digest and application-level checks answer different questions. [file layout] [page CRC]

One bit, nine read cases

We generated two files with the same 128 synthetic records: an id column and a measurement column, both required 64-bit integers. Two row groups contain 64 rows each. One file has page checksums; the other does not. Neither file uses compression, encryption or dictionary encoding.

In a copy of each file, we flipped the low bit of measurement row 17, changing 100017 to 100016. These numbers are invented test inputs, not sensor data. The injector verifies that exactly one bit changed and locates the full known payload inside the column-chunk range returned by the library. Parquet's PLAIN INT64 encoding uses eight little-endian bytes per value. [encoding]

The 3 fresh-fixture repetitions produced identical results. We compare every returned selected cell with the expected table, rather than accepting a successful call or the correct number of rows.

First repetition. Unless stated otherwise, the corrupted file has CRCs and the operation reads both columns and both row groups.
Case	Verify CRC	Observed outcome
Clean file, CRC present	Enabled	Selected values match
Corrupt file, default reader	Default	One wrong value returned
Corrupt file, check disabled	Disabled	One wrong value returned
Corrupt file, check enabled	Enabled	Checksum error
Read only unmodified id column	Enabled	Selected values match
Read only unmodified row group	Enabled	Selected values match
Read metadata only	Not applicable	Metadata readable
Corrupt file, no CRC written	Enabled	One wrong value returned
Clean file, no CRC written	Enabled	Selected values match

The default and explicitly disabled reads each returned all 128 rows, with one wrong measurement. The checked full read of the checksummed file raised a checksum error. A clean checked file passed, providing the positive control.

Verification enabled is not checksum presence enforced. The deliberately damaged file written without CRCs still returned the wrong measurement with verification enabled. This agrees with Arrow's versioned source: the comparison is guarded by both the reader setting and the presence of the CRC field. [checksum implementation]

The clean column and second row group also read successfully from the damaged checksummed file. Their selected values were correct; they simply did not establish anything about the excluded damaged chunk. We did not trace physical I/O, so this result concerns validation coverage, not whether prefetch happened to transfer additional bytes.

Metadata inspection reported 128 rows and two row groups without validating the altered payload. It is a useful structural observation with a narrower scope than a data scan.

Intact pages, missing query results

A fourth concern sits outside page checksums: can the metadata used to select data be trusted? In a separate PyArrow 21.0.0 experiment, a CRC-enabled query returned zero rows even though its target record remained intact. Reading the entire file with the same verification setting returned all 128 original rows; filtering that decoded table recovered the target.

Parquet stores column-chunk statistics in file metadata. Arrow's dataset reader can use them to exclude row groups whose declared bounds cannot satisfy a predicate. Page CRCs do not cover that footer. We connected the format definition to the versioned pruning code, then tested the consequence rather than inferring it from the API option alone. [format schema] [pruning implementation]

This optimization is a min-max index, or zone map: bounds can rule out a region, but an overlapping interval cannot prove that a matching value exists. Zeng et al.'s columnar-format study explains the mechanism and why clustering affects its usefulness. That performance study is context, not evidence for our injected corruption result. [Zeng et al., sections 3.5 and 5.6]

For example, valid bounds [10, 30] can reject a search for 40, but not a search for 20, even if the region contains only 10 and 30. Looser valid bounds cost extra candidate reads; invalid narrower bounds can discard a true match. This is a logical example, not a measured speedup.

The fresh fixture again contains two 64-row groups and two required INT64 columns. We use the official Parquet 2.10.0 Thrift schema and a pinned parser to change only the first group's measurement minimum from 100000 to 100032, leaving its maximum at 100063. Both legacy min and modern min_value fields are changed. This is deliberately inconsistent metadata, not a valid alternative description of the data and not a model of a random single-bit error.

The original footer must roundtrip byte-for-byte before editing. The narrowed copy differs at exactly two footer byte positions. All bytes before the footer are identical. Independently recomputed CRC32 values match all four stored page CRCs, and all decoded values match the generated input. An altered-page positive control still raises a checksum error, confirming that checking was active.

First of 3 identical fresh-fixture repetitions. Returned and expected row counts. Page verification is enabled throughout. Filter: measurement = 100017, expected id = 17.
Operation	Read	Expected
Original statistics, filtered read	1	1
Narrowed minimum, filtered read	0	1
Statistics removed, filtered read	1	1
Wider valid bounds, filtered read	1	1
Narrowed minimum, full scan	128	128
Full scan, then filter in memory	1	1
Changed page body, full scan	CRC error	128

The row-group selection API selects group 0 for the original file and no groups for the narrowed copy. Removing only the affected statistics, or replacing them with wider valid bounds, restores the filtered result. These controls isolate the metadata-driven exclusion; we did not measure physical reads, prefetch traffic or query performance.

A full checked scan is not automatically a statistics audit. The narrowed file's full read succeeds because its data pages are intact. An additional comparison detects that stored minimum 100032 excludes decoded minimum 100000. For this required integer fixture, valid bounds must enclose the decoded minimum and maximum. Equality is not always required: the format allows conservative bounds, and our wider-bound control remains correct. Nulls, NaNs, logical types, ordering and truncated string bounds require additional rules outside this experiment. [format schema]

The practical inference is to distinguish byte integrity, metadata/data consistency and query-result completeness in an archive validation record. A trusted whole-file digest can reveal a change from a known artifact; it cannot establish that the producer's original statistics were correct. Do not treat this controlled invalid-footer example as a claim that Arrow mishandles valid Parquet files, that all readers behave identically, or that production archives suffer this fault at a measured rate.

Download the supplementary 21 observations, raw evidence and provenance and field definitions. The dataset's reproduction section provides the harness, pinned requirements and hash-checked Apache schema.

What an archive validation record should say

Our engineering conclusion is to record three separate facts: which artifacts were written with checksums, which reader policy was used, and which data the validation actually covered. A successful preview or sample query should not be reported as whole-file verification.

Writer evidence: record the writer version, checksum configuration and immutable artifact identity. A setting used for new files says nothing about older files from another producer.
Reader evidence: record the actual verification setting. In this tested implementation, checked reading is not a strict rejection policy for missing CRCs.
Coverage evidence: distinguish metadata inspection, projected columns, selected row groups and a scan of all logical data. Complete consumption with verification can check present CRCs on pages read; it is still not proof of checksum presence on every page.
Independent identity: retain a trusted whole-artifact digest or manifest when unchanged bytes matter. A digest calculated only after corruption cannot identify the earlier correct artifact. Application checks must separately validate schema, units and relationships.

These are complementary checks, not reasons to disable column pruning in normal queries. Separate routine query execution from a deliberately scoped validation job. On large archives, schedule and record validation coverage instead of silently equating the latest successful query with a scrub.

The same distinction connects this result to Hesela's checksum identity analysis, backup verification limits and HDF5 application-consistency experiment: each check answers a particular question. None automatically validates every other layer.

Reproduce and inspect the evidence

The dataset page provides all 27 observations as CSV, raw JSON, environment, source hashes and exact fault offsets. Download the Python harness and pinned requirements together:

python3 -m venv .venv
.venv/bin/pip install --only-binary=:all: -r parquet-integrity-requirements.txt
.venv/bin/python parquet-integrity.py --repetitions 3 --output observations.json

The runner creates only disposable temporary fixtures and refuses an existing output path. It uses explicit checks that remain active under Python optimization and rejects an unexpected result instead of publishing it as a passing observation. The pin identifies the tested release, not the latest available version.

Environment: Python 3.12.3, PyArrow and Arrow C++ 21.0.0, Linux 6.8.0-146-generic, x86_64. The checksummed fixture was 3077 bytes; its altered byte offset was 791. Those offsets describe this fixture, not a general recipe for editing Parquet files.

What this does not establish

This is a controlled demonstration of API behavior on one host, not a field-error sample, benchmark, corruption-detection probability or power-loss test. Repeating the same chosen fault does not estimate how often real corruption occurs. No confidence interval is warranted.

The payload and statistics experiments did not test compressed, encrypted, nested or dictionary pages, other Parquet implementations, page-header damage, arbitrary footer faults, malicious tampering, concurrent mutation or remote storage. Some other corruptions may fail to decode even without CRC verification. The bounded result is that this validly decodable changed value did not fail under the specified unchecked or absent-CRC conditions.

Primary sources

Source review: 17 October 2026. Experimental outputs are Hesela's own synthetic-fixture observations.