Parquet row groups, column chunks and pages

Two Parquet files with the same rows can differ by two orders of magnitude in scan time. The difference is usually internal layout: how many row groups the file has, and whether the reader can skip most of them.

The outer frame

A Parquet file begins and ends with the 4-byte magic number PAR1. Immediately before the trailing magic is a 4-byte little-endian integer giving the length of the file metadata, and before that is the metadata itself, Thrift-encoded.

A Parquet reader therefore needs two reads before it touches any data. The official layout description states: "Readers are expected to first read the file metadata to find all the column chunks they are interested in. The columns chunks should then be read sequentially."

Three levels, not one

Inside the data section the file is split three ways. A row group is a horizontal slice of the table. Within a row group, each column occupies one column chunk, stored contiguously. Each column chunk is divided into pages, which are the unit of compression and encoding.

A column is contiguous within a row group, not across the file. Selecting one column from a 200-row-group file means 200 separate reads, so row group count matters as much as row group size.

Where the statistics live, and what they buy

The footer carries, for each column chunk, its byte offset and optional statistics: min, max, null count and distinct count. Later versions of the format added min_value and max_value, which deprecate min/max because the originals were compared with signed byte ordering rather than the column's logical-type ordering. A reader evaluating WHERE score > 0.9 can compare the predicate against each chunk's maximum and skip the entire chunk when it cannot match. This is predicate pushdown, and it is the mechanism behind most large Parquet speedups.

Skipping only works if the data is clustered. If score is randomly distributed, every row group's min/max spans nearly the full range and nothing is skippable. Sorting by the column you filter on, before writing, is what makes the statistics usable.

import pyarrow.parquet as pq

f = pq.ParquetFile("data.parquet")
print(f.metadata.num_row_groups, f.metadata.num_rows)

rg = f.metadata.row_group(0)
for i in range(rg.num_columns):
    col = rg.column(i)
    s = col.statistics
    print(col.path_in_schema,
          col.total_compressed_size,
          (s.min, s.max) if s and s.has_min_max else None)

Choosing a row group size

The Parquet configuration guidance recommends row groups of 512 MB to 1 GB and 8 KB data pages. Both are recommendations rather than defaults: parquet-mr's actual default page size is 1 MB. Large row groups permit larger sequential reads of each column chunk.

That 512 MB to 1 GB figure is HDFS-era advice, aimed at making one row group fill one HDFS block. Current writers are more conservative: PyArrow defaults to roughly one million rows per group, and 128 MB is a common target in Spark and DuckDB practice. The official number is an upper bound rather than a target.

The failure mode at the other end is many small files, each with a single small row group. The footer is then a large fraction of every file, per-file open costs dominate, and the column chunks are too short for the sequential reads the format is built around. Compaction into fewer, larger files usually recovers more time than any encoding change.

Checking a real file

Before tuning anything, look at what a writer actually produced: row group count, rows per group, and compressed size per column chunk. The snippet above reports all three, as does the Parquet viewer, which reads the footer locally in the browser.