Parquet vs Arrow vs Feather

Parquet is an encoded format optimized for storage, Arrow is an uncompressed memory layout optimized for computation, and Feather is Arrow serialized to disk. Two of the three are not file formats in the same sense, which is where most of the confusion starts.

The actual distinction

Arrow's in-memory layout and its file format are the same bytes, so a process can memory-map an Arrow file and operate on it without parsing. Parquet cannot work this way, because its values are dictionary-encoded, bit-packed and compressed, so they must be reconstructed before use.

Feather is Arrow IPC

Feather v2 is the Arrow IPC file format. "Feather" is effectively a filename convention for it; pyarrow.feather and pyarrow.ipc write compatible files. Feather v1 was a separate, now-superseded format, which is the source of some older advice claiming Feather is unstable. That advice applied to v1.

import pyarrow as pa, pyarrow.feather as ft, pyarrow.parquet as pq

table = pa.table({"id": range(1_000_000), "label": ["a"] * 1_000_000})

pq.write_table(table, "t.parquet")              # encoded + compressed

# write_feather defaults to LZ4 when available, which would mean a decode pass
# on read. Ask for uncompressed explicitly to get the memory-mappable form.
ft.write_feather(table, "t.arrow", compression="uncompressed")

# Reading Parquet decodes; reading uncompressed Arrow maps.
t1 = pq.read_table("t.parquet")
with pa.memory_map("t.arrow") as src:
    t2 = ft.read_table(src)                     # no decode pass

The label column above is a million copies of one string. Parquet stores it as a dictionary of one entry plus a run-length encoded index, which comes to a handful of bytes. Arrow does not deduplicate values: a plain string array holds a million-and-one 32-bit offsets plus a million copies of the byte a, about 5 MB. The gap is four to five orders of magnitude, and it is an encoding difference rather than a compression one.

Choosing between them

• Storing a dataset that will be read many times, by many tools, possibly over a network: Parquet. Smaller files, wide ecosystem support, statistics that let readers skip data.
• Passing data between processes or languages on one machine, or caching an intermediate result you will re-read in seconds: Arrow IPC / Feather. No decode cost, and memory-mapping makes it near-free to open.
• Neither, exclusively. The common pipeline stores Parquet and materialises Arrow in memory. pyarrow.parquet.read_table returns an Arrow table; the two are designed to be used together, not chosen between.

Inspecting either one

The schema, row counts and column types of both can be read without loading the data. The Parquet viewer and Arrow viewer do this locally in the browser; pq.ParquetFile(...).schema_arrow and pa.ipc.open_file(...).schema do it from Python.