Notes on how Parquet, Arrow, HDF5, Zarr and the other formats this site opens are laid out on disk, and the decisions that follow. Each one is self-contained, cites its sources, and sticks to what the specifications say.
Why selecting one column can mean 200 separate reads, and what row group size actually trades off.
When to store Parquet, when to store Feather, and why converting between them costs a decode rather than a copy.
How a string column becomes three buffers, why a null costs one bit, and where the 64-byte padding comes from.
Why a 40 KB slice can cost 400 MB of I/O, and how to pick a chunk shape for HDF5 and Zarr.
Why Zarr writes chunks in parallel with no coordination while HDF5 needs an MPI build, and which to pick for object storage.
The renamed metadata fields, the single zarr.json, and the codec pipeline that replaced filters and compressor.
The complete layout in three fields, and why a pickle checkpoint can execute code while this one cannot.
Why float16 needs loss scaling and bfloat16 does not, with the bit layouts and ranges for both.
The header layout, the 64-byte padding rule, and why .npy can be memory-mapped but .npz cannot.
Sixteen bytes of framing per record, why there is no random access, and how to read one without TensorFlow.
How samples are grouped inside a tar, and why sequential shard reads keep the GPU fed.