HDF5 vs Zarr

HDF5 and Zarr describe almost the same thing: a hierarchy of groups, containing N-dimensional arrays, stored as compressed chunks, with attributes attached to any node. The data models are nearly identical; the storage strategies are not, and every practical consequence follows from that.

One file with an internal index, or a directory of objects

HDF5 packs the group hierarchy, attributes, dataset metadata and every chunk into a single file, with an internal B-tree index mapping chunk coordinates to byte offsets. Reading a chunk means traversing that index, then seeking.

Zarr stores metadata as JSON and each chunk as a separate key in a store. On a filesystem those are files in a directory; on S3 they are individual objects. There is no index to traverse, because a chunk's key is computed directly from its coordinates.

Concurrency

Because Zarr chunks are independent objects, N writers can write N different chunks with no coordination and no locking. Nothing is shared, so nothing needs protecting.

HDF5 has a single-file structure with shared metadata, so concurrent modification requires coordination. The C library is not thread-safe unless built in its thread-safe configuration, which serializes API calls behind a global lock. Parallel writing is possible through the MPI-based parallel HDF5 build, but that is a deployment decision, not a flag. SWMR (single-writer/multiple-reader) mode allows readers during appends, but does not permit structural changes while open.

Object storage

On S3 or similar, HDF5 requires the reader to fetch pieces of one large object: the superblock, then index nodes, then chunk bytes, each as a range request, with each step depending on the last. Latency accumulates because the traversal is sequential.

Zarr reads one small JSON object and can then issue every chunk request in parallel, since keys are computable. On high-latency storage this is the dominant factor. The same property becomes a liability on very large arrays: 100,000 chunks means 100,000 objects, with per-request overhead and listing costs to match.

A reasonable default

• Local or network filesystem, one process writing, files that get moved around as units: HDF5. A single file is easier to move, and the tooling is mature.
• Object storage, or many workers writing concurrently: Zarr. Chunk independence is what makes the concurrency work.
• Existing HDF5 in the cloud that you cannot rewrite: build a chunk index and read it with the Zarr machinery. kerchunk and virtualizarr generate a manifest of byte ranges so the HDF5 chunks can be fetched in parallel without conversion.

For either format, chunk shape matters more than the choice between them. See chunk shape and read amplification. A well-chunked HDF5 file beats a badly-chunked Zarr store by a wide margin, and the reverse is equally true.

To see what an existing file uses, the HDF5 viewer and Zarr viewer show the group tree, chunk shape and compression for each array without loading the data.