The .npy and .npz formats, byte by byte

.npy is the smallest useful array format: enough metadata to reconstruct dtype and shape, and then the array bytes exactly as they sit in memory. That last property is what allows np.load(..., mmap_mode="r") to open a 100 GB file instantly.

The header

The header is a Python dictionary literal with exactly three keys, in alphabetical order: descr (a dtype descriptor), fortran_order (a bool), and shape (a tuple). It is terminated by a newline and padded with spaces so that the magic, version, length field and header together are divisible by 64.

93 4e 55 4d 50 59 01 00 76 00 7b 27 64 65 73 63
 \x93  N  U  M  P  Y  1  0  118    {  '  d  e  s  c

{'descr': '<f4', 'fortran_order': False, 'shape': (1000, 256), }
<- padded with spaces, ending with a newline as the final byte ->

The 64-byte alignment means the data section starts on a boundary suitable for SIMD loads, so the file can be mapped and used directly rather than copied into an aligned buffer.

Versions

• 1.0: 2-byte header length, so headers up to 65,535 bytes.
• 2.0: 4-byte header length, for structured dtypes with very many fields. np.save upgrades automatically when the header will not fit.
• 3.0: UTF-8 header, for structured arrays with non-ASCII field names.

Reading the header without NumPy

import ast, struct

with open("array.npy", "rb") as f:
    assert f.read(6) == b"\x93NUMPY"
    major, minor = f.read(2)
    n = struct.unpack("<H" if major == 1 else "<I",
                      f.read(2 if major == 1 else 4))[0]
    header = ast.literal_eval(f.read(n).decode())

print(header)   # {'descr': '<f4', 'fortran_order': False, 'shape': (1000, 256)}

.npz is a ZIP file

An .npz is a ZIP archive containing one .npy member per array, named after the keyword passed to np.savez. Standard ZIP tools will list and extract them.

import numpy as np, zipfile

np.savez("bundle.npz", train=np.zeros((100, 8)), test=np.ones((20, 8)))
print(zipfile.ZipFile("bundle.npz").namelist())   # ['train.npy', 'test.npy']

# np.savez is stored (uncompressed); np.savez_compressed deflates.
z = np.load("bundle.npz")        # lazy: members read on access
print(z["train"].shape)

np.savez stores members uncompressed, so members inside the archive can be memory-mapped by tools that understand the offsets; members written by np.savez_compressed cannot be. np.load on an .npz returns a lazy mapping: each member is read, and inflated if it was compressed, on first access rather than upfront, so touching one key in a large bundle is cheap.

Memory mapping

import numpy as np

a = np.load("big.npy", mmap_mode="r")   # nothing read yet
print(a.shape, a.dtype)
window = a[5000:5100]                   # only these pages fault in

To check shape, dtype and Fortran order of a file without loading it, the snippet above is sufficient; the NumPy viewer does the same in the browser and also lists the members of an .npz.