Arrow's columnar layout

Arrow specifies exact memory layouts so that two processes, possibly in different languages, can share a buffer without serializing. Zero-copy reads, cheap null checks and SIMD-friendly scans all follow from three buffer types.

The validity bitmap

Nullness is stored separately from values, one bit per element. The specification defines it as: "A 1 (set bit) for index j indicates that the value is not null, while a 0 (bit not set) indicates that it is null."

Bits are numbered least-significant-first within each byte, so the validity test is bitmap[j / 8] & (1 << (j % 8)). An array with no nulls may omit the bitmap entirely, which is a real saving on wide tables of non-nullable columns.

Offsets and values

Variable-length types (strings, binary, lists) do not store lengths per element. They store a buffer of length + 1 offsets, and the element at j occupies bytes offsets[j] to offsets[j+1] of the values buffer.

Offsets are monotonically increasing even across nulls, hence the repeated 3, so the length of any element is a subtraction with no branching. The values buffer contains no placeholder for nulls at all; a null contributes zero bytes.

import pyarrow as pa

arr = pa.array(["joe", None, None, "mark"])
validity, offsets, values = arr.buffers()

print(bytes(validity)[0] & 0b1111)     # 9  -> 0b1001
print(memoryview(offsets).cast("i").tolist())
print(bytes(values))                   # b'joemark'

Fixed-width types

Numeric arrays have no offsets buffer: element j is at byte j * itemsize. A float64 column with no nulls can therefore be handed to NumPy as a view rather than a copy. With nulls, the values buffer is still valid, but the consumer needs the bitmap to know which entries are meaningful.

Padding and alignment

The specification recommends allocating buffers on 8- or 64-byte aligned addresses and padding their length to a multiple of 8 or 64 bytes. The stated reason for 64 specifically is that it "matches the largest SIMD instruction registers available on widely deployed x86 architecture (Intel AVX-512)".

The practical effect is that a vectorized loop can process a buffer in full register-width steps without a scalar tail and without reading past an allocation. This is also why a small Arrow array can occupy more bytes than its contents suggest. The padding is what makes the scan loop branch-free.

Why this shows up in practice

• A string column with many short values is dominated by its offsets buffer: 4 bytes per element for string, 8 for large_string. For very short strings, the offsets can exceed the data.
• Filtering does not compact the values buffer unless you ask for it. A filtered Arrow array can hold a values buffer far larger than the surviving elements.
• Nulls cost one bit, not a sentinel value. There is no NaN-as-null ambiguity of the kind NumPy float arrays have.

Opening an Arrow or Feather file in the Arrow viewer shows the schema and per-column types, which is usually enough to spot an unexpected large_string or a column that is nullable when it should not be.