TFRecord's record framing

TFRecord is a container with almost nothing in it: a stream of length-prefixed byte strings with checksums. It has no header, no footer, no schema and no index. The framing is why it streams well, and why reaching record 40,000 means reading the 39,999 records before it.

One record

uint64 length
uint32 masked_crc32_of_length
byte   data[length]
uint32 masked_crc32_of_data

The checksums are CRC-32C (the Castagnoli polynomial), and they are stored masked rather than raw:

masked_crc = ((crc >> 15) | (crc << 17)) + 0xa282ead8

The rotation exists so that the checksum of a block is not itself a valid CRC of that block, which avoids a class of errors where a checksum field is accidentally computed over data that already contains it. Both operations are on 32-bit unsigned values, so the addition wraps.

Reading one without TensorFlow

The framing has no dependency on TensorFlow, protobuf, or the payload type. A reader is about fifteen lines.

import struct
from crc32c import crc32c            # pip install crc32c

def mask(crc):
    return (((crc >> 15) | (crc << 17)) + 0xA282EAD8) & 0xFFFFFFFF

def records(path, verify=True):
    with open(path, "rb") as f:
        while True:
            head = f.read(8)
            if len(head) < 8:
                return                        # clean end of file
            (len_crc,) = struct.unpack("<I", f.read(4))
            if verify and mask(crc32c(head)) != len_crc:
                raise ValueError("corrupt length field")
            (length,) = struct.unpack("<Q", head)
            data = f.read(length)
            (data_crc,) = struct.unpack("<I", f.read(4))
            if verify and mask(crc32c(data)) != data_crc:
                raise ValueError("corrupt record")
            yield data

The payload is almost always a serialized tf.train.Example protobuf, but nothing in the container requires that. The framing carries JPEG bytes or JSON equally well; only the consumer decides.

from tensorflow.core.example import example_pb2

for raw in records("shard-00000-of-00064.tfrecord"):
    ex = example_pb2.Example()
    ex.ParseFromString(raw)
    print(dict(ex.features.feature).keys())
    break

The consequences of having no index

• No random access. Reaching record n means reading the framing of every record before it. There is no offset table to consult.
• No count without a full pass. len(dataset) is not answerable cheaply; the number of records is only known after reading to the end.
• No schema. Field names and types live in the payload, repeated in every record. Discovering the schema means parsing at least one record.
• Corruption is bounded. Because records are independent and checksummed, a damaged region can be skipped without invalidating the rest.

The first two are why TFRecord datasets are conventionally sharded. Nothing in the format requires it, and single-file TFRecords are common at small scale. Splitting into files named train-00000-of-00064 gives back the parallelism the format lacks internally: each worker reads whole shards sequentially, and shuffling happens across shards plus a buffer within them, rather than by seeking.

Practical notes

• Shard size of roughly 100 to 200 MB is the common recommendation, balancing parallelism against per-file overhead.
• Small records are wasteful. At 16 bytes of framing plus protobuf overhead, records of a few hundred bytes spend a noticeable fraction on structure. Batch several examples per record if payloads are tiny.
• Compression is per-file, via GZIP or ZLIB options, not per-record. A compressed shard cannot be partially read.

To read the record count, record sizes and the feature keys of the first examples in a shard, the TFRecord viewer walks the framing locally in the browser, which is often faster than setting up a TensorFlow environment to answer one question.