float16, bfloat16 and fp8

A checkpoint listing F16 and one listing BF16 occupy identical bytes per element and are not interchangeable. Both spend 16 bits; they divide those bits differently, and the division determines which one silently produces zeros and infinities during training.

The bit layouts

Every one of these formats is sign, then biased exponent, then mantissa. The exponent width sets the range; the mantissa width sets the precision.

Converting float32 to bfloat16 keeps the high 16 bits; serious implementations round to nearest even rather than simply discarding the low half, so the carry can reach the bits that are kept. Converting back is a zero-fill. No rescaling, no range check, and the same exponent range at both ends.

What the split costs

float16 runs out of range long before it runs out of precision. Gradients in a transformer routinely fall below 6.1 × 10⁻⁵, where float16 enters subnormals and then flushes to zero. This is what loss scaling exists to fix: multiply the loss by a large constant so gradients land inside float16's window, then divide it out before the optimizer step.

bfloat16 has the same range as float32, so gradients underflow at the same point they would in full precision, which in practice does not happen. No loss scaling is required. The cost is 7 mantissa bits, roughly two to three significant decimal digits, which is tolerable for weights and activations and is why bfloat16 became the default on hardware that supports it.

The two fp8 formats

FP8 is standardized as two formats rather than one, because inference activations and training gradients have different requirements. Micikevicius et al. (2022) specify E4M3 for weights and activations, favoring precision, and E5M2 for gradients, favoring range.

E5M2 follows IEEE 754 conventions, with infinities and the usual NaN encodings. Both are almost always used with a per-tensor or per-block scale factor stored alongside, since eight bits cannot span a realistic dynamic range on their own.

How they appear in files

bfloat16 and fp8 have no NumPy dtype, so they cannot be read with np.frombuffer directly. In safetensors they appear as the dtype strings BF16, F8_E4M3 and F8_E5M2; the bytes are there, but reconstructing values needs either a framework or a manual widen.

import numpy as np

def bf16_to_f32(raw: bytes) -> np.ndarray:
    """bfloat16 is the high 16 bits of float32; widen by zero-filling the low half."""
    u16 = np.frombuffer(raw, dtype="<u2").astype("<u4")
    return (u16 << 16).view("<f4")

def f32_to_bf16(a: np.ndarray) -> np.ndarray:
    """Narrow with round-to-nearest-even, guarding NaN."""
    u32 = a.astype("<f4").view("<u4")
    rounded = (u32 + 0x7FFF + ((u32 >> 16) & 1)) >> 16
    # A NaN whose payload sits only in the low 16 bits would round to 0x7F80,
    # which is +inf. Detect NaN first and set a mantissa bit in the result.
    is_nan = (u32 & 0x7FFFFFFF) > 0x7F800000
    rounded = np.where(is_nan, (u32 >> 16) | 0x0040, rounded)
    return rounded.astype("<u2")

import torch

t = torch.randn(4, dtype=torch.float32)
print(t.to(torch.bfloat16).to(torch.float32) - t)   # rounding error
print(t.to(torch.float16).to(torch.float32) - t)    # smaller, until it overflows

print(torch.finfo(torch.float16).max)               # 65504.0
print(torch.finfo(torch.bfloat16).max)              # 3.3895e+38

Choosing

• Training on hardware with bfloat16 support: bfloat16, no loss scaling. This is the default for most current training stacks.
• Training on hardware without it: float16 with dynamic loss scaling, and a float32 master copy of the weights.
• Inference: float16 is usually fine, because activations are better conditioned than gradients, and its extra mantissa bits are worth having.
• fp8: only with framework support for the scale factors. The formats alone do not have the range to be used unscaled.

To check what a checkpoint actually stores, the header is enough, because dtype is recorded per tensor. The safetensors viewer lists every tensor with its dtype and shape without loading weights, which is the quickest way to confirm a file is bfloat16 rather than float16.