NF4 (NormalFloat4) Quantization — Hurray Format Specification

Scheme tag: 0x04 | Tier: 2

This section uses RFC 2119 key words: MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL.

Description

A non-linear 4-bit quantization scheme introduced by the QLoRA paper. Each storage code in [0, 15] decodes to one of 16 fixed real-valued levels chosen to be information-theoretically optimal for weights drawn from a standard normal distribution. Each block carries a single absmax scale; there is no per-element zero point.

The block index computation follows the same rule as Per-Block Affine (see quantization/per-block-affine.md § Block Layout).

Binary Encoding

Total descriptor length: 16 bytes.

All multi-byte fields MUST be encoded in little-endian byte order.

Lookup Table

The 16 NF4 levels are fixed by this specification and MUST NOT be altered by readers or writers. Indexed by the unsigned 4-bit storage code q:

The table values are given here as float32 decimal expansions of the exact levels from the QLoRA reference implementation. Implementations MUST use these float32 values verbatim; deriving the table from first principles at runtime is NOT RECOMMENDED and MUST match these values bit-for-bit if attempted.

Referenced Buffer

The scale buffer MUST contain exactly num_blocks consecutive float32 absmax values in little-endian byte order, starting at byte offset 0 within the referenced buffer. num_blocks is computed identically to the Per-Block Affine scheme. Its byte size MUST be exactly num_blocks * 4.

Dequantization Formula

Let b be the block index for a storage element q at logical position [i_0, ..., i_{rank-1}] (computed as in Per-Block Affine). Let s = scale[b].

x_real = s * nf4[q]

The multiplication is performed in float32 arithmetic.

Validity Constraints

• axis MUST satisfy axis < rank.
• block_size MUST be a power of two and MUST be greater than or equal to 8.
• When shape[axis] is greater than 0, block_size MUST be less than or equal to shape[axis]. A reader MUST reject a descriptor whose block_size exceeds a non-zero shape[axis].
• When shape[axis] equals 0 (an empty quantization axis, per ADR-007), the upper-bound check is waived. The block_size field MUST still be a power of two greater than or equal to 8, but its value has no effect on buffer sizing: num_blocks evaluates to 0, and the absmax scale buffer MUST have byte size 0.
• shape[axis] MUST NOT equal 0xFFFFFFFFFFFFFFFF.
• Every scale value MUST be a finite, non-negative float32.
• scale_buffer_index MUST be a valid index into the buffer table and MUST NOT refer to the tensor data buffer.

Note (non-normative): The minimum block size of 8 reflects the information-theoretic minimum for NF4: below 8 elements per block, the 16-point quantization grid provides no statistical benefit over a lower-bit format.

Valid Storage Types

The storage type (type_tag in the tensor descriptor) MUST be uint4 (0x49).

A reader MUST reject an NF4 descriptor whose storage type is not uint4.

Note (non-normative): NF4 storage codes are conceptually unsigned (they index into a signed-valued lookup table); uint4 is the correct storage type. The packing order follows the standard uint4 rule from element-types.md: element 2k in the low nibble of byte k, element 2k+1 in the high nibble.