Metadata — Hurray Format Specification

Status: Draft

Scope

This section defines the binary encoding of the tensor descriptor: the self-describing header that precedes every tensor data buffer in the Hurray format. The tensor descriptor encodes all information required to interpret a tensor: its element type, rank, shape, memory layout, buffer table, and optional quantization and shard annotations.

Note (non-normative): The tensor descriptor is designed to be self-delimiting: a receiver can determine its total byte length from the first 10 bytes (the descriptor_length field occupies bytes 6–9) and MAY skip the descriptor entirely without parsing any layout-specific fields. Skipping past the whole descriptor is what only requires the first 10 bytes; locating or skipping a particular section within the descriptor (e.g. the buffer table or the quantization section) requires reading through the preceding fields. This property is essential for streaming readers.

Normative Requirements

This section uses RFC 2119 key words: MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL.

Overall Descriptor Structure

A tensor descriptor consists of a fixed header, followed by variable-length core fields, followed by layout-specific fields, followed by a buffer table, followed by zero or more optional sections selected by the flags field.

[Fixed Header]         20 bytes
[shape]                8 × rank bytes
[byte_offset]          8 bytes
[Layout-specific]      variable (layout-dependent)
[Buffer Table]         variable
[Quantization]         variable, present if HAS_QUANTIZATION flag is set
[Shard]                variable, present if HAS_SHARD flag is set
[Statistics]           72 bytes, present if HAS_STATISTICS flag is set
[Extension Type]       20 bytes, present if HAS_EXTENSION_TYPE flag is set
[Composite Member]     16 bytes, present if HAS_COMPOSITE_MEMBER flag is set

All multi-byte fields MUST be encoded in little-endian byte order (least significant byte at the lowest address).

Fixed Header

The fixed header occupies the first 20 bytes of every tensor descriptor.

A reader MUST reject a descriptor whose magic field is not 0x48 0x52 0x52 0x59.

A reader MUST reject a descriptor whose version_major exceeds the reader's supported major version.

A reader MUST reject a descriptor whose descriptor_length is less than 20 (the minimum valid descriptor size).

A reader MUST NOT read beyond descriptor_length bytes when parsing a descriptor.

Flags

Core Variable Fields

Immediately following the fixed header:

shape

uint64[rank] — the size of each dimension, in ascending dimension order (dimension 0 first). Each value MUST be greater than or equal to 0. A dimension size of 0 indicates an empty tensor.

The value 0xFFFFFFFFFFFFFFFF (UINT64_MAX) is the dynamic dimension sentinel: it indicates that the dimension's size is not statically known. A reader MUST NOT compute buffer sizes, strides, or element counts for a dimension carrying this sentinel without first resolving it to a concrete value.

For a scalar tensor (rank = 0), this field is absent (zero bytes).

byte_offset

uint64 — the byte offset from the start of buffer 0 in the buffer table to the element at logical index [0, 0, ..., 0]. MUST be less than or equal to the byte size of buffer 0.

For sub-byte types (bool, int4, uint4, int2, uint2), byte_offset MUST point to a byte boundary. The first element begins at bit 0 of that byte.

Layout-Specific Fields

Immediately following byte_offset, the layout-specific fields for the layout identified by layout_tag are encoded. If the layout has no additional fields, this section is absent.

Row-Major (0x01) and Column-Major (0x02)

No additional fields. Strides are implicit and computed as defined in memory-layout.md.

Strided (0x03)

Tiled / Blocked (0x04)

If outer_layout is 0x03 (strided):

If inner_layout is 0x03 (strided):

If inner_layout is 0x04 (recursive tiling), the tiled layout-specific fields are encoded recursively at this point, beginning with tile_shape.

A reader MUST enforce a maximum recursion depth for nested tiled descriptors. The RECOMMENDED limit is 8 levels. A reader MUST reject a descriptor that exceeds its configured recursion limit.

Morton (Z-Order Curve) (0x05)

COO (0x06)

See layouts/coo.md for buffer table composition, storage order, and validity constraints.

CSR (0x07)

See layouts/csr.md for buffer table composition, storage invariants, and rank-2 restriction.

CSC (0x08)

See layouts/csc.md for buffer table composition, storage invariants, and rank-2 restriction.

Hilbert Curve (0x40)

Composite / Virtual (0x0B)

The composite head is a virtual (data-less) descriptor. Its layout-specific fields encode the composition rule. All multi-byte fields are little-endian.

A composite head MUST have buffer_count = 0x00 and byte_offset = 0x0000000000000000. See layouts/composite.md for members, binding, composition semantics, and validation.

Extension Layouts (0xF0–0xFE)

A reader that does not recognise extension_layout_id MUST reject the descriptor, unless operating in permissive mode.

Buffer Table

Immediately following the layout-specific fields, the buffer table is encoded.

The maximum value is 255, imposed by the uint8 wire type. This limit applies to the sum of data and quantization-parameter buffers. Implementations that require more than 255 buffers MUST use multiple tensor descriptors.

Followed by buffer_count buffer handles, each encoded as 16 bytes:

The _reserved bytes MUST be 0x00. A conforming reader in strict mode MUST reject a descriptor containing any buffer handle whose _reserved bytes are not all 0x00.

Note (non-normative): For sparse layout tags (COO 0x06, CSR 0x07, CSC 0x08, CSF 0x09), buffer_count exceeds 1 — each entry holds a distinct component array (values, indices, pointers). For CSF the count is rank-dependent, 2·rank + 1 (one values buffer plus a pos/crd pair per level). For quantized dense tensors, quantization-parameter buffers (scales, zero-points) extend the buffer table beyond the layout baseline. The layout-defined minimum is always 0x01 for dense layouts; quantization schemes append their parameter buffers on top.

Quantization Section

Present if and only if the HAS_QUANTIZATION flag (bit 0) is set.

A reader that encounters HAS_QUANTIZATION but does not support quantized types MUST reject the descriptor unless operating in permissive mode.

Shard Section

Present if and only if the HAS_SHARD flag (bit 1) is set.

The constraint shard_offset[k] + shape[k] <= parent_shape[k] MUST hold for every dimension k. A reader MUST reject a shard descriptor that violates this constraint.

Statistics Section

Present if and only if the HAS_STATISTICS flag (bit 3) is set.

The statistics section is a fixed-size 72-byte block. All statistics are advisory: they reflect the tensor data at write time. A reader MUST NOT rely on any statistic for correctness; statistics MAY be used only as optimization hints (algorithm selection, memory pre-allocation, routing decisions).

Note (non-normative): A streaming writer that has not processed the entire tensor buffer before emitting the descriptor (e.g., a pipeline stage forwarding data on the fly) MUST omit the statistics section (HAS_STATISTICS not set) rather than emitting invalid statistics. A writer that knows only a subset of statistics (e.g., nnz is known from a sparse format but value_mean was not computed) MUST mark the unknown fields as not valid in computed_mask.

computed_mask

The computed_mask field (first 4 bytes of the section) is a bitmask indicating which statistics fields contain valid data. A reader MUST check the relevant bit before using any field. Fields whose bit is not set MUST be treated as unknown, regardless of their encoded value.

A conforming reader MUST reject a descriptor whose computed_mask has any reserved bit set (any bit greater than or equal to 6, given the six defined statistics fields above).

Field Encoding

The 72-byte statistics block is encoded as follows. All multi-byte fields are little-endian.

Note (non-normative): value_min, value_max, value_abs_max, value_mean, and value_stddev are always expressed in float64 regardless of the tensor's element type. For quantized tensors, these values reflect dequantized (real-valued) statistics, not the raw quantized storage values. For bool types, the statistics are defined over the integer domain {0, 1}.

Note (non-normative): N:M structured sparsity is particularly relevant for NVIDIA Ampere/Ada/Hopper Tensor Cores, which provide hardware-accelerated 2:4 sparsity (2 non-zeros in every group of 4 consecutive elements). Declaring the N:M pattern in the descriptor lets a receiver select the sparse kernel path without scanning the buffer.

Extension Type Section

Present if and only if the HAS_EXTENSION_TYPE flag (bit 2) is set. This flag MUST be set whenever type_tag is in the range 0xF0–0xFE, and MUST NOT be set otherwise.

Per ADR-001, extension type tags MUST carry an inline descriptor providing at minimum the bit width and packing parameters required to compute buffer sizes.

The extension type descriptor is 20 bytes:

Note (non-normative): The sign of a float extension type is carried by sign_bits; is_signed applies to integer types only. Requiring is_signed to be 0x00 for float types does not mean float extension types are unsigned — it means a reader has exactly one field to consult. A signed float sets is_float = 0x01 and sign_bits = 0x01; an unsigned float, such as a private analogue of the exponent-only float8_e8m0 (0x42), sets is_float = 0x01 and sign_bits = 0x00.

Sub-byte element widths that are not a power of two (notably 6-bit) are reserved to the built-in type tag space. Implementors requiring an interchange-portable non-power-of-two sub-byte type MUST request a built-in tag assignment through the specification governance process rather than encoding the type in the private extension range. The extension descriptor's whole-byte and power-of-two sub-byte width restriction ensures that buffer-size computation remains a single integer formula (ceil(N / packing_factor) for sub-byte, N * (bit_width / 8) for whole-byte) without rational arithmetic.

Note (non-normative): The 6-bit float6_e2m3 (0x44) and float6_e3m2 (0x45) types are built-in (Tier 2) and use a dedicated 4-elements-per-3-bytes packing defined in element-types.md. Their packing rule is not expressible as 8 / bit_width and is therefore not delegable to the generic extension descriptor. The extension descriptor is designed for private, implementation-defined types whose layout fits the simple "elements per byte" model; richer packings remain the prerogative of the standardized type system.

A reader MUST use bit_width and packing_factor to compute buffer sizes for tensors with extension type tags, even if it does not interpret the numeric semantics of the type.

Composite Member Section

Present if and only if the HAS_COMPOSITE_MEMBER flag (bit 4) is set. It appears after the Extension Type section. The section is a fixed-size 16-byte block. All multi-byte fields are little-endian.

A reader MUST reject a Composite Member section whose member_role is not 0x00 or 0x01, or whose _reserved bytes are not all 0x00.

This section is carried by the members of an overlay composite (composition_rule = 0x02). Partition and group members MUST NOT set HAS_COMPOSITE_MEMBER. The full semantics — base-vs-correction roles, precedence, and the combine operation — are defined in layouts/composite.md.

Note (non-normative): v1.0 uses member_role only. The 15 reserved bytes hold room for a future member_version field (versioned overlay), added additively without reallocating the section. See layouts/composite.md § Deferred.

Version Compatibility

A reader MUST reject a descriptor whose version_major exceeds the reader's supported major version.

A reader encountering a version_minor greater than its supported minor version SHOULD accept the descriptor but MUST NOT interpret fields beyond what its supported minor version defines. The descriptor_length field allows the reader to skip the descriptor entirely if desired.

Note (non-normative): Minor version increments add optional fields or new flag bits. A reader built against version 1.0 will correctly skip a 1.1 descriptor by consuming exactly descriptor_length bytes, because all reserved flag bits were required to be 0 in 1.0. Major version increments signal backward-incompatible changes; a reader MUST NOT attempt to parse a descriptor with an unsupported major version.

Worked Example

A rank-2 float32 tensor with shape [3, 4] in row-major layout, one buffer of 192 bytes aligned to 64 bytes on CPU, no optional sections:

Offset  Value (hex)                   Field
------  ----------------------------  -----
0       48 52 52 59                   magic = "HRRY"
4       01                            version_major = 1
5       00                            version_minor = 0
6       3D 00 00 00                   descriptor_length = 61
10      00 00 00 00                   flags = 0x00000000 (no optional sections)
14      03                            type_tag = 0x03 (float32)
15      01                            layout_tag = 0x01 (row-major)
16      02 00 00 00                   rank = 2
20      03 00 00 00 00 00 00 00       shape[0] = 3
28      04 00 00 00 00 00 00 00       shape[1] = 4
36      00 00 00 00 00 00 00 00       byte_offset = 0
                                      (no layout-specific fields for row-major)
44      01                            buffer_count = 1
45      C0 00 00 00 00 00 00 00       buffer[0].byte_size = 192
53      40 00 00 00                   buffer[0].alignment = 64
57      00                            buffer[0].device_tag = 0x00 (CPU)
58      00 00 00                      buffer[0]._reserved

Total: 61 bytes. Fixed header (20) + shape (16) + byte_offset (8) + buffer table (1 + 16) = 61.

Interaction with Other Sections

• Element Types (element-types.md): defines type_tag values and the bit-width, packing, and alignment properties used during buffer size computation.
• Memory Layout (memory-layout.md): defines layout_tag values and the layout-specific fields encoded in this descriptor.
• Quantization (quantization.md): defines the binary format of the quantization_descriptor payload in the quantization section.
• Buffer Protocol (buffer-protocol.md): defines buffer ownership, device memory semantics, and release callback conventions referenced by the buffer table entries.
• Interchange (interchange.md): the tensor descriptor defined here is transmitted verbatim in TENSOR_DESCRIPTOR and TENSOR_PUT message payloads, followed by transport-specific fields (total_data_bytes, shard_index, total_shards).

Open Questions

[OQ-1]: Should the descriptor include a CRC-32 checksum field? Resolved: No checksum in the descriptor. Integrity is delegated to the transport/storage layer (TCP, TLS, ECC, ZFS). Adding 4 bytes and a full-pass CRC on every descriptor would penalise in-process and IPC interchange where corruption is not a realistic threat. If file-level integrity is needed, it belongs in the file format footer (see file-format.md OQ-3).

[OQ-2]: The binary encoding of the quantization descriptor is deferred to quantization.md. Resolved: The encoding is fully defined in quantization.md: a fixed 4-byte header (scheme_tag, scheme_version, flags) followed by a per-scheme payload. Complete byte-level layouts are specified in quantization/per-tensor-affine.md, quantization/per-channel-affine.md, quantization/per-block-affine.md, quantization/nf4.md, and quantization/mxfp.md. The quantization_length prefix in metadata.md allows readers to skip unrecognised schemes safely.