Hurray Format Specification

Version: 0.1.0-draft

Hurray is a language-agnostic, zero-copy runtime interchange format for multi-dimensional tensor data, optimized for the memory layout diversity, quantization schemes, and access patterns of modern AI/ML inference pipelines and scientific arrays.

Scope and Goals

• Define a binary tensor descriptor encoding that is language- and runtime-agnostic.
• Enable zero-copy buffer sharing across runtimes, processes, and devices.
• Support the full range of quantization schemes used in modern inference.
• Be streamable (streaming format): a reader MUST be able to start processing tensor data without buffering the entire input, and a writer MUST be able to emit tensor data incrementally without buffering the entire output. Tensor descriptors always precede their data buffers; the format is self-delimiting; back-references are not permitted. File format writers operate in a single forward pass and append a footer index at the end.
• Extensible and evolvable: extension points are stable across 1.x; new named values go through the spec amendment process; backward and forward-additive compatibility is guaranteed within major version 1.x. See Versioning § Evolvability Contract.
• Use the standard numeric dtype vocabulary (shared by NumPy and the Python Array API Standard) for Tier 1 element types, enabling zero-copy interoperability without dtype translation. This is an interop convenience, not an Array API conformance claim. See Python Bindings for binding-level requirements.
• Serve as the storage foundation of an array database engine: the file format, tiled/blocked layout, Morton and Hilbert curve layouts, and footer index are designed to be compatible with sub-array queries, tile-skipping, and range-based retrieval. A concrete target use case is an embeddable SQL/MDA query engine (ISO 9075-15) backed by Hurray buffers, with zero-copy handoff from query results to an ML inference pipeline. Spec decisions that would foreclose chunk-based access, spatial locality, dimension-range indexing, or SQL/MDA interoperability MUST be evaluated against this use case before being adopted.

RFC 2119 Notice

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in these documents are to be interpreted as described in RFC 2119.

Versioning

This specification follows semantic versioning. A reader MUST reject a tensor descriptor whose major version field exceeds the reader's supported major version.

Table of Contents