Interchange — Hurray Format Specification

Status: Draft

This section uses RFC 2119 key words: MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL.

Scope

This section defines how Hurray tensors are exchanged between producers and consumers. Three interchange modes are in scope:

This section focuses primarily on the network transport mode, as it is the most complex and the most relevant to distributed inference pipelines. In-process and IPC modes are covered in In-Process and IPC.

A Hurray stream MAY contain zero or more tensors. Back-to-back concatenation of self-delimiting descriptor+data pairs is the canonical multi-tensor encoding: a reader processes one tensor at a time, advancing to the next descriptor after the current tensor's data is consumed. No container header, index, or tensor names are defined at this level (see docs/adr/ADR-010-multi-tensor-collections-deferred.md).

Note (non-normative): The design of the Hurray network transport protocol draws inspiration from Apache Arrow Flight, but differs in several key respects: it is tensor-focused (not columnar), it supports layout and device negotiation, it defines on-the-fly transcoding, and its data plane is designed for large buffer transfers where gRPC framing overhead is impractical.

In-Process and IPC

In-Process

Within a single address space, tensor interchange is accomplished by passing a tensor descriptor (see metadata.md) and a buffer handle (see buffer-protocol.md) by value. No serialization is required. The buffer handle carries a release callback (see buffer-protocol.md § Buffer Ownership and Lifetime and ADR-009); the receiver MUST retain the handle for the duration of its use and call the release callback exactly once when done.

IPC

For cross-process interchange on a single machine, two mechanisms are supported:

1. Shared memory: the producer maps a shared memory segment and places the data buffer there. The tensor descriptor is transmitted over any IPC channel (pipe, Unix domain socket, etc.). The byte_offset field in the descriptor identifies the buffer's position within the shared segment.
2. Unix domain socket streaming: the producer serializes the tensor using the network transport framing defined below, transmitted over a Unix domain socket. This is slower than shared memory but requires no shared-memory setup.

In both cases, buffer alignment requirements from buffer-protocol.md apply.

Network Transport Protocol

Overview

The Hurray network transport protocol is a client-server streaming protocol for transferring tensor data over a reliable, ordered byte stream (e.g., TCP). It consists of:

• A control plane: session establishment, capability negotiation, tensor requests, and error signalling. Uses message framing defined in this section.
• A data plane: tensor descriptor and buffer transmission. Uses the same message framing, but data frame payloads MAY be transferred over a separate high-throughput channel (e.g., RDMA) by prior agreement during session establishment.

The protocol is half-duplex per stream: within a single stream, the client sends a request and the server responds with a sequence of messages. Multiple streams MAY be multiplexed over a single connection.

Note (non-normative): The stream_id field is an opaque per-stream identifier. Implementations MAY use one TCP connection per stream, or MAY multiplex multiple streams over a single connection by demultiplexing on stream_id. Messages are self-framing (fixed-size header + payload_length bytes), so a receiver can always advance to the next message regardless of stream_id. Normative multiplexing rules are not defined in this version of the spec.

Message Framing

Every message on the wire consists of a message header followed by a payload. All multi-byte fields are little-endian.

Message header (12 bytes):

The payload follows immediately after the header. A receiver MUST read exactly payload_length bytes as the payload. A receiver MUST reject messages whose payload_length exceeds the receiver's configured maximum message size, and MUST send an ERROR message in response.

Message Types

A receiver that encounters an unrecognised message_type MUST send an ERROR message and close the stream.

Message type values in the range 0x000000F0–0x000000FE are reserved for implementation-private extensions. Implementations MAY use these values for private extension message types; messages using private extension types MUST NOT be sent to a peer that has not agreed to the extension out of band. The value 0x000000FF is reserved and MUST NOT be used.

Session Establishment

CLIENT_HELLO Payload

SERVER_HELLO Payload

Note (non-normative): supported_memory_classes advertises which memory access classes a peer can produce or consume across all devices. A server that supports CUDA UNIFIED (cudaMallocManaged) and CPU STANDARD would list [0x00, 0x02]. A client that cannot handle UNIFIED buffers lists only [0x00]; the server MUST NOT send a UNIFIED buffer to such a client. Memory class negotiation follows the same fallback logic as device negotiation: the server SHOULD prefer the client's advertised classes and MUST report the actual memory_class in the TENSOR_DESCRIPTOR buffer handle.

A client MUST send a CLIENT_HELLO as the first message on every new connection. The server MUST respond with a SERVER_HELLO before any other message. If the server cannot satisfy the minimum requirements (e.g., protocol version mismatch), it MUST respond with an ERROR message instead and close the connection.

Capability Flags

Device Tags

The canonical definition of device tags lives in buffer-protocol.md § Device Tags. The table below is reproduced for transport-protocol convenience and MUST stay consistent with buffer-protocol.md.

Layout Entry Encoding

Layout lists appear in CLIENT_HELLO, SERVER_HELLO, and TENSOR_REQUEST. Each list is preceded by a uint16 count, followed by that many layout entries. A layout entry is variable-length and encoded as follows:

1. layout_tag (uint8): the layout tag as defined in memory-layout.md.
2. If layout_tag is in the extension range (0xF0–0xFE):
   • ext_metadata_length (uint16): byte length of the opaque metadata that follows. MAY be 0.
   • ext_metadata (byte sequence): opaque hardware- or implementation-specific metadata of ext_metadata_length bytes.
3. If layout_tag is not in the extension range, the entry consists of the single layout_tag byte only. No length or metadata fields follow.

A reader MUST skip any extension entry whose ext_metadata it does not understand, using ext_metadata_length to advance past it. A reader MUST NOT reject a layout list solely because it contains unrecognised extension entries.

Note (non-normative): The primary use case for extension layout entries in negotiation is hardware-specific panel/pack formats for BLAS kernels. A client advertising such a format would include an extension tag with opaque metadata encoding its hardware parameters (e.g. panel width, register block dimensions, SIMD width, cache line size). The server either recognises the profile and transcodes accordingly, or skips the entry and falls back to the next preference. The packed buffer travels to the client and is handed directly to the BLAS kernel — it is never forwarded or reinterpreted by generic tensor code.

Layout Negotiation

Request

When sending a TENSOR_REQUEST, the client specifies its layout preferences. The server MUST honor the negotiation rules below.

TENSOR_REQUEST Payload

Request flags:

Server Layout Selection

Upon receiving a TENSOR_REQUEST, the server MUST select a layout for the response according to the following rules, in order:

1. If the client supplied a non-empty preferred_layouts list, the server MUST iterate through the list in order and select the first layout tag that satisfies one of: a. The tensor is already stored in that layout (no transcoding needed), or b. ALLOW_TRANSCODE is set and the server has TRANSCODING capability for that layout tag.
2. If no preferred layout can be satisfied (list exhausted or empty), the server MUST serve the tensor in its native stored layout.

The server MUST indicate the chosen layout in the TENSOR_DESCRIPTOR message. The client MUST be prepared to receive any layout that appeared in the server's supported_layouts advertisement, even if it was not among the client's preferences.

On-the-Fly Transcoding

When the server transcodes a tensor to satisfy a layout preference, the transcoding MUST be element-preserving: the logical tensor (same rank, same shape, same element values at every index) MUST be identical before and after transcoding. Only the memory layout differs.

A server MUST NOT transcode if doing so would require materialising a buffer larger than the server's configured transcoding memory limit. In that case the server MUST fall through to the next preferred layout or the native layout.

Note (non-normative): Transcoding is inherently a memory and compute cost on the server side. Servers SHOULD document their transcoding capabilities and limits. Clients SHOULD list lightweight layouts (e.g. row-major 0x01) later in their preference list as a fallback, rather than requiring the server to transcode into a complex layout first.

Device Negotiation

The preferred_device field of TENSOR_REQUEST drives device selection for the response buffer. Device selection is distinct from layout negotiation: device selection has no ordered preference list on the wire (a single tag is supplied), and the rules below define the server's response when the preferred device cannot be served.

Server Device Selection

Upon receiving a TENSOR_REQUEST, the server MUST select a placement device for the response buffer according to the following rules, in order:

1. If preferred_device appears in the server's supported_devices list (from SERVER_HELLO) and the server has resources to satisfy it at request time, the server MUST place the response buffer on preferred_device.
2. If preferred_device == 0x00 (CPU) and the server cannot satisfy CPU, the server MUST send ERROR with error_code = DEVICE_UNAVAILABLE and close the stream. CPU is the universal fallback; if a server cannot satisfy CPU, no silent fallback is meaningful.
3. If the preferred device is a non-CPU device the server cannot serve, the server MUST send ERROR with error_code = DEVICE_UNAVAILABLE and close the stream. The server MUST NOT silently fall back to a different device.
4. Exception to rule 3: if preferred_device was advertised by the server in SERVER_HELLO but is transiently unavailable (e.g., out of device memory), the server MAY fall back to CPU (0x00) if and only if the client also advertised CPU in its CLIENT_HELLO supported_devices list. This is the only permitted silent fallback.
5. The server MUST report the actual placement device in the device_tag field of every buffer handle in TENSOR_DESCRIPTOR. Per buffer-protocol.md § Device Colocation, all buffers of a single tensor MUST share the same device_tag.
6. A client that receives a TENSOR_DESCRIPTOR whose buffer device_tag differs from its preferred_device MUST be prepared to either accept the placement (if the tag is in its supported_devices list) or close the stream with ERROR.

Note (non-normative): The "no silent fallback" design is motivated by the performance characteristics of inference workloads: a model running on CUDA that silently lands on CPU produces correct output but performance collapses catastrophically (often by orders of magnitude). The narrow CPU-fallback exception in rule 4 is provided only as a graceful degradation path for clients that explicitly opt in by advertising CPU in their supported_devices list.

Note (non-normative): Clients that want graceful CPU fallback should advertise CPU (0x00) in CLIENT_HELLO supported_devices. The DEVICE_UNAVAILABLE error gives the client enough information to retry with a different preferred_device if it has alternatives available.

Streaming: Tensor Descriptor and Data Frames

Ordering Invariant

For every tensor transferred, the server MUST send messages in the following order:

TENSOR_DESCRIPTOR
TENSOR_DATA  (zero or more frames)
TENSOR_DATA_END

A tensor whose total_data_bytes is greater than 0 MUST send one or more TENSOR_DATA frames. A tensor whose total_data_bytes is 0 — an empty tensor (ADR-007) or a composite head, which owns no data — MUST send zero TENSOR_DATA frames: the server sends TENSOR_DESCRIPTOR immediately followed by TENSOR_DATA_END. A receiver MUST accept this frame-free sequence when total_data_bytes = 0.

A receiver MUST NOT attempt to interpret data frames before receiving the TENSOR_DESCRIPTOR. This invariant holds for each shard in a parallel transfer.

TENSOR_DESCRIPTOR Payload

The payload is a serialized tensor descriptor as defined in metadata.md, followed by the following transport-specific fields:

The sync_mode field of each buffer handle inside TENSOR_DESCRIPTOR declares the producer's synchronisation guarantee for that buffer. Consumers MUST observe the sync_mode value and apply the matching rule per buffer-protocol.md § Stream and Event Synchronisation before accessing the buffer's bytes.

TENSOR_DATA Payload

Data frames for a single tensor MUST be sent in ascending byte_offset_in_buffer order with no gaps and no overlaps. The sum of all data frame lengths MUST equal the total_data_bytes declared in the TENSOR_DESCRIPTOR.

TENSOR_DATA_END Payload

The TENSOR_DATA_END message has an empty payload (payload_length = 0). It signals that all data frames for the current tensor (or shard) have been sent on this stream.

Composite Tensor Streaming

A composite tensor (head + members; see layouts/composite.md) is streamed as a forward-adjacency sequence with no back-reference. The head is an ordinary tensor descriptor with layout_tag = 0x0B and buffer_count = 0, so it is sent as a TENSOR_DESCRIPTOR message immediately followed by a TENSOR_DATA_END (an empty data plane — the head owns no buffers). The head's member_count = N binds the next N self-delimiting tensors on the stream, in order, as its members:

TENSOR_DESCRIPTOR        (head, layout_tag = 0x0B, member_count = N)
TENSOR_DATA_END          (head has no data buffers)
  TENSOR_DESCRIPTOR      (member 0)
  TENSOR_DATA  (zero or more frames)
  TENSOR_DATA_END
  ...
  TENSOR_DESCRIPTOR      (member N-1)
  TENSOR_DATA  (zero or more frames)
  TENSOR_DATA_END        (composite "close": the Nth member's TENSOR_DATA_END)

The head's TENSOR_DESCRIPTOR MUST precede its members' descriptors, which MUST precede their data — a forward promise, never a back-reference. The composite closes on the Nth member's TENSOR_DATA_END; at that point a receiver runs the close-time validation for the composition rule (layouts/composite.md § Validation).

A member that is itself a composite head (a nested composite) recursively binds its own members before the enclosing composite's member count advances; the sequence is parsed pre-order, subject to the depth limit in layouts/composite.md § Binding.

A receiver MUST reject a composite whose per-member checks fail (send ERROR and close the stream). A torn composite — the stream ends before all N members' TENSOR_DATA_END messages arrive — is incomplete: a strict receiver MUST reject it; a permissive receiver MAY expose the arrived members as independent shard tensors but MUST NOT present the composite as complete (layouts/composite.md § Validation).

Note (non-normative): The head carries no data, so its total_data_bytes in the TENSOR_DESCRIPTOR transport fields is 0 and no TENSOR_DATA frame is sent for it. Binding is purely positional (member count + stream order); no tensor names or member identifiers are introduced.

Note (non-normative): A composite head therefore sends zero TENSOR_DATA frames — TENSOR_DESCRIPTOR immediately followed by TENSOR_DATA_END — as does any tensor whose total_data_bytes is 0. This is governed by the general rule in § Ordering Invariant ("zero or more frames"); the head is not a special case.

Parallel Transfers

Overview

A tensor MAY be transferred as a set of shards delivered simultaneously over multiple independent streams (e.g., multiple TCP connections or RDMA queue pairs). Each shard is a rectangular sub-region of the logical tensor, described by the shard descriptor mechanism defined in memory-layout.md (fields parent_shape and shard_offset).

The client indicates willingness to receive parallel shards by setting the PARALLEL_OK flag in the TENSOR_REQUEST. The server indicates parallel transfer support via the PARALLEL_STREAMS capability flag in SERVER_HELLO.

Parallel Transfer Flow

1. The client sends a single TENSOR_REQUEST with PARALLEL_OK set.
2. The server selects a sharding strategy (number of shards, shard boundaries) and responds on N separate streams, one per shard. Each stream carries an independent TENSOR_DESCRIPTOR → TENSOR_DATA → TENSOR_DATA_END sequence.
3. Each TENSOR_DESCRIPTOR payload MUST include:
   • A tensor descriptor with a shard descriptor (parent_shape, shard_offset, and the shard's own shape) embedded as defined in metadata.md.
   • shard_index and total_shards fields in the transport header.
4. The client reassembles the logical tensor by placing each shard at the position indicated by its shard_offset within a buffer sized for parent_shape.

Shard Consistency

All shards of a single tensor MUST share the same:

• parent_shape
• element type
• layout tag (outer layout; the shard's inner layout MAY vary in future extensions)

The union of all shard bounding boxes (defined by shard_offset and shape) MUST exactly cover the full parent_shape without overlap, satisfying the coverage and non-overlap constraints from memory-layout.md.

A client MUST validate shard consistency upon receiving all TENSOR_DESCRIPTOR messages. A client MUST reject a parallel transfer where any shard descriptor violates these constraints.

Note (non-normative): Sharding along the batch dimension (dimension 0) is the simplest and most common case — each shard is a contiguous slice of rows. The protocol does not restrict sharding to any particular dimension or sharding scheme.

RDMA Data Plane

Overview

When both client and server advertise the RDMA_DATA_PLANE capability flag during session establishment, the data plane for individual tensor transfers MAY use RDMA rather than TCP TENSOR_DATA frames. The control plane (TCP) is still used for all session management, descriptor exchange, and completion signalling.

Note (non-normative): The RDMA data plane bypasses TCP framing for the tensor buffer itself. For GB-scale tensors this eliminates CPU copies and TCP serialisation overhead, achieving near-line-rate GPU-to-GPU transfer via GPUDirect RDMA. The underlying RDMA operations are performed by an RDMA library such as UCX (ucp_put_nb / ucp_get_nb) or libibverbs (ibv_post_send). The Hurray protocol specifies the handshake messages; it does not mandate a specific RDMA library.

Handshake Flow (Server → Client Tensor Transfer)

When both parties have advertised RDMA_DATA_PLANE, the server MAY substitute the TENSOR_DATA / TENSOR_DATA_END sequence with an RDMA handshake. The client MUST be prepared to handle either path.

Client                          Server
  |                               |
  |--- TENSOR_REQUEST ----------->|
  |<-- TENSOR_DESCRIPTOR ---------|
  |<-- RDMA_REGISTER -------------|  (server registers source buffer, shares rkey + addr)
  |--- RDMA_REGISTER ------------>|  (client registers destination buffer; only if RDMA_GPUDIRECT)
  |--- RDMA_READY --------------->|  (client is ready; RDMA operation may begin)
  |                               |
  |   [RDMA Write executes outside the TCP control plane]
  |                               |
  |<-- TENSOR_DATA_END -----------|  (server signals that buffer is ready on client side)

The client's RDMA_REGISTER step is OPTIONAL and only occurs when both peers advertised the RDMA_GPUDIRECT capability flag in their respective HELLO messages. See GPUDirect Destination Registration.

The server MUST send TENSOR_DESCRIPTOR before RDMA_REGISTER. The RDMA_REGISTER message MUST be sent on the same stream as the corresponding TENSOR_DESCRIPTOR.

RDMA_REGISTER Payload

The RDMA_REGISTER message is sent in one or two roles per transfer:

• The source-buffer owner sends RDMA_REGISTER to declare its source memory region. This is the server for TENSOR_REQUEST transfers and the client for TENSOR_PUT transfers.
• When both peers advertised the RDMA_GPUDIRECT capability flag, the destination-buffer owner MAY also send RDMA_REGISTER to declare a pre-pinned destination memory region. This is the client for TENSOR_REQUEST transfers and the server for TENSOR_PUT transfers. See GPUDirect Destination Registration.

The payload table is identical in both roles; the role is determined by the sender and the position of the message in the handshake sequence.

A receiver that cannot complete RDMA setup (e.g., memory pinning failed, no RDMA hardware available on the required path) MUST respond with an ERROR message instead of RDMA_READY. The source-buffer owner MUST then fall back to transmitting the tensor via TENSOR_DATA frames on the control plane.

GPUDirect Destination Registration

When both peers advertised the RDMA_GPUDIRECT capability flag in their respective HELLO messages, the destination-buffer owner (the receiver) MAY pre-register a destination memory region and share its rkey with the sender via a second RDMA_REGISTER message. This enables the sender to write directly into the receiver's device memory (e.g., GPU memory), eliminating an intermediate host-to-device copy.

1. The receiver MUST send its RDMA_REGISTER (destination) after the sender's RDMA_REGISTER (source) and before RDMA_READY, if it intends to use GPUDirect.
2. The length field of the destination RDMA_REGISTER MUST equal total_data_bytes from the preceding TENSOR_DESCRIPTOR.
3. The receiver MUST register a buffer whose device type is consistent with the device_tag reported in TENSOR_DESCRIPTOR. If the server fell back to a different device per Device Negotiation, the receiver MUST NOT send a destination RDMA_REGISTER for a buffer on the originally requested device. The receiver MUST either accept the fallback and proceed without GPUDirect, or close the stream with ERROR.
4. If the receiver does NOT send RDMA_REGISTER, the sender MUST use its own buffer management strategy for the RDMA destination — typically landing data in the receiver's host memory via the RDMA library's managed staging area.
5. A sender that receives a destination RDMA_REGISTER but cannot perform a GPUDirect write (e.g., topology mismatch) MUST send ERROR instead of RDMA_READY.

Note (non-normative): GPUDirect requires the NIC and GPU to share a PCIe root complex (or to be NVLink-connected). The protocol cannot validate this topology; the receiver is responsible for ensuring it before advertising RDMA_GPUDIRECT and before sending a destination RDMA_REGISTER.

RDMA_READY Payload

The RDMA_READY message has an empty payload (payload_length = 0). It signals that the receiver has processed RDMA_REGISTER and is ready for the RDMA transfer to begin.

Completion

After the RDMA operation completes on the sender's side, the sender MUST send TENSOR_DATA_END over the control plane. The receiver MUST NOT read from the transferred buffer before receiving TENSOR_DATA_END.

For cross-machine transport, the sender MUST set sync_mode = SYNC_PRODUCER_SYNCED (0x00) in every buffer handle within the transmitted TENSOR_DESCRIPTOR. TENSOR_DATA_END is the cross-machine equivalent of SYNC_PRODUCER_SYNCED per buffer-protocol.md § Stream and Event Synchronisation: the receiver MUST NOT read from the transferred buffer before receiving TENSOR_DATA_END, and upon receipt of TENSOR_DATA_END the buffer is considered producer-synced. The sync_mode values SYNC_EVENT (0x01) and SYNC_CONSUMER_STREAM (0x02) are FORBIDDEN on cross-machine transports because device event and stream handles are not valid in a different driver context on a different host. A receiver MUST reject a cross-machine TENSOR_DESCRIPTOR whose buffer handle declares any other mode.

Note (non-normative): RDMA Write completion on the sender does not imply that the receiver has observed the data without an explicit memory fence or signal. TENSOR_DATA_END serves as that authoritative "buffer is ready" signal. The sender MUST ensure the RDMA operation has completed (e.g., via a completion queue event) before sending TENSOR_DATA_END.

TENSOR_PUT with RDMA

For TENSOR_PUT (client → server), roles are reversed relative to the server-to-client flow: the client is the source-buffer owner and the server is the receiver.

Client                          Server
  |                               |
  |--- TENSOR_PUT --------------->|  (tensor key + descriptor)
  |--- RDMA_REGISTER ------------>|  (client registers source buffer, shares rkey + addr)
  |<-- RDMA_REGISTER -------------|  (server registers destination buffer; only if RDMA_GPUDIRECT)
  |<-- RDMA_READY ----------------|  (server is ready; RDMA operation may begin)
  |                               |
  |   [RDMA Write executes outside the TCP control plane]
  |                               |
  |--- TENSOR_DATA_END ---------->|  (client signals data placed)
  |<-- TENSOR_PUT_ACK ------------|  (server confirmed receipt)

The server's RDMA_REGISTER (destination) step is OPTIONAL and only occurs when both peers advertised the RDMA_GPUDIRECT capability flag in their respective HELLO messages.

In TENSOR_PUT, the client unilaterally declares the destination device_tag in the descriptor; the server has no negotiation opportunity (there is no analogue of preferred_device for PUT transfers). If the server cannot accept the tensor on the declared device, it MUST send ERROR with error_code = DEVICE_UNAVAILABLE before any RDMA exchange and close the stream.

The rules in GPUDirect Destination Registration apply symmetrically with roles inverted: the server is the destination-buffer owner, and its destination RDMA_REGISTER declares a region whose device type MUST match the device_tag declared by the client in the descriptor.

If the client (source-buffer owner) cannot perform a GPUDirect write after receiving the server's destination RDMA_REGISTER (e.g., topology mismatch), the client MUST send ERROR and close the stream before TENSOR_DATA_END.

TENSOR_PUT

Overview

TENSOR_PUT allows a client to push a tensor to the server. The protocol defines the wire exchange only — the server-side storage model (lifetime, eviction, collision handling) is an implementation concern and is intentionally out of scope.

TENSOR_PUT Flow

Client                          Server
  |--- TENSOR_PUT --------------->|  (tensor key + descriptor)
  |--- TENSOR_DATA (one or more)->|  (data frames)
  |--- TENSOR_DATA_END ---------->|
  |<-- TENSOR_PUT_ACK ------------|  (server confirmed receipt)

With the RDMA data plane, TENSOR_DATA frames are replaced by the RDMA handshake described in TENSOR_PUT with RDMA.

TENSOR_PUT Payload

TENSOR_PUT_ACK Payload

The TENSOR_PUT_ACK message has an empty payload (payload_length = 0). It signals that the server has received and accepted the complete tensor buffer. It does not imply anything about how the server stores, forwards, or uses the tensor.

If the server cannot accept the tensor for any reason (e.g., policy rejection, resource exhaustion), it MUST send an ERROR message instead of TENSOR_PUT_ACK and close the stream.

Note (non-normative): Server-side storage semantics — including tensor lifetime, eviction policy, and key collision handling — are deliberately unspecified. A server implementation is free to store the tensor for the session, forward it immediately to another peer, or discard it after use. The protocol's role is delivery confirmation, not storage coordination.

Error Handling

ERROR Payload

Upon sending or receiving an ERROR message, both parties MUST close the affected stream. The connection MAY remain open for other streams.

Open Questions Summary

[OQ-1]: Endianness negotiation: should the transport protocol allow a client to request big-endian wire encoding? Resolved: No endianness negotiation. The wire format is always little-endian; big-endian clients MUST byte-swap on receipt. Rationale: all AI/ML inference hardware targeted by Hurray is little-endian; negotiation would add protocol complexity with zero practical benefit. Consistent with Arrow, DLPack, and SafeTensors.

[OQ-2]: RDMA data plane handshake. Resolved: RDMA_REGISTER (0x0000000C) and RDMA_READY (0x0000000D) message types are now defined. The party owning the source buffer registers its memory region and sends RDMA_REGISTER (rkey + remote address + length) over the control plane; the peer responds with RDMA_READY; the RDMA operation executes out-of-band; TENSOR_DATA_END is sent over the control plane as the authoritative completion signal. See RDMA Data Plane.

[OQ-3]: Multiplexing scheme. Resolved: The stream_id field is defined as an opaque per-stream identifier. Implementations MAY multiplex multiple streams over a single TCP connection using stream_id for demultiplexing, but the protocol does not mandate a normative multiplexing scheme. Each stream MAY equivalently run on its own connection. Normative multiplexing rules are deferred to a future revision once the format is stable.

[OQ-4]: TENSOR_PUT semantics. Resolved: Server-side storage model is explicitly out of scope. TENSOR_PUT_ACK means "received and accepted"; it carries no implication about persistence, lifetime, or collision handling. Those are implementation concerns. The server sends ERROR to reject a PUT for any reason. See TENSOR_PUT.

[OQ-5]: Device negotiation and GPUDirect RDMA destination registration. Resolved: A normative Device Negotiation section defines server device selection rules. GPUDirect destination registration uses bidirectional RDMA_REGISTER gated by the new RDMA_GPUDIRECT capability flag (bit 3). See docs/adr/ADR-011-server-device-selection.md (server device selection) and docs/adr/ADR-012-gpudirect-rdma.md (GPUDirect RDMA).

Interaction with Other Sections

• Memory Layout (memory-layout.md): defines the layout tag space used in capability advertisement and layout negotiation. Shard descriptors (parent_shape, shard_offset) are the logical basis for parallel transfers.
• Metadata (metadata.md): defines the binary encoding of the tensor descriptor transmitted in TENSOR_DESCRIPTOR messages.
• Buffer Protocol (buffer-protocol.md): defines buffer alignment requirements that the client expresses via min_alignment in TENSOR_REQUEST, and device memory semantics relevant to the preferred_device field.
• Element Types (element-types.md): defines the little-endian wire encoding of tensor element data, which is the data transmitted in TENSOR_DATA frames.