SPEAD: Streaming Protocol for Exchanging Astronomical Data

references

1 Scope

This document describes a datastream format suitable for use by radio astronomy instruments. The data stream is distinct from the KATCP control protocol which follows a format described in [.

2 Protocol Layers

The data is transmitted via a network interface, or stored on disk.

Although this format describes only application and presentation layers of the OSI model and data transmission can take place over any medium (serial data lines, ethernet networks or other), the preffered interface for network communications consists of the following layers, from top to bottom:

Application and Presentation: The data format as described in later sections of this document.
Session and Transport: UDP, noting that the receiving party can not request retransmission of lost, duplicated or corrupt packets.
Network: IP (Internet protocol)
Link: 10 Gbps, 1 Gbps or 100 Mbps Ethernet
Physical: SFP+, CX4 or other 10 Gigabit ethernet cable, Cat6, Cat5e etc.

3 Overview

Many instruments output data over ethernet. These streams are often high speed, unsuitable for TCP/IP links. UDP is often prefered in the cases where partial data-loss can be accomodated. This document aims to standardise the output format of such UDP packets.

An essential feature of the datastream is that it is self describing. This self description extends to both the content and interpretation of the option fields in the packet header, and of the data payload itself.

The same receive and transmit code may be used for multiple instruments as the layout of the packets are well defined.

It is possible to record the raw datastream directly to disk for later parsing.

It is flexible to allow metadata and descriptors to be injected into the datastream at runtime and recorded or processed along with the primary datastream.

The format of the packetised stream is described in this document.

Lower speed and ad-hoc data products may be transferred using the KATCP control interface [.

Each user datagram contains an application header, zero or more optional fields (items) and a packet payload.

Items are used to store user-variables. Data products (items) can be single values, multi-dimensional data or can point to other items (or multiples of other items) thereby allowing hierarchical data structures. These items can either be declared directly (if they fit into the allotted number of bytes), or can point to an address within an assembled heap of data where the data-products are stored, allowing unlimited lengths for a given item. The heap is constructed by assembling all packet payloads prior to data unpacking. All items are then unpacked and presented to the user’s application.

The address space you to allocate to the number of items and also the length of your pointers in your heap space defines the flavour of SPEAD employed. It is expressed as /textitSPEAD-XX-YY where YY is the number of bytes allocated for heap address space, and XX is the sum of those bytes and the address space for the items themselves. SPEAD-64-40 is currently employed, allowing for data heaps of up to 1TB and up to 8388608 variables. The heap size limit does not limit the total data storage capacity, mearly the total variables that are able to change simultaneously.

The SPEAD header is always 8 octets (64 bits) long. Items can have a user-defined length. This defines the flavour of the SPEAD protocol and should remain static for a given implementation.

4 Packet Data Representation

Data is sent Most Significant Byte first. The most significant bit is first on the wire (network byte order). This same ordering is used for on-disk storage, allowing data to be streamed directly between the network and storage.

All data stream packets have a header of the following form:

 Number of bits Description (MSB) 8 Magic Number (constant 0x53, ascii S) 8 Version identifier (this version is 4) 8 Item width (number of bytes per item) 8 Heap address width (number of bytes in the heap address space) 16 Reserved (LSB) 16 Number of items
Magic number: This is a constant value. All packets will start with the following 8 bit pattern: (msb) 0101 0011 (lsb). This allows basic identification of the packet as a SPEAD packet.
Version identifier: This field identifies the version of the packet format. Designed in case future packet structures differ. The version described here is 004.
Item width: SPEAD allows for various lengths of the user-data item fields, depending on your particular requirements. This field describes the number of bytes of each item (described in below section). Initial implementations have 64 bit fields, of which 24 bits are the item ID and 40 bits are the data or address space. Item width is thus set to ${\displaystyle 64/8=8bytes}$. This is configuration defines the version of SPEAD and is referenced as SPEAD-64-40.
Heap address widthThis field describes the length of the portion of the item width field used for the actual data word or heap address. Current implemenations of SPEAD-64-40 have this field set to 6. The remaining 24 bits are used as an item ID field.
Reserved: This field should be ignored. The space is reserved for future allocation.
Number of items: The value stored in this field indicates the number of optional 64-bit fields following the header. Any attached data follows directly after this number of 64-bit fields.

4.2 Items: optional fields

Each packet can contain zero or more data fields, called items. Each item consists of some bits of identifier (MSb) and some bits of data (LSb).

The number of items in a packet is defined by the Number of items field in the SPEAD header. The length of each item field is defined in the SPEAD header option item width in bytes and the number of bytes of the item field containing the actual data is stored in the header field heap address width. The difference between these values accounts for the item identifier field. The width of the field and the portion of that field used for data and identifier must remain fixed for a given implementation.

Items can store the user data value directly, or can point to an offset address (in bytes) where the start of the data is to be found. Thus, any length of data can be accomodated. In standard SPEAD-64-40, items are 64 bits long, of which 24 bits used to identify the item and the remaining 40 bits are used . This allows for up to

If a packet containing an unknown item is received, it might be recorded, ignored or otherwise the receiver might request a metadata reissue to explain the content (see field 0x30). In any case, the reception of unknown item should not be considered illegal and should be handled cleanly by a receiver.

Generally the items will be self defining and thus do not need strict identification in this document. However, it is useful to define a number of standard options that are likely to be used across a wide range of deployments.

The following table defines these standardised options identifier codes and their meaning.

 Dec Hex Description 0 0000 NULL - Ignore 1 0001 Instrument type 4 0004 Data payload length 5 0005 Data payload offset 13 000D Stream control 14 000E Metadata packet counter 48 0030 Option descriptor 49 0031 Payload descriptor

In addition to these standardised options, a number of reserved options are defined. These are options that are currently in use in systems, including those using previous versions of this protocol. Overloading of these should be avoided where practical.

User defined options are allowed, but these should have the MSb of the option ID set (i.e. option ID > 32767)

4.3 Standard Options

Instrument Type

The instrument type field defines the source of the data. It also includes the payload id identifying the decoding to be used to interpret the data.

The 48-bit field is subdivided into 16 bits representing instrument type and 32 bits representing the payload id (currently only the 16lsb are used). The instrument type is used to provide scope to particular option and payload id’s. Generally the instrument type will be fairly broad (e.g. KAT-7 which includes streamed data from all KAT-7 facility instruments).

The following are defined existing instrument types (16 MSbs):

 Hex identifier Instrument 0000 NULL instrument 0001 Pocket correlator (KAT Fringe Finder) 0002 Single-dish spectrometer

Note that header and options are specified to be transmitted MSb first, regardless of instrument type.

This is the size in bytes of the packet payload length (excluding the header and optional fields).

The position, relative to the integration, at which the payload of the current packet starts.

This is used for re-assembling data which was split across multiple packets. It is reset back to zero for the following integration (use in conjunction with the timestamp to determine if data loss occurred).

Stream control and metadata packet counter

Indicates the status of the stream. This control packet is contained in a stand-alone packet not containing any actual data products. If present, the data field of the packet (ie after all the 64bit options) might contain text descriptors related to option are payload ids’ in the stream. See fields 48 (0x30) and 49 (0x31) for further details.

The most significant bit of the stream control field indicates that this is the last metadata packet. Used in conjunction with the metadata packet counter, this allows you to check if all the metadata was successfully received.

The lower 47 bits are a number representing:

0: Stream start
2: Stream stop
3: Metadata update (some values have changed, probably in response to a command)
4: Metadata description packet (see Stream descriptors)
>=5: Reserved for future use. Ignore.

If only one packet is sent, the Metadata packet counter will be zero and the MSb in Stream control will be set.

4.4 Stream descriptors

Stream descriptors are used to provide receiving clients with the metadata required to decode and interpret options and data sent as part of a particular SPEAD stream. These descriptors are sent in a special metadata-only packet (stream_control==4), and must never appear in a packet where ordinary instrument data is present, lest the description strings be confused with actual data. In general these packets will be sent at the start of a session before data and general metadata packets.

Provision is made for both option (describe the meaning and interpretation of header options), and payload (the structure of the streamed data) descriptors. Although they share a similar syntax, it is important to understand the distinction between the two types.

This option describes the starting position of a string in the data field where the description for a given payload id may be found.

 Number of bits Representation (MSB) 16 Payload ID that we are describing. 16 Index in bytes of string start position in data field. Indexed from zero. (LSB) 16 Length of string in bytes including a null-terminator.

The data field of the descriptor packet then contains strings of the form:

The unpack string tells the parser how to unpack the payload, and follows a C style printf convention, with the addition of referenced data types. The unpack string can contain one or more 24-bit unpack directives, with each directive referring to a consecutive data type in the payload. Each 24-bit directive has the following form: unpack_type[bit_length | payload_id]

 Unpack character Representation 0 interpret remaining 16 bits as a Payload ID i signed integer u unsigned integer f IEEE float c ASCII character b boolean

This allows the unpack string to reference another payload type, thus allowing hierarchical constructs. For non payload ID types, the remaining 16 bits are the bit length of a single field of the specified type.

For example an unpack string could contain: u[32]u[32]f[64]0[002]. This would translate into payload segments that contain two 32-bit unsigned integers, followed by a double, followed by a structure that is defined in Payload ID 002.

The count string allows multiple data constructs of the type specified in the unpack string to be placed contiguously in the data stream. It has the following form (56 bits): count_source[count | option_id]

 Unpack character Representation 0 the remaining 48 bits provide the count directly. (cast to an unsigned integer) 1 the count is read as the 48-bit value associated with the specified Option ID.

Payload names are intended to be parsed and presented in the receiver namespace and thus should not include whitespace. They may not contain [LF] either, as this is used for argument demarkation when parsing.

As an example, consider a descriptor packet containing the 16bit field id 0x31 with 48 bit content 0x0005 0000 0035.

 Unpack size Unpacked value 16 msb 048 decimal. 16b 005 decimal. 16b 000. 16b lsb 53 decimal.

This would describe a Payload ID of 0005 starting at offset 0 and having a length of 53 bytes. And the start of the packet’s datafield could contain the string (with additional payload id’s included that have not been shown in the above header.)

0[2][LF]1[9][LF]frequency[LF]A single frequency bin.[NULL]0[3][LF]1[11][LF]baseline[LF]A single baseline. Baseline ordering starts with the cross correlation between A0 and A1. Thereafter all cross correlations in natural order. Auto correlations follow.[NULL]0[4][LF]0[4][LF]polarisation[LF]The polarisation products. Order is XX, YY, XY, YX[NULL]u[32][LF]0[2][LF]complex_data[LF]The complex data point. Two uint32, real followed by imaginary[NULL].

This above would describe the current v0002 correlator output format. Essentially it shows a hierarchy the begins with two uint32’s the form a single complex sample. Each 4 complex samples describe a set of polarisation products. The polarisation products are then grouped into a number of baselines (that is specified in option 0x0011. The baselines are finally grouped into frequency channels, the number of which is specified in option 0x0009. This fully describes the data payload for a single correlator dump.

Option descriptor

In exactly the same fashion as the payload descriptor, the option descriptor describes the starting position of a string in the packet data where the description for a given option field may be found. The syntax is the same, but only a subset of the descriptive features are used since the format of the option field is much more tightly defined.

 Number of bits Representation (MSB) 16 Option ID that we are describing. 16 Index in bytes of string start position in data field. Indexed from zero. (LSB) 16 Length of string in bytes including a null-terminator.

The data field of the metadata packet then contains strings of the form:

unpack_string>[LF]0[1][LF]Option_name[LF]Description

The unpack string is interpreted as before, with the exclusion of the indirect referencing of other descriptors. Further, the option field has a fixed 48-bit length. This implies, for instance, that a float can only be a maximum of 32-bits long. The count string is not used and is set to 0[1] to indicate a single instance of the unpack string is to be used. An unpack string might look like: u[4]b[1]f[32]c[8]b[1]b[1]b[1].

As an example, consider a metadata packet containing the 16bit field id 0x30 with 48 bit content of 12884902140, this would decode as:

 Unpack size Unpacked value 16 msb 048 decimal. 16b 003 decimal. 16b 000. 16b lsb 252 decimal.

And the start of the packet’s datafield could contain the string u[48][LF]0[1][LF]timestamp[LF]This field timestamps all data for any DBE instrument. Time zero corresponds to the time at which the system was last sync’d to a 1PPS signal. Each increment of this field corresponds to a change of timestamp_scale_factor x n_ADC_samples.[NULL]

In the same packet, we might have a second metadata field with ID 48 and 48bit content of 128865534254. This would decode as:

 Unpack size Unpacked value 16 msb 048 decimal. 16b 030 decimal. 16b 252 decimal. 16b 302 decimal.

And at offset 252 in the datafield, we might find: u[16]b[1]u[31][LF]0[1][LF]FFT Shift pattern[LF]Indicates if a divide-by-2 was done between butterfly stages in the FFT. LSB is last FFT stage, with every higher bit representing the previous FFT stage. Only the <FFT Length> LSbs are valid. Struct is input number (16b msb), polarisation (1bit), FFT shift pattern (31 bits). [NULL]

A single metadata description should never straddle a packet boundary. Each metadata packet should be self-contained. This effectively limits the length of a single description string. Jumbo frames typically allow for over 8KiB of data, so these strings can still be reasonably long.

A Example Packet Exchange

This section provides an example packet exchange that could occur between a simple correlator and a SPEAD receiver instance.

A.1 Packet Sequence

At the start of the data transmission, initial metadata packets are sent. These should setup any stream descriptors required to describe options and payloads. They will also include non-payload specific option fields that do not require transmission synchronously with the data packets.

During transmission of the data stream, packets should contain only a limited number of optional fields, in order to maximise the utilisation of the link. Certain options are sent if parameters change or errors are encountered. Full metadata packets can be regenerated upon request, allowing a receiver to process a stream which had been started earlier.

When a data stream terminates, a futher metadata packet is sent, this may not contain a full description of the data stream, but does include stream control options to indicate that the transmission has completed.

The metadata packets may not arrive from the same source as the data packets. They may not even arrive from the same subnet. The receiver should not care about the source of the packet, as the contents of a stream are sufficient to fully identify the sender (barring any rogue packets on the network).

It is recommended that unknown options be recorded, in order to facilitate debugging and forward compatibility, even if these options are not understood.

A.2 Descriptor packets

Revision 3 of the protocol (as described here) allows for initial descriptor packets to describe the nature of the variables in the stream and also detail how to unpack the data for the receiving parser. Revisions 1 and 2 did not include this facility, though the protocol was designed to be forwards and backwards compatible. Earlier receivers might have these decode values hard-coded.

A transmitter conforming to Revision 3 could send a descriptor packet similar to the following:

 Packet contents Representation Significant meaning 0x4B52 0003 0000 000B Header 11 options will follow. 0x000D 0000 0000 0004 Stream control Metadata descr, not the last. 0x000E 0000 0000 0000 Metadata pkt cnt First metadata packet in this stream. 0x0001 0001 C020 000C Instrument type PoCo, 12 complex 32b unsgnd ints, msb first per data item. 0x0003 0000 0000 0000 Instance ID Stream 0 from engine 0. 0x0030 001E 00FC 012E Option descr Option 0x1E descr. at payload offset 252 is 302 bytes long. 0x0030 0003 0000 00FC Option descr Option 0x03 descr. at payload start is 252 bytes long. … … … 0x0031 0001 0352 011D Payload descr Paylod 0x01 descr. at payload offset 850 is 285 bytes long.

Directly following those options, we’d have the following ASCII string, where values in brackets correspond to raw binary representations:

u[48][LF]0[1][LF]Timestamp[LF]This field timestamps all data for any DBE instrument. Time zero corresponds to the time at which the system was last sync’d to a 1PPS signal. Each increment of this field corresponds to a change of timestamp_scale_factor x n_ADC_samples.[NULL]u[16]b[1]u[31][LF]0[1][LF]FFT Shift pattern[LF]Indicates if a divide-by-2 was done between butterfly stages in the FFT. LSB is last FFT stage, with every higher bit representing the previous FFT stage. Only the <FFT Length> LSbs are valid. Struct is input number (16b msb), polarisation (1bit), FFT shift pattern (31 bits). [NULL]....0[2][LF]1[9][LF]frequency[LF]A single frequency bin.[NULL]

Thus the receiver would know how to interpret option ID 0x1E and 0x03 as well as payload ID 0x01.

These descriptions are intended to supplement this document. Although every effort is made to describe all required fields, it is understood that future systems may need to supplement these fields. These descriptors provide a mechanism for achieving this. It will still be necessary for a receiver to understand certain standard fields a-priori (eg Stream control, Metadata packet counter, Instrument type, Instance ID, Metadata descriptor etc), in order to decode adescriptor packet.

It is recommended that all option fields used in a given stream be described this way and that a full list of descriptors be pre-pended to the datastream.

Once the descriptors have been transmitted the receiver will be in a position to decode additional option fields. These are then sent in a number of metadata packets before the start of payload transmission. For example,the poco and KAT-7 corr instruments (and any other instrument implementing per-channel gain control) will need to issue many metadata packets describing the digital gain setting on each frequency channel for each ADC.

 Number of bytes Option ID Representation 8 Header 8 0x000D Stream control 8 0x000E Metadata packet counter 8 0x0001 Instrument type 8 0x0003 Instance ID 8 0x000F Start time, whole seconds 8 0x0011 Center frequency (Hz) 8 0x0013 Bandwidth (Hz) 8 0x0007 ADC Sample Rate (Hz) 8 0x002F Timestamp Scale Factor 8 0x0009 Number of channels 8 0x000A Number of antennas 8 0x000B Number of baselines 8 0x0015 Number of accumulations per integration 8 0x0016 Quantisation scalar X-pol, real 8 0x0016 Quantisation scalar X-pol, real 8 0x0016 Quantisation scalar X-pol, real … … … 8 0x0016 Quantisation scalar X-pol, real

Many such metadata packets will be sent to describe all the quantisation scalars for all ADCs, all polarisations, all channels. Each subsequent packet will increment the metadata packet counter field’s value by one, which allows the receiver to check that no metadata packets were missed. The length of each packet can be calculated based on the number of options in the preamble. The last packet will have the MSb in the Stream Control field set, indicating the end of the metadata bundle.

If only one packet is sent, the metadata packet counter will be zero and the MSb in Stream Control will be set.

During the normal course of operation, additional metadata packets may be received with updated values. In this case, the Stream Control field will have a value of 3 and the updated values should be recorded.

A.4 Data packets

Data packets will generally have only a small number of option fields that are required to unambiguously describe the payload of the particular packet in question.

Both the poco mode and packetised corr (correlator) instruments will emit packets with a data payload typically containing the following options, noting that the ordering of options may vary.

 Number of bytes Option ID Representation (MSB) 8 Header 8 0x0001 Instrument type 8 0x0002 Instance ID 8 0x0003 Timestamp 8 0x0004 Data payload length 8 0x0000 NULL 8 0x0005 Data payload offset (in bytes) (LSB) 4096 Data The receiver will use the payload ID that forms part of the instrument type to correctly decode the data payload. Note that a payload descriptor refers to a complete, unfragmented data block. Blocks of data that are too large for a single packet will get fragmented. The receiver can re-assemble these fragments using the payload length and payload offset information. Once unfragmented that block can be parsed using the supplied descriptor. The length of the payload may vary, but is likely to be 2kiB, 4kiB or 8kiB. The last packet of a data block may be shorter if there is not enough data to fill the packet. ie. ${\displaystyle (n\_bls*n\_chans*n\_stokes*2/n_{xeng})\%(pkt\_len)!=0\,\!}$ The fifth field is kept in reserve for future use. Once data transmission is complete a Stream Stop packet is issued.