Difference between revisions of "Spead specification"

From AstroBaki
Jump to navigationJump to search
Line 153: Line 153:
 
All packets have a header of the following form:
 
All packets have a header of the following form:
  
\begin{tabular}{|r|l|}
+
\begin{tabular}{|c|c|c|c|c|c|}
 +
\emph{MSb} &&&&& \emph{LSb} \\
 
\hline
 
\hline
\emph{Number of bits} & \emph{Description} \\
+
Magic \# (8b) & Version (8b) & Item Width (8b) & Heap Addr Width (8b) & Reserved (16b) & \# Items (16b)\\
\hline
 
(MSB) 8 & Magic Number (constant 0x53, ascii \emph{S}) \\
 
      8 & Version identifier (this is version 4) \\
 
      8 & Item width (number of bytes per item) \\
 
      8 & Heap address width (number of bytes in the heap address space) \\
 
      16 & Reserved \\
 
(LSB) 16 & Number of items \\
 
 
\hline
 
\hline
 
\end{tabular}
 
\end{tabular}

Revision as of 09:13, 1 April 2010

SPEAD: Streaming Protocol for Exchanging Astronomical Data

references

1 Scope

This document describes a datastream format suitable for use by radio astronomy instruments. The data stream is distinct from the KATCP control protocol which follows a format described in [.

Many instruments output data over ethernet. These streams are often high speed and unsuitable for TCP/IP links. UDP is often prefered in the cases where partial data-loss can be accomodated. This document aims to standardise the output format of such UDP packets.

An essential feature of the datastream is that it is self describing. This self description extends to both the content and interpretation of the option fields in the packet header, and of the data payload itself. The same receive and transmit code may be used for multiple instruments as the layout of the packets are well defined.

Emphasis is placed on lightweight implementation, suitable for data packetisation by hardware devices or by software processes, while retaining flexibility for complex, dynamically changing data exchanges.

Metadata and variable descriptors can be injected into the datastream at runtime and recorded or processed along with the primary datastream. It is also possible to record the raw datastream directly to disk for later parsing.

The format of the application-layer packetised stream is described in this document. Lower speed and ad-hoc data products may be transferred using the TCP-based KATCP control interface [.

2 Protocol Layers

The data is transmitted via a network interface, or stored on disk.

Although this format describes only application and presentation layers of the OSI model and data transmission can take place over any medium (serial data lines, ethernet networks or other), the preffered interface for network communications consists of the following layers, from top to bottom:

Application and Presentation: The data format as described in later sections of this document.
Session and Transport: UDP, noting that the receiving party can not request retransmission of lost, duplicated or corrupt packets.
Network: IP (Internet protocol)
Link: 10 Gbps, 1 Gbps or 100 Mbps Ethernet
Physical: SFP+, CX4 or other 10 Gigabit ethernet cable, Cat6, Cat5e etc.

3 Overview

Each user datagram contains a SPEAD header, three or more optional fields (item pointers) and a packet payload.

Items are used to store user-variables. These items can contain single valued data products, multi-dimensional data or can point to other items (or multiples of other items) thereby allowing hierarchical data structures.

A heap is a collection of data packets, assembled prior to data unpacking. Item pointers point to an address within an assembled heap of data where the item’s data-products are stored, allowing virtually unlimited data lengths for a given item. All items can be automatically unpacked and interpreted before presentation to the user’s application. Item pointers can also contain single values themselves (immediate addressing) instead of a pointer to a location containing the data (absolute addressing). This is more efficient for items which contain short integer values.

Different flavours of SPEAD are available depending on the number of addressable variables and total variable-space size. The SPEAD flavour is expressed as SPEAD-XX-YY where XX is the number of bits allocated for heap address space (total variable-space), and YY is the number of bits allocated to numerical item identifiers (number of addressable variables). The format is self-defining and these widths are configurable in the SPEAD header. The allocated sizes should remain static for a given implementation.

SPEAD-40-24 is currently deployed, allowing for data heaps of up to 1TB (40 bits) and up to 8388608 variables (23 bits, with one bit allocated to indicate immediate addressing or absolute addressing). The heap size limit does not limit the total size of the SPEAD stream, mearly the total variables (items) that are able to change simultaneously.

4 Data Flow

4.1 Overview

The SPEAD protocol is designed to propagate changes in a set of variables from one location to another. We define an ItemGroup to be a set of Items, where each Item is a variable with a value and a set of metadata that includes the variable’s name, description, type, shape, and numerical identifier. ItemGroups may be changed by adding Items, modifying an Item’s metadata, or changing an Item’s value. Once Items in an ItemGroup have been modified to a self-consistent state, the change in the state of the ItemGroup can be encoded as a Heap.

The process of encoding changes in Items as a Heap proceeds as follows:

  1. For each Item whose value has changed, the binary representation of the new value is appended to the Heap’s payload, and an ItemPointer is appended to the header of the Heap. An ItemPointer encodes the Item’s numerical identifier, and an offset (in bytes) into the Heap’s payload where the Item’s value begins. For the special case where an Item’s value fits within the field allocated for the ItemPointer address, the value is not appended to the Heap, but is instead placed in the address field, and a one-bit IMMEDIATE flag in this ItemPointer is set to indicate that the value can be accessed directly.
  2. For each new Item that has been added to the ItemGroup, or whose metadata has changed, an ItemDescriptor is generated. The value of an ItemDescriptor is a SpeadStream that may contain such metadata Items as NAME, DESCRIPTION, TYPE, SHAPE, and ID. The binary representation of this SpeadStream is appended to the payload of the Heap, and a special ItemPointer is added to the header of the Heap that has its numerical identifier set to DESCRIPTOR_ID, and points to the start of the ItemDescriptor’s value in the Heap’s payload.
  3. This Heap is now converted into a series of SPEAD packets. Each packet consists of a SPEAD header, a number of ItemDescriptors, and a payload. With the exception of a some ItemDescriptors added to aid in packet decoding and reassembly, ItemDescriptors are drawn from the header of the Heap. Each packet payload contains a portion of the Heap payload. Taken together, the generated series of packets will contain the entirety of the Heap’s header and payload, along with a small number of IMMEDIATE ItemDescriptors that allow each packet to be associated with reconstructing a single Heap on the receive side.
  4. After packets pass through a physical transport layer (a network interface, POSIX pipe, file, etc.), a receiver sorts incoming packets by associated Heap and begins the process of reconstructing the Heap. Packet headers are concatenated into a Heap header, and packet payloads are concatenated in the correct order to reconstruct the Heap payload.
  5. Next, each ItemDescriptor in the Heap header is processed, associating a numerical identifier with a the binary value. For standard ItemDescriptors, this involves seeking within the Heap payload to the specified address offset and extracting data from that point to the start of the next ItemDescriptor. For IMMEDIATE ItemDescriptors, the binary value is extracted directly from the address field.
  6. Finally, the receiver’s ItemGroup associates each (numerical identifier, binary value) pair with an Item having the same identifier. First, all ItemDescriptors are processed, creating new Items when appropriate. Next, each binary value is decoded to an actual value by the Item referenced by the associated numerical identifier.

5 Packet Data Representation

Data is always sent and stored Most Significant Byte first. The most significant bit is first on the wire (network byte order). Because this same ordering is used for on-disk storage, data can be streamed directly between the network and storage.

At a minimum, a SPEAD packet must contain:

  • A SPEAD header
  • Three item pointers describing
    • the length of the packet’s payload (to locate packet boundaries for on-disk storage),
    • the offset of this packet’s contents within the heap,
    • and the heap to which it belongs (a counter or timestamp).

The SPEAD header is always 8 octets (64 bits) long.

Item pointers can have a user-defined length, which defines the flavour of the SPEAD protocol and should remain static for a given implementation. Different flavours of SPEAD are available depending on the number of addressable variables and total variable-space size. The SPEAD flavour is expressed as SPEAD-XX-YY where XX is the number of bits allocated for heap address space (total variable-space), and YY is the number of bits allocated to numerical item identifiers (number of addressable variables). The format is self-defining and these widths are configurable in the SPEAD header. The allocated sizes should remain static for a given implementation.

SPEAD-40-24 is currently deployed, allowing for data heaps of up to 1TB (40 bits) and up to 8388608 variables (23 bits, with one bit allocated to indicate immediate addressing or absolute addressing). The heap size limit does not limit the total size of the SPEAD stream, mearly the total variables (items) that are able to change simultaneously.

5.1 SPEAD header

All packets have a header of the following form:

MSb

LSb

Magic # (8b)

Version (8b)

Item Width (8b)

Heap Addr Width (8b)

Reserved (16b)

  1. Items (16b)
Magic number: This is a constant value. All packets will start with the following 8 bit pattern: (msb) 0101 0011 (lsb). This allows basic identification of the packet as a SPEAD packet.
Version identifier: This field identifies the version of the packet format. Designed in case future packet structures differ. The version described here is 004.
Heap address width and item identifier width: SPEAD allows for various lengths of the item pointer fields, depending on your particular requirements. These fields describe the number of bytes of the heap addresses and the numerical item identifiers. Initial implementations have 64 bit item pointer fields, of which 24 bits represent a numeric item identifier and 40 bits address the data in a heap. Heap address width is thus set to and item identifier width set to . This configuration defines the flavour of SPEAD and is referenced as SPEAD-40-24. These numbers need to remain static for a given configuration or implementation.
Reserved: This field should be ignored. The space is reserved for future allocation.
Number of items: The value stored in this field indicates the number of item pointers contained in this packet. Item pointers follow the SPEAD header. The packet payload follows directly after these item pointer fields.

5.2 Items pointers: optional fields

Items are used to transfer variables. Each packet can contain a variable number of data fields, called items pointers, which follow directly after the SPEAD header. Each item pointer consists of some bits of identifier (the most significant bits) and some bits of address (the least significant bits), a pointer to the data location in the heap.

The number of items in a packet is defined by the Number of items field in the SPEAD header. The length of each item pointer field is defined by the sum of the SPEAD header options heap address width and item identifier width (each in bytes). Both the heap address width and the item identifier width fields must remain static for a given SPEAD implementation.

Item pointers’ lower bits (the address field) can store the item’s value directly (immediate addressing), or can point to a start address (in bytes) within the heap where the start of the item’s data is to be found (direct addressing). Using a pointer in this fashion, any length of data can be accomodated. The addressing mode and interpretation of these bits is determined by the first bit in the item pointer. A set bit indicates immediate addressing, while a zero indicates direct addressing. The use of one bit to indicate addressing mode halves the number of addressable items.

In SPEAD-40-24, item pointers are 64 bits long, of which the most significant bit selects the addressing mode, 23 bits form a numerical identifier used to identify the item and the 40 least significant bits contains the direct address or immediate value.

If a packet containing an unknown item is received, it might be recorded, ignored or otherwise the receiver might request a metadata reissue to explain the content (see field 0x30, the ITEM_DESCRIPTOR. In any case, the reception of unknown item should not be considered illegal and should be handled cleanly by a receiver.

Generally the items will be self defining and thus do not need strict identification in this document. However, it is necessary to define a few items which will be used for stream configuration and control.

The following table defines these standardised item identifiers and their meaning. Note that these items should require certain address modes.

Dec

Hex

Description

Addr mode

0

0x0000

NULL - Ignore

Ignored

4

0x0004

Packet payload length

Immediate

5

0x0005

Heap offset

Immediate

13

0x000D

Stream control

Immediate

14

0x000E

Metadata packet counter

Not specified

48

0x0030

Item descriptor

Absolute

49

0x0031

Item descriptor: ID

Not specified

50

0x0032

Item descriptor: name

Not specified

51

0x0033

Item descriptor: description

Not specified

52

0x0034

Item descriptor: type

Not specified

53

0x0035

Item descriptor: shape

Not specified

The remaining item identifier address space is available for user use. A separate document contains a list of recommended item identifiers for common astronomy applications.

Herewith follows an explanation for these standard items...

5.3 Standard Items

NULL

This item is used for optional data padding and can be ignored.

Packet payload length

Numerical ID: 0x04
Name: pkt_len
Description: The length (in bytes) of this packet’s payload.
Type: unsigned integer, immediate addressing (ie with item pointer address number of bits).
Shape: 1

This is the size in bytes of the packet payload length (excluding the header and item fields).

Heap offset

Numerical ID: 0x05
Name: heap_offset
Description: The position (in bytes) relative to the start of this heap at which the payload of the current packet starts.
Type: unsigned integer, immediate addressing (ie with item pointer address number of bits).
Shape: 1

The position (in bytes) relative to the start of this data transfer at which the payload of the current packet starts. It is used for re-assembling data which was split across multiple packets. It is reset back to zero for the following heap (use in conjunction with the HEAP_COUNTER to determine if data loss occurred).

For example, in a correlator, this field would typically measure the byte offset from the start of an integration, as each integration would constitute the start of another heap.

Stream control

Numerical ID: 0x0D
Name: stream_ctrl
Description:
Type: unsigned integer, immediate addressing (ie with item pointer address number of bits).
Shape: 1

Indicates the status of the stream.

The most significant bit of the stream control field indicates that this is the last metadata packet. Used in conjunction with the metadata packet counter, this allows you to check if all the metadata was successfully received.

The lower bits are a number representing:

0: Stream start
1: Metadata reissue (response to command requesting metadata reissue)
2: Stream stop
3: Metadata update (some values have changed, probably in response to a command)
4: Metadata description packet (see Stream descriptors)
>=5: Reserved for future use. Ignore.

Metadata packet counter

Numerical ID: 0x0E
Name: metadata_cnt
Description: Counts metadata packets.
Type: unsigned integer, unspecified addressing mode.
Shape: 1

This item counts metadata packets. When used in conjunction with stream control item’s most significant bit, it can be used to determine if all data was received.

If only one metadata packet is sent, the Metadata packet counter will be zero and the MSb in Stream control will be set.

5.4 Stream descriptors

Stream descriptors are used to provide receiving clients with the metadata required to decode and interpret and unpack data items as part of a particular SPEAD stream. In general these packets will be sent at the start of a session before data and general metadata packets.

Payload descriptor

This option describes the starting position of a string in the data field where the description for a given payload id may be found.

Number of bits

Representation

(MSB) 16

Payload ID that we are describing.

16

Index in bytes of string start position in data field. Indexed from zero.

(LSB) 16

Length of string in bytes including a null-terminator.

The data field of the descriptor packet then contains strings of the form:

unpack_string>[LF]count[LF]Payload_name[LF]Description

The unpack string tells the parser how to unpack the payload, and follows a C style printf convention, with the addition of referenced data types. The unpack string can contain one or more 24-bit unpack directives, with each directive referring to a consecutive data type in the payload. Each 24-bit directive has the following form: unpack_type[bit_length | payload_id]

Unpack character

Representation

0

interpret remaining 16 bits as a Payload ID

i

signed integer

u

unsigned integer

f

IEEE float

c

ASCII character

b

boolean

This allows the unpack string to reference another payload type, thus allowing hierarchical constructs. For non payload ID types, the remaining 16 bits are the bit length of a single field of the specified type.

For example an unpack string could contain: u[32]u[32]f[64]0[002]. This would translate into payload segments that contain two 32-bit unsigned integers, followed by a double, followed by a structure that is defined in Payload ID 002.

The count string allows multiple data constructs of the type specified in the unpack string to be placed contiguously in the data stream. It has the following form (56 bits): count_source[count | option_id]

Unpack character

Representation

0

the remaining 48 bits provide the count directly. (cast to an unsigned integer)

1

the count is read as the 48-bit value associated with the specified Option ID.

Payload names are intended to be parsed and presented in the receiver namespace and thus should not include whitespace. They may not contain [LF] either, as this is used for argument demarkation when parsing.

As an example, consider a descriptor packet containing the 16bit field id 0x31 with 48 bit content 0x0005 0000 0035.

Unpack size

Unpacked value

16 msb

048 decimal.

16b

005 decimal.

16b

000.

16b lsb

53 decimal.

This would describe a Payload ID of 0005 starting at offset 0 and having a length of 53 bytes. And the start of the packet’s datafield could contain the string (with additional payload id’s included that have not been shown in the above header.)

0[2][LF]1[9][LF]frequency[LF]A single frequency bin.[NULL]0[3][LF]1[11][LF]baseline[LF]A single baseline. Baseline ordering starts with the cross correlation between A0 and A1. Thereafter all cross correlations in natural order. Auto correlations follow.[NULL]0[4][LF]0[4][LF]polarisation[LF]The polarisation products. Order is XX, YY, XY, YX[NULL]u[32][LF]0[2][LF]complex_data[LF]The complex data point. Two uint32, real followed by imaginary[NULL].

This above would describe the current v0002 correlator output format. Essentially it shows a hierarchy the begins with two uint32’s the form a single complex sample. Each 4 complex samples describe a set of polarisation products. The polarisation products are then grouped into a number of baselines (that is specified in option 0x0011. The baselines are finally grouped into frequency channels, the number of which is specified in option 0x0009. This fully describes the data payload for a single correlator dump.

Option descriptor

In exactly the same fashion as the payload descriptor, the option descriptor describes the starting position of a string in the packet data where the description for a given option field may be found. The syntax is the same, but only a subset of the descriptive features are used since the format of the option field is much more tightly defined.

Number of bits

Representation

(MSB) 16

Option ID that we are describing.

16

Index in bytes of string start position in data field. Indexed from zero.

(LSB) 16

Length of string in bytes including a null-terminator.

The data field of the metadata packet then contains strings of the form:

unpack_string>[LF]0[1][LF]Option_name[LF]Description

The unpack string is interpreted as before, with the exclusion of the indirect referencing of other descriptors. Further, the option field has a fixed 48-bit length. This implies, for instance, that a float can only be a maximum of 32-bits long. The count string is not used and is set to 0[1] to indicate a single instance of the unpack string is to be used. An unpack string might look like: u[4]b[1]f[32]c[8]b[1]b[1]b[1].

As an example, consider a metadata packet containing the 16bit field id 0x30 with 48 bit content of 12884902140, this would decode as:

Unpack size

Unpacked value

16 msb

048 decimal.

16b

003 decimal.

16b

000.

16b lsb

252 decimal.

And the start of the packet’s datafield could contain the string u[48][LF]0[1][LF]timestamp[LF]This field timestamps all data for any DBE instrument. Time zero corresponds to the time at which the system was last sync’d to a 1PPS signal. Each increment of this field corresponds to a change of timestamp_scale_factor x n_ADC_samples.[NULL]

In the same packet, we might have a second metadata field with ID 48 and 48bit content of 128865534254. This would decode as:

Unpack size

Unpacked value

16 msb

048 decimal.

16b

030 decimal.

16b

252 decimal.

16b

302 decimal.

And at offset 252 in the datafield, we might find: u[16]b[1]u[31][LF]0[1][LF]FFT Shift pattern[LF]Indicates if a divide-by-2 was done between butterfly stages in the FFT. LSB is last FFT stage, with every higher bit representing the previous FFT stage. Only the <FFT Length> LSbs are valid. Struct is input number (16b msb), polarisation (1bit), FFT shift pattern (31 bits). [NULL]

A single metadata description should never straddle a packet boundary. Each metadata packet should be self-contained. This effectively limits the length of a single description string. Jumbo frames typically allow for over 8KiB of data, so these strings can still be reasonably long.

A Example Packet Exchange

This section provides an example packet exchange that could occur between a simple correlator and a SPEAD receiver instance.

A.1 Packet Sequence

At the start of the data transmission, initial metadata packets are sent. These should setup any stream descriptors required to describe options and payloads. They will also include non-payload specific option fields that do not require transmission synchronously with the data packets.

During transmission of the data stream, packets should contain only a limited number of optional fields, in order to maximise the utilisation of the link. Certain options are sent if parameters change or errors are encountered. Full metadata packets can be regenerated upon request, allowing a receiver to process a stream which had been started earlier.

When a data stream terminates, a futher metadata packet is sent, this may not contain a full description of the data stream, but does include stream control options to indicate that the transmission has completed.

The metadata packets may not arrive from the same source as the data packets. They may not even arrive from the same subnet. The receiver should not care about the source of the packet, as the contents of a stream are sufficient to fully identify the sender (barring any rogue packets on the network).

It is recommended that unknown options be recorded, in order to facilitate debugging and forward compatibility, even if these options are not understood.

A.2 Descriptor packets

Revision 3 of the protocol (as described here) allows for initial descriptor packets to describe the nature of the variables in the stream and also detail how to unpack the data for the receiving parser. Revisions 1 and 2 did not include this facility, though the protocol was designed to be forwards and backwards compatible. Earlier receivers might have these decode values hard-coded.

A transmitter conforming to Revision 3 could send a descriptor packet similar to the following:

Packet contents

Representation

Significant meaning

0x4B52 0003 0000 000B

Header

11 options will follow.

0x000D 0000 0000 0004

Stream control

Metadata descr, not the last.

0x000E 0000 0000 0000

Metadata pkt cnt

First metadata packet in this stream.

0x0001 0001 C020 000C

Instrument type

PoCo, 12 complex 32b unsgnd ints, msb first per data item.

0x0003 0000 0000 0000

Instance ID

Stream 0 from engine 0.

0x0030 001E 00FC 012E

Option descr

Option 0x1E descr. at payload offset 252 is 302 bytes long.

0x0030 0003 0000 00FC

Option descr

Option 0x03 descr. at payload start is 252 bytes long.

0x0031 0001 0352 011D

Payload descr

Paylod 0x01 descr. at payload offset 850 is 285 bytes long.

Directly following those options, we’d have the following ASCII string, where values in brackets correspond to raw binary representations:

u[48][LF]0[1][LF]Timestamp[LF]This field timestamps all data for any DBE instrument. Time zero corresponds to the time at which the system was last sync’d to a 1PPS signal. Each increment of this field corresponds to a change of timestamp_scale_factor x n_ADC_samples.[NULL]u[16]b[1]u[31][LF]0[1][LF]FFT Shift pattern[LF]Indicates if a divide-by-2 was done between butterfly stages in the FFT. LSB is last FFT stage, with every higher bit representing the previous FFT stage. Only the <FFT Length> LSbs are valid. Struct is input number (16b msb), polarisation (1bit), FFT shift pattern (31 bits). [NULL]....0[2][LF]1[9][LF]frequency[LF]A single frequency bin.[NULL]

Thus the receiver would know how to interpret option ID 0x1E and 0x03 as well as payload ID 0x01.

These descriptions are intended to supplement this document. Although every effort is made to describe all required fields, it is understood that future systems may need to supplement these fields. These descriptors provide a mechanism for achieving this. It will still be necessary for a receiver to understand certain standard fields a-priori (eg Stream control, Metadata packet counter, Instrument type, Instance ID, Metadata descriptor etc), in order to decode adescriptor packet.

It is recommended that all option fields used in a given stream be described this way and that a full list of descriptors be pre-pended to the datastream.

A.3 Metadata packets

Once the descriptors have been transmitted the receiver will be in a position to decode additional option fields. These are then sent in a number of metadata packets before the start of payload transmission. For example,the poco and KAT-7 corr instruments (and any other instrument implementing per-channel gain control) will need to issue many metadata packets describing the digital gain setting on each frequency channel for each ADC.

Number of bytes

Option ID

Representation

8

Header

8

0x000D

Stream control

8

0x000E

Metadata packet counter

8

0x0001

Instrument type

8

0x0003

Instance ID

8

0x000F

Start time, whole seconds

8

0x0011

Center frequency (Hz)

8

0x0013

Bandwidth (Hz)

8

0x0007

ADC Sample Rate (Hz)

8

0x002F

Timestamp Scale Factor

8

0x0009

Number of channels

8

0x000A

Number of antennas

8

0x000B

Number of baselines

8

0x0015

Number of accumulations per integration

8

0x0016

Quantisation scalar X-pol, real

8

0x0016

Quantisation scalar X-pol, real

8

0x0016

Quantisation scalar X-pol, real

8

0x0016

Quantisation scalar X-pol, real

Many such metadata packets will be sent to describe all the quantisation scalars for all ADCs, all polarisations, all channels. Each subsequent packet will increment the metadata packet counter field’s value by one, which allows the receiver to check that no metadata packets were missed. The length of each packet can be calculated based on the number of options in the preamble. The last packet will have the MSb in the Stream Control field set, indicating the end of the metadata bundle.

If only one packet is sent, the metadata packet counter will be zero and the MSb in Stream Control will be set.

During the normal course of operation, additional metadata packets may be received with updated values. In this case, the Stream Control field will have a value of 3 and the updated values should be recorded.

A.4 Data packets

Data packets will generally have only a small number of option fields that are required to unambiguously describe the payload of the particular packet in question.

Both the poco mode and packetised corr (correlator) instruments will emit packets with a data payload typically containing the following options, noting that the ordering of options may vary.

Number of bytes

Option ID

Representation

(MSB) 8

Header

8

0x0001

Instrument type

8

0x0002

Instance ID

8

0x0003

Timestamp

8

0x0004

Data payload length

8

0x0000

NULL

8

0x0005

Data payload offset (in bytes)

(LSB) 4096

Data

The receiver will use the payload ID that forms part of the instrument type to correctly decode the data payload. Note that a payload descriptor refers to a complete, unfragmented data block. Blocks of data that are too large for a single packet will get fragmented. The receiver can re-assemble these fragments using the payload length and payload offset information. Once unfragmented that block can be parsed using the supplied descriptor.

The length of the payload may vary, but is likely to be 2kiB, 4kiB or 8kiB.

The last packet of a data block may be shorter if there is not enough data to fill the packet. ie.

The fifth field is kept in reserve for future use.

Once data transmission is complete a Stream Stop packet is issued.