Difference between revisions of "Spead specification"

From AstroBaki
Jump to navigationJump to search
Line 139: Line 139:
 
This field identifies the version of the packet format. Designed in case future packet structures differ. The version described here is 004.
 
This field identifies the version of the packet format. Designed in case future packet structures differ. The version described here is 004.
  
\item[Heap address width and item identifier width]
+
\item[Heap address width and item identifier width]:
 
SPEAD allows for various lengths of the \textit{item pointer} fields, depending on your particular requirements. These fields describe the number of bytes of the \textit{heap} addresses and the numerical \textit{item identifiers}. Initial implementations have 64 bit \textit{item pointer} fields, of which 24 bits represent a numeric \textit{item identifier} and 40 bits address the data in a \textit{heap}. \textit{Heap address width} is thus set to $40/8=5 bytes$ and \textit{item identifier width} set to $24/8=3 bytes$. This configuration defines the flavour of SPEAD and is referenced as SPEAD-40-24. These numbers need to remain static for a given configuration or implementation.
 
SPEAD allows for various lengths of the \textit{item pointer} fields, depending on your particular requirements. These fields describe the number of bytes of the \textit{heap} addresses and the numerical \textit{item identifiers}. Initial implementations have 64 bit \textit{item pointer} fields, of which 24 bits represent a numeric \textit{item identifier} and 40 bits address the data in a \textit{heap}. \textit{Heap address width} is thus set to $40/8=5 bytes$ and \textit{item identifier width} set to $24/8=3 bytes$. This configuration defines the flavour of SPEAD and is referenced as SPEAD-40-24. These numbers need to remain static for a given configuration or implementation.
  
Line 146: Line 146:
  
 
\item[Number of items]:
 
\item[Number of items]:
The value stored in this field indicates the number of optional 64-bit fields following the header. Any attached data follows directly after this number of 64-bit fields.
+
The value stored in this field indicates the number of \textit{items} following the SPEAD header. The packet payload follows directly after this number of \textit{item pointer} fields.
  
 
\end{description}
 
\end{description}
  
\subsection{\emph{Items}: optional fields}
+
\subsection{\emph{Items pointers}: optional fields}
  
Each packet can contain zero or more data fields, called \emph{items}. Each item consists of some bits of identifier (MSb) and some bits of data (LSb).  
+
Each packet can contain zero or more data fields, called \emph{items pointers}, which follow directly after the SPEAD header. Each item consists of some bits of identifier (the most significant bits) and some bits of data or pointer (the least significant bits).  
  
The number of items in a packet is defined by the \emph{Number of items} field in the SPEAD header. The length of each \textit{item} field is defined in the SPEAD header option \textit{item width} in bytes and the number of bytes of the \emph{item} field containing the actual data is stored in the header field \emph{heap address width}. The difference between these values accounts for the \emph{item identifier} field. The width of the field and the portion of that field used for data and identifier must remain fixed for a given implementation.
+
The number of items in a packet is defined by the \emph{Number of items} field in the SPEAD header. The length of each \textit{item pointer} field is defined by the sum of the SPEAD header options \textit{heap address width} and \textit{item identifier width} (each in bytes). The size of these fields must remain fixed for a given implementation.
  
\textit{Items} can store the user data value directly, or can point to an offset address (in bytes) where the start of the data is to be found. Thus, any length of data can be accomodated. In standard SPEAD-64-40, \textit{items} are 64 bits long, of which 24 bits used to identify the \textit{item} and the remaining 40 bits are used . This allows for up to 
+
\textit{Item pointers}' lower bits can store the data value directly (immediate addressing), or can point to an offset address (in bytes) where the start of the data is to be found in the \textit{heap} (direct addressing). Thus, any length of data can be accomodated. The addressing mode and interpretation of these bits is determined by the first bit in the \textit{item pointer}. A set bit indicates immediate addressing, while a zero indicates direct addressing. This halves the number of addressable items.
 +
 
 +
In standard SPEAD-40-24, \textit{item pointers} are 64 bits long, of which the most significant bit selects the addressing mode, 23 bits form a numerical identifier used to identify the \textit{item} and the 40 least significant bits contains the direct address or immediate value.  
  
 
If a packet containing an unknown \textit{item} is received, it might be recorded, ignored or otherwise the receiver might request a metadata reissue to explain the content (see field 0x30). In any case, the reception of unknown \textit{item} should not be considered illegal and should be handled cleanly by a receiver.
 
If a packet containing an unknown \textit{item} is received, it might be recorded, ignored or otherwise the receiver might request a metadata reissue to explain the content (see field 0x30). In any case, the reception of unknown \textit{item} should not be considered illegal and should be handled cleanly by a receiver.

Revision as of 15:39, 31 March 2010

SPEAD: Streaming Protocol for Exchanging Astronomical Data

references

1 Scope

This document describes a datastream format suitable for use by radio astronomy instruments. The data stream is distinct from the KATCP control protocol which follows a format described in [.

Many instruments output data over ethernet. These streams are often high speed and unsuitable for TCP/IP links. UDP is often prefered in the cases where partial data-loss can be accomodated. This document aims to standardise the output format of such UDP packets.

An essential feature of the datastream is that it is self describing. This self description extends to both the content and interpretation of the option fields in the packet header, and of the data payload itself. The same receive and transmit code may be used for multiple instruments as the layout of the packets are well defined.

Emphasis is placed on lightweight implementation, suitable for data packetisation by hardware devices or by software processes, while retaining flexibility for complex, dynamically changing data exchanges.

Metadata and variable descriptors can be injected into the datastream at runtime and recorded or processed along with the primary datastream. It is also possible to record the raw datastream directly to disk for later parsing.

The format of the application-layer packetised stream is described in this document. Lower speed and ad-hoc data products may be transferred using the TCP-based KATCP control interface [.

2 Protocol Layers

The data is transmitted via a network interface, or stored on disk.

Although this format describes only application and presentation layers of the OSI model and data transmission can take place over any medium (serial data lines, ethernet networks or other), the preffered interface for network communications consists of the following layers, from top to bottom:

Application and Presentation: The data format as described in later sections of this document.
Session and Transport: UDP, noting that the receiving party can not request retransmission of lost, duplicated or corrupt packets.
Network: IP (Internet protocol)
Link: 10 Gbps, 1 Gbps or 100 Mbps Ethernet
Physical: SFP+, CX4 or other 10 Gigabit ethernet cable, Cat6, Cat5e etc.

3 Overview

Each user datagram contains a SPEAD header, three or more optional fields (item pointers) and a packet payload.

Items are used to store user-variables. These items can contain single valued data products, multi-dimensional data or can point to other items (or multiples of other items) thereby allowing hierarchical data structures. Item pointers point to an address within an assembled heap of data where those data-products are stored, allowing virtually unlimited lengths for a given item value. The heap is constructed by assembling all packet payloads prior to data unpacking. All items can be automatically unpacked and interpretted before presentation to the user’s application. Item pointers can contain single values themselved (immediate addressing) rather than a pointer to a location containing the data (absolute addressing). This is more efficient for items which contain short integer values.

Different flavours of SPEAD are available depending on the complexity of an application. This is determined by the address-space that the user chooses to allocate to the items and also the length of the pointers to (and hence address space in) the heap. It is expressed as /textitSPEAD-XX-YY where XX is the number of bits allocated for heap address space, and YY is the number of bits allocated to numerical item identifiers. The format is self-defining and these widths are configurable in the SPEAD header. The allocated sizes should remain static for a given implementation.

SPEAD-40-24 is currently employed, allowing for data heaps of up to 1TB (40 bits) and up to 8388608 variables (23 bits, with one bit allocated to indicate immediate addressing or absolute addressing). The heap size limit does not limit the total size of the SPEAD stream, mearly the total variables (items) that are able to change simultaneously.

4 Packet Data Representation

Data is always sent and stored Most Significant Byte first. The most significant bit is first on the wire (network byte order). Because this same ordering is used for on-disk storage, data can be streamed directly between the network and storage.

At a minimum, a SPEAD packet must contain:

  • A SPEAD header
  • Three item pointers describing
    • the length of the packet’s payload,
    • the offset of this packet within the larger heap,
    • and the heap to which it belongs.

The SPEAD header is always 8 octets (64 bits) long. Item pointers can have a user-defined length, which defines the flavour of the SPEAD protocol and should remain static for a given implementation.

4.1 SPEAD header

All data stream packets have a header of the following form:

Number of bits

Description

(MSB) 8

Magic Number (constant 0x53, ascii S)

8

Version identifier (this version is 4)

8

Item width (number of bytes per item)

8

Heap address width (number of bytes in the heap address space)

16

Reserved

(LSB) 16

Number of items

Magic number: This is a constant value. All packets will start with the following 8 bit pattern: (msb) 0101 0011 (lsb). This allows basic identification of the packet as a SPEAD packet.
Version identifier: This field identifies the version of the packet format. Designed in case future packet structures differ. The version described here is 004.
Heap address width and item identifier width: SPEAD allows for various lengths of the item pointer fields, depending on your particular requirements. These fields describe the number of bytes of the heap addresses and the numerical item identifiers. Initial implementations have 64 bit item pointer fields, of which 24 bits represent a numeric item identifier and 40 bits address the data in a heap. Heap address width is thus set to and item identifier width set to . This configuration defines the flavour of SPEAD and is referenced as SPEAD-40-24. These numbers need to remain static for a given configuration or implementation.
Reserved: This field should be ignored. The space is reserved for future allocation.
Number of items: The value stored in this field indicates the number of items following the SPEAD header. The packet payload follows directly after this number of item pointer fields.

4.2 Items pointers: optional fields

Each packet can contain zero or more data fields, called items pointers, which follow directly after the SPEAD header. Each item consists of some bits of identifier (the most significant bits) and some bits of data or pointer (the least significant bits).

The number of items in a packet is defined by the Number of items field in the SPEAD header. The length of each item pointer field is defined by the sum of the SPEAD header options heap address width and item identifier width (each in bytes). The size of these fields must remain fixed for a given implementation.

Item pointers’ lower bits can store the data value directly (immediate addressing), or can point to an offset address (in bytes) where the start of the data is to be found in the heap (direct addressing). Thus, any length of data can be accomodated. The addressing mode and interpretation of these bits is determined by the first bit in the item pointer. A set bit indicates immediate addressing, while a zero indicates direct addressing. This halves the number of addressable items.

In standard SPEAD-40-24, item pointers are 64 bits long, of which the most significant bit selects the addressing mode, 23 bits form a numerical identifier used to identify the item and the 40 least significant bits contains the direct address or immediate value.

If a packet containing an unknown item is received, it might be recorded, ignored or otherwise the receiver might request a metadata reissue to explain the content (see field 0x30). In any case, the reception of unknown item should not be considered illegal and should be handled cleanly by a receiver.

Generally the items will be self defining and thus do not need strict identification in this document. However, it is useful to define a number of standard options that are likely to be used across a wide range of deployments.

The following table defines these standardised options identifier codes and their meaning.

Dec

Hex

Description

0

0000

NULL - Ignore

1

0001

Instrument type

4

0004

Data payload length

5

0005

Data payload offset

13

000D

Stream control

14

000E

Metadata packet counter

48

0030

Option descriptor

49

0031

Payload descriptor

In addition to these standardised options, a number of reserved options are defined. These are options that are currently in use in systems, including those using previous versions of this protocol. Overloading of these should be avoided where practical.

User defined options are allowed, but these should have the MSb of the option ID set (i.e. option ID > 32767)

Dec

Hex

Description

2

0002

Instance ID and sub-instance source (24 bit instance + 24 bit engine num)

3

0003

Timestamp

6

0006

Previous packet data error counter

7

0007

ADC Sample rate (Hz)

8

0008

ADC Sample rate fractional (femto-Hz)

9

0009

Number of frequency channels

10

000A

Number of antennas

11

000B

Number of baselines

12

000C

FPGA clock cycles per sync pulse (F engine)

15

000F

Last DBE system sync time (Unix time since epoch, seconds)

16

0010

Last DBE system sync time (fractional seconds, femto-seconds)

17

0011

Center frequency (Hz)

18

0012

Center frequency fractional (femto-Hz)

19

0013

Bandwidth (Hz)

20

0014

Bandwidth fractional (femto-Hz)

21

0015

Number of accumulations per integration

22

0016

Quantisation scalar X-polarisation, real component

23

0017

Quantisation scalar X-polarisation, imaginary component

24

0018

Quantisation scalar Y-polarisation, real component

25

0019

Quantisation scalar Y-polarisation, imaginary component

26

001A

Phase

27

001B

Phase timestamp (ADC sample number)

28

001C

Delay

29

001D

Delay timestamp (ADC sample number)

30

001E

FFT shift pattern

31

001F

X engine accumulation length

32

0020

F engine quantised bits per sample

33

0021

F engine packet payload length

34

0022

10GbE destination port (X engine)

35

0023

10GbE destination port (F engine)

36

0024

10GbE destination IP address (X engine)

37

0025

10GbE destination IP address (F engine)

38

0026

Processing engine clock rate (Hz)

39

0027

ADC overflow

40

0028

ADC overflow Timestamp (ADC sample number)

41

0029

Quantisation overflow

42

002A

Quantisation overflow Timestamp (ADC sample number)

43

002B

FFT overflow

44

002C

FFT overflow Timestamp (ADC sample number)

45

002D

Accumulator overflow

46

002E

Accumulator overflow Timestamp (ADC sample number)

47

002F

Timestamp scale factor

4.3 Standard Options

Instrument Type

The instrument type field defines the source of the data. It also includes the payload id identifying the decoding to be used to interpret the data.

The 48-bit field is subdivided into 16 bits representing instrument type and 32 bits representing the payload id (currently only the 16lsb are used). The instrument type is used to provide scope to particular option and payload id’s. Generally the instrument type will be fairly broad (e.g. KAT-7 which includes streamed data from all KAT-7 facility instruments).

The following are defined existing instrument types (16 MSbs):

Hex identifier

Instrument

0000

NULL instrument

0001

Pocket correlator (KAT Fringe Finder)

0002

Single-dish spectrometer

Note that header and options are specified to be transmitted MSb first, regardless of instrument type.

Data Payload Length

This is the size in bytes of the packet payload length (excluding the header and optional fields).

Data Payload Offset

The position, relative to the integration, at which the payload of the current packet starts.

This is used for re-assembling data which was split across multiple packets. It is reset back to zero for the following integration (use in conjunction with the timestamp to determine if data loss occurred).

Stream control and metadata packet counter

Indicates the status of the stream. This control packet is contained in a stand-alone packet not containing any actual data products. If present, the data field of the packet (ie after all the 64bit options) might contain text descriptors related to option are payload ids’ in the stream. See fields 48 (0x30) and 49 (0x31) for further details.

The most significant bit of the stream control field indicates that this is the last metadata packet. Used in conjunction with the metadata packet counter, this allows you to check if all the metadata was successfully received.

The lower 47 bits are a number representing:

0: Stream start
1: Metadata reissue (response to command requesting metadata reissue)
2: Stream stop
3: Metadata update (some values have changed, probably in response to a command)
4: Metadata description packet (see Stream descriptors)
>=5: Reserved for future use. Ignore.

If only one packet is sent, the Metadata packet counter will be zero and the MSb in Stream control will be set.

4.4 Stream descriptors

Stream descriptors are used to provide receiving clients with the metadata required to decode and interpret options and data sent as part of a particular SPEAD stream. These descriptors are sent in a special metadata-only packet (stream_control==4), and must never appear in a packet where ordinary instrument data is present, lest the description strings be confused with actual data. In general these packets will be sent at the start of a session before data and general metadata packets.

Provision is made for both option (describe the meaning and interpretation of header options), and payload (the structure of the streamed data) descriptors. Although they share a similar syntax, it is important to understand the distinction between the two types.

Payload descriptor

This option describes the starting position of a string in the data field where the description for a given payload id may be found.

Number of bits

Representation

(MSB) 16

Payload ID that we are describing.

16

Index in bytes of string start position in data field. Indexed from zero.

(LSB) 16

Length of string in bytes including a null-terminator.

The data field of the descriptor packet then contains strings of the form:

unpack_string>[LF]count[LF]Payload_name[LF]Description

The unpack string tells the parser how to unpack the payload, and follows a C style printf convention, with the addition of referenced data types. The unpack string can contain one or more 24-bit unpack directives, with each directive referring to a consecutive data type in the payload. Each 24-bit directive has the following form: unpack_type[bit_length | payload_id]

Unpack character

Representation

0

interpret remaining 16 bits as a Payload ID

i

signed integer

u

unsigned integer

f

IEEE float

c

ASCII character

b

boolean

This allows the unpack string to reference another payload type, thus allowing hierarchical constructs. For non payload ID types, the remaining 16 bits are the bit length of a single field of the specified type.

For example an unpack string could contain: u[32]u[32]f[64]0[002]. This would translate into payload segments that contain two 32-bit unsigned integers, followed by a double, followed by a structure that is defined in Payload ID 002.

The count string allows multiple data constructs of the type specified in the unpack string to be placed contiguously in the data stream. It has the following form (56 bits): count_source[count | option_id]

Unpack character

Representation

0

the remaining 48 bits provide the count directly. (cast to an unsigned integer)

1

the count is read as the 48-bit value associated with the specified Option ID.

Payload names are intended to be parsed and presented in the receiver namespace and thus should not include whitespace. They may not contain [LF] either, as this is used for argument demarkation when parsing.

As an example, consider a descriptor packet containing the 16bit field id 0x31 with 48 bit content 0x0005 0000 0035.

Unpack size

Unpacked value

16 msb

048 decimal.

16b

005 decimal.

16b

000.

16b lsb

53 decimal.

This would describe a Payload ID of 0005 starting at offset 0 and having a length of 53 bytes. And the start of the packet’s datafield could contain the string (with additional payload id’s included that have not been shown in the above header.)

0[2][LF]1[9][LF]frequency[LF]A single frequency bin.[NULL]0[3][LF]1[11][LF]baseline[LF]A single baseline. Baseline ordering starts with the cross correlation between A0 and A1. Thereafter all cross correlations in natural order. Auto correlations follow.[NULL]0[4][LF]0[4][LF]polarisation[LF]The polarisation products. Order is XX, YY, XY, YX[NULL]u[32][LF]0[2][LF]complex_data[LF]The complex data point. Two uint32, real followed by imaginary[NULL].

This above would describe the current v0002 correlator output format. Essentially it shows a hierarchy the begins with two uint32’s the form a single complex sample. Each 4 complex samples describe a set of polarisation products. The polarisation products are then grouped into a number of baselines (that is specified in option 0x0011. The baselines are finally grouped into frequency channels, the number of which is specified in option 0x0009. This fully describes the data payload for a single correlator dump.

Option descriptor

In exactly the same fashion as the payload descriptor, the option descriptor describes the starting position of a string in the packet data where the description for a given option field may be found. The syntax is the same, but only a subset of the descriptive features are used since the format of the option field is much more tightly defined.

Number of bits

Representation

(MSB) 16

Option ID that we are describing.

16

Index in bytes of string start position in data field. Indexed from zero.

(LSB) 16

Length of string in bytes including a null-terminator.

The data field of the metadata packet then contains strings of the form:

unpack_string>[LF]0[1][LF]Option_name[LF]Description

The unpack string is interpreted as before, with the exclusion of the indirect referencing of other descriptors. Further, the option field has a fixed 48-bit length. This implies, for instance, that a float can only be a maximum of 32-bits long. The count string is not used and is set to 0[1] to indicate a single instance of the unpack string is to be used. An unpack string might look like: u[4]b[1]f[32]c[8]b[1]b[1]b[1].

As an example, consider a metadata packet containing the 16bit field id 0x30 with 48 bit content of 12884902140, this would decode as:

Unpack size

Unpacked value

16 msb

048 decimal.

16b

003 decimal.

16b

000.

16b lsb

252 decimal.

And the start of the packet’s datafield could contain the string u[48][LF]0[1][LF]timestamp[LF]This field timestamps all data for any DBE instrument. Time zero corresponds to the time at which the system was last sync’d to a 1PPS signal. Each increment of this field corresponds to a change of timestamp_scale_factor x n_ADC_samples.[NULL]

In the same packet, we might have a second metadata field with ID 48 and 48bit content of 128865534254. This would decode as:

Unpack size

Unpacked value

16 msb

048 decimal.

16b

030 decimal.

16b

252 decimal.

16b

302 decimal.

And at offset 252 in the datafield, we might find: u[16]b[1]u[31][LF]0[1][LF]FFT Shift pattern[LF]Indicates if a divide-by-2 was done between butterfly stages in the FFT. LSB is last FFT stage, with every higher bit representing the previous FFT stage. Only the <FFT Length> LSbs are valid. Struct is input number (16b msb), polarisation (1bit), FFT shift pattern (31 bits). [NULL]

A single metadata description should never straddle a packet boundary. Each metadata packet should be self-contained. This effectively limits the length of a single description string. Jumbo frames typically allow for over 8KiB of data, so these strings can still be reasonably long.

5 Data Flow

5.1 Overview

The SPEAD protocol is designed to propagate changes in a set of variables from one location to another. We define an ItemGroup to be a set of Items, where each Item is a variable with a value and a set of metadata that includes the variable’s name, description, type, shape, and numerical identifier. ItemGroups may be changed by adding Items, modifying an Item’s metadata, or changing an Item’s value. Once Items in an ItemGroup have been modified to a self-consistent state, the change in the state of the ItemGroup can be encoded as a Heap.

The process of encoding changes in Items as a Heap proceeds as follows:

  1. For each Item whose value has changed, the binary representation of the new value is appended to the Heap’s payload, and an ItemPointer is appended to the header of the Heap. An ItemPointer encodes the Item’s numerical identifier, and an offset (in bytes) into the Heap’s payload where the Item’s value begins. For the special case where an Item’s value fits within the field allocated for the ItemPointer offset, the value is not appended to the Heap, but is instead placed in the offset field, and a one-bit IMMEDIATE flag in this ItemPointer is set to indicate that the value can be accessed directly.
  2. For each new Item that has been added to the ItemGroup, or whose metadata has changed, an ItemDescriptor is generated. The value of an ItemDescriptor is a SpeadStream that may contain such metadata Items as NAME, DESCRIPTION, TYPE, SHAPE, and ID. The binary representation of this SpeadStream is appended to the payload of the Heap, and a special ItemPointer is added to the header of the Heap that has its numerical identifier set to DESCRIPTOR_ID, and points to the start of the ItemDescriptor’s value in the Heap’s payload.
  3. This Heap is now converted into a series of SPEAD packets. Each packet consists of a SPEAD header, a number of ItemDescriptors, and a payload. With the exception of a some ItemDescriptors added to aid in packet decoding and reassembly, ItemDescriptors are drawn from the header of the Heap. Each packet payload contains a portion of the Heap payload. Taken together, the generated series of packets will contain the entirety of the Heap’s header and payload, along with a small number of IMMEDIATE ItemDescriptors that allow each packet to be associated with reconstructing a single Heap on the receive side.
  4. After packets pass through a physical transport layer (a network interface, POSIX pipe, file, etc.), a receiver sorts incoming packets by associated Heap and begins the process of reconstructing the Heap.

A Example Packet Exchange

This section provides an example packet exchange that could occur between a simple correlator and a SPEAD receiver instance.

A.1 Packet Sequence

At the start of the data transmission, initial metadata packets are sent. These should setup any stream descriptors required to describe options and payloads. They will also include non-payload specific option fields that do not require transmission synchronously with the data packets.

During transmission of the data stream, packets should contain only a limited number of optional fields, in order to maximise the utilisation of the link. Certain options are sent if parameters change or errors are encountered. Full metadata packets can be regenerated upon request, allowing a receiver to process a stream which had been started earlier.

When a data stream terminates, a futher metadata packet is sent, this may not contain a full description of the data stream, but does include stream control options to indicate that the transmission has completed.

The metadata packets may not arrive from the same source as the data packets. They may not even arrive from the same subnet. The receiver should not care about the source of the packet, as the contents of a stream are sufficient to fully identify the sender (barring any rogue packets on the network).

It is recommended that unknown options be recorded, in order to facilitate debugging and forward compatibility, even if these options are not understood.

A.2 Descriptor packets

Revision 3 of the protocol (as described here) allows for initial descriptor packets to describe the nature of the variables in the stream and also detail how to unpack the data for the receiving parser. Revisions 1 and 2 did not include this facility, though the protocol was designed to be forwards and backwards compatible. Earlier receivers might have these decode values hard-coded.

A transmitter conforming to Revision 3 could send a descriptor packet similar to the following:

Packet contents

Representation

Significant meaning

0x4B52 0003 0000 000B

Header

11 options will follow.

0x000D 0000 0000 0004

Stream control

Metadata descr, not the last.

0x000E 0000 0000 0000

Metadata pkt cnt

First metadata packet in this stream.

0x0001 0001 C020 000C

Instrument type

PoCo, 12 complex 32b unsgnd ints, msb first per data item.

0x0003 0000 0000 0000

Instance ID

Stream 0 from engine 0.

0x0030 001E 00FC 012E

Option descr

Option 0x1E descr. at payload offset 252 is 302 bytes long.

0x0030 0003 0000 00FC

Option descr

Option 0x03 descr. at payload start is 252 bytes long.

0x0031 0001 0352 011D

Payload descr

Paylod 0x01 descr. at payload offset 850 is 285 bytes long.

Directly following those options, we’d have the following ASCII string, where values in brackets correspond to raw binary representations:

u[48][LF]0[1][LF]Timestamp[LF]This field timestamps all data for any DBE instrument. Time zero corresponds to the time at which the system was last sync’d to a 1PPS signal. Each increment of this field corresponds to a change of timestamp_scale_factor x n_ADC_samples.[NULL]u[16]b[1]u[31][LF]0[1][LF]FFT Shift pattern[LF]Indicates if a divide-by-2 was done between butterfly stages in the FFT. LSB is last FFT stage, with every higher bit representing the previous FFT stage. Only the <FFT Length> LSbs are valid. Struct is input number (16b msb), polarisation (1bit), FFT shift pattern (31 bits). [NULL]....0[2][LF]1[9][LF]frequency[LF]A single frequency bin.[NULL]

Thus the receiver would know how to interpret option ID 0x1E and 0x03 as well as payload ID 0x01.

These descriptions are intended to supplement this document. Although every effort is made to describe all required fields, it is understood that future systems may need to supplement these fields. These descriptors provide a mechanism for achieving this. It will still be necessary for a receiver to understand certain standard fields a-priori (eg Stream control, Metadata packet counter, Instrument type, Instance ID, Metadata descriptor etc), in order to decode adescriptor packet.

It is recommended that all option fields used in a given stream be described this way and that a full list of descriptors be pre-pended to the datastream.

A.3 Metadata packets

Once the descriptors have been transmitted the receiver will be in a position to decode additional option fields. These are then sent in a number of metadata packets before the start of payload transmission. For example,the poco and KAT-7 corr instruments (and any other instrument implementing per-channel gain control) will need to issue many metadata packets describing the digital gain setting on each frequency channel for each ADC.

Number of bytes

Option ID

Representation

8

Header

8

0x000D

Stream control

8

0x000E

Metadata packet counter

8

0x0001

Instrument type

8

0x0003

Instance ID

8

0x000F

Start time, whole seconds

8

0x0011

Center frequency (Hz)

8

0x0013

Bandwidth (Hz)

8

0x0007

ADC Sample Rate (Hz)

8

0x002F

Timestamp Scale Factor

8

0x0009

Number of channels

8

0x000A

Number of antennas

8

0x000B

Number of baselines

8

0x0015

Number of accumulations per integration

8

0x0016

Quantisation scalar X-pol, real

8

0x0016

Quantisation scalar X-pol, real

8

0x0016

Quantisation scalar X-pol, real

8

0x0016

Quantisation scalar X-pol, real

Many such metadata packets will be sent to describe all the quantisation scalars for all ADCs, all polarisations, all channels. Each subsequent packet will increment the metadata packet counter field’s value by one, which allows the receiver to check that no metadata packets were missed. The length of each packet can be calculated based on the number of options in the preamble. The last packet will have the MSb in the Stream Control field set, indicating the end of the metadata bundle.

If only one packet is sent, the metadata packet counter will be zero and the MSb in Stream Control will be set.

During the normal course of operation, additional metadata packets may be received with updated values. In this case, the Stream Control field will have a value of 3 and the updated values should be recorded.

A.4 Data packets

Data packets will generally have only a small number of option fields that are required to unambiguously describe the payload of the particular packet in question.

Both the poco mode and packetised corr (correlator) instruments will emit packets with a data payload typically containing the following options, noting that the ordering of options may vary.

Number of bytes

Option ID

Representation

(MSB) 8

Header

8

0x0001

Instrument type

8

0x0002

Instance ID

8

0x0003

Timestamp

8

0x0004

Data payload length

8

0x0000

NULL

8

0x0005

Data payload offset (in bytes)

(LSB) 4096

Data

The receiver will use the payload ID that forms part of the instrument type to correctly decode the data payload. Note that a payload descriptor refers to a complete, unfragmented data block. Blocks of data that are too large for a single packet will get fragmented. The receiver can re-assemble these fragments using the payload length and payload offset information. Once unfragmented that block can be parsed using the supplied descriptor.

The length of the payload may vary, but is likely to be 2kiB, 4kiB or 8kiB.

The last packet of a data block may be shorter if there is not enough data to fill the packet. ie.

The fifth field is kept in reserve for future use.

Once data transmission is complete a Stream Stop packet is issued.