.. _HotRIO Protocol: HotRIO Protocol =============== Introduction ^^^^^^^^^^^^ The HotRIO communication protocol defines: - The physical layer - The way data is packed - The topology and the way data is routed - A clock/distribution mechanism. Scope ----- The scope of this document is the requirements definition for the FPGA firmware of the data packaging protocol of HotRIO communication links (layer 2 protocol) connecting one Master HotRIO and one or more Slave HotRIO modules (daisy chain connection). Hardware overview and physical layer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The protocol is not dependent on any vendor specific technology (hardware, software or firmware). As of today, the HotRIO communication protocol will be used in three different HW platforms, which are: - HotRIO modules with Zynq Ultrascale+ FPGA from Xilinx (XCZU9EG-1FFVC900E) - HotRIO modules with ECP5 FPGA from Lattice (ECP5 / ECP5-5G | Ultra Low Power FPGA | Lattice Semiconductor) - HotRIO modules with ECP5 FPGA from Lattice pluggable into a National Instruments CRIO chassis. These HW platforms will use the same physical layer for the HotRIO communication protocol: SFP+ fiber optic transceivers with following characteristics: - Cables: SM fiber, MM fibers and SFP+ DAC Twinax cables - Full duplex: SFP+ fiber pair - Half duplex: Just one fiber used Data packing ^^^^^^^^^^^^ Some characteristics of the HotRIO communications are common to Ethernet Gigabit communications: - 1Gb/s raw throughput (800Mbs net throughput) - 1.25GHz clock with 8b/10b encoding (8bits at 125MHz) Data will be continuously sent at 125MB/s (125MHz clock embedded) in frames of 50B following the structure defined in Figure 1. .. figure:: figures/Fig1_HotRIO_Frame_50B.png :align: center Figure 1.- Data Structure for HotRIO Protocol Communication Each frame will have: - A Frame Marker or sync character: 1B - A protocol ID marker: 1B - 4 Packets: 12B each The frame maker has two purposes: - To mark the beginning of each frame - To provide synchronization + timing features. The frame marker values are 8b10b valid sequences that do not correspond to any 8b valid data. The frame marker values for HotRIO communications are: - K28.5: used for sync or to flag a Start of Frame (SoF). - K28.1: used to mark 2 microseconds flag. - K28.2: used to mark millisecond flag. - K28.3: used to mark a second flag. The second byte of the frame will be the protocol ID: - "AA": for hotRIO protocol - "00": for any other protocol. After the protocol ID byte, 4 packets of 12B each will be sent, 48B in overall. Each packet will contain: - Address: 1B - Payload: 10B - Control: 1B Addresses conventions --------------------- Each slave node will have a unique address identifier. How this address is configured/assigned to each slave node is out of the scope of this document. The address of the packet defines to which node the data payload is destined for. - Address = 0 is reserved to address unconfigured modules (after power cycle) - Address = 128 is broadcast address to update clock. It is also used to create empty messages. When a slave node receives a packet with its own address, the node will transmit a new packet whose address will be its own plus an offset. The addresses of the packets included in a single HotRIO frame can be different. Control Byte ------------ Each Control Byte will have: - One parity bit for the odd bits of the address and packet payload (Po) - One parity bit for the even bits of the address and packet payload (Pe) - 2 reserved bits for future applications - 4 bits for side band communication for information to be exchanged between master micros and slave micros of the daisy chain. The bits structure of the control byte is shown in Figure 2 .. figure:: figures/Fig2_HotRIO_CONTROL_Byte_50B.png :align: center Figure 2.-Structure of Control Byte The four packets of each HotRIO frame will provide 16bits of sideband communication: - [15:12]: Arbitration bits - [11:8]: Control bits - [7:0]: Data There are no address bits in the sideband communication. The address will be inferred by each module (master and slave) with the ms flag of the HotRIO frame. Everytime the master or slave modules generates/receives the ms flag, an internal address counter going from '0' to '49' will be reset to '0'. During the ms flag, the sideband commincation will be meant for address '0' or broadcast. Next HotRIO frame for master will be meant for slave node with address '1', next HotRIO frame will be addressed to node '2' and so on. This internal counters also get reset when the address value reaches '49', i.e., every 50 HotRIO frames (20us). This imposes a limitation in the number of slaves (maximum 49). In case, lower number of slaves were connected to the daisy chain, this counter could be reduced to '0' to '24' (25 frames) and it would still be synchronous with the 1ms flag. The arbitration bits are: - SEL: controlled by the Master Node to enable/disable communication between Master and a given slave node. The Master FPGA will set this bit to '1' when the internal address counter of Master matches the slave address of interest and one or more of following conditions are met: * Outgoing queues of Master FPGA are not empty * Master wants to know RDY status of slave node to receive data from master. * Master wants to send data to the selected slave after having received a RDY status from that slave. - RDY: controlled by the Slave Node and used to inform Master that the Slave is ready to receive data. The slave sets this bit to '1' in outgoing HotRIO frame when the next 3 conditions are met: * Internal counter address matches the slave node address. * Incoming HR frame has SEL = '1' * Queue to receive data from sideband communication is NOT almost full, i.e., ready to receive information - REQ: controlled by the Slave Node and used to send data from Slave to Master. The slave sets this bit to '1' in outgoing HotRIO frame when the next 3 conditions are met: * Internal counter address matches the slave node address. * Incoming HR frame has SEL = '1' * Queue to send data from sideband communication is NOT empty, i.e., the slave micro has sent some information to the slave FPGA to be sent to master. When a slave address is selected (SEL = '1'), all the outgoing sideband DATA and CONTROL bits are meant for that address until: * SEL = '1' in a different address * SEL = '0' in selected address. The control bits (CONT) are used to indicate: - If the node is ready to receive data (either master or slave) - Whether there is data being transmitted or not. - Parity (odd/even) of transmitted DATA bits. Next table shows the values of CONT bits depending on each case: .. csv-table:: Table 1.- CONT bits of HotRIO sideband communication :file: tables/20240701_HotRIO_CONT_Bits_Sideband.csv :align: left :header-rows: 1 Master and Slave HotRIO ^^^^^^^^^^^^^^^^^^^^^^^ The HotRIO communication protocol will have two communication modes: - Master - Slave The Master HotRIO should generate the clock for the data transmission, while the Slave HotRIO should recover the clock from the incoming data. The Master HotRIO should also generate the marker (sync, 2us, 1ms and 1s) while the Slave HotRIO should transmit the marker that it has received. Figure 3 shows a representation of the connection between the Master and Slave modules with a daisy chain connection. .. figure:: figures/Fig3_HotRIO_DaisyChain_Connection.png :align: center Figure 3.- Master and Slave HotRIO modules connection (daisy chain) Master HotRIO ------------- The Master HotRIO transmitter needs to: - Provide transmission data stream to specific addresses (nodes). The generation of this data stream is out of the scope of this document. - Trigger specific data production in the slave nodes by transmitting the corresponding address node. The Master HotRIO receiver needs to: - Collect data from a set of addresses (nodes) (out of scope of this document). - Handle potentially different clock domains (transmitting and receiving). A possible FW implementation for the Master HotRIO Transmitter is provided as reference in next points. Other equivalent or better approaches might be proposed: Marker Generation ................. The Master HotRIO will generate its own clock (125MHz) and the markers for the “Start of Frame” (SoF) and synchronization (2us, ms and s). A process with a counter could be implemented in the FPGA to identify when the different time or synchronization marks should be sent. Also, external PPS input can/will be used to reset the timing flags counters (1s, 1ms, 2us). Control generation: parity bits ............................... There should be a process or equivalent to calculate the two parity bits of the address + payload of each packet of the frame. Master TX State Machine ....................... The main goal of this state machine is to sequentially send the 50 bytes composing the HotRIO frame to the PCS/SERDES of the FPGA. Possible names for this state machine could be: - FRAME: to send the required marker - ADDRESS: to send the address of the node. During this state, the payload should be sampled - DATA: to send the payload, one byte at a time - CONTROL: to send the control byte with updated parity bits. To produce a HotRIO Frame, the flow should be: FRAME -> 4 x [ADDRESS -> DATA (x10) -> CONTROL] Master RX State Machine ....................... A similar state machine could be built for the reception of frames in the master side: - FRAME: to identify the input marker - ADDRESS: to identify which node is providing data. During this state, the payload should be sampled. - DATA: to read the payload, byte by byte. - CONTROL: to check the SPI bits and to check parity bits to disregard/accept the received payload. The receiver clock will have the same frequency than the transmitter clock, although the phase might defer due to the different lenghts of fibers. Slave HotRIO ------------ Each Slave HotRIO module connected to the same daisy chain will have a unique address or identifier. The configuration of the nodes addresses is out of the scope of this document. The Slave HotRIO will continuously forward the stream of data from the RX to the TX. For doing so, the Slave HotRIO RX should use the retrieved clock from the incoming data and use it in the Slave HotRIO TX. When there is no incoming data in the Slave HotRIO Rx, the Slave Tx will start self-transmisting with its own clock and self-generated timing flags. When the protocol ID of the incoming frame is not HotRIO, the next 48B will be forwarded from RX to TX. When the protocol ID is HotRIO, the state machine will check the address of each packet before forwarding data from RX to TX. When the address of the incoming data does not match the address of the node, the Slave HotRIO TX will forward the received address + payload. The parity bits of the CONTROL byte are the only ones that are always re-calculated after reception and before re-transmission. If the incoming address matches the node address, then the incoming payload is replaced by the data of the Slave HotRIO and an offset is added to the TX address. A State Machine for the Slave HotRIO with similar states than the Master HotRIO could be implemented for the reception/transmission of the Slave HotRIO frames. The possible states could be: - FRAME: to receive/forward the input marker. - ADDRESS: if the incoming address is different from the node address, the system should forward the ADDRESS information to the TX. In case, the ADDRESS does match the node address, the offset will be added to the transmitted address. - DATA: Depending on the address value, in this state the system would forward/replace the incoming data. Payload should be transferred byte by byte. - CONTROL: In this state, the incoming parity bits should be checked and provide a “data_valid” signal. Also, the outcoming parity bits should be re-calculated based on incoming data. In case the TX is just forwarding the incoming data and one (or two) of the incoming parity bits is not correct, both transmitted parity bits of the CONTROL TX byte should be wrong. Tests ----- The generation of the data payload to be transmitted from Master to Slave HotRIO is out of the scope of this document. However for tests purposes, a bist function should be included in the FW for following test cases: - Pattern generation in Master HotRIO to be forwarded to the Slave module and from the Slave, back to the Master HotRIO. A check function of the received pattern generation should also be implemented in the Master. - Pattern generation in Slave HotRIO to be sent back to Master HotRIO when the Master sends the slave address. A check function of the received pattern in the Master should also be implemented.