Volume 06 Advanced 5 sub-modules ~60 min read

Zynq: Processing System + Programmable Logic

A Zynq is two computers on one die: hard ARM cores with their own DDR controller and peripherals, and an ordinary FPGA fabric. Neither half is unusual. All of the interesting engineering - and essentially all of the bugs - live in the seam between them, and most of that seam is about caches.

6.1 The architecture, and what boots first

The PS (processing system) is a hardened SoC: CPU cluster, caches, a DDR controller, and a fixed set of peripherals - UART, Ethernet, USB, SD, QSPI, timers. It exists whether or not you put anything in the fabric, and it is not configurable logic. The PL (programmable logic) is the FPGA from Volume 01, unchanged.

Zynq-7000 Zynq UltraScale+ MPSoC
Application cores 2 x Cortex-A9, up to ~1 GHz 2 or 4 x Cortex-A53, 64-bit
Real-time cores None 2 x Cortex-R5 (lock-step capable)
Cache 32 KB L1 each, 512 KB shared L2 32 KB L1 each, 1 MB shared L2
DDR DDR2/3/3L, 32-bit DDR3/4/LPDDR4, 32 or 64-bit
PS-PL AXI ports 2 GP master, 2 GP slave, 4 HP, 1 ACP Extended set, plus cache-coherent CCI ports
Fabric interrupts 16 into the GIC Larger, with more routing options
Zynq block diagram showing the processing system with CPU cluster, caches and DDR controller connected to programmable logic through general purpose, high performance and accelerator coherency ports PROCESSING SYSTEM (hard) CPU 0 + L1 32 KB D / 32 KB I CPU 1 + L1 32 KB D / 32 KB I SCU + L2 cache (512 KB) DDR controller GIC interrupt controller UART Eth USB SD QSPI timers runs with an empty fabric - the PS is not optional PROGRAMMABLE LOGIC your accelerator AXI DMA AXI4-Lite regs LUTs, BRAM, DSP M_AXI_GP S_AXI_HP S_AXI_ACP IRQ_F2P HP goes straight to the DDR controller. ACP goes through the SCU, so it is cache coherent.
Figure 6.1 - The seam. Note where each port lands: GP and HP bypass the caches entirely, ACP does not. That single difference is the source of most Zynq data-corruption bugs.
The PS boots the PL, not the other way round On power-up the on-chip BootROM loads the FSBL (first-stage boot loader) from QSPI, SD or JTAG. The FSBL initialises DDR and clocks, then programs the PL with the bitstream embedded in the boot image, then hands control to your application or U-Boot. Consequences worth internalising: the fabric is unconfigured for the first fraction of a second, PL logic must tolerate a PS that is already running, and if the FSBL fails, the fabric never gets programmed at all - which looks exactly like a broken bitstream.

6.2 GP, HP and ACP: choosing the right door

Port Direction Width Peak (7000, 150 MHz) Cache coherent?
M_AXI_GP0/1 PS is master, PL is slave 32-bit ~600 MB/s N/A
S_AXI_GP0/1 PL is master 32-bit ~600 MB/s No
S_AXI_HP0..3 PL is master 64-bit ~1200 MB/s each No
S_AXI_ACP PL is master 64-bit ~1200 MB/s Yes - through the SCU
The selection rule M_AXI_GP for control registers - it is a 32-bit path optimised for single accesses, and your AXI4-Lite slave from Volume 05 hangs off it. S_AXI_HP for bulk data - four independent 64-bit paths with their own FIFOs, straight into the DDR controller. S_AXI_ACP only when transfers are small, frequent and shared with CPU working data, because coherency is free there but the port steals L2 bandwidth from the cores and can evict their lines.

Do not put control registers on an HP port. HP is optimised for long bursts and has deep buffering; a single 32-bit register read through it has noticeably worse latency than the same read through GP, and you will have burned one of only four high-bandwidth paths on traffic that needed none of it.

6.3 Address maps and bare-metal drivers

Everything the PL exposes appears in the ARM's physical address space. Vivado's address editor assigns the ranges, exports them into xparameters.h, and from there your C code addresses your custom IP exactly like any other peripheral.

Range (Zynq-7000) Contents
0x0000_0000 - 0x3FFF_FFFF DDR memory
0x4000_0000 - 0x7FFF_FFFF PL slaves via M_AXI_GP0
0x8000_0000 - 0xBFFF_FFFF PL slaves via M_AXI_GP1
0xE000_0000 - 0xE02F_FFFF PS peripherals: UART, SPI, I2C, GPIO, Ethernet
0xF800_0000 - 0xF800_0BFF SLCR - clocks, resets, PS-PL configuration
0xFFFC_0000 - 0xFFFF_FFFF 256 KB on-chip memory (OCM)

#include "xparameters.h"
#include "xil_io.h"
#include "xil_cache.h"

/* Register map of the AXI4-Lite slave written in Volume 05. Offsets are
   byte addresses; the slave decoded bits [3:2] as the word index.       */
#define MYIP_BASE     XPAR_AXIL_REGS_0_S_AXI_BASEADDR   /* e.g. 0x43C00000 */
#define MYIP_CFG_A    0x00
#define MYIP_CFG_B    0x04
#define MYIP_CMD      0x08
#define MYIP_STATUS   0x0C

#define CMD_START     0x00000001u
#define STATUS_DONE   0x00000001u
#define STATUS_ERR    0x00000002u

/* volatile matters: without it the compiler will hoist the status read
   out of the polling loop and the function never returns.              */
static inline void  reg_write(u32 off, u32 v) { Xil_Out32(MYIP_BASE + off, v); }
static inline u32   reg_read (u32 off)        { return Xil_In32(MYIP_BASE + off); }

int myip_run(u32 a, u32 b, u32 timeout_iters)
{
    u32 status;

    reg_write(MYIP_CFG_A, a);
    reg_write(MYIP_CFG_B, b);
    reg_write(MYIP_CMD,   CMD_START);   /* self-clearing pulse in the PL */

    while (timeout_iters--) {
        status = reg_read(MYIP_STATUS);
        if (status & STATUS_ERR)  return -1;
        if (status & STATUS_DONE)  return 0;
    }
    return -2;                           /* never poll without a bound */
}
If every register reads back 0xFFFFFFFF or hangs Three causes, in order of likelihood. The PL is not programmed - the FSBL skipped the bitstream, so there is nothing at that address. The AXI clock is not running - FCLK_CLK0 is gated until the PS enables it, and a slave with no clock never asserts ARREADY. The address is outside any mapped range - the interconnect returns DECERR, which the ARM reports as a data abort or as all-ones. Check them in that order before touching your RTL.

6.4 DMA between the PL and DDR

Register access through GP moves tens of megabytes per second at best - every transaction costs a full AXI round trip and a CPU instruction. For real data you need DMA: the PL becomes an AXI master and writes DDR directly, at HP port speed, with the CPU uninvolved.

Mode How it works Use when
Simple (direct register) mode CPU writes a source address and a length, then waits One contiguous buffer at a time; simple, easy to debug
Scatter-gather mode CPU builds a linked list of descriptors in memory; the DMA walks it Many buffers, or continuous streaming with no CPU in the loop
Datamover / custom master Your own AXI4 master issues bursts directly Access patterns no packaged DMA expresses well

The bug that costs everyone a day

The ARM cores have write-back data caches. When your program fills a buffer, those writes may still be sitting in L1 or L2 - DDR has not been updated. An HP-port DMA reads DDR directly and sees stale data. In the other direction, after the DMA writes DDR, the CPU may still hold cached copies of those lines and read the old contents.

Cache coherency problem showing CPU writes held in cache while a high-performance port DMA reads stale data from DDR, and the flush and invalidate calls that fix each direction HP PORTS DO NOT SNOOP THE CACHES CPU writes buf[0..1023] = data L1 / L2 cache holds the NEW data DDR still holds OLD data not yet written back PL DMA via HP reads DDR directly reads stale bytes before PL reads: Xil_DCacheFlushRange((UINTPTR)buf, len); before CPU reads: Xil_DCacheInvalidateRange((UINTPTR)buf, len); the ACP port avoids all of this - at the cost of L2 bandwidth shared with the cores
Figure 6.4 - Flush before the fabric reads, invalidate before the CPU reads. Two lines of code, and the symptom of omitting them is data that is correct in simulation, correct with the cache disabled, and wrong on the board.

#include "xaxidma.h"
#include "xil_cache.h"

static XAxiDma dma;

/* ---- PL reads a buffer the CPU just filled -------------------------- */
int send_to_pl(u8 *buf, u32 len)
{
    int status;

    /* Push our dirty cache lines out to DDR FIRST. Skip this and the
       DMA reads whatever was in DDR before - often zeros, sometimes
       last frame's data, always confusing.                             */
    Xil_DCacheFlushRange((UINTPTR)buf, len);

    status = XAxiDma_SimpleTransfer(&dma, (UINTPTR)buf, len,
                                    XAXIDMA_DMA_TO_DEVICE);
    if (status != XST_SUCCESS) return status;

    while (XAxiDma_Busy(&dma, XAXIDMA_DMA_TO_DEVICE)) { }
    return XST_SUCCESS;
}

/* ---- CPU reads a buffer the PL just wrote --------------------------- */
int recv_from_pl(u8 *buf, u32 len)
{
    int status;

    status = XAxiDma_SimpleTransfer(&dma, (UINTPTR)buf, len,
                                    XAXIDMA_DEVICE_TO_DMA);
    if (status != XST_SUCCESS) return status;

    while (XAxiDma_Busy(&dma, XAXIDMA_DEVICE_TO_DMA)) { }

    /* Throw away any cached copies of these lines so the next load
       actually goes to DDR.                                            */
    Xil_DCacheInvalidateRange((UINTPTR)buf, len);
    return XST_SUCCESS;
}
Align your buffers to a cache line Xil_DCacheInvalidateRange operates on whole 32-byte cache lines. If your buffer starts mid-line, invalidating it discards the other variables sharing that line - silently corrupting data that had nothing to do with the transfer. Declare DMA buffers with __attribute__((aligned(64))) and round the length up. This is a real bug, it is intermittent, and it is almost impossible to find by reading code.

6.5 Interrupts from fabric to CPU

Polling a status register wastes a CPU core and adds latency. An interrupt from the PL lets the fabric tell the processor when something happened.

The path is: your logic drives a bit of IRQ_F2P on the Zynq PS block; that maps onto a shared peripheral interrupt in the GIC; the GIC dispatches to a handler you registered. On Zynq-7000 the mapping is fixed:

IRQ_F2P bit GIC interrupt ID Notes
[0][7] 61 … 68 The usual choice
[8][15] 84 … 91 Note the gap - the IDs are not contiguous

// Level-sensitive interrupt: assert on completion, hold until the ISR
// clears it by writing 1 to the status bit. This is the shape the GIC
// expects for a peripheral with a status register.
always_ff @(posedge clk) begin
  if (!rstn) begin
    irq        <= 1'b0;
    done_latch <= 1'b0;
  end else begin
    if (engine_done)              done_latch <= 1'b1;   // set on event
    else if (clear_done_w1c)      done_latch <= 1'b0;   // cleared by SW

    irq <= done_latch && irq_enable;
  end
end

#include "xscugic.h"

static XScuGic gic;

/* The handler runs in interrupt context. Keep it short: clear the
   source, set a flag, return. Never block, never printf.              */
static void myip_isr(void *arg)
{
    u32 status = reg_read(MYIP_STATUS);

    if (status & STATUS_DONE) {
        reg_write(MYIP_STATUS, STATUS_DONE);   /* write-1-to-clear     */
        transfer_complete = 1;                  /* volatile global      */
    }
}

int irq_init(void)
{
    XScuGic_Config *cfg = XScuGic_LookupConfig(XPAR_SCUGIC_SINGLE_DEVICE_ID);
    XScuGic_CfgInitialize(&gic, cfg, cfg->CpuBaseAddress);

    /* Priority 0xA0, trigger 0x3 = rising edge. Use 0x1 for level-high
       if your PL holds irq until software clears it.                  */
    XScuGic_SetPriorityTriggerType(&gic, XPAR_FABRIC_MYIP_IRQ_INTR,
                                   0xA0, 0x3);

    XScuGic_Connect(&gic, XPAR_FABRIC_MYIP_IRQ_INTR,
                    (Xil_ExceptionHandler)myip_isr, NULL);
    XScuGic_Enable(&gic, XPAR_FABRIC_MYIP_IRQ_INTR);

    Xil_ExceptionRegisterHandler(XIL_EXCEPTION_ID_INT,
        (Xil_ExceptionHandler)XScuGic_InterruptHandler, &gic);
    Xil_ExceptionEnable();
    return XST_SUCCESS;
}
The interrupt storm A level-sensitive interrupt that the handler does not clear fires again the instant the handler returns, forever. The board appears frozen; the CPU is in fact executing your ISR several million times a second. Two ways to avoid it: clear the source before returning, or make the fabric interrupt a single-cycle pulse and configure the GIC for rising-edge trigger. Mixing them - a pulse with a level-configured GIC - gives the opposite failure, where the interrupt is missed entirely if the pulse lands while the GIC is busy.
Interview grilling - "Design the PS-PL interface for a 1080p60 video accelerator."

Do the bandwidth arithmetic out loud before proposing anything:

  • 1920 × 1080 × 60 fps × 4 bytes/pixel ≈ 498 MB/s in one direction.
  • Read a frame and write a frame: ~1 GB/s total.
  • One HP port peaks at ~1.2 GB/s. Running a single port at 83% of theoretical is not realistic once DDR refresh and CPU traffic are included - so use two HP ports, one for read and one for write.

Then the structure:

  • Control: AXI4-Lite on M_AXI_GP0 - frame pointers, size, enable, status.
  • Data in: VDMA or AXI DMA on S_AXI_HP0, converted to AXI-Stream for the processing pipeline.
  • Data out: a second DMA on S_AXI_HP1.
  • Sync: one IRQ_F2P line per frame completion, so software can flip buffers.
  • Buffers: triple-buffered in DDR, 64-byte aligned, with flush before the PL reads and invalidate after it writes - or mark the frame buffers non-cacheable outright, which for streaming video the CPU never touches is usually the better call.

The point that separates a strong answer: "I would not use ACP here. Coherency sounds convenient, but pushing 1 GB/s of video through the snoop control unit would evict the CPUs' working set continuously and make everything else on the system slower."

Volume 06 recap

Concept The one thing to remember
Boot order BootROM → FSBL → bitstream → app. The PS programs the PL.
GP ports 32-bit, for control registers. Not for data.
HP ports 64-bit to DDR, ~1.2 GB/s each, not coherent.
ACP port Coherent, but steals L2 bandwidth from the cores.
Cache Flush before the PL reads. Invalidate before the CPU reads.
Buffer alignment 64-byte aligned, or invalidation corrupts a neighbour.
Interrupts IRQ_F2P[7:0] → IDs 61-68. Clear the source in the ISR.