On-Hardware Debug & Bring-Up
A simulator shows you every signal in the design and proves nothing about the silicon.
Hardware proves everything and shows you almost nothing - a handful of LEDs and whatever
you had the foresight to route to a probe. This volume is about closing that gap
deliberately, instead of by adding $display statements to a testbench that
already passes.
7.1 The ILA: insertion, probes and depth
The Integrated Logic Analyzer is a small logic analyser you build into the fabric. It watches signals on a clock you nominate, writes them into block RAM every cycle, stops on a trigger condition, and ships the captured window out over JTAG.
Two ways to insert one
Netlist insertion (MARK_DEBUG) |
IP instantiation | |
|---|---|---|
| How | Attribute in RTL, then Set Up Debug after synthesis | ila_0 instance wired in your source |
| Re-run cost | Implementation only | Full synthesis |
| Signals reachable | Anything surviving optimisation | Whatever you wired |
| Best for | Exploratory debug - you do not yet know what to look at | Permanent instrumentation you want in every build |
// MARK_DEBUG implies DONT_TOUCH on the net, so the signal survives
// optimisation and is selectable in the Set Up Debug wizard.
(* MARK_DEBUG = "TRUE" *) logic [ 7:0] fsm_state;
(* MARK_DEBUG = "TRUE" *) logic [31:0] axi_awaddr_dbg;
(* MARK_DEBUG = "TRUE" *) logic axi_awvalid_dbg;
(* MARK_DEBUG = "TRUE" *) logic axi_awready_dbg;
// Instantiated form, for instrumentation you keep permanently.
ila_0 u_ila (
.clk (axi_clk), // MUST be the clock these signals live on
.probe0 (fsm_state), // 8 bits
.probe1 (axi_awaddr_dbg), // 32 bits
.probe2 (axi_awvalid_dbg), // 1 bit
.probe3 (axi_awready_dbg) // 1 bit
);
What an ILA actually costs
ASYNC_REG on a clock crossing, or a state element with no defined
reset value. Go and check those three rather than concluding the bug was imaginary.
7.2 Triggers, capture control and storage
A 1024-sample window at 200 MHz covers 5.1 microseconds. The fault you are chasing happens once every four minutes. Triggering is the entire game.
| Control | What it does | When it saves you |
|---|---|---|
| Basic trigger | AND/OR across per-probe comparisons | You know exactly what the bad state looks like |
| Trigger position | Where in the window the trigger sits | Set it to ~75% to see the cause, not just the effect |
| Storage qualification | Only store a sample when a condition holds | Sparse traffic - the single highest-leverage setting |
| Advanced trigger FSM | Up to 16 states with counters and flags | "The third error after a reset", sequence-dependent faults |
| Cross-trigger | One ILA arms another, or the CPU | Correlating fabric events with software |
# Driving the ILA from Tcl instead of the GUI - scriptable, repeatable,
# and the only sane way to run the same capture fifty times overnight.
set ila [get_hw_ilas hw_ila_1]
# ---- storage qualification: only record real transactions ------------
set_property CONTROL.CAPTURE_MODE BASIC $ila
set_property CAPTURE_COMPARE_VALUE eq1'b1 [get_hw_probes u_dut/tvalid -of_objects $ila]
# ---- trigger: an error flag, with 75% of the window BEFORE it --------
set_property CONTROL.TRIGGER_POSITION 768 $ila ;# of 1024
set_property TRIGGER_COMPARE_VALUE eq1'b1 [get_hw_probes u_dut/err_sticky -of_objects $ila]
# ---- arm, wait, save --------------------------------------------------
run_hw_ila $ila
wait_on_hw_ila $ila
display_hw_ila_data [upload_hw_ila_data $ila]
write_hw_ila_data -force capture_run1.ila $ila
7.3 VIO and runtime control
An ILA observes. Two other cores let you act on a running design without rebuilding the bitstream, and together they turn a board into something you can poke interactively.
| Core | What it gives you | Limitations |
|---|---|---|
| VIO (Virtual I/O) | Drive values into the fabric and read values out, from the Vivado hardware manager | Updates at JTAG speed - milliseconds, not cycles. Control only, never data. |
| JTAG-to-AXI Master | Issue real AXI reads and writes from the Tcl console | One transaction at a time; slow, but it is genuine AXI traffic |
| Cross-trigger | Fabric event halts the CPU, or a software breakpoint arms the ILA | Zynq only; needs the debug bridge in the block design |
// A VIO gives you a soft reset, a mode switch and a live error counter
// without rebuilding. On a board with no buttons, this is the difference
// between a five-minute experiment and a forty-minute rebuild.
vio_0 u_vio (
.clk (sys_clk),
.probe_in0 (err_count), // 16 bits, read back in the GUI
.probe_in1 (fsm_state), // 8 bits
.probe_out0 (vio_soft_rst), // 1 bit, you drive it
.probe_out1 (vio_test_mode) // 2 bits
);
// IMPORTANT: a VIO output is asynchronous to your logic - it changes
// whenever the JTAG chain happens to update. Treat it as a clock domain
// crossing, exactly as Volume 02 insisted.
(* ASYNC_REG = "TRUE" *) logic rst_meta, rst_sync;
always_ff @(posedge sys_clk) begin
rst_meta <= vio_soft_rst;
rst_sync <= rst_meta;
end
# Read and write your AXI4-Lite slave with no CPU, no software build and
# no bitstream change. Invaluable for proving an address map before the
# firmware team has anything to run.
set jtag [get_hw_axis hw_axi_1]
# Write 0xDEADBEEF to the config register at 0x43C0_0000
create_hw_axi_txn wr $jtag -address 43C00000 -data DEADBEEF -type write
run_hw_axi wr
# Read the status register back
create_hw_axi_txn rd $jtag -address 43C0000C -type read
run_hw_axi rd
puts "status = [get_property DATA [get_hw_axi_txns rd]]"
7.4 Catching rare and intermittent faults
The hard bugs are not the ones that fail every time. They are the ones that fail once an hour, under load, on one board out of six. Four techniques cover almost all of them.
1. Make the fault sticky in RTL
An ILA cannot trigger on something that lasted one cycle four seconds ago. Latch it. Sticky error flags cost a handful of flip-flops and are the single most useful piece of permanent instrumentation you can build.
// Permanent instrumentation. Leave this in the shipping design - it is
// tiny, and the first question after any field failure is "which of
// these bits is set?"
logic [7:0] err_sticky;
always_ff @(posedge clk) begin
if (!rstn || err_clear) begin
err_sticky <= 8'h00;
end else begin
if (fifo_wr && fifo_full) err_sticky[0] <= 1'b1; // overflow
if (fifo_rd && fifo_empty) err_sticky[1] <= 1'b1; // underflow
if (axi_bresp != 2'b00) err_sticky[2] <= 1'b1; // slave error
if (crc_fail) err_sticky[3] <= 1'b1;
if (state == S_ILLEGAL) err_sticky[4] <= 1'b1; // FSM fell out
if (timeout_expired) err_sticky[5] <= 1'b1;
if (pkt_len_bad) err_sticky[6] <= 1'b1;
if (drop_count != 16'd0) err_sticky[7] <= 1'b1;
end
end
// Expose err_sticky through the AXI4-Lite status register AND probe it
// with the ILA. Trigger on any bit going high.
2. Count, do not just capture
Free-running counters - transactions accepted, packets dropped, cycles stalled, maximum FIFO occupancy seen - cost almost nothing and answer questions no waveform can. "The FIFO reached depth 61 of 64 at some point in the last hour" is a fact a 1024-sample window will never give you.
3. Use the advanced trigger state machine
"Trigger on the third underflow after a soft reset" is one screen of trigger-FSM code and impossible with a basic trigger. The ILA's trigger state machine supports counters, flags and up to sixteen states - enough to describe most sequence-dependent faults exactly.
4. Suspect timing before you suspect logic
| Symptom | Most likely cause | First check |
|---|---|---|
| Fails on one board, works on five | Marginal timing, or a real signal-integrity issue | WNS/WHS margin; then swap the board and the cable |
| Fails after minutes to hours | Metastability on a clock crossing | Every CDC has ASYNC_REG and a clock group |
| Fails only when hot | Timing margin closing up over temperature | Was the design signed off at the right speed grade and corner? |
| Fails only at high traffic | Back-pressure path never exercised in simulation | FIFO full/empty handling; the tready logic |
| Works with the ILA, fails without | Placement-dependent timing | The failing path's slack, and any missing reset |
| Fails right after power-up only | Reset released before clocks are stable | MMCM locked gating the reset release |
7.5 A systematic bring-up method
A new board that does nothing is not a debugging problem, it is an ordering problem. Each layer depends on the one below it, and reaching for an ILA before the clock is confirmed wastes an afternoon staring at waveforms that are all wrong for the same reason.
| Step | Question | How to answer it |
|---|---|---|
| 1. Power | Are the rails up and is DONE high? |
The DONE LED. If it is low, configuration failed - nothing else can work. |
| 2. Clock | Is the clock present, and at the frequency you constrained? | A counter dividing to 1 Hz on an LED. If it blinks twice as fast as expected, your create_clock is wrong. |
| 3. Reset | Has reset actually been released? | Route it to an LED. Active-low resets get inverted by accident constantly. |
| 4. Configuration | Can you read back a known value? | A scratch register that returns 0xC0DE1234. Proves clock, reset, decode and AXI in one read. |
| 5. Loopback | Does the shortest possible path through the design work? | Write a value in, read it straight back out, bypassing the processing. |
| 6. Datapath | Where does correct data become incorrect? | Now the ILA, probing at the boundary you suspect. |
# In build.tcl, before synthesis: bake the git hash into a parameter.
set githash [string range [exec git rev-parse HEAD] 0 7]
set_property generic "BUILD_ID=32'h$githash" [current_fileset]
# In RTL:
# parameter logic [31:0] BUILD_ID = 32'hDEADBEEF;
# ... 2'd3: s_rdata <= BUILD_ID;
#
# On the bench:
# create_hw_axi_txn rd $jtag -address 43C0000C -type read
# run_hw_axi rd
# -> if this does not match your working tree, stop debugging.
Interview grilling - "It works in simulation but not on hardware. Walk me through it."
The strongest answer names the categories, because the sentence itself tells you the bug is in something simulation cannot see. There are only five such categories:
- Timing. RTL simulation has no delays. Check WNS and WHS first - and check that the constraints describe the design, which is a different question from whether they are met.
- Clock domain crossings. Simulation resolves races deterministically; silicon does not. Every crossing needs a synchroniser and a constraint.
- Reset and initialisation. Testbenches reset everything neatly. Real hardware powers up with an MMCM that is not yet locked and a reset that may be released asynchronously.
- The real world. The external device does not behave the way your behavioural model did, the board has a floating pin, the I/O standard is wrong, or a trace is not terminated.
- Synthesis-simulation mismatch. An incomplete sensitivity list, a
latch you did not intend, an
initialblock that is simulation-only, or an X that simulation optimistically resolved to a real value.
The close: "And I would treat the fact that it passes in simulation as information - it tells me the logic is probably right, which moves timing, CDC and reset to the top of the list rather than the bottom."
Volume 07 recap
| Concept | The one thing to remember |
|---|---|
| ILA cost | width × depth bits of real BRAM. Budget it. |
| ILA clocking | One clock per ILA. Two domains means two ILAs. |
| Heisenbug | If probing hides it, suspect timing, CDC or reset. |
| Storage qualification | Worth more than 10× the capture depth on idle buses. |
| Trigger position | 75% pre-trigger. You want the cause, not the effect. |
| Sticky flags | Latch every error. Ship them. Read them first. |
| Bring-up order | Power → clock → reset → readback → loopback → datapath. |
| Build ID | 32 bits that end the "which bitstream is this" argument. |