Volume 01 Intermediate 5 sub-modules ~55 min read

FPGA Architecture from the Inside Out

An ASIC designer starts with a blank wafer and a standard cell library. You do not. Your silicon was fabricated years ago, and it is a fixed grid of six-input lookup tables, hardened carry chains, 36-kilobit memories and 48-bit multiply-accumulate slices. RTL that matches that grid closes timing on the first run. RTL that ignores it burns four times the area and fails timing by 3 ns - for exactly the same logical function.

Where this volume sits This course assumes you can already write clean synchronous RTL - if always_ff, non-blocking assignment and reset strategy are not yet automatic, work through the Verilog & SystemVerilog masterclass first. Everything here is about the physical target: what the fabric already contains, and what your code has to look like to land on it.

1.1 LUTs, flip-flops, slices and CLBs

There are no gates in an FPGA. There is no AND gate, no OR gate, no XOR gate anywhere in the fabric. What exists is a lookup table: a tiny block of SRAM that stores the complete truth table of whatever function you asked for, addressed by that function's inputs.

A 6-input LUT (LUT6) is 64 SRAM cells and a 64-to-1 multiplexer tree. The six inputs are the select lines. At configuration time the bitstream writes your truth table into those 64 cells; at run time the inputs pick one out. The consequence is the single most important fact in FPGA design:

Every 6-input function costs the same y = a & b & c & d & e & f and y = (a ^ b) | (c & ~d) ^ (e & f) both occupy exactly one LUT6 and incur exactly one LUT delay. Boolean simplification that reduces gate count in an ASIC buys you nothing here unless it reduces the number of inputs. Count inputs, not operators.
A 6-input lookup table shown as 64 SRAM configuration cells feeding a multiplexer tree CONFIGURATION SRAM 64 x 1-bit cells INIT[0] = 1 INIT[1] = 0 INIT[2] = 0 . . . INIT[41] = 1 . . . INIT[63] = 0 written once by the bitstream MULTIPLEXER TREE 64 : 1 6 stages A[5:0] the address = your inputs O6 FF optional clk Q combinational bypass
Figure 1.1 - A LUT6 is 64 configuration bits and a mux tree. Your six inputs are the address; the output is whichever truth-table bit they select. The flip-flop that follows it is separate silicon and can be used or bypassed independently.

Slices, CLBs and what "one LUT" really costs

LUTs are not scattered loose. In 7-series and UltraScale devices four LUT6s and eight flip-flops are packaged into a slice, and slices are grouped into CLBs (configurable logic blocks). Two flavours exist:

Resource Contents Extra capability
SLICEL 4 x LUT6, 8 x FF, carry chain, wide-mux (F7/F8) Logic only - the majority of the fabric
SLICEM Everything a SLICEL has LUTs can also be distributed RAM or a 32-bit shift register (SRL32)
CLB 1 slice (UltraScale) or 2 slices (7-series) The unit the placer actually moves around

Each LUT6 is internally fracturable. It can implement one 6-input function, or two independent 5-input functions that share the same five inputs, exposed as the O6 and O5 outputs. This is why a utilisation report can show more logic functions than LUTs: the tool packed two small ones into a single site.

Wider functions are built with the dedicated wide multiplexers. A 7-input function costs two LUT6s plus one MUXF7; an 8-input function costs four LUT6s plus two MUXF7 and one MUXF8. Those muxes are hard silicon inside the slice, so the extra levels are cheap - but they are still extra levels of delay.


module lut_cost (
  input  logic [7:0] a,
  output logic       f6, f7, f8
);

  // ---- 6 inputs: ONE LUT6, one LUT delay -------------------------------
  // The expression looks expensive. It is not. Six inputs => one truth
  // table => one LUT. The tool computes INIT[63:0] and moves on.
  assign f6 = (a[0] ^ a[1]) | (a[2] & ~a[3]) ^ (a[4] & a[5]);

  // ---- 7 inputs: TWO LUT6 + one MUXF7 ----------------------------------
  // a[6] cannot fit in the same truth table, so the tool builds the
  // function twice (once for a[6]=0, once for a[6]=1) and selects.
  assign f7 = (a[0] ^ a[1]) | (a[2] & ~a[3]) ^ (a[4] & a[5]) ^ a[6];

  // ---- 8 inputs: FOUR LUT6 + two MUXF7 + one MUXF8 ---------------------
  assign f8 = ^a;   // 8-input XOR reduction

endmodule
The fan-in trap A priority encoder over 64 requests, a 64-way one-hot mux, a "flag OR" across a hundred status bits - all of these look like one line of RTL and all of them build a tree of LUTs six inputs wide per level. Sixty-four inputs needs ceil(log₆64) = 3 levels minimum, more once routing gets involved. If a path with a huge fan-in is failing timing, the fix is a pipeline register in the middle of the tree, not a cleverer Boolean expression.
Interview grilling - "Why is LUT count a bad proxy for design complexity?"

Because the LUT is a fixed-size container, not a measure of work. Three separate realities hide behind one number:

  • Packing. Two 5-input functions sharing inputs land in one LUT6. The same two functions with disjoint inputs need two. Identical "complexity", double the count.
  • Control sets. Flip-flops in a slice share clock, clock-enable and set/reset. A design with many distinct enable signals cannot fill its slices, so LUT utilisation stays low while the device is effectively full. Vivado reports this separately as control set count - that is the number to watch.
  • Routing. Congestion, not logic, is what usually stops a large design. A 60%-utilised device can be unroutable if the connectivity is bad.

The answer that lands: "LUT count tells me whether the design fits. Control sets and routing congestion tell me whether it will close timing."

1.2 Carry chains and hard arithmetic

Volume 02 of the Verilog course spends considerable time on carry-lookahead adders - the classic answer to ripple-carry delay. On an FPGA, writing one is a mistake.

Every slice contains a dedicated carry chain (CARRY4 in 7-series, CARRY8 in UltraScale). It is hard silicon: a chain of muxes and XORs with a private, ultra-fast vertical route from one slice to the slice directly above it. The carry does not go through the general routing fabric at all, and it does not go through a LUT. Propagation from one bit to the next costs on the order of 10-30 picoseconds.

Dedicated carry chain propagating vertically through stacked slices, compared with a LUT-built adder using general routing DEDICATED CARRY CHAIN CARRY4 bits 3:0 LUTs generate propagate CARRY4 bits 7:4 same column, next slice CARRY4 bits 11:8 CI to CO, no fabric ~20 ps ~20 ps 32-bit add: ~1.2 ns total HAND-WRITTEN CLA IN LUTS LUT6 LUT6 LUT6 LUT6 ~600 ps routing each hop leaves the slice 32-bit add: 3-6 ns, 3x the LUTs
Figure 1.2 - The carry chain is a private vertical wire between stacked slices. A carry-lookahead structure written by hand is built out of ordinary LUTs joined by ordinary routing, which is where all the delay actually lives on an FPGA.

// Textbook carry-lookahead. Correct,
// and wrong for this target.
logic [31:0] g, p, c;
assign g = a & b;
assign p = a ^ b;

always_comb begin
  c[0] = cin;
  for (int i = 0; i < 31; i++)
    c[i+1] = g[i] | (p[i] & c[i]);
end
assign sum = p ^ c;

// Synthesis result: ~96 LUTs, four
// levels of general routing, and the
// carry chain sits completely unused.

// One line. The tool maps this onto
// CARRY4/CARRY8 primitives directly.
assign {cout, sum} = a + b + cin;

// Synthesis result: 32 LUTs (for the
// propagate terms) + 8 CARRY4 sites,
// and the whole 32-bit carry resolves
// in about 1.2 ns.
//
// Same rule for subtract, compare,
// increment and accumulate - write
// the operator, not the structure.

What else rides the carry chain

The chain is not just for +. Vivado maps a surprising range of operators onto it, and knowing which ones are cheap changes how you write control logic:

RTL Maps to Cost for 32 bits
a + b, a - b Carry chain ~32 LUT + 8 CARRY4, ~1.2 ns
a < b, a >= b Carry chain (subtract, keep only carry-out) Same as an adder
a == b LUT XOR tree - not the carry chain ~11 LUT, 2 levels
cnt <= cnt + 1 Carry chain with one operand tied high Cheapest sequential structure on the device
a * b DSP48 slice (see §1.4) Zero LUTs if it fits the DSP
Why a wide counter is hard to place badly A carry chain is physically vertical. A 64-bit counter must occupy 16 consecutive CARRY4 sites in one column, and the placer has no freedom to spread them out. That is normally a gift - it guarantees a short path - but it also means a very wide counter can force a tall, thin placement that drags its surrounding logic along. If a 128-bit counter is distorting your floorplan, split it into two 64-bit halves joined by a registered carry.
Interview grilling - "Your 32-bit comparator fails timing. What do you do?"

First establish which comparison it is, because the two map to completely different silicon:

  • Magnitude (<, >) uses the carry chain and is already near-optimal. If it fails, the problem is almost certainly the logic feeding it or consuming its result, not the comparator.
  • Equality (==) is a LUT XOR tree, roughly two levels for 32 bits. Failing here usually means high fan-in on the result - one comparator driving thirty destinations.

Then the actual fixes, in order of preference: register the operands so the compare starts at a flop; pipeline the compare into two stages (upper half, lower half, combine); or, if it is an address decode, restructure so only the bits that matter are compared.

What interviewers listen for is whether you check the timing report path before changing code. "I would open the failing path in report_timing and see whether the delay is in logic levels or in routing" is the answer that separates people who have closed timing from people who have read about it.

1.3 Distributed RAM, block RAM and UltraRAM

There are three completely different ways to store data in an FPGA, and choosing wrongly is one of the most common causes of a design that "should fit" but does not. The choice is driven by depth, not by how the memory feels conceptually.

Registers Distributed RAM Block RAM UltraRAM
Built from Slice flip-flops SLICEM LUTs Hard BRAM36 tiles Hard URAM288 tiles
Typical size < 64 bits total 16-256 words Up to 36 Kb per tile 288 Kb per tile
Read latency 0 (it is the register) 0 - asynchronous read 1 cycle (+1 if output reg) 1 cycle (+1 output reg, effectively mandatory)
Write Synchronous Synchronous Synchronous Synchronous
Ports Unlimited reads 1 write + 1-3 reads True dual-port Two ports, same clock only
Initial contents Yes Yes Yes (INIT / $readmemh) No - powers up as zeros
Available on Every device Every device Every device UltraScale+ only

A block RAM in 7-series is a 36 Kb tile that can also be split into two independent 18 Kb halves. Its width and depth trade off against each other - 32K x 1, 4K x 9, 1K x 36 and so on. That aspect-ratio flexibility is why a 1024-deep by 32-wide memory costs exactly one BRAM36 while a 1025-deep by 32-wide memory costs two: you crossed a boundary the silicon cannot bend.

Round down to the tile, not up to the nearest power of two Memory sizing on an FPGA is quantised. A 33-bit wide, 1024-deep buffer needs two BRAM36 tiles because 33 x 1024 = 33,792 bits exceeds one tile's usable 32 Kb in the x36 aspect ratio. Dropping one bit of width - often a parity or valid flag you could carry separately - halves the memory cost. Always check the utilisation report against what you expected; a surprise 2x usually means a boundary crossed by a single bit.

The one coding rule that decides everything

Whether you get a block RAM or a pile of registers comes down to a single property of your RTL: is the read address registered? A block RAM physically cannot do an asynchronous read. If your code demands one, the tool has no choice but to build the memory out of LUTs and flip-flops instead.


logic [31:0] mem [0:1023];
logic [31:0] rdata;

always_ff @(posedge clk)
  if (we) mem[waddr] <= wdata;

// Asynchronous read: rdata changes
// the instant raddr changes.
assign rdata = mem[raddr];

// BRAM cannot do this. Vivado builds
// 1024 x 32 = 32768 flip-flops plus a
// 1024-way mux. On a mid-range part
// that alone is most of the device.

logic [31:0] mem [0:1023];
logic [31:0] rdata;

always_ff @(posedge clk) begin
  if (we) mem[waddr] <= wdata;
  rdata <= mem[raddr];      // 1 cycle
end

// Exactly one BRAM36. Zero LUTs,
// zero flip-flops from the array.
//
// The cost is one cycle of latency,
// which your datapath must absorb --
// that is the whole trade.

Block RAM output also has an optional output register inside the tile. Using it adds a second cycle of read latency and buys a substantially better clock-to-out, which is frequently the difference between 200 MHz and 400 MHz on a memory-heavy design. Vivado infers it when you register the memory output one more time in RTL and the register has no reset:


module bram_pipelined #(
  parameter int AW = 10,
  parameter int DW = 32
) (
  input  logic          clk,
  input  logic          we,
  input  logic [AW-1:0] addr,
  input  logic [DW-1:0] wdata,
  output logic [DW-1:0] rdata      // valid 2 cycles after addr
);

  (* ram_style = "block" *)
  logic [DW-1:0] mem [0:(1<<AW)-1];

  logic [DW-1:0] mem_q;   // BRAM internal output latch
  logic [DW-1:0] out_q;   // BRAM optional output register

  always_ff @(posedge clk) begin
    if (we) mem[addr] <= wdata;
    mem_q <= mem[addr];
    out_q <= mem_q;       // NO reset here - a reset forces this
  end                     // register out into the fabric

  assign rdata = out_q;

endmodule
Never reset a block RAM output register you want inside the tile The BRAM output register does support a reset, but attaching a global asynchronous reset to it, or resetting it to a value the primitive cannot produce, forces Vivado to pull the register out of the tile and rebuild it in slice flip-flops. You lose the timing benefit and gain 32 flops. If you need the data path cleared, gate the consumer with a valid flag instead of resetting the memory output. We return to reset discipline in §1.5 and again in Volume 04.
Interview grilling - "When is distributed RAM the right answer?"

Three situations, and they are all about shape rather than size:

  • Shallow and wide. A 16-entry x 64-bit register file costs one BRAM (99% wasted) or eight SLICEM LUTs. Distributed RAM wins on area by a wide margin.
  • Asynchronous read genuinely required. A CPU register file that must present two operands in the same cycle as the decode cannot tolerate BRAM latency. Distributed RAM reads combinationally.
  • You have run out of BRAM. On a congested design, moving small FIFOs and lookup tables to distributed RAM frees whole tiles for the big buffers that actually need them.

The counter-point to raise yourself: distributed RAM consumes SLICEM sites, and SLICEMs are roughly half the fabric. Filling them with memory removes the sites the placer wanted for your shift registers and wide muxes. It is a trade, not a free lunch.

1.4 DSP slices and hardened macros

The DSP slice is the most under-used resource on most FPGAs. It is not a multiplier - it is a small fixed-function ALU with a multiplier in the middle, and it will happily do work that would otherwise consume hundreds of LUTs.

A DSP48E1 (7-series) or DSP48E2 (UltraScale+) contains, in order along its datapath:

DSP48 slice datapath showing pre-adder, multiplier, ALU, accumulator and pattern detector with pipeline register stages A D B C PRE-ADD A + D MULT 25 x 18 signed ALU 48-bit + - and or xor P REG 48-bit P accumulate feedback (P into ALU) AREG/BREG MREG PREG
Figure 1.4 - The DSP48 datapath. The gold dashed lines are the optional pipeline register stages. Using all three is what takes the slice from roughly 250 MHz to its full rated speed.

The multiplier is asymmetric: 25 bits by 18 bits on 7-series, 27 by 18 on UltraScale+. A 32-bit by 32-bit multiply does not fit and is decomposed into four DSP slices plus adders. That is fine - but it is worth knowing before you write logic [63:0] prod = a32 * b32; in an inner loop and wonder where 40 DSPs went.


// A fully pipelined multiply-accumulate that maps to ONE DSP48
// with every internal register stage used.
module mac_pipelined (
  input  logic               clk,
  input  logic               clr,      // clear the accumulator
  input  logic signed [24:0] a,
  input  logic signed [17:0] b,
  output logic signed [47:0] acc
);

  logic signed [24:0] a_q;
  logic signed [17:0] b_q;
  logic signed [42:0] m_q;

  always_ff @(posedge clk) begin
    a_q <= a;                    // AREG  - DSP input register
    b_q <= b;                    // BREG  - DSP input register
    m_q <= a_q * b_q;            // MREG  - DSP pipeline register
    if (clr) acc <= '0;          // PREG  - DSP output register
    else     acc <= acc + m_q;   //          and the accumulator
  end

endmodule
Latency is free, area is not Those four register stages cost zero extra flip-flops - they already exist inside the DSP tile and are switched on by the USE_MULT / AREG / MREG / PREG attributes the inference engine sets for you. Not using them does not save anything; it just leaves the slice running at a fraction of its rated fmax. Write the pipeline. Absorb the latency in your control logic.

What silently falls out of the DSP

RTL pattern Result Why
a * b, both signals DSP slice Exactly what the primitive is for
a * 8'd5 (constant) Usually LUTs Constant multiply becomes shift-and-add, often cheaper
a * b with an async reset on the product DSP + external flops DSP registers only support synchronous reset
Product feeding combinational logic before a flop DSP with PREG off Output register is bypassed; fmax collapses
a / b (variable divisor) Hundreds of LUTs There is no divider primitive. Use a restoring-division IP or reciprocal multiply

Force the issue when inference guesses wrong. The use_dsp attribute is module-scoped or signal-scoped and is honoured by Vivado synthesis:


(* use_dsp = "yes" *)   module heavy_filter (...);   // force DSP mapping
(* use_dsp = "no"  *)   module tiny_scaler  (...);   // force LUT mapping

// Or per-signal, which is usually what you actually want:
(* use_dsp = "yes" *) logic signed [47:0] partial_sum;

1.5 Clock regions, BUFGs and MMCMs

Clocking is where FPGA design departs most sharply from ASIC design. An ASIC has a clock tree synthesised specifically for that netlist. An FPGA has a pre-built clock distribution network that was etched into the silicon before your design existed, and your only decision is which resources to use.

The device is divided into a grid of clock regions. Each region contains a fixed number of CLB columns, BRAM columns and DSP columns, and - critically - a horizontal clock spine (the HROW) that can carry a limited number of distinct clocks. A global clock buffer (BUFG) drives every region on the device with tightly matched delay; a regional buffer (BUFR, BUFH) drives only its own region but costs less of the global budget.

Clock region grid with a global buffer distributing one clock across all regions via horizontal clock spines X0Y2 X1Y2 X2Y2 X0Y1 X1Y1 X2Y1 X0Y0 X1Y0 X2Y0 HROW HROW HROW BUFG MMCM 100 -> 200 MHz clk pin one BUFG reaches every region with matched delay - skew measured in tens of ps
Figure 1.5 - Clock distribution: pin into an MMCM, MMCM output into a BUFG, BUFG onto the global spine, spine into every region's HROW. Nothing you write in RTL creates this network - you can only choose to be on it or off it.

The divided-clock mistake

The single most damaging clocking habit is generating a slower clock with a flip-flop and using it as a clock port. It is the natural thing to write, it simulates perfectly, and it creates a permanent timing problem on hardware.


logic clk_div2 = 1'b0;

always_ff @(posedge clk)
  clk_div2 <= ~clk_div2;

// clk_div2 now leaves the global
// network and rides general routing.
always_ff @(posedge clk_div2)
  count <= count + 1;

// Consequences:
//  * a second clock domain STA must
//    analyse, with unknown skew
//  * every path between the two
//    domains becomes a CDC path
//  * Vivado will warn, then route it
//    on fabric, then you will chase
//    a phantom bug for two days

logic tick = 1'b0;

always_ff @(posedge clk)
  tick <= ~tick;        // 50% duty

// ONE clock domain. tick is data.
always_ff @(posedge clk)
  if (tick) count <= count + 1;

// Consequences:
//  * one clock, one STA problem
//  * no CDC anywhere
//  * the enable rides the slice's
//    dedicated CE pin, costing zero
//    extra logic
//
// If you genuinely need a clock OUT
// of the chip at half rate, use ODDR
// - never a fabric-routed flop.
When you do need a real second clock, use the MMCM An MMCM (mixed-mode clock manager) or PLL is a hardened analogue block that multiplies and divides a reference clock, de-skews it against the fabric, and can shift its phase in fine steps. Its outputs are properly characterised, drive BUFGs directly, and are understood by static timing analysis. One MMCM can produce up to seven related clocks. That is the supported way to get 25 MHz, 100 MHz and 200 MHz in the same design - we set the constraints for exactly this case in Volume 03.
Interview grilling - "What is a control set, and why should I care?"

A control set is the tuple {clock, clock-enable, set/reset} that a group of flip-flops shares. Within one slice, all eight flip-flops must share a single control set - the silicon has one CE pin and one SR pin per half-slice, not eight.

The practical effect: if your design has hundreds of distinct enable signals, the placer cannot pack flops densely. You end up with slices holding two flip-flops instead of eight, LUT utilisation looks fine, and yet the device is full and routing is congested.

Fixes, in order:

  • Remove resets from flops that do not need them. Most datapath registers do not - let them power up as zeros from the bitstream and clear the control path instead.
  • Use synchronous reset rather than asynchronous where you do need one.
  • Merge enables that are logically equivalent, and push rarely-changing enables into the data path (d = en ? new : old) instead of onto the CE pin.

report_control_sets -verbose is the command. A design with more than a few hundred unique control sets on a mid-range part is worth investigating.

Volume 01 recap

Concept The one thing to remember
LUT An SRAM truth table. Count inputs, never operators.
Slice packing Control sets, not LUT count, are what fill a device.
Carry chain Write a + b. Never hand-build a lookahead adder.
Block RAM Register the read address, or you get flip-flops instead.
BRAM output reg Free speed - unless you reset it out of the tile.
DSP48 25x18 signed. Use all four pipeline stages; they cost nothing.
Clocking One clock plus enables. A flop is never a clock source.