I recently took on a project to implement a language model on an FPGA. In simple terms, a Field Programmable Gate Array (FPGA) is an IC that can be reconfigured to implement hardware circuit designs written in a Hardware Description Language (HDL) such as SystemVerilog. They are particularly interesting because you don't run software on them the way you would using microcontrollers. Instead, you reconfigure the chip into a hardware design. This gives you things that you couldn't get in software such as true parallelism in combinational logic circuits. This is how projects like MiSTer are able to end up with implementations of consoles that run extremely close to the original hardware.

My goal was to:

1. Implement a hardware design capable of running a modern Transformer based language model without a softcore (CPU implemented in the design that can run software compiled with toolchains like GCC).
2. Design and train a model that can generate coherent output but is small enough to fit on an FPGA platform - I chose the iCESugar which has an iCE40UP5K
3. Do it all with Opensource tooling

I eventually ended up with a 1.07M parameter model trained using the TinyChat dataset. It's able to generate mostly coherent output at ~10 tokens per second.

Here's an interactive port of the model that runs in the browser you can play with to get a feel for the model's behavior.

You can find the project files on GitHub and model checkpoint on Hugging Face.

Hardware Constraints

The FPGA platform I was working with is the iCE40UP5K on an iCESugar. This is a heavily resource constrained platform.

Looking at the specs, the most important features are:

• 5,280 logic cells (a 4-input LUT and a flip-flop each) - This is where the control logic of my design would go.
• 128 KB of SPRAM and 15 KB of block RAM - Memory to hold activations, kv cache etc
• 8 DSPs that each do a 16x16 multiply - Integer only arithmetic
• 8 MB of SPI flash on the board - Slow memory to hold model weights
• a 12 MHz clock - Clock to power the design

I effectively have 144KB and no native floating point to work with. This meant designing my model from the onset to be quantized for fixed-point math.

V1 - 238K Params

I started very simple, with a single layer transformer with self attention, quantization aware trained for Int4. The tokenizer is BPE trained on the TinyChat for a Vocab size of 2048.

I took some inspiration from tinyworldformer to try a looped transformer.

The idea is to feed the outputs back through some or all layers. This gives you a higher effective depth per token without increasing the number of layers in the model.

Testing showed that I got the best performance in my model with 6 loops however this is only on the dataset I was working with.

The final architecture looks like

Ice40GPT(
  (tok): Embedding(2048, 64)
  (pos): Embedding(32, 96)
  (up): QLinear(in_features=64, out_features=96, bias=False)
  (blocks): ModuleList(
    (0): Block(
      (norm1): ModuleList(
        (0-5): 6 x RMSNorm()
      )
      (norm2): ModuleList(
        (0-5): 6 x RMSNorm()
      )
      (qkv): QLinear(in_features=96, out_features=144, bias=False)
      (proj): QLinear(in_features=96, out_features=96, bias=False)
      (up): QLinear(in_features=96, out_features=384, bias=False)
      (down): QLinear(in_features=384, out_features=96, bias=False)
    )
  )
  (norm): RMSNorm()
)

The design is a state machine that processes each token. It follows the expected process of embed → (norm → attention → feed-forward) × 6 loops → logits. When the board boots up, the model weights are copied into SPRAM while Block RAM holds the activations and KV cache. Communication with the device is done over UART at 115200 baud.

To save on resources:

1. I precompute lookup tables for exp 1/x and 1/sqrt(x).
2. Tokenizer encoding/decoding is done off of the board

Scaling to 1M Params

Now that I had a working design, I wanted to scale the model to > 1M params which was ~5x the size of my current model. This meant looking for opportunities to grow the model while respecting my hardware constraints and maintaining performance.

So in no particular order, here are improvements I implemented:

Mixture of Experts (MoE) architecture

Moving from a dense model allowed me to increase the size of the feed forward network without significantly increasing the size of the matmuls. I also moved the model weights to flash memory and dynamically copy experts that are needed for each token. This allowed me to increase the size of the model to 1M params but maintain only ~200k active params per token (1M-A200k).

RoPE to replace positional embeddings

Keeping a second embedding table for token positions was costly. ~1.5 KB. Replacing it with Rotary Positional Embedding (RoPE) allowed me to replace the table with a few more dynamic calculations.

1. Split each head's 24 channels into 12 adjacent pairs.
2. Rotate pair i by an angle that grows with the position. The first pair rotates more than the last pair.
3. Do the rotation with a lookup table of cosines and sines

Hybrid Gated delta nets for longer context

I wanted the model to get better at longer context responses but couldn't scale the context window because of limited RAM, so I opted for Gated delta nets which maintain a fixed memory budget. In my design, I found that it acted a bit more like long term memory, so the self attention attends to tokens present in the context window where delta net output residuals maintain a summary of tokens that have been pushed out.

This is what my model architecture looked like after these changes.

Ice40GPT(
  (tok): Embedding(2048, 64)
  (up): QLinear(in_features=64, out_features=96, bias=False)
  (blocks): ModuleList(
    (0): Block(
      (norm1): ModuleList(
        (0-5): 6 x RMSNorm()
      )
      (norm2): ModuleList(
        (0-5): 6 x RMSNorm()
      )
      (qkv): QLinear(in_features=96, out_features=144, bias=False)
      (proj): QLinear(in_features=96, out_features=96, bias=False)
      (delta): DeltaMixer(
        (qkv): QLinear(in_features=96, out_features=96, bias=False)
        (gate): QLinear(in_features=96, out_features=4, bias=False)
        (proj): QLinear(in_features=32, out_features=96, bias=False)
      )
      (moe): Experts(
        (router): ModuleList(
          (0-5): 6 x QLinear(in_features=96, out_features=18, bias=False)
        )
      )
    )
  )
  (norm): RMSNorm()
)

And this is the current utilization report.

Info: Device utilisation:
Info: 	         ICESTORM_LC:    5215/   5280    98%
Info: 	        ICESTORM_RAM:      29/     30    96%
Info: 	               SB_IO:      12/     39    30%
Info: 	               SB_GB:       6/      8    75%
Info: 	        ICESTORM_PLL:       0/      1     0%
Info: 	         SB_WARMBOOT:       0/      1     0%
Info: 	        ICESTORM_DSP:       8/      8   100%
Info: 	      ICESTORM_HFOSC:       0/      1     0%
Info: 	      ICESTORM_LFOSC:       0/      1     0%
Info: 	              SB_I2C:       0/      2     0%
Info: 	              SB_SPI:       0/      2     0%
Info: 	              IO_I3C:       0/      2     0%
Info: 	         SB_LEDDA_IP:       0/      1     0%
Info: 	         SB_RGBA_DRV:       0/      1     0%
Info: 	      ICESTORM_SPRAM:       4/      4   100%
Info: 	      SB_FILTER_50NS:       0/      2     0%

Info: Placed 12 cells based on constraints.
Info: Max frequency for clock 'clk$SB_IO_IN_$glb_clk': 22.50 MHz (PASS at 12.00 MHz)
Info: Max frequency for clock 'clk$SB_IO_IN_$glb_clk': 21.81 MHz (PASS at 12.00 MHz)

Conclusion

Overall, this was a lot of fun. Working on this project has given me a much better understanding of what goes on under the hood. Learning to build with constraints is always instructive.

Looking forward, I think there's still a lot of room to optimize in every part of the system. Things I'm thinking about include:

1. Exploring much more aggressive quants (1 bit and ternary quants for example).
2. There's also an opportunity to make the datasets much more information dense. It needs to have a limited vocabulary and domain for the size, but there's room to explore how much it's possible to pack in such a small model.