info@embecosm.com

This is part of a blog post series on enabling IREE on bare metal systems. The full series can be found here.

In the last post we built a Verilator model of a four-core CV32E40Pv2 cluster, ran our first programs on it and measured what memory costs. In this post, we follow a matrix multiplication from a dozen lines of MLIR down to RISC-V instructions, run it on the cluster, and measure what it costs against the same calculation written by hand in C.

As before, everything here is reproducible. The code is in three repositories on GitHub, all tagged iree-blogpost-3: the cluster model from Part 2, the build scripts and the IREE demo. Every number below comes from a log saved alongside the demo, in matmul/results/iree-blogpost-3.

Two programs, not one

The first thing to understand about IREE is that it is two programs that ship in one project. There is a compiler, and there is a runtime. They share a file format and an interface, and nothing else.

The compiler, iree-compile, is an ordinary program that runs on your device. It reads a model and writes out code for the target. The runtime is a small C library that is linked into the program running on the target. Nothing on the target ever compiles anything.

That split is not a detail. It is the reason IREE suits a target with no operating system: all the heavy machinery stays on the workstation, and what reaches the target is plain code and a library.

The compiler: from a matmul to RISC-V

IREE’s compiler is built on MLIR, the Multi-Level Intermediate Representation from the LLVM project. An intermediate representation is the form a compiler holds a program in while it works on it. MLIR’s idea is that there need not be just one: it is a framework for defining many, called dialects, each at a different level of abstraction, and for rewriting a program from one dialect down to the next. IREE is a particular stack of those dialects, lowered in a fixed order.

The clearest way to see it is to follow a real program through. Our example is a single matrix multiplication of two f32 matrices whose sizes are only known at run time:

%1 = linalg.fill ins(%cst : f32) outs(%0 : tensor<?x?xf32>) -> tensor<?x?xf32>
%2 = linalg.matmul ins(%arg0, %arg1 : tensor<?x?xf32>, tensor<?x?xf32>)
                   outs(%1 : tensor<?x?xf32>) -> tensor<?x?xf32>

matmul.1.input.mlir. tensor<?x?xf32> is a two-dimensional array of 32-bit floats whose dimensions (?) are not known until run time.

This is the level a framework like PyTorch exports at: a description of the maths, with no mention of memory, cores or instructions. iree-compile can save the program after each stage of lowering (the --dump-compilation-phases-to option), and that is what the excerpts below are taken from.

Flow decides what becomes a kernel. It groups operations into dispatch regions, each of which will be compiled into one piece of native code. Here the whole matmul becomes one region, and the function that used to contain it now just calls it:

%6 = flow.dispatch @matmul_f32f32f32_dispatch_0::@matmul_f32f32f32_dispatch_0_matmul_like_DxDxD_f32
       [%1, %3, %0, %4](%2, %5, %1, %3, %0, %4) : ...

Stream makes execution explicit. Tensors become buffers, the dispatch says which buffers it reads (ro) and writes (wo), and the program waits for the result before handing it back:

%27 = stream.cmd.execute ... await(%result_timepoint) => with(...) {
  stream.cmd.dispatch @matmul_f32f32f32_dispatch_0::...(...) {
    ro %arg2[%c0 for %3] : !stream.resource<external>{%3},
    ro %arg3[%c0 for %8] : !stream.resource<external>{%8},
    wo %arg4[%c0 for %10] : !stream.resource<external>{%10}
  }
} => !stream.timepoint
%28 = stream.timepoint.await %27 => %result : ...

HAL, the Hardware Abstraction Layer, turns that into calls on a C interface that every IREE backend implements: create a command buffer, record the dispatch into it, submit it to the device’s queue, and wait on a fence (a flag that is set when the work has finished):

%cmd = hal.command_buffer.create device(%__device_0 : !hal.device) mode(OneShot) ...
hal.command_buffer.dispatch<%cmd : !hal.command_buffer> target(...)[%c0] workgroups([%42, %c1, %c1]) ...
hal.command_buffer.finalize<%cmd : !hal.command_buffer>
hal.device.queue.execute<%__device_0 : !hal.device> ... wait(%fence) signal(%fence_1) commands(%cmd)
%status = hal.fence.await until([%fence_1]) ...

VM is the last step. IREE has its own small virtual machine, and the control program becomes a list of calls into the runtime’s HAL module, vm.call.variadic @hal.command_buffer.dispatch, vm.call @hal.device.queue.execute and so on.

Two things come out of the compiler. The first is that control program, and the important point about it is that it does not contain the model’s arithmetic. It allocates buffers, records the dispatch and waits. The second is one native kernel per dispatch region, generated by LLVM for our target, and that is where the multiply-adds live.

The kernel is also where the decisions that matter on a small target are made. Before generating code, IREE picks a tiling for the dispatch region:

#config1 = #iree_cpu.lowering_config<cache_parallel = [64, 64, 0], 
distribution = [64, 64, 0],
vector_common_parallel = [8, 32, 0], vector_reduction = [0, 0, 1]>

Read that as: split the output matrix into workgroups of 64×64 elements, each an independent unit of work, and inside each workgroup compute 8×32 elements at a time. Both numbers come back later in this post.

What it can emit, and the format we use

iree-compile produces two things, both of which appeared in the diagram at the start: the kernel, the RISC-V code that does the arithmetic, and the control program, which sets up the buffers, calls the kernel and waits for it to finish. IREE can package each of them in more than one format. On a microcontroller the choice matters, because some formats need extra software on the target to load or run them.

Format	For	What it is	What the target needs
Embedded ELF	Kernel	The default: the kernel in a small, portable library file	IREE’s own loader, to map it into memory at run time. This is what IREE’s own bare-metal guide uses
Static library	Kernel	--iree-llvmcpu-link-static: a .o and a .h	Nothing extra: you link it into your image like any other object file
VM bytecode	Control program	A .vmfb file of instructions for IREE’s virtual machine	IREE’s interpreter, to step through the instructions at run time
VM as C	Control program	--output-format=vm-c: the same program as C source	Nothing extra: it is compiled with the rest of your program

The output formats that matter here. IREE has others, for operating systems with a dynamic loader, which this target does not have.

We use the static library for the kernel and C for the control program. The kernel is linked like any other object file, and the control program is compiled by the same toolchain as everything else in the image. Nothing is interpreted and nothing is loaded at run time. These are the options our build passes to iree-compile:

--output-format=vm-c
--iree-hal-target-device=local
--iree-hal-local-target-device-backends=llvm-cpu
--iree-llvmcpu-target-triple=riscv32-corev-elf
--iree-llvmcpu-target-cpu=generic
--iree-opt-level=O2
--iree-llvmcpu-target-cpu-features=+m
--iree-llvmcpu-target-abi=ilp32
--iree-llvmcpu-link-embedded=false
--iree-llvmcpu-link-static
--iree-llvmcpu-static-library-output-path=matmul_lib.o
--iree-vm-target-index-bits=32

The runtime

On the target, the runtime is three layers. iree/vm runs the control program; here that program is native C, so this layer mostly provides its data types and calling convention. iree/hal is the C interface for devices, buffers, command buffers and fences, and it is IREE’s retargeting point: supporting new hardware means implementing this interface. iree/runtime is a thin layer on top that bundles the other two into an instance, a session and a call, which is all our demo uses.

For running kernels on a CPU there are two HAL devices, and the choice between them shapes everything that follows:

• local-sync runs work inline, on the calling core, one workgroup after another. It needs no threads, no scheduler and no operating system.
• local-task hands workgroups to a pool of threads through IREE’s own task system. It is how IREE normally uses several CPU cores.

local-task is not available here. It needs threads and a way for one thread to wake another, and this cluster has no threading library to provide them. So we use local-sync, which means one core runs the runtime and the kernel, and the other three must be kept out of it. What it takes to give the other three cores useful work is the subject of the rest of the series.

If you know ExecuTorch, this configuration is where the two differ most. ExecuTorch’s runtime always walks a list of instructions at run time, calling a pre-written kernel, or a backend delegate, for each one. Here both the kernel and the control program are compiled to native code, so there is no interpreter on the target at all. Part 1 covers the wider comparison.

Getting the tools

Building iree-compile from source takes hours. You do not need to: IREE publishes nightly builds of the compiler as a Python package, and the one matching the runtime source we use is 3.10.0rc20251128:

python3 -m venv .venv
.venv/bin/pip install iree-base-compiler==3.10.0rc20251128 \
    -f https://iree.dev/pip-release-links.html
.venv/bin/iree-compile --version

That version is not on PyPI, only on IREE’s own package index, hence the -f. It reports commit 4bb0c1298d, which is the commit IREE’s iree-3.10.0rc20251128 tag points to. We checked it against a compiler built from source at that tag: for our matmul it produces an identical control program and an identical kernel, instruction for instruction.

The runtime is built from the IREE source at the same tag, iree-3.10.0rc20251128, with the same CORE-V GCC 14.1.0 toolchain as the rest of the image, using tooling/build-iree-corev.sh. That script turns off everything a bare metal target cannot use (threads, file I/O, the local-task device, the compiler itself) and keeps local-sync. Building it needs two small host tools from IREE’s runtime, iree-flatcc-cli and iree-c-embed-data, which the pip package does not include. They build from the same source in seconds, without the compiler; the commands are in “Try it yourself” below.

Using one toolchain for everything matters on bare metal: an instruction set or calling convention mismatch between two libraries shows up as a dead core, not an error message. The compiler records its version in every object it builds, so this is easy to check:

$ riscv32-corev-elf-readelf -p .comment matmul_demo
String dump of section '.comment':
  [     0]  GCC: ('corev-openhw-gcc-ubuntu2204-20240530') 14.1.0

The first inference, and the crash before it

The demo, matmul_demo.c, follows IREE’s usual pattern: create a runtime instance and a device, load the compiled module into a session, push two input matrices, invoke the function and read back the result. As we saw in Part 2, the board support package starts all four cores and all four call main. Built and run exactly like that, this is what happens:

[TB] loading shared firmware image: benchmarks/matmul/build/matmul_demo.hex
CV32E40P BSP: illegal instruction exception handler entered
CCVV3322EE4400PP  BBSSPP::  iilllleeggaall  iinnssttrruucctt
iioonn  eexxcceeppttiioonn  hhaannddlleerr  eenntteerreedd
...
CV32E40P BSP: unknown exception handler entered
...
[TB] max cycle limit reached (50000000)

The doubled characters are the same effect as the stutter in Part 2’s hello world: several cores writing to the one output register at the same time. Here they are all reporting faults. It starts with illegal instruction exceptions, unknown exceptions soon join them, and it never recovers: when the testbench stopped the run at 50 million cycles, the cores were still faulting. The log holds more than 1,400 unknown-exception reports.

The cause is not IREE and not the generated kernel. It is two reasonable assumptions meeting at main. The board support package releases every core into main, because the cluster’s own parallel demos need all four there. The application assumes it is the only thing running, because a single-threaded runtime reasonably would. So four cores each build a complete IREE runtime at once, all allocating from the same heap. That heap has no protection against this: in our image, the C library’s __malloc_lock and __malloc_unlock are each a single ret. We have not traced the corruption instruction by instruction, but four cores sharing an unlocked heap is the likely cause, and keeping three of them out of it removes the crash.

Neither assumption is wrong, and the board support package cannot settle it. Whoever writes the application has to decide which cores run the runtime. For now the answer is core 0, and the fix is one line at the top of main:

static inline uint32_t read_mhartid(void) {
  uint32_t hart;
  __asm__ volatile("csrr %0, mhartid" : "=r"(hart));
  return hart;
}
int main() {
  // The board support package releases all four cores into main(), but the
  // IREE runtime with the local-sync driver assumes it owns the machine.
  // Only core 0 runs it; the others return and park in crt0.
  if (read_mhartid() != 0) return 0;
  ...

Returning is safe: the startup code parks every core except core 0 when main returns. With that line, and the timing we add in the next section, the program gives:

[TB] loading shared firmware image: benchmarks/matmul/build-n4/matmul_demo.hex
35.000000 35.000000 30.000000 20.000000
66.000000 55.000000 39.000000 53.000000
48.000000 47.000000 41.000000 30.000000
51.000000 32.000000 43.000000 49.000000
TIMING n=4 startup=492851 call1=1204525 call2=1202005 c_loop=64461
Passed!
[TB] CLUSTER EXIT SUCCESS

That is an operation of the kind PyTorch exports, lowered through MLIR, compiled for a CORE-V target and run on a bare metal cluster with no operating system. It is also using one core out of four.

What it costs

Getting the right answer is the start. To see what IREE costs we extended the demo to take the matrix size at build time, timed each stage with the cluster’s cycle counter (the one at 0x1500_1004 from Part 2’s address map), and ran the same multiplication through a plain C triple loop for comparison. The inputs are small whole numbers, so every sum is exact in f32 and the two results can be compared bit for bit; they matched at every size.

Size	Runtime start-up	One IREE call	Plain C loop	Faster
4×4	492,851	1,202,005	64,461	C, by 18.6×
16×16	492,856	6,535,215	3,904,653	C, by 1.7×
32×32	492,856	23,898,480	30,996,521	IREE, by 1.3×

Three things stand out:

Start-up costs about half a million cycles, whatever the size. Creating the instance, the device and the session and loading the module is a fixed cost, paid once.

At small sizes, the call overhead dominates. A 4×4 matmul is 64 multiply-adds. Even at the soft-float prices below, that is well under 50,000 cycles of arithmetic, so more than a million of the 1.2 million cycles in an IREE call go on something other than arithmetic: the runtime recording, submitting and waiting for the work, and the kernel’s own setup.

At 32×32, IREE’s kernel beats the hand-written loop. Divide the cycles by the 32,768 multiply-adds: the C loop spends about 946 cycles on each, IREE about 729. One soft-float add plus one multiply costs about 663 cycles (below), so IREE’s kernel spends roughly nine-tenths of its time in the arithmetic itself, while the plain loop spends a good deal on everything around it. The 8×32 inner tile is why. The kernel’s main loop is one straight-line block that works on an 8×32 block of results: 256 multiply-adds, 512 soft-float calls, and a single branch at the end. Each input value it loads is used 8 or 32 times, and the loop overhead is paid once per 256 multiply-adds instead of once per one.

Where the time goes: no floating point unit

The CV32E40P in this cluster has no floating point unit, so every f32 operation is a call into the compiler’s software floating point routines in libgcc. We measured them with a small benchmark that times each operation over 1,000 iterations and subtracts the same loop with an integer add:

Operation	Cycles	Instructions
float add (__addsf3)	222	66
float multiply (__mulsf3)	441	100
float divide (__divsf3)	583	104
int32 to float (__floatsisf)	193	46

Each figure is an average over 1,000 operations on 16 different inputs, so read it as typical rather than exact.

The generated kernel shows the same thing from the other side: its assembly contains 731 calls to __mulsf3 and 728 to __addsf3. An integer add takes one cycle. This is why quantising a model to 8-bit integers matters so much on parts like this: it shrinks the data, and it also steps around a software path that costs hundreds of cycles per operation.

Where the bytes go

The complete image is 1,105,394 bytes of code and initialised data, plus 11,536 bytes of zero-initialised data:

Nearly two-thirds of the image is the IREE runtime. The kernel is 116 KB, large for one matmul, probably because of how far that inner tile is unrolled; the soft-float routines are only 39 KB. The runtime’s share is the obvious target for trimming, and how small it can be made on a part like this is a question for a later post.

Why one core

It would be easy to read the crash as a bug and the working run as the end of the story. It is the other way round. Even with a runtime that could use all four cores, this matmul would run on one, because of the 64×64 workgroups chosen earlier. The compiler also emits the formula for how many workgroups a call needs:

count(%arg0: !hal.device, %arg1: index, %arg2: index, %arg3: index, %arg4: index) -> (index, index, index) {
  %c64 = arith.constant 64 : index
  ...                                   // ceil(M / 64) and ceil(N / 64)
  %16 = arith.muli %7, %15 overflow<nsw> : index
  hal.return %16, %c1_3, %c1_4 : index, index, index
}

Because the sizes are only known at run time, the count is worked out at run time, as ceil(M/64) × ceil(N/64). For every size we ran, that is one workgroup. IREE never split the work, so there was nothing to share out.

Getting four cores doing useful work is therefore three problems:

1. More than one workgroup. Either a larger problem, or smaller tiles chosen for this target.
2. Something to hand workgroups to the other cores. Either a HAL device built on the cluster’s barrier and its sleeping cores, or an application that dispatches them itself.
3. Work of similar size for each core. The tiles have to be shaped so that four cores get comparable work, and the memory measurements in Part 2 decide what “comparable” means.

None of that is visible from a 4×4 matmul on one core. But that matmul proves every other part of the toolchain is in place.

Try it yourself

You need the CORE-V GCC toolchain and Verilator from Part 2, plus Python 3, CMake and Ninja. The build scripts expect the repositories side by side in one directory, under the folder names below (tooling, benchmarks and so on, not the repository names), with the toolchain in it as corev-openhw-gcc-ubuntu2204-20240530:

mkdir iree-bare-metal && cd iree-bare-metal
ln -s /opt/corev corev-openhw-gcc-ubuntu2204-20240530
git clone --recursive --branch iree-blogpost-3 https://github.com/pietraferreira/cv32e40p-verilator-cluster4.git cv32e40p-verilator-cluster4
git clone --branch iree-blogpost-3 https://github.com/pietraferreira/iree-bare-metal-tooling.git tooling
git clone --branch iree-blogpost-3 https://github.com/pietraferreira/iree-bare-metal-benchmarks.git benchmarks
git clone https://github.com/iree-org/iree.git
git -C iree checkout iree-3.10.0rc20251128
git -C iree submodule update --init

Install the compiler:

python3 -m venv .venv
.venv/bin/pip install iree-base-compiler==3.10.0rc20251128 -f https://iree.dev/pip-release-links.html
export PATH="$PWD/.venv/bin:$PATH"

Build the two host tools the runtime build needs. This takes seconds, because it skips the compiler:

cmake -G Ninja -S iree -B build-iree-host-tools -DCMAKE_BUILD_TYPE=Release \
    -DIREE_BUILD_COMPILER=OFF -DIREE_BUILD_TESTS=OFF -DIREE_BUILD_SAMPLES=OFF \
    -DIREE_BUILD_PYTHON_BINDINGS=OFF \
    -DIREE_HAL_DRIVER_DEFAULTS=OFF -DIREE_HAL_DRIVER_LOCAL_SYNC=ON
cmake --build build-iree-host-tools --target iree-flatcc-cli iree-c-embed-data
mkdir -p install-iree-host/bin
cp build-iree-host-tools/tools/iree-flatcc-cli \
   build-iree-host-tools/tools/iree-c-embed-data install-iree-host/bin/

Build the cluster model and its board support library, which the runtime links against. Then build the runtime and the demo, and run it:

make -C cv32e40p-verilator-cluster4 \
    TOOLCHAIN_ROOT=$PWD/corev-openhw-gcc-ubuntu2204-20240530 verilate firmware
tooling/build-iree-corev.sh
MATMUL_N=4 tooling/build-corev-matmul-demo.sh
cv32e40p-verilator-cluster4/build/verilator/obj_dir/Vtb_top_verilator \
    +firmware=benchmarks/matmul/build-n4/matmul_demo.hex

The last command should print the output shown earlier, ending in Passed!.

MATMUL_N=16 and MATMUL_N=32 give the other rows of the timing table; 32 takes noticeably longer to simulate. The soft-float benchmark runs from the cluster repository like any other demo:

cd cv32e40p-verilator-cluster4
make TOOLCHAIN_ROOT=../corev-openhw-gcc-ubuntu2204-20240530 APP=softfloat-bench run

Next

We now have IREE compiling a model, a runtime on bare metal, and a first inference with its costs measured: about half a million cycles to start, more than a million cycles of overhead on a small call, a few hundred cycles per floating point operation, and a kernel that beats hand-written C once the problem is big enough to amortise the rest. All of it on one core.

Next time we turn to the other three. Part 4 builds out the board support package for multi-core work: holding cores, handing them work and getting results back, and what the cluster’s barrier and DMA engine allow and do not allow. That is the ground IREE will need before it can use more than one core.

This is Part 3 of a blog post series on enabling IREE on bare metal systems. Part 1 is an overview of IREE on a bare metal microcontroller, and Part 2 builds an open reference platform for IREE on a bare metal microcontroller. The full series can be found here.

If you would like to know more about how we can help you get AI models onto your own hardware, please contact Embecosm at info@embecosm.com.