info@embecosm.com

This is part of a blog post series on enabling IREE on bare metal systems. The full series can be found here.

In the last post we ran a matrix multiplication through IREE on one core of our four-core CV32E40Pv2 cluster, and parked the other three. In this post we wake them up. We look at what the cluster gives us for making cores work together, and build the two ways of sharing work that IREE will need. Then we measure what four cores actually buy us.

We said last time that this post would also cover the DMA engine. It will be in Part 5, where it is first needed for moving data into the scratchpads.

As before, everything here is reproducible. The code is in the cluster repository, tagged iree-blogpost-4, and every number comes from a log saved in results/iree-blogpost-4.

What the hardware gives you

Before writing any parallel code, it is worth listing what the cluster provides for cores to coordinate. It is a short list.

Every core runs main. All four cores start together, and the start-up code sends every one of them into main(). There is no way to start just one core, or to start a particular function on one. When core 0 returns from main, the simulation ends, so core 0 has to wait for the others before it returns.

There are no atomic instructions. The cores implement rv32imc, without the RISC-V “A” extension. There is no instruction that reads and updates a memory word in one indivisible step. Without that, you cannot build an ordinary lock, and two cores adding into the same variable will lose updates.

There is one barrier. A barrier is a meeting point: each core that reaches it waits until all the expected cores have arrived, and then they all continue. The cluster’s event unit implements exactly one. Software uses it through three memory-mapped registers, which we listed in Part 2: one to say which cores take part, one to say “I have arrived”, and one to say “I have seen the release”.

The wake-up goes to every core. A core waiting at the barrier goes to sleep with wfi (“wait for interrupt”). When the last core arrives, the event unit raises an interrupt to wake the sleepers, and that interrupt line is wired to all four cores. There is no way to wake one particular core.

Here is the whole barrier, from sw/cluster_sync.h:

static inline void cl_barrier_wait(cl_barrier_t *bar) {
    while (bar->initialized == 0u) {}
    cl_fence();
    cl_barrier_notify(bar->barrier_id);
    cl_evt_wait();
    cl_gpevt_clear(bar->barrier_id);
    cl_fence();
}

Wait until the barrier has been set up, make sure our own writes have reached memory, say we have arrived, sleep until everyone has, and acknowledge the wake-up. The sleep is a small trick: cl_evt_wait() enables the barrier’s interrupt only around the wfi, while interrupts as a whole stay off. The core never jumps to an interrupt handler. The interrupt is used purely as an alarm clock.

So the toolkit is one barrier and nothing else. No locks, no queues, no way to wake a single core. Everything in the rest of this post is built from that one barrier.

Pattern 1: every core takes a slice

The simplest way to use four cores falls straight out of “every core runs main“. Every core runs the same loop, each over its own part of the data, and they meet at the barrier at the end.

The only real decision is how to split the loop. A loop count is rarely a multiple of four, and the obvious split, a quarter each with the remainder going to the last core, gives the last core up to three extra iterations. Everyone waits at the barrier for the slowest core, so those extra iterations are paid for by all four. The better split gives the leftover iterations to the first few cores, one each:

static inline cl_range_t cl_range_split(uint32_t n, uint32_t core,
                                        uint32_t ncores) {
    cl_range_t r;
    uint32_t chunk = n / ncores;
    uint32_t rem = n % ncores;
    uint32_t extra = (core < rem) ? core : rem;
    r.begin = core * chunk + extra;
    r.end = r.begin + chunk + ((core < rem) ? 1u : 0u);
    return r;
}

From sw/cluster_team.h. Each core calls this with its own number and gets back the half-open range [begin, end) it should work on.

parallel-for-demo splits a 101-element loop this way, and checks that the four slices fit together with no gap and no overlap:

  core 0: [0, 26) len=26
  core 1: [26, 51) len=25
  core 2: [51, 76) len=25
  core 3: [76, 101) len=25
n=101 cores=4 imbalance=1 iteration(s)
PARALLEL FOR DEMO PASS

This pattern is fine for a self-contained program. It is not what a runtime looks like.

Pattern 2: one core hands out work

A runtime such as IREE’s is ordinary sequential code. It loads a model, sets up buffers, and works through a list of steps. Some of those steps are parallel: the IREE compiler splits a model into dispatches, and each dispatch runs the same function over a grid of workgroups. In Part 3, the runtime ran every workgroup one after another on core 0. To use the other cores, the runtime needs to be able to say “run this function on every core, and tell me when it is done”, many times over, with sequential code in between.

So we turn the cluster round. Core 0 runs the program. Cores 1 to 3 enter main, and immediately call a loop they never leave:

static inline void cl_team_worker_loop(cl_dispatch_t *d) {
    for (;;) {
        cl_team_barrier(&d->team);
        cl_task_fn_t fn = d->fn;
        if (fn == 0) {
            return;
        }
        fn(d->arg, cl_core_idx());
        cl_fence();
        cl_team_barrier(&d->team); /* join */
    }
}

They sleep at the barrier until core 0 has work. Core 0 writes a function pointer and an argument to shared memory, then meets them at the barrier, which releases them. Every core, core 0 included, calls the function with its own core number, and they meet at the barrier again. Then the workers go back to sleep, and core 0 carries on.

From core 0’s side, a parallel step is two calls, and the code around it stays sequential:

    scale_args_t sargs = {3u};
    cl_team_dispatch(&disp, kernel_scale, &sargs);
    cl_team_join(&disp);
    cl_team_dispatch(&disp, kernel_partial_sum, 0);
    cl_team_join(&disp);

From sw/dispatch-demo.c.

Two details come from the hardware. The usual design would give each core its own queue of work and an interrupt to wake it when something arrives, but this cluster cannot wake one core, so dispatch is built on the barrier, twice. And because there are no atomic instructions, the cores cannot add into one shared total: kernel_partial_sum has each core write its own partial sum, and core 0 adds the four together after the join.

What a dispatch costs

dispatch-demo also times the mechanism itself, using the free-running cluster counter rather than the core’s own mcycle, which stops while the core sleeps (the trap from Part 2):

dispatch round trip = 1192 cycles (empty kernel, 4 cores)
team barrier        = 471 cycles
sum=98688 expected=98688
DISPATCH DEMO PASS

An empty dispatch, which starts the workers and waits for them without doing any work, costs 1,192 cycles. A single barrier on its own costs 471, so most of a dispatch is its two barriers. For scale, 1,192 cycles is about fifty accesses to shared memory, or a handful of the soft-float operations we measured in Part 3.

That sets a lower limit on what is worth parallelising. A piece of work much shorter than a dispatch should stay on one core, because sharing it out would cost more than doing it.

A barrier for some of the cores

There is only one barrier, and its wake-up goes to every core, so we had assumed it could only ever include all four. That turns out to be wrong. The register that sets the barrier up takes a mask of which cores take part, and the event unit honours it: the barrier releases when the cores in the mask have arrived, and only those cores are released.

subteam-barrier-test sets the mask to cores 0 and 1, makes core 1 arrive late on purpose, and checks that core 0 waited for it while cores 2 and 3 carried on:

core 1 arrived 157124 cycles after core 0
core 0 was held for 157410 cycles, core 1 for 308
SUBTEAM BARRIER TEST PASS

What the hardware cannot do is two such barriers at the same time, because there is only one mask. Two pairs of cores each meeting on their own barrier is out of reach. For now that is not a limit we hit: IREE’s dispatches use every core.

Four cores are not four times faster

With work spread over the cores, the obvious question is how much faster it goes. scaling-bench runs the same total amount of work on 1, 2, 3 and 4 cores, split with cl_range_split. The cores not in use sleep at the barrier. There are two workloads:

• compute: a short loop of arithmetic that never touches data in memory;
• memory: each core adds up its share of a 16,384-word array in shared memory.

workload cores   cycles  speed-up%
compute      1   320439       100
compute      2   240391       133
compute      3   240464       133
compute      4   240549       133
memory       1   508392       100
memory       2   360976       140
memory       3   361025       140
memory       4   361066       140
SCALING BENCH PASS

A second core gives about a third more speed. After that the lines are flat. The reason is the way the four cores reach shared memory.

Instructions. There is no instruction cache, and code cannot run from the scratchpads, so every instruction every core executes is fetched from shared memory. All four cores’ fetches go through one arbiter, which passes on one request per cycle. That is enough to keep one core busy, with some cycles to spare. It is not enough for four. The compute loop, which never touches data, takes 8.0 cycles per iteration on one core. On two or more cores the whole cluster together manages one iteration every 6.0 cycles, however many cores share the work: the instruction path is busy every cycle, and adding cores only adds queueing.

Data. Only one access to shared memory can be in flight at a time, across the whole cluster, and each one takes about 22 cycles, as we measured in Part 2. The memory workload makes one load per iteration. On one core it takes 31 cycles per word: the load plus the loop around it. On two or more cores the cluster as a whole takes exactly 22.0 cycles per word. The cores take turns on the data path, back to back, and no number of cores can make it go faster.

One caution about the exact figures. Because these loops are limited by instruction fetch, their speed depends on exactly which instructions the compiler chooses: a register allocated differently, or an instruction that loses its compressed 2-byte form, changes the result. Small edits to this benchmark during development moved the speed-ups anywhere between 1.1× and 1.45×. What never changed was the shape: a modest gain from the second core, and nothing after that. The numbers above are exactly reproducible from the tagged code with the same compiler and flags.

What this means for IREE

Three things carry forward into the posts that follow.

First, the dispatch mechanism works, and it is the shape IREE needs: a sequential program that hands a function to every core and waits.

Second, a dispatch costs about 1,200 cycles, so a workgroup has to be well above that to be worth sending to another core. IREE’s choice of tile size, which in Part 3 gave us a single workgroup, will matter twice over.

Third, and most important, this cluster does not scale with cores, because of how they share memory. Some of that is within reach of software. Data the cores keep in their own scratchpads stays off the shared data path entirely. The instruction path is different: every core has to fetch its code from shared memory, and no software change avoids that. Getting close to four times the speed of one core would mean changing the model itself, for example giving each core its own instruction port or adding an instruction cache. We will come back to that once IREE is running on all four cores and we can see how much it matters for a real workload.

Try it yourself

You need the CORE-V GCC toolchain and Verilator from Part 2. As in Part 3, put the toolchain and the repository side by side in one directory. --recursive matters here: it also fetches the CV32E40P core, which the model is built from. Then build the model:

mkdir iree-bare-metal && cd iree-bare-metal
ln -s /opt/corev corev-openhw-gcc-ubuntu2204-20240530
git clone --recursive --branch iree-blogpost-4 https://github.com/pietraferreira/cv32e40p-verilator-cluster4.git
cd cv32e40p-verilator-cluster4
make TOOLCHAIN_ROOT=../corev-openhw-gcc-ubuntu2204-20240530 verilate

Then run each program:

make TOOLCHAIN_ROOT=../corev-openhw-gcc-ubuntu2204-20240530 APP=parallel-for-demo run
make TOOLCHAIN_ROOT=../corev-openhw-gcc-ubuntu2204-20240530 APP=dispatch-demo run
make TOOLCHAIN_ROOT=../corev-openhw-gcc-ubuntu2204-20240530 APP=subteam-barrier-test run
make TOOLCHAIN_ROOT=../corev-openhw-gcc-ubuntu2204-20240530 APP=scaling-bench run

Each prints the output shown above and ends with a line saying it passed. The team functions they use are in sw/cluster_team.h, about 120 lines of C on top of the barrier, and are meant to be copied into your own programs.

Next

All four cores now do useful work, coordinated through one barrier: either every core takes a slice of a loop, or core 0 hands a function to the others and waits. We know the price of handing out work, about 1,200 cycles, and we know the limit on what four cores can gain, set by the instruction and data paths they share.

Next time we write the baseline that IREE has to beat: a hand-written integer matrix multiplication on one and four cores, with its data in shared memory and in the scratchpads. That is also where the DMA engine comes in, for moving data into the scratchpads while the cores work.

This is Part 4 of a blog post series on enabling IREE on bare metal systems. Part 1 is an overview of IREE on a bare metal microcontroller, Part 2 builds an open reference platform for IREE on a bare metal microcontroller, and Part 3 runs IREE on one core. The full series can be found here.

If you would like to know more about how we can help you get AI models onto your own hardware, please contact Embecosm at info@embecosm.com.