5. Implementing an ExecuTorch runtime for the CV32E40Pv2

The runtime processing in ExecuTorch is essentially quite similar. For each operator, we need to load the method associated with an operator and connect it to its data. Then we need to execute the method. In the case of a delegated operator, the loading and execution will be handed off to custom code. This is summarized in the following diagram (based on this diagram in the ExecuTorch online user guide).

Since we are not supporting any delegation at this stage, we need to just provide empty functions for the delegation activities at this stage.

5.1. Files to be changed

The changes to the executorch repository described here can be found in the commit tagged ean16-1.0-runtime. It can be selected using

git switch --detach ean16-1.0-runtime

Runtime support for the CORE-V backend will be placed in the runtime subdirectory of backends/corev. We will implement the CoreVBackend C++ class as a derived class of BackendInterface. This declared and defined in the following files.

• backends/corev/runtime/CoreVBackend.h

• backends/corev/runtime/CoreVBackend.cpp

In addition, we need to define the Physical Abstraction Layer (PAL) for our platform. This goes in the following file.

• backends/corev/runtime/corev_pal.cpp

As with the AoT, we then need a driver for the new backend runtime in the examples directory. We already have a corev subdirectory, the runtime driver is placed in the executor_runner subdirectory

• examples/corev/executor_runner/corev_executor_runner.cpp

We will need to add a CMakeLists.txt file to control building the CORE-V backend, which is placed the backends/corev directory, not the runtime subdirectory.

• backends/corev/CMakeLists.txt

The detailed compilation parameters for this will be picked up from a CMake configuration file at the top level of the tooling repository. There is considerable complexity to the compilation of the runtime, so we also provide a script to run then the entire process, also at the top level of the tooling repository. These files are respectively as follows.

• riscv32-corev-elf.cmake

• build-executorch-runtime.sh

5.2. Modifying the build system for CORE-V

We need to create two files: i) in the executorch repository a CMakeLists.txt file in the backends/corev directory; and ii) in the tooling repository, a local CMake parameters file, riscv32-corev-elf.cmake`.

The first of these specifies ExecuTorch configuration specific to the CORE-V backend. The second is focused on compiling C/C++ code for the specific target. It is where we will experiment with things like optimization flags for C and C++ compilation.

5.2.1. The general ExecuTorch runtime CMake configuration for CORE-V

The file we are modifying is backends/corev/CMakeLists.txt. There is no template in the examples directory, so we base our implementation on existing backends.

We start by giving our project a name, corev_backend and ensuring all the compile commands are exported to a JSON file for use elsewhere in the build system.

project(corev_backend)

set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

We need to establish an absolute path to the ExecuTorch source code, and include the general tooling CMake utilities. We can then from this create a variable (_corev_backend_dir) referencing the CORE-V backend runtime directory, and hence the list of source files (in the variable _corev_backend_srcs).

# Source root directory for executorch.
if(NOT EXECUTORCH_ROOT)
  cmake_path(SET EXECUTORCH_ROOT ${CMAKE_CURRENT_SOURCE_DIR}/../..)
  cmake_path(ABSOLUTE_PATH EXECUTORCH_ROOT NORMALIZE)
endif()

include(${EXECUTORCH_ROOT}/tools/cmake/Utils.cmake)

add_compile_definitions(C10_USING_CUSTOM_GENERATED_MACROS)

set(_corev_backend_dir backends/corev/runtime)
list(TRANSFORM _corev_backend_dir PREPEND "${EXECUTORCH_ROOT}/")

set(_corev_backend_srcs ${_corev_backend_dir}/CoreVBackend.cpp)

We will create the runtime as a library, which can eventually be linked into the runtime. We do the initial set up of that library, in the variable corev_delegate and add the associated include directories: those specific to the CORE-V backend and the general include directories.

add_library(corev_delegate STATIC ${_corev_backend_srcs})
target_include_directories(corev_delegate PUBLIC
    ${_corev_backend_dir}
    ${_common_include_directories}
)

The remainder of the CMake file is contingent on actually building the runtime for CORE-V. This is enabled by the CMake variable, EXECUTORCH_BUILD_COREV_RUNNER. The CORE-V backend will only be built if this variable is defined on the CMake command line.

if(EXECUTORCH_BUILD_COREV_RUNNER)

At this point, we include standard CMake utilities for code generation. We then define the libraries that are needed for this backend in the variable _corev_runner_libs. We also specify that the interface to the executorch general library will need the common ExecuTorch include directories.

  include(${EXECUTORCH_ROOT}/tools/cmake/Codegen.cmake)

  # Baseline libraries that corev_runner will link against.
  set(_corev_runner_libs
    executorch
    program_schema
    common_schema
    extension_runner_util
    "-Wl,--whole-archive"
    corev_delegate
    "-Wl,--no-whole-archive"
  )
  target_include_directories(executorch INTERFACE ${_common_include_directories})

Note

The corev_delegate library includes registration code, which superficially appears unused. To ensure this code is included in the library, we wrap the library reference with -Wl,--whole-archive and -Wl,--no-whole-archive.

We then specify the sources that will be needed to create the actual CORE-V runtime program. The actual CORE-V backend functionality is in the corev_delegate library. For the full program we will need the PAL abstraction code, the driver itself, the main ExecuTorch Program class and support utilities for printing values. These are defined in the variable _corev_runner_srcs

  set(_corev_runner_srcs
    ${EXECUTORCH_ROOT}/backends/corev/runtime/corev_pal.cpp
    ${EXECUTORCH_ROOT}/examples/corev/executor_runner/corev_executor_runner.cpp
    ${EXECUTORCH_ROOT}/runtime/executor/program.cpp
    ${EXECUTORCH_ROOT}/extension/evalue_util/print_evalue.cpp
  )

There are full native libraries to provide I/O for the PAL. However these require operating system features such as threads and mutexes, which are not available on a bare metal platform such as ours. Instead we use a portable library, to use just basic PAL features.

  # NOTE: CORE-V Runner cannot support optimized_native_cpu_ops_lib (aka
  #       EXECUTORCH_BUILD_KERNELS_OPTIMIZED)
  #       while threadpool and std::mutex aren't supported. Just use the
  #       generic kernel implementations for now.
  list(APPEND _corev_runner_libs portable_ops_lib)

At this stage we have no quantization in our example, but if were to enable quantization, we would need to add in the standard libraries for handling quantization. This is controlled from the command line by the CMake parameter EXECUTORCH_BUILD_KERNELS_QUANTIZED.

  # Register quantized ops if enabled
  if(EXECUTORCH_BUILD_KERNELS_QUANTIZED)
    list(APPEND _corev_runner_libs quantized_ops_lib)
  endif()

At this point we can define our executable, which we name corev_runner. We add a definition to enable or disable logging as appropriate, and finally add the include directories for the executable.

  add_executable(corev_runner ${_corev_runner_srcs})
  if(EXECUTORCH_ENABLE_LOGGING)
    target_compile_definitions(corev_runner PUBLIC ET_LOG_ENABLED=1)
  else()
    target_compile_definitions(corev_runner PUBLIC ET_LOG_ENABLED=0)
  endif()
  target_include_directories(corev_runner PUBLIC
    ${CMAKE_BINARY_DIR}
    ${_common_include_directories}
  )

The final step is to decide on the general compile and link options we will use. To keep the binary small, we use the --gc-sections link option to remove used sections of code, unless we are debugging. We then specify the libraries we are linking (see libraries definition) and specify the common compile options.

  if(CMAKE_BUILD_TYPE STREQUAL "Release" OR CMAKE_BUILD_TYPE STREQUAL "RelWithDebInfo")
      target_link_options(corev_runner PRIVATE "LINKER:--gc-sections")
  endif()
  target_link_libraries(corev_runner PRIVATE ${_corev_runner_libs})
  target_compile_options(corev_runner PUBLIC ${_common_compile_options})
endif()

5.2.2. The CORE-V target specific configuration

This file, riscv32-corev-elf.cmake` is found in the ``tooling repository. It is used to give specifics of the tool chain. It is specified in the CMAKE_TOOLCHAIN_FILE environment variable, and automatically picked up by CMake. We’ll see later in the script we use to build the CORE-V backend (see The ExecuTorch runtime build script), the following command.

build_target=riscv32-corev-elf
export CMAKE_TOOLCHAIN_FILE="${tooldir}/${build_target}.cmake"

One of the major features of this configuration is pulling values from environment variables which are set in the build script, so they can be passed into the C/C++ programs as defined constants. We start with two of these, the location of the .pte file generated by the ExecuTorch AoT and the location of the crt0.o for the target platform. These are passed respectively in the environment variables MODEL_PATH and CRT0_PATH. Of these MODEL_PATH is mandatory.

if (NOT DEFINED ENV{MODEL_PATH})
  message(FATAL_ERROR "Environment Variable MODEL_PATH not set!")
endif()

set(MODEL_PATH $ENV{MODEL_PATH})
set(CRT0_PATH $ENV{EXECUTORCH_CRT0})

We start by defining where the model we created during the AoT. We define this as a generic system (since we have no operating system) and mark this as cross compiling.

set(CMAKE_SYSTEM_NAME Generic)

set(CMAKE_CROSSCOMPILING TRUE)

We then have a placeholder, were we can add specific assembly and compilation flags for the three possible systems we can build (Release, ReleaseWithDebugInfo and Debug).

set(CMAKE_ASM_FLAGS_RELEASE "")
set(CMAKE_ASM_FLAGS_RELWITHDEBINFO "")
set(CMAKE_ASM_FLAGS_DEBUG "")
set(CMAKE_C_FLAGS_RELEASE "")
set(CMAKE_C_FLAGS_RELWITHDEBINFO "")
set(CMAKE_C_FLAGS_DEBUG "")
set(CMAKE_CXX_FLAGS_RELEASE "")
set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "")
set(CMAKE_CXX_FLAGS_DEBUG "")

We then have a pair of conditional declarations to enable Position Independent Code (PIC) and to enable Link Time Optimization (LTO. This latter is now more correctly known as Interprocedural Optimization (IPO).

# Gated on 'DBG_KEEP_PIC' env variable so we can compare it on and off
if (NOT DEFINED ENV{DBG_KEEP_PIC})
  set(CMAKE_POSITION_INDEPENDENT_CODE OFF)
endif()

# Gated on 'DBG_NO_IPO' env variable so we can compare it on and off
if (NOT DEFINED ENV{DBG_NO_IPO})
  set(CMAKE_INTERPROCEDURAL_OPTIMIZATION_RELEASE TRUE) # Enables LTO for release builds
endif()

Next come the definitions of the various tools being used for compilation and the C and C++ standards which must be supported.

set(CMAKE_C_COMPILER riscv32-corev-elf-gcc)
set(CMAKE_ASM_COMPILER riscv32-corev-elf-gcc)
set(CMAKE_CXX_COMPILER riscv32-corev-elf-g++)
set(CMAKE_LINKER riscv32-corev-elf-ld)

set(CMAKE_OBJCOPY riscv32-corev-elf-objcopy)
set(CMAKE_OBJDUMP riscv32-corev-elf-objdump)
set(CMAKE_AR riscv32-corev-elf-ar)
set(CMAKE_RANLIB riscv32-corev-elf-ranlib)

set(CMAKE_C_STANDARD 23)
set(CMAKE_CXX_STANDARD 20)

Next come modifiers to the compile command lines, driven by environment variables. These include the RISC-V ISA and ABI, the location of the linker script, the board support library and the location of the compiled constructor (crtbegin.o) and destructor (crtend.o) code.

set(ISA "$ENV{EXECUTORCH_RV_ISA}")
set(ABI "$ENV{EXECUTORCH_RV_ABI}")
set(LINKER_SCRIPT "$ENV{EXECUTORCH_LDSCRIPT}")
set(BSP_LIBDIR "$ENV{EXECUTORCH_BSP_LIBDIR}")
set (CRTBEGIN "$ENV{EXECUTORCH_CRTBEGIN}")
set (CRTEND "$ENV{EXECUTORCH_CRTEND}")

These are then used to extend the C++, C and assembly command flags generically, and the conditionally depending on the type of release.

set(CMAKE_CXX_FLAGS_INIT "${CMAKE_CXX_FLAGS_INIT} -march=${ISA} -mabi=${ABI} -fno-strict-aliasing")
set(CMAKE_C_FLAGS_INIT "${CMAKE_C_FLAGS_INIT} -march=${ISA} -mabi=${ABI} -fno-strict-aliasing")
set(CMAKE_ASM_FLAGS_INIT "${CMAKE_ASM_FLAGS_INIT} -march=${ISA} -mabi=${ABI} -fno-strict-aliasing")

add_compile_options(
    $<$<CONFIG:Release>:-Os>

    $<$<CONFIG:RelWithDebInfo>:-Os>
    $<$<CONFIG:RelWithDebInfo>:-g>

    $<$<CONFIG:Debug>:-O1> # O0 code size too big
    $<$<CONFIG:Debug>:-g3>
    $<$<CONFIG:Debug>:-ggdb>
)

Next, we optionally enable logging messages.

# Propagate log disabled definition to all targets (similar to upstream da2223fcf99fc2214a61321d842e0c953dc5dd16)
if(NOT EXECUTORCH_ENABLE_LOGGING)
  add_compile_definitions(ET_LOG_ENABLED=0)
else()
  add_compile_definitions(ET_LOG_ENABLED=1)
endif()

Finally we add some defines to the compile line (VERILATOR, RUNNER_MODEL_PATH) add the options needed for effective garbage collection of sections (--ffunction-sections, --fdata-sections). We also specify (`-nostartfiles), since we are going to define our own crt0.o, crtbegin.o and crtend.o. We then build up the final set of link options based on all the variables defined earlier in the CMake file.

add_compile_definitions(VERILATOR)
add_compile_definitions(RUNNER_MODEL_PATH="${MODEL_PATH}")
add_compile_options(-ffunction-sections -fdata-sections -nostartfiles -fno-PIC)
add_link_options(-march=${ISA} -mabi=${ABI} -lm -v -static --specs=nosys.specs --specs=nano.specs -Wl,--gc-sections -nostartfiles ${CRTBEGIN} ${CRTEND} -T${LINKER_SCRIPT} -L${BSP_LIBDIR} -lcv-verif )

With the output of a final status message, this completes the CORE-V target specific build system configuration.

5.3. The CORE-V runtime backend implementation

5.3.1. The CORE-V runtime backend declaration

This is found in backends/corev/runtime/CoreVBackend.h. It is a derived class of BackendInterface (defined in runtime/backend/interface.h). The definition makes use of the error return values (defined in runtime/core/error.h) and EValues (defined in runtime/core/evalue.h). Having included these headers, we are generally doing to be using functions from the executorch::runtime namespace, for which we need using declaration. We then define our class in the corev namespace within the executorch::backends namespace.

#include <executorch/runtime/backend/interface.h>
#include <executorch/runtime/core/error.h>
#include <executorch/runtime/core/evalue.h>

using namespace executorch::runtime;

namespace executorch::backends::corev {

Note

Good programming practice deprecates the use of using declarations in headers. However it seems to be common in ExecuTorch coding.

The class is derived from BackendInterface

class CoreVBackend final : public BackendInterface {

It then defines four public interface functions. An initialization function to allocate any resources needed and perform any other initialization. This function will return a handle to the resources as an opaque pointer, which can be passed to the execution function (see the diagram at the start of this chapter).

  Result<DelegateHandle*> init(
      BackendInitContext& context,
      FreeableBuffer* processed,
      ArrayRef<CompileSpec>
          compile_specs) const override;

Note

As much of the initialization work as possible should be moved in the AoT.

Then the function to actually execute the behavior of the function. It returns an error code indicating success, or the type of failure.

  Error execute(
      ET_UNUSED BackendExecutionContext& context,
      DelegateHandle* handle,
      Span<EValue*> args) const override;

A function to release resources at the end of execution. This is a void function. Any failures are ignored.

  void destroy(DelegateHandle* handle) const override;

And finally a predicate to indicate if the backend is available to process delegation calls.

  bool is_available() const override;

5.3.2. The CORE-V runtime backend definition

This is found in backends/corev/runtime/CoreVBackend.cpp. This will be a minimal implementation at this stage, since we are providing no delegation support at this stage.

We start by including the declaration header. We again using interface for the executorch::runtime namespace. Then our implementations for the CoreVBackend class are within the executorch::backends::corev namespace.

#include <executorch/runtime/core/error.h>
#include "CoreVBackend.h"

using namespace executorch::runtime;

The initialization function need do nothing, and since we will be doing no execution can return a nullptr as the handle to pass on.

Result<DelegateHandle*> CoreVBackend::init(
    BackendInitContext& context,
    FreeableBuffer* processed,
    ArrayRef<CompileSpec> compile_specs) const {
  return nullptr;

The execution function, similarly need do nothing. We choose to return a success error code, although it should never actually be called.

Error CoreVBackend::execute(
    ET_UNUSED BackendExecutionContext& context,
    DelegateHandle* handle,
    Span<EValue*> args) const {
  return Error::Ok;

Since we have no resources to release, the destruction function need do nothing.

void CoreVBackend::destroy(DelegateHandle* handle) const {
  return;

Finally the availability predict can return true. Again it doesn’t really matter what it returns, since it will never be called.

There is one final part of this source file, which is the code to register the backend with the wider ExecuTorch system. This is done in an unnamed namespace to avoid polluting other namespaces, since nothing here is needed elsewhere.

We instantiate an instance of the CoreVBackend class we have just defined. We then instantiate an instance of the ExecuTorch Backend class, passing in the name of our class ("CoreVBackend") and a reference to the instance of the CoreVBackend class we just created.

Finally we pass the new ExecuTorch Backend instance to the ExecuTorch register_backend function. We capture the result of that call, but do nothing with it.

namespace {
auto cls = executorch::backends::corev::CoreVBackend();
executorch::runtime::Backend backend{"CoreVBackend", &cls};
volatile static auto success_with_compiler =
    executorch::runtime::register_backend(backend);

Because the code is purely for side-effect, and the values are in an unnamed space, there is a risk the code may not be linked. This is why we wrapped the linking of the CORE-V backend library with -Wl,--whole-archive and -Wl,--no-whole-archive earlier (see why we use –whole-archive. There is a risk that even this may not be enough if LTO is enabled. In this case the result variable, success_with_compiler should be declared volatile.

5.3.3. The Platform Abstraction Layer

The CORE-V PAL implementation can be found in backends/corev/runtime/corev_pal.cpp. This provides implementation of the functions in the PAL API. This is declared (and well documented) in the runtime/platform/platform.h header. The implementation for a bare metal platform can be very simple.

We include the platform.h header and then define the initialization function for the PAL. This need do nothing.

#include <executorch/runtime/platform/compiler.h>
#include <executorch/runtime/platform/platform.h>
#include <cstdlib>
#include <stdio.h>

On an embedded system, aborting has limited meaning. For now we trigger a trap, and the code will just loop permanently in the trap handler code.

ET_NORETURN void et_pal_abort(void) {
  __builtin_trap();

We can provide a meaningful implementation of the timing function et_pal_current_ticks. We have access to the RISC-V cycle count CSRs, mcycle and mcycleh, which gives us a 64-bit cycle count we can use as the result.

et_timestamp_t et_pal_current_ticks(void) {
  uint32_t cycle_lo, cycle_hi;
  uint64_t cycles;
  asm volatile(
      "csrr t1, mcycle\n"
      "sw t1, %0\n"
      "csrr t1, mcycleh\n"
      "sw t1, %1\n"
      : "=m"(cycle_lo), "=m"(cycle_hi)
      :
      : "t1", "memory");
  cycles = static_cast<uint64_t> (cycle_hi) << 32;
  cycles |= static_cast<uint64_t> (cycle_lo);
  return static_cast<et_timestamp_t> (cycles);

The PAL converts ticks to nanoseconds using a ratio, thus avoiding use of floating point arithmetic. Since we are modeling a 400MHz system, 5 ticks is 2 cycles, and we can use the ratio of 5/2 to convert.

et_tick_ratio_t et_pal_ticks_to_ns_multiplier(void) {
  return {5, 2};

The PAL output function is relatively complex, since it is intended for categorized log messages. Even though our target platform is embedded it has support for writing to standard output, which will appear on the console when run on a Verilator model of the target. We can take a simple standard implementation based on printf to produce the message.

void et_pal_emit_log_message(
    et_timestamp_t timestamp,
    et_pal_log_level_t level,
    const char* filename,
    const char* function,
    size_t line,
    const char* message,
    ET_UNUSED size_t length) {
  const char *level_name;
  switch (level)
    {
    case 'D': level_name = "Debug"; break;
    case 'I': level_name = "Info"; break;
    case 'E': level_name = "ERROR"; break;
    case 'F': level_name = "*** FATAL ERROR ***"; break;
    default: level_name = "??? UNKNOWN ???"; break;
    }
  printf ("%lu: %s %s:%s:%d %s\n", timestamp, level_name, filename, function,
	  line, message);

Lastly we have functions to allocate and free memory. Hopefully not often used, since this is not a good practice in resource constrained systems. We just wrap C malloc and free functions for this.

void* et_pal_allocate(size_t size) {
  return malloc(size);
}

void et_pal_free(void* ptr) {
  free(ptr);

With this we now have everything to create a library of backend runtime functionality for CORE-V

5.4. The CORE-V runtime backend driver

So far we have built a minimal ExecuTorch runtime library for CORE-V. We now need to create a driver using that library, so we have a complete application.

As with the AoT driver, this is placed in the examples/corev directory, this time in an executor_runner subdirectory.

• examples/corev/executor_runner/corev_executor_runner.cpp

We can use examples/portable/executor_runner/executor_runner.cpp as a starting point for our code.

5.4.1. Including the AoT generated model in the runtime

If we have a file system, we would just read the model in from its .pte file. However in a bare metal system this is not possible. Instead we want to include the file image as a byte array. In the C23 standard, there is now a directive, #embed, to just include a binary file as a byte array.

However most Linux distributions do not have a compiler which supports #embed, so we use an alternative approach. We can use the Linux xxd hex dump utility to convert the .pte file. The -i flag to this command generates the output as a series of byte values in a format suitable for inclusion in a C/C++ byte array.

xxd -i < model.pte > model.pte.inc

We can inspect model.pte.inc and see it is in the appropriate format.

more model.pte.inc
0x24, 0x00, 0x00, 0x00, 0x45, 0x54, 0x31, 0x32, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x18, 0x00,
...
0x66, 0x6f, 0x72, 0x77, 0x61, 0x72, 0x64, 0x00

This will then be used to embed data in the source of the runtime driver (see the next section).

alignas(alignof(std::max_align_t)) const uint8_t model_pte[] = {
  #include "model.pte.inc"
};

With a modern compiler, we could omit the conversion using xxd and just use instead.

alignas(alignof(std::max_align_t)) const uint8_t model_pte[] = {
  #embed "model.pte"
};

5.4.2. The main driver program

The main driver program, corev_executor_runner.cpp creates a main function which invokes the functionality of the runtime library we have created. The program is placed in the examples/corev/executor_runner. It is heavily based on the generic runtime driver in examples/portable/executor_runner/executor_runner.cpp.

We start by bringing in some standard C++ headers to give us the fixed size integer declarations (cstdint), Standard Template Library (STL) vectors (vector) and STL smart pointers (memory).

#include <cstdint>
#include <memory>
#include <vector>

We then bring in the headers for the functionality we need from the ExecuTorch runtime.

#include <executorch/extension/data_loader/buffer_data_loader.h>
#include <executorch/extension/evalue_util/print_evalue.h>
#include <executorch/extension/runner_util/inputs.h>
#include <executorch/runtime/core/event_tracer.h>
#include <executorch/runtime/executor/method.h>
#include <executorch/runtime/executor/program.h>
#include <executorch/runtime/platform/log.h>
#include <executorch/runtime/platform/platform.h>
#include <executorch/runtime/platform/runtime.h>

There is one more header, and we include it separately. This is to allow us direct use of printf, which will pass output to the console when we run our target as a Verilator model.

// So we can use printf to stdout when simulating
#include <cstdio>

At this point we create an array by including the model generated by the ExecuTorch AoT (see Running the ExecuTorch AoT). The actual path to the model should have been defined in preprocessor variable RUNNER_MODEL_PATH. We check first that this actually exists, and if not throw an error.

#ifndef RUNNER_MODEL_PATH
  #error "RUNNER_MODEL_PATH definition has not been specified!"
#endif

alignas(alignof(std::max_align_t)) const uint8_t model_pte[] = {
  #include RUNNER_MODEL_PATH
};

Note

We align this to the maximum alignment of the architecture. This is not actually needed at this stage, but it will be useful later in this application note (see why align the memory buffers).

Dynamically allocating memory in embedded systems is not a good idea in general. You really want to know that your program is going to fit. To address this, ExecuTorch allocates two fixed buffers for use by models at the start of execution. One is for the various data structures declared for the model (method_allocator_pool), the second is for temporary intermediate data structures needed while evaluating individual operators (temp_allocator_pool). We are running on a 4MiB platform, so we keep these small.

// These are pretty small pools, but we are a 4MB system!
static uint8_t method_allocator_pool[1 * 1024U * 1024U]; // 1 MB
static uint8_t temp_allocator_pool[1024U * 256U]; // 256KB

The standard code we are basing this on allows a model to be run multiple times, perhaps with different data. The main loop is controlled by a count FLAGS_num_executions, which in our implementation we just define as constant 1, so we only ever execute a model once.

constexpr uint32_t FLAGS_num_executions = 1;

For ease of reading there are then using declarations for common classes from the ExecuTorch runtime.

using executorch::runtime::Error;
using executorch::runtime::EValue;
using executorch::runtime::HierarchicalAllocator;
using executorch::runtime::MemoryAllocator;
using executorch::runtime::MemoryManager;
using executorch::runtime::Method;
using executorch::runtime::MethodMeta;
using executorch::runtime::Program;
using executorch::runtime::Result;
using executorch::runtime::Span;

We now come to the main program. It starts by calling the standard ExecuTorch runtime initialization.

int main() {
  executorch::runtime::runtime_init();

We need to load the model created by the AoT. This is a two stage process. First we need to create a loader suitable for the representation of the model. In our case a byte vector in memory.

  // Create a loader to get the data of the program file. There are other
  // DataLoaders that use mmap() or point to data that's already in memory, and
  // users can create their own DataLoaders to load from arbitrary sources.
  constexpr size_t program_data_len = sizeof(model_pte);
  ET_LOG(Info, "Size of model is %zu.", sizeof(model_pte));
  BufferDataLoader loader = BufferDataLoader(&model_pte, program_data_len);

Note

The ET_LOG macro is defined in runtime/platform/log.h and provides the main interface to the logging infrastructure (documented in that file). For us one useful feature is that if logging is not enabled at configuration time, to code is generated at all.

Then we need to load the program, parsing it in the process.

  // Parse the program file. This is immutable, and can also be reused between
  // multiple execution invocations across multiple threads.
  Result<Program> program = Program::load(&loader);
  if (!program.ok()) {
    ET_LOG(Error, "Failed to parse model file %s", model_pte);
    return 1;
  }
  ET_LOG(Info, "Model file %s is loaded.", model_pte);

Important

We see here a standard programming paradigm in ExecuTorch, in which we wrap results with an error code, using the Result class (defined in executorch/runtime/core/result.h). The class holds a union of the value (of the template type) and a result (of class Error, defined in executorch/runtime/core/error.h). The various operators are overloaded, so that indirection on an instance will yield the underlying value.

Thus a result of this single type can hold either a value or an error code. This makes for more compact coding when testing for success or failure of a method which returns a result. In this case we expect Program::load to return the loaded program if it parsed successfully, or else an error code.

Having loaded the program, we can get its first method. We still use a Result wrapper, but for convenience we also keep a copy of its contents, since it is used several times later in the main function.

  // Use the first method in the program.
  const char* method_name = nullptr;
  {
    const auto method_name_result = program->get_method_name(0);
    ET_CHECK_MSG(method_name_result.ok(), "Program has no methods");
    method_name = *method_name_result;
  }
  ET_LOG(Info, "Using method %s", method_name);

If the test fails, the supplied message is reported through the logging system and then the PAL abort method is called.

Note

The ET_CHECK_MSG macro is one of several convenience macros in runtime/platform/assert.h for handling values wrapped in Result.

Having obtained the method, we need to look at its metadata to see the resources it requires. The metadata is kept in its own structure, so that it can be inspected without the initialization cost of initializing the full method.

  // MethodMeta describes the memory requirements of the method.
  Result<MethodMeta> method_meta = program->method_meta(method_name);
  ET_CHECK_MSG(
      method_meta.ok(),
      "Failed to get method_meta for %s: 0x%" PRIx32,
      method_name,
      (uint32_t)method_meta.error());

We now need to allocate the memory for the method. This will be used for all the dynamic C++ metadata and objects used to represent the loaded method. If there is not enough memory, we will fail with an error at this point.

The ExecuTorch runtime does not use malloc or new. It instead uses a memory manager (class MemoryManager) provided for the particular target. In our case, we use a memory allocator based on the statically allocated pool, method_allocator_pool, declared earlier (see declaration of memory pools).

In addition we may need to allocate temporary memory, for use by the kernels which implement the operators of the method. In our case, we use a memory allocator based on the statically allocated pool, temp_allocator_pool, declared earlier (see declaration of memory pools).

  // In this example we use a statically allocated memory pool.
  MemoryAllocator method_allocator{
      MemoryAllocator(sizeof(method_allocator_pool), method_allocator_pool)};

  // Temporary memory required by kernels
  MemoryAllocator temp_allocator{
      MemoryAllocator(sizeof(temp_allocator_pool), temp_allocator_pool)};

Note

At this point, we have not allocated any memory, just created the memory allocators we need for such allocation.

The actual memory allocation is more complex, because we can associate arbitrarily complex memory hierarchies to the target. These are captured in the meta data of the model as a series of memory blocks of different sizes. It is up to the memory manager for the target to determine how to do the allocation.

We will represent the sequence of memory areas for the method using the ExecuTorch HierarchicalAllocator class. We extract the sequence of buffer sizes from the method metadata, and for each buffer create a unique pointer to an array of bytes of that size. We can then pass the vector of pointers and vector of corresponding sizes to the constructor for an instance of HierarchicalAllocator.

  // Each buffer typically corresponds to a different hardware memory bank. Most
  // mobile environments will only have a single buffer. Some embedded
  // environments may have more than one for, e.g., slow/large DRAM and
  // fast/small SRAM, or for memory associated with particular cores.
  std::vector<std::unique_ptr<uint8_t[]>> planned_buffers; // Owns the memory
  std::vector<Span<uint8_t>> planned_spans; // Passed to the allocator
  size_t num_memory_planned_buffers = method_meta->num_memory_planned_buffers();
  for (size_t id = 0; id < num_memory_planned_buffers; ++id) {
    // .get() will always succeed because id < num_memory_planned_buffers.
    size_t buffer_size =
        static_cast<size_t>(method_meta->memory_planned_buffer_size(id).get());
    ET_LOG(Info, "Setting up planned buffer %zu, size %zu.", id, buffer_size);
    planned_buffers.push_back(std::make_unique<uint8_t[]>(buffer_size));
    planned_spans.push_back({planned_buffers.back().get(), buffer_size});
  }
  HierarchicalAllocator planned_memory(
      {planned_spans.data(), planned_spans.size()});

Note

At this point, we have still not allocated any memory, but we have a detailed description of the memory areas that are available to the runtime.

We can then create a memory manager for the method. It needs the memory allocator for the method memory, and its associated description (as a HierarchicalAllocator), along with the allocator for temporary memory (which has no detailed structure).

  // Assemble all of the allocators into the MemoryManager that the Executor
  // will use.
  MemoryManager memory_manager(
      &method_allocator, &planned_memory, &temp_allocator);

Note

The memory manager has still not allocated any memory, but it is passed to the method when it is loaded, so that it can allocate memory as needed.

A this point, we can load the actual method, passing it the memory manager to allocate memory as needed.

  Result<Method> method = program->load_method(
      method_name, &memory_manager, nullptr);
  ET_CHECK_MSG(
      method.ok(),
      "Loading of method %s failed with status 0x%" PRIx32,
      method_name,
      (uint32_t)method.error());
  ET_LOG(Info, "Method loaded.");

With the model loaded, we can now run it. As noted earlier (see defintion of FLAGS_num_executions, we could have multiple invocations. In our program we have just a for loop, controlled by a constant value, FLAGS_num_executions, whose value is 1. So we will only execute our model once.

  // Set up the inputs and run the model.
  for (uint32_t i = 0; i < FLAGS_num_executions; i++) {
    ET_LOG(Debug, "Preparing inputs.");

At this point we need to set up the input data for this model run. Our model adds two 5-element vectors of type float32. We are not concerned with specific data at this stage, so for now, we can just set all the inputs to 1.

For this we can use the prepare_input_tensors method. We pass in the method, and from its metadata, it can see the number and shape of the inputs. With default values of its remaining parameters, it will just set each value to 1.

    // Allocate input tensors and set all of their elements to 1. The `inputs`
    // variable owns the allocated memory and must live past the last call to
    // `execute()`.
    //
    // NOTE: we have to re-prepare input tensors on every execution
    // because inputs whose space gets reused by memory planning (if
    // any such inputs exist) will not be preserved for the next
    // execution.
    auto inputs = executorch::extension::prepare_input_tensors(*method);
    ET_CHECK_MSG(
        inputs.ok(),
        "Could not prepare inputs: 0x%" PRIx32,
        (uint32_t)inputs.error());
    ET_LOG(Debug, "Inputs prepared.");

Hint

We will see how to set specific input values later in this application note (see setting explicit input values).

So now there is only one thing left to do, which is to execute the model and check it completed successfully.

    Error status = method->execute();
    ET_CHECK_MSG(
        status == Error::Ok,
        "Execution of method %s failed with status 0x%" PRIx32,
        method_name,
        (uint32_t)status);

Having completed our execution, we can put out a message to this effect. The correct way to do this is to use the logging infrastructure, but as we shall see shortly (see Running the ExecuTorch runtime on the Verilator model), this is something we do not want to enable for a very small microcontroller. Instead, we cheat and use printf to print the output message.

  // Horrible kludge, because ET logging is too big!
  printf("Model executed successfully %" PRIu32 " time(s).\n",
	 FLAGS_num_executions);
  std::vector<EValue> outputs(method->outputs_size());
  ET_LOG(Info, "%zu outputs: ", outputs.size());
  Error status = method->get_outputs(outputs.data(), outputs.size());
  ET_CHECK(status == Error::Ok);

Having completed everything, we can return successfully.

  return 0;
}

In a real embedded system, returning from main is not useful. However with our Verilator model this is detected and causes the model to terminate, reporting execution statistics.

5.5. Building and running ExecuTorch runtime for CORE-V

As we saw in Modifying the build system for CORE-V, there are a great many parameters configuring how to build the ExecuTorch runtime. We use a script for convenience.

5.5.1. The ExecuTorch runtime build script

The script is build-executorch-runtime.sh in the main tooling repository.

We start with a helper shell function used to report errors or if the --help option is specified.

# Help message
display_usage() {
    cat <<EOF
Usage for $0:
  [--clean]                       Erase build & install directories
  [--mode <mode>]                 Build type: Debug, RelWithDebInfo, Release
  [--verbose]                     More detail in the log
  [--model <path>]                Set .pte or .pte.inc path for executor_runner
  [--executorch-logging <level>]  Executoch loging: Debug, Info, Error, Fatal
  [-h | --help]                   Present this message and exit
EOF
}

To avoid embarrassment, we provide a simple check we are running in a Python virtual environment.

# Give up if we don't have a Python virtual environment
if [[ ! -v VIRTUAL_ENV ]]
then
    echo "No Python virtual environment set." >&2
    echo "Please create or activate your venv before running this script!" >&2
    echo "Example: " >&2
    echo "   python3 -m venv .venv" >&2
    echo "   source .venv/bin/activate" >&2
    echo "   pip install --upgrade pip" >&2
    exit 2
fi

We then identify the top level directory—the one in the directory about the one containing this script. From this we can derive the other directories of interest.

# Top level and derived directories
topdir="$(dirname "$(cd "$(dirname "$0")" && echo "$PWD")")"
tooldir=${topdir}/tooling
builddir="${topdir}"/build
logdir="${topdir}"/logs
installdir="${topdir}"/install
executorchdir="${topdir}"/executorch

Our target is specified using a standard GNU triplet. We put this in a local variable. It is the only target supported in this project, but in principle we could set this using a command line option to generalize the script for multiple architectures.

# Fixed parameters
build_target=riscv32-corev-elf

We then set defaults for the various parameters that can be set from the command line.

# Default parameter values that can be changed on the command line
build_cmake_type=RelWithDebInfo
doclean=false
is_verbose=OFF
et_logging=OFF
et_log_level=Fatal
model_path="${executorchdir}"/model.pte

We can now process the command line, to override any of the parameter defaults. We’ll process any unrecognized parameters with an error message.

set +u
until
    opt="$1"
    case ${opt} in
	"--clean")
	    doclean=true;
	    ;;
	"--mode")
	    shift
	    build_cmake_type=$1
	    ;;
	"--verbose")
	    is_verbose=ON
	    ;;
	"--model"*)
	    shift
	    model_path=$1
	    ;;
	"--executorch-logging"*)
	    shift
	    et_log_level=$1
	    et_logging=ON
	    ;;
	"-h|--help")
	    display_usage
	    exit 0
	    ;;
	?*)
	    echo "Unknown argument ${opt}" >&2
	    display_usage >&2
	    exit 1
	    ;;
    esac
    [[ "${opt}" == "" ]]
do
    shift
done
set -u

As described in Modifying the build system for CORE-V, we have a target specific CMake configuration file, whose name is derived from the target triplet. We check this is found.

# Check that we have a valid target
if [ ! -f "${tooldir}"/"${build_target:?}".cmake ]
then
    echo "ERROR: ${build_target:?}.cmake not found"
    display_usage >&2
    exit 1
fi

A number of the parameters configuring CMake are set by environment variables. We assign these and export them.

# Parameters set by environment variable
export EXECUTORCH_RV_ABI=ilp32
export EXECUTORCH_RV_ISA=rv32imc_zicsr_zifencei_xcvhwlp_xcvmem_xcvbitmanip_xcvalu_xcvbi_xcvmac_xcvsimd
export EXECUTORCH_LDSCRIPT="${topdir}/core-v-verif/cv32e40p/bsp/link.ld"
export EXECUTORCH_BSP_LIBDIR="${topdir}/core-v-verif/cv32e40p/bsp"
export EXECUTORCH_CRT0="${topdir}/core-v-verif/cv32e40p/bsp/crt0.S"
export CMAKE_TOOLCHAIN_FILE="${tooldir}/${build_target}.cmake"
EXECUTORCH_CRTBEGIN=$(riscv32-corev-elf-gcc -mabi=${EXECUTORCH_RV_ABI} \
    -march=${EXECUTORCH_RV_ISA} -### /dev/null 2>&1 | grep crtbegin.o | \
    sed -e 's/^.* \([^ ]\+crtbegin.o\).*$/\1/')
EXECUTORCH_CRTEND=$(riscv32-corev-elf-gcc -mabi=${EXECUTORCH_RV_ABI} \
    -march=${EXECUTORCH_RV_ISA} -### /dev/null 2>&1 | grep crtend.o | \
    sed -e 's/^.* \([^ ]\+crtend.o\).*$/\1/')
export EXECUTORCH_CRTBEGIN
export EXECUTORCH_CRTEND

We create the build and install directories if they do not exist. If a clean build has been requested, the contents of these directories is deleted.

# Create build directory (Use :? to verify path as specified in SC2115)
mkdir -p "${builddir}"

if "${doclean}"
then
    rm -rf "${builddir:?}"/*
fi

# Create an install directory
mkdir -p "${installdir}"

if "${doclean}"
then
  rm -rf "${installdir:?}"/*
fi

We create a unique log file for each run, timestamped when the run occurred. This is placed in the log directory, which is created if necessary.

# Set up log files
mkdir -p "${logdir}"
logf="${logdir}"/"executorch-runtime-$(date +%Y%m%d-%H%M%S)".log
touch "${logf}"

Having done all the setup, we start the build process, with a time stamp to the log file and switching to the build directory.

cd "${builddir}"
echo "Starting at $(date)" 2>&1 | tee -a "${logf}"

The first step is to check we have a model file.

# Check we have a model and then get it into the right format if not already
# done.
if [ ! -f "${model_path}" ]
then
  echo "ERROR: Model does not exist! (${model_path})" >&2
  exit 1
fi

We then convert it to a C array of bytes using xxd as described in Including the AoT generated model in the runtime.

if [ -f "${model_path}" ]
then
    if [[ ${model_path} == *.pte.inc ]]
    then
	echo "Specified model is already a C byte array." 2>&1 | \
	    tee -a "${logf}"
	export MODEL_PATH="${model_path}"
    elif [[ ${model_path} == *.pte ]]
    then
	echo "Generating C byte array from model." 2>&1 | tee -a "${logf}"
	modelname=$(basename "${model_path}")
	xxd -i < "${model_path}" > "${builddir}"/"${modelname}".inc
	export MODEL_PATH="${builddir}"/"${modelname}".inc
    else
	echo "ERROR: Could not create model include file" 2>&1 \
	    | tee -a "${logf}"
	exit 1
    fi
fi

We can now configure the ExecuTorch runtime ready to be built. There are a number of CMake parameters on the command line which we have not described, but which are defined. Most will be self-evident, but they are all described within the ExecuTorch online documentation. Some are controlled with the parameters set within the script.

# Build the Executorch runtime components and executor runner
echo "Configuring Executorch Runtime..." 2>&1 | tee -a "${logf}"

cmake -G"Ninja" \
      -DCMAKE_INSTALL_PREFIX="${installdir}" \
      -DCMAKE_BUILD_TYPE="${build_cmake_type}" \
      -DPYTHON_EXECUTABLE=python3 \
      -DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
      -DEXECUTORCH_BUILD_EXECUTOR_RUNNER=OFF \
      -DEXECUTORCH_BUILD_COREV=ON \
      -DEXECUTORCH_BUILD_COREV_RUNNER=ON \
      -DEXECUTORCH_USE_DL=OFF \
      -DEXECUTORCH_XNNPACK_SHARED_WORKSPACE=OFF \
      -DEXECUTORCH_XNNPACK_ENABLE_KLEIDI=OFF \
      -DEXECUTORCH_BUILD_PTHREADPOOL=OFF \
      -DEXECUTORCH_BUILD_CPUINFO=OFF \
      -DEXECUTORCH_BUILD_GFLAGS=OFF \
      -DEXECUTORCH_ENABLE_LOGGING="${et_logging}" \
      -DEXECUTORCH_LOG_LEVEL="${et_log_level}" \
      -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=OFF \
      -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
      -DEXECUTORCH_BUILD_PORTABLE_OPS=ON \
      -DFLATC_EXECUTABLE="$(which flatc)" \
      -DCMAKE_VERBOSE_MAKEFILE="${is_verbose}" \
      -B "${builddir}" -S "${executorchdir}" >> "${logf}" 2>&1

Having configured the runtime, we can then use standard CMake build and install commands to build the runtime and install it. However at this stage we install in the build directory, not the install directory.

echo "Building Executorch Runtime..." 2>&1 | tee -a "${logf}"
cmake --build "${builddir}" -j "$(nproc)" >> "${logf}" 2>&1

echo "Installing Executorch Runtime in ${installdir}" 2>&1 tee -a "${logf}"
cmake --install "${builddir}" >> "${logf}" 2>&1

We then move the installed executable from the build directory to the install directory.

echo "Installing Executor Runner in ${installdir}" 2>&1 tee -a "${logf}"
cp "${builddir}"/backends/corev/corev_runner \
   "${installdir}"/executor_runner >> "${logf}" 2>&1

The reason for this, is that the install process puts the executable down in the file hierarchy, and this way we can put it at the top of the install directory.

The final step is to take the executable and put it into Verilog hexdump format, suitable for use with the Verilator model. We conclude with a timestamp in the log.

echo "Creating hexdump in ${installdir}" 2>&1 | tee -a "${logf}"
${build_target}-objcopy -O verilog \
   "${installdir}/executor_runner" "${installdir}/executor_runner.hex"

echo Finishing at "$(date)" 2>&1 | tee -a "${logf}"

5.5.2. Running the ExecuTorch runtime on the Verilator model

We can use our script from the tooling repository to build the ExecuTorch runtime. It is tempting to turn on logging, so we can see detail as everything runs.

./build-executorch-runtime.sh --executorch-logging Info

The problem is that the ExecuTorch runtime has a lot of logging, and this adds a few MiB to the binary size. Given we only have a 4MiB micro-controller we end up with a program that will not fit. So we build without logging (the default), and use explicitly printf wherever we want to produce some output.

./build-executorch-runtime.sh

Assuming this all builds correctly, we can now just pass the binary in hexdump format to the Verilator model.

../core-v-verif/cv32e40p/sim/core/cobj_dir/Vtb_top_verilator \
    +firmware=../install/executor_runner.hex

The output will be as follows:

Model executed successfully 1 time(s).
TOP.tb_top_verilator @ 113333120: EXIT SUCCESS
- /home/jeremy/gittrees/ean16/core-v-verif/cv32e40p/tb/core/tb_top_verilator.sv:83: Verilog $finish

The initial output is some logging from the Verilator model. It is the highlighted lines that matter. The model executed successfully, taking 113,333,120 ps of simulated time and existed the Verilator model.

The Verilog of this core is defined with a nominal clock period of 10ps (i.e. 100GHz). However this is just a simulation, and we can divide the time by 10 to get the number of cycles. A typical low power silicon implementation of this MCU would run at 400MHz, and from the cycle count we can work out the real world execution time.

Verilator time	113,333,120 ps
Cycle count	11,333,312
Time @ 400 MHz	28.33 ms

Now 28.33 ms does not seem impressive to add 5 single precision floating point values, even without a floating point unit. There is a lot of overhead in ExecuTorch. We’ll look at this in much more detail in Benchmarking ExecuTorch.