.. Embecosm Application Note "Howto: Bringing up ExecuTorch on a Bare Metal
   Microcontroller". ExecuTorch runtime implementation.

   Copyright (C) 2026 Embecosm Limited
   Contributor: Jeremy Bennett
   Contributor: Pietra Ferreira
   Contributor: William Jones
   Contributor: Shane Slattery

   SPDX-License-Identifier: CC-BY-SA-4.0

.. _executorch_runtime:

*****************************************************
Implementing an ExecuTorch runtime for the CV32E40Pv2
*****************************************************

.. index:: ! ExecuTorch runtime
.. index:: ExecuTorch; runtime see: ExecuTorch runtime
.. index:: ! ExecuTorch runtime; stages

The runtime processing in :term:`ExecuTorch` is essentially quite similar.
For each operator, we need to load the method associated with an operator and
connect it to its data.  Then we need to execute the method.  In the case of a
delegated operator, the loading and execution will be handed off to custom
code.  This is summarized in the following diagram (based on `this diagram
<https://docs.pytorch.org/executorch/stable/compiler-delegate-and-partitioner.html>`_
in the :term:`ExecuTorch` online user guide).

.. image:: images/executorch-runtime-stages.png
   :align: center

Since we are not supporting any :term:`delegation` at this stage, we need to
just provide empty functions for the :term:`delegation` activities at this
stage.

Files to be changed
===================

.. index:: ean16-1.0-runtime (git tag)

The changes to the ``executorch`` repository described here can be found in the
commit tagged ``ean16-1.0-runtime``.  It can be selected using

.. code-block:: bash

   git switch --detach ean16-1.0-runtime

.. index:: BackendInterface (class)
.. index:: CoreVBackend (class)
.. index:: CoreVBackend.cpp (file)
.. index:: CoreVBackend.h (file)

Runtime support for the :term:`CORE-V` backend will be placed in the
``runtime`` subdirectory of ``backends/corev``. We will implement the
``CoreVBackend`` C++ class as a derived class of ``BackendInterface``.  This
declared and defined in the following files.

* ``backends/corev/runtime/CoreVBackend.h``
* ``backends/corev/runtime/CoreVBackend.cpp``

.. index:: ExecuTorch runtime; physical abstraction layer (PAL)
.. index:: corev_pal.cpp (file)

In addition, we need to define the Physical Abstraction Layer (:term:`PAL`)
for our platform.  This goes in the following file.

* ``backends/corev/runtime/corev_pal.cpp``

.. index:: corev_executor_runner.cpp (file)

As with the :term:`AoT`, we then need a driver for the new backend runtime in
the ``examples`` directory.  We already have a ``corev`` subdirectory, the
runtime driver is placed in the ``executor_runner`` subdirectory

* ``examples/corev/executor_runner/corev_executor_runner.cpp``

.. index:: ! CMakeLists.txt (file)

We will need to add a ``CMakeLists.txt`` file to control building the
:term:`CORE-V` backend, which is placed the ``backends/corev`` directory,
*not* the ``runtime`` subdirectory.

* ``backends/corev/CMakeLists.txt``

The detailed compilation parameters for this will be picked up from a
:term:`CMake` configuration file at the top level of the ``tooling``
repository.  There is considerable complexity to the compilation of the
runtime, so we also provide a script to run then the entire process, also at
the top level of the ``tooling`` repository.  These files are respectively as
follows.

.. index:: riscv32-corev-elf.cmake (file)
.. index:: build-executorch-runtime.sh (file)

* ``riscv32-corev-elf.cmake``
* ``build-executorch-runtime.sh``

.. _modifying_runtime_build_system:

Modifying the build system for CORE-V
=====================================

.. index:: ExecuTorch runtime; building

We need to create two files: i) in the ``executorch`` repository a
``CMakeLists.txt`` file in the ``backends/corev`` directory; and ii) in the
``tooling`` repository, a local CMake parameters file,
``riscv32-corev-elf.cmake```.

The first of these specifies :term:`ExecuTorch` configuration specific to the
:term:`CORE-V` backend.  The second is focused on compiling C/C++ code for the
specific target.  It is where we will experiment with things like optimization
flags for C and C++ compilation.

The general ExecuTorch runtime CMake configuration for CORE-V
-------------------------------------------------------------

.. index:: CMakeLists.txt (file)
.. index:: ExecuTorch runtime; CMake configuration

The file we are modifying is ``backends/corev/CMakeLists.txt``.  There is no
template in the ``examples`` directory, so we base our implementation on
existing backends.

We start by giving our project a name, ``corev_backend`` and ensuring all the
compile commands are exported to a :term:`JSON` file for use elsewhere in the
build system.

.. literalinclude:: code-samples/runtime/CMakeLists.txt
   :language: CMake
   :lines: 12-14

We need to establish an absolute path to the :term:`ExecuTorch` source code,
and include the general tooling :term:`CMake` utilities.  We can then from
this create a variable (``_corev_backend_dir``) referencing the :term:`CORE-V`
backend runtime directory, and hence the list of source files (in the variable ``_corev_backend_srcs``).

.. literalinclude:: code-samples/runtime/CMakeLists.txt
   :language: CMake
   :lines: 16-29

We will create the runtime as a library, which can eventually be linked into
the runtime.  We do the initial set up of that library, in the variable
``corev_delegate`` and add the associated include directories: those specific
to the :term:`CORE-V` backend and the general include directories.

.. literalinclude:: code-samples/runtime/CMakeLists.txt
   :language: CMake
   :lines: 31-35

The remainder of the :term:`CMake` file is contingent on actually building the
runtime for :term:`CORE-V`.  This is enabled by the :term:`CMake` variable,
``EXECUTORCH_BUILD_COREV_RUNNER``.  The :term:`CORE-V`  backend will only be
built if this variable is defined on the :term:`CMake` command line.

.. literalinclude:: code-samples/runtime/CMakeLists.txt
   :language: CMake
   :lines: 37

.. _libraries_definition:

At this point, we include standard :term:`CMake` utilities for code
generation.  We then define the libraries that are needed for this backend in
the variable ``_corev_runner_libs``.  We also specify that the interface to
the ``executorch`` general library will need the common :term:`ExecuTorch`
include directories.

.. literalinclude:: code-samples/runtime/CMakeLists.txt
   :language: CMake
   :lines: 38-50

.. _runtime_use_of_whole_archive:

.. note::

   The ``corev_delegate`` library includes registration code, which
   superficially appears unused.  To ensure this code is included in the
   library, we wrap the library reference with ``-Wl,--whole-archive`` and
   ``-Wl,--no-whole-archive``.

We then specify the sources that will be needed to create the actual CORE-V
runtime program.  The actual :term:`CORE-V` backend functionality is in the
``corev_delegate`` library.  For the full program we will need the :term:`PAL`
abstraction code, the driver itself, the main :term:`ExecuTorch` ``Program``
class and support utilities for printing values.  These are defined in the
variable ``_corev_runner_srcs``

.. literalinclude:: code-samples/runtime/CMakeLists.txt
   :language: CMake
   :lines: 52-57

There are full native libraries to provide I/O for the :term:`PAL`. However
these require operating system features such as threads and mutexes, which are
not available on a bare metal platform such as ours.  Instead we use a
portable library, to use just basic :term:`PAL` features.

.. literalinclude:: code-samples/runtime/CMakeLists.txt
   :language: CMake
   :lines: 59-63

At this stage we have no :term:`quantization` in our example, but if were to
enable :term:`quantization`, we would need to add in the standard libraries
for handling :term:`quantization`.  This is controlled from the command line
by the :term:`CMake` parameter ``EXECUTORCH_BUILD_KERNELS_QUANTIZED``.

.. literalinclude:: code-samples/runtime/CMakeLists.txt
   :language: CMake
   :lines: 65-68

At this point we can define our executable, which we name ``corev_runner``.
We add a definition to enable or disable logging as appropriate, and finally
add the include directories for the executable.

.. literalinclude:: code-samples/runtime/CMakeLists.txt
   :language: CMake
   :lines: 70-79

The final step is to decide on the general compile and link options we will
use.  To keep the binary small, we use the ``--gc-sections`` link option to
remove used sections of code, unless we are debugging.  We then specify the
libraries we are linking (see :ref:`libraries definition
<libraries_definition>`) and specify the common compile options.

.. literalinclude:: code-samples/runtime/CMakeLists.txt
   :language: CMake
   :lines: 81-86

The CORE-V target specific configuration
----------------------------------------

.. index:: ExecuTorch runtime; CMake configuration

This file, ``riscv32-corev-elf.cmake` is found in the ``tooling`` repository.
It is used to give specifics of the tool chain.  It is specified in the
``CMAKE_TOOLCHAIN_FILE`` environment variable, and automatically picked up by
:term:`CMake`.  We'll see later in the script we use to build the
:term:`CORE-V` backend (see :ref:`runtime_build_script`), the following
command.

.. code-block:: bash

   build_target=riscv32-corev-elf
   export CMAKE_TOOLCHAIN_FILE="${tooldir}/${build_target}.cmake"

.. index:: MODEL_PATH (environment variable)
.. index:: CRT0_PATH (environment variable)

One of the major features of this configuration is pulling values from
environment variables which are set in the build script, so they can be passed
into the C/C++ programs as defined constants.  We start with two of these, the
location of the ``.pte`` file generated by the :term:`ExecuTorch` :term:`AoT`
and the location of the :term:`crt0.o` for the target platform.  These are
passed respectively in the environment variables ``MODEL_PATH`` and
``CRT0_PATH``.  Of these ``MODEL_PATH`` is mandatory.

.. literalinclude:: code-samples/runtime/riscv32-corev-elf.cmake
   :language: CMake
   :lines: 13-18

We start by defining where the model we created during the :term:`AoT`.  We
define this as a generic system (since we have no operating system) and mark
this as cross compiling.

.. literalinclude:: code-samples/runtime/riscv32-corev-elf.cmake
   :language: CMake
   :lines: 20-22

We then have a placeholder, were we can add specific assembly and compilation
flags for the three possible systems we can build (``Release``,
``ReleaseWithDebugInfo`` and ``Debug``).

.. literalinclude:: code-samples/runtime/riscv32-corev-elf.cmake
   :language: CMake
   :lines: 24-32

We then have a pair of conditional declarations to enable Position Independent
Code (:term:`PIC`) and to enable Link Time Optimization (:term:`LTO`.  This
latter is now more correctly known as Interprocedural Optimization
(:term:`IPO`).

.. literalinclude:: code-samples/runtime/riscv32-corev-elf.cmake
   :language: CMake
   :lines: 34-42

Next come the definitions of the various tools being used for compilation and
the C and C++ standards which must be supported.

.. literalinclude:: code-samples/runtime/riscv32-corev-elf.cmake
   :language: CMake
   :lines: 44-55

Next come modifiers to the compile command lines, driven by environment
variables.  These include the RISC-V :term:`ISA` and :term:`ABI`, the location
of the linker script, the board support library and the location of the
compiled constructor (``crtbegin.o``) and destructor (``crtend.o``) code.

.. literalinclude:: code-samples/runtime/riscv32-corev-elf.cmake
   :language: CMake
   :lines: 57-62

These are then used to extend the C++, C and assembly command flags
generically, and the conditionally depending on the type of release.

.. literalinclude:: code-samples/runtime/riscv32-corev-elf.cmake
   :language: CMake
   :lines: 63-76

Next, we optionally enable logging messages.

.. literalinclude:: code-samples/runtime/riscv32-corev-elf.cmake
   :language: CMake
   :lines: 78-83

Finally we add some defines to the compile line (``VERILATOR``,
``RUNNER_MODEL_PATH``) add the options needed for effective garbage collection
of sections (``--ffunction-sections``, ``--fdata-sections``).  We also specify
(```-nostartfiles``), since we are going to define our own :term:`crt0.o`,
``crtbegin.o`` and ``crtend.o``.  We then build up the final set of link
options based on all the variables defined earlier in the :term:`CMake` file.

.. literalinclude:: code-samples/runtime/riscv32-corev-elf.cmake
   :language: CMake
   :lines: 85-88

With the output of a final status message, this completes the :term:`CORE-V`
target specific build system configuration.

The CORE-V runtime backend implementation
=========================================

.. index:: ExecuTorch runtime; CORE-V backend

The CORE-V runtime backend declaration
--------------------------------------

.. index:: ExecuTorch runtime; CORE-V backend declaration
.. index:: ! CoreVBackend.h (file)
.. index:: ! CoreVBackend (class)
.. index:: BackendInterface (class)
.. index:: error.h (file)
.. index:: evalue.h (file)
.. index:: executorch::runtime (namespace)
.. index:: executorch::backends::corev (namespace)
.. index:: EValue


This is found in ``backends/corev/runtime/CoreVBackend.h``.  It is a derived
class of ``BackendInterface`` (defined in ``runtime/backend/interface.h``).
The definition makes use of the error return values (defined in
``runtime/core/error.h``) and :term:`EValue`\ s (defined in
``runtime/core/evalue.h``).  Having included these headers, we are generally
doing to be using functions from the ``executorch::runtime`` namespace, for
which we need ``using`` declaration. We then define our class in the
``corev`` namespace within the ``executorch::backends`` namespace.

.. literalinclude:: code-samples/runtime/CoreVBackend.h
   :language: C++
   :lines: 15-21

.. note::

   Good programming practice deprecates the use of ``using`` declarations in
   headers.  However it seems to be common in :term:`ExecuTorch` coding.

The class is derived from ``BackendInterface``

.. literalinclude:: code-samples/runtime/CoreVBackend.h
   :language: C++
   :lines: 23

.. index:: CoreVBackend (class); init (method)

It then defines four public interface functions.  An initialization function
to allocate any resources needed and perform any other initialization. This
function will return a handle to the resources as an opaque pointer, which can
be passed to the execution function (see the diagram at the start of this
chapter).

.. literalinclude:: code-samples/runtime/CoreVBackend.h
   :language: C++
   :lines: 25-29

.. note::

   As much of the initialization work as possible should be moved in the
   :term:`AoT`.

.. index:: CoreVBackend (class); execute (method)

Then the function to actually execute the behavior of the function.  It
returns an error code indicating success, or the type of failure.

.. literalinclude:: code-samples/runtime/CoreVBackend.h
   :language: C++
   :lines: 31-34

.. index:: CoreVBackend (class); destroy (method)

A function to release resources at the end of execution. This is a ``void``
function.  Any failures are ignored.

.. literalinclude:: code-samples/runtime/CoreVBackend.h
   :language: C++
   :lines: 36

.. index:: CoreVBackend (class); is_available (method)

And finally a predicate to indicate if the backend is available to process
:term:`delegation` calls.

.. literalinclude:: code-samples/runtime/CoreVBackend.h
   :language: C++
   :lines: 38


The CORE-V runtime backend definition
-------------------------------------

.. index:: ExecuTorch runtime; CORE-V backend definition
.. index:: ! CoreVBackend.cpp (file)
.. index:: CoreVBackend (class)

This is found in ``backends/corev/runtime/CoreVBackend.cpp``.  This will be a
minimal implementation at this stage, since we are providing no
:term:`delegation` support at this stage.

We start by including the declaration header.  We again ``using`` interface
for the ``executorch::runtime`` namespace.  Then our implementations for the
``CoreVBackend`` class are within the ``executorch::backends::corev``
namespace.

.. literalinclude:: code-samples/runtime/CoreVBackend.cpp
   :language: C++
   :lines: 13-17

.. index:: CoreVBackend (class); init (method)

The initialization function need do nothing, and since we will be doing no
execution can return a ``nullptr`` as the handle to pass on.

.. literalinclude:: code-samples/runtime/CoreVBackend.cpp
   :language: C++
   :lines: 19-24

.. index:: CoreVBackend (class); execute (method)

The execution function, similarly need do nothing.  We choose to return a
success error code, although it should never actually be called.

.. literalinclude:: code-samples/runtime/CoreVBackend.cpp
   :language: C++
   :lines: 26-31

.. index:: CoreVBackend (class); destroy (method)

Since we have no resources to release, the destruction function need do
nothing.

.. literalinclude:: code-samples/runtime/CoreVBackend.cpp
   :language: C++
   :lines: 33-35

.. index:: CoreVBackend (class); is_available (method)

Finally the availability predict can return ``true``.  Again it doesn't really
matter what it returns, since it will never be called.

.. index:: CoreVBackend (class); registration with ExecuTorch runtime

There is one final part of this source file, which is the code to register the
backend with the wider :term:`ExecuTorch` system.  This is done in an unnamed
namespace to avoid polluting other namespaces, since nothing here is needed
elsewhere.

.. index:: Backend (class)

We instantiate an instance of the ``CoreVBackend`` class we have just defined.
We then instantiate an instance of the :term:`ExecuTorch` ``Backend`` class,
passing in the name of our class (``"CoreVBackend"``) and a reference to the
instance of the ``CoreVBackend`` class we just created.

.. index:: register_backend (function)

Finally we pass the new :term:`ExecuTorch` ``Backend`` instance to the
:term:`ExecuTorch` ``register_backend`` function.  We capture the result of
that call, but do nothing with it.

.. literalinclude:: code-samples/runtime/CoreVBackend.cpp
   :language: C++
   :lines: 43-48

.. index:: ExecuTorch runtime; linking without side-effects
.. index:: ExecuTorch runtime; --whole-archive

Because the code is purely for side-effect, and the values are in an unnamed
space, there is a risk the code may not be linked.  This is why we wrapped the
linking of the :term:`CORE-V` backend library with ``-Wl,--whole-archive`` and
``-Wl,--no-whole-archive`` earlier (see :ref:`why we use --whole-archive
<runtime_use_of_whole_archive>`.  There is a risk that even this may not be
enough if :term:`LTO` is enabled. In this case the result variable,
``success_with_compiler`` should be declared ``volatile``.

The Platform Abstraction Layer
------------------------------

.. index:: ! ExecuTorch runtime; physical abstraction layer
.. index:: corev_pal.cpp (file)
.. index:: platform.h (file)

The :term:`CORE-V` :term:`PAL` implementation can be found in
``backends/corev/runtime/corev_pal.cpp``.  This provides implementation of the
functions in the :term:`PAL` :term:`API`.  This is declared (and well
documented) in the ``runtime/platform/platform.h`` header.  The implementation
for a bare metal platform can be very simple.

.. index:: platform.h (file)
.. index:: et_pal_init (function)

We include the ``platform.h`` header and then define the initialization
function for the :term:`PAL`.  This need do nothing.

.. literalinclude:: code-samples/runtime/corev_pal.cpp
   :language: C++
   :lines: 13-17

.. index:: et_pal_abort (function)

On an embedded system, aborting has limited meaning.  For now we trigger a
trap, and the code will just loop permanently in the trap handler code.

.. literalinclude:: code-samples/runtime/corev_pal.cpp
   :language: C++
   :lines: 19-21

.. index:: et_pal_current_ticks (function)

We can provide a meaningful implementation of the timing function
``et_pal_current_ticks``.  We have access to the :term:`RISC-V` cycle count
:term:`CSR`\ s, ``mcycle`` and ``mcycleh``, which gives us a 64-bit cycle
count we can use as the result.

.. literalinclude:: code-samples/runtime/corev_pal.cpp
   :language: C++
   :lines: 23-37

.. index:: et_pal_ticks_to_ns_multiplier (function)

The :term:`PAL` converts ticks to nanoseconds using a ratio, thus avoiding use
of floating point arithmetic.  Since we are modeling a 400MHz system, 5 ticks
is 2 cycles, and we can use the ratio of 5/2 to convert.

.. literalinclude:: code-samples/runtime/corev_pal.cpp
   :language: C++
   :lines: 39-41

.. index:: et_pal_emit_log_message (function)

The :term:`PAL` output function is relatively complex, since it is intended
for categorized log messages. Even though our target platform is embedded it
has support for writing to standard output, which will appear on the console
when run on a :term:`Verilator` model of the target.  We can take a simple
standard implementation based on ``printf`` to produce the message.

.. literalinclude:: code-samples/runtime/corev_pal.cpp
   :language: C++
   :lines: 43-62

.. index:: et_pal_allocate (function)
.. index:: et_pal_free (function)

Lastly we have functions to allocate and free memory.  Hopefully not often
used, since this is not a good practice in resource constrained systems.  We
just wrap C ``malloc`` and ``free`` functions for this.

.. literalinclude:: code-samples/runtime/corev_pal.cpp
   :language: C++
   :lines: 64-70

With this we now have everything to create a library of backend runtime
functionality for :term:`CORE-V`

The CORE-V runtime backend driver
=================================

.. index:: ! ExecuTorch runtime; backend driver

So far we have built a minimal :term:`ExecuTorch` runtime library for
:term:`CORE-V`.  We now need to create a driver using that library, so we have
a complete application.

.. index:: corev_executor_runner.cpp (file)

As with the :term:`AoT` driver, this is placed in the ``examples/corev``
directory, this time in an ``executor_runner`` subdirectory.

* ``examples/corev/executor_runner/corev_executor_runner.cpp``

We can use ``examples/portable/executor_runner/executor_runner.cpp`` as a
starting point for our code.

.. _including_the_aot_generated_model_in_the_runtime:

Including the AoT generated model in the runtime
------------------------------------------------

.. index:: ! corev_executor_runner.cpp (file)

If we have a file system, we would just read the model in from its ``.pte``
file.  However in a bare metal system this is not possible.  Instead we want
to include the file image as a byte array.  In the C23 standard, there is now
a directive, ``#embed``, to just include a binary file as a byte array.

.. index:: ExecuTorch runtime; model conversion

However most :term:`Linux` distributions do not have a compiler which supports
``#embed``, so we use an alternative approach.  We can use the :term:`Linux`
*xxd* hex dump utility to convert the ``.pte`` file.  The ``-i`` flag to this
command generates the output as a series of byte values in a format suitable
for inclusion in a C/C++ byte array.

.. code-block:: bash

   xxd -i < model.pte > model.pte.inc

We can inspect ``model.pte.inc`` and see it is in the appropriate format.

.. code-block:: bash

   more model.pte.inc
   0x24, 0x00, 0x00, 0x00, 0x45, 0x54, 0x31, 0x32, 0x00, 0x00, 0x00, 0x00,
   0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x18, 0x00,
   ...
   0x66, 0x6f, 0x72, 0x77, 0x61, 0x72, 0x64, 0x00

.. index:: ExecuTorch runtime; model inclusion

This will then be used to embed data in the source of the runtime driver (see
the next section).

.. code-block:: C++

   alignas(alignof(std::max_align_t)) const uint8_t model_pte[] = {
     #include "model.pte.inc"
   };

With a modern compiler, we could omit the conversion using *xxd* and just use
instead.

.. code-block:: C++
   :emphasize-lines: 2

   alignas(alignof(std::max_align_t)) const uint8_t model_pte[] = {
     #embed "model.pte"
   };

The main driver program
-----------------------

The main driver program, ``corev_executor_runner.cpp`` creates a ``main``
function which invokes the functionality of the runtime library we have
created.  The program is placed in the ``examples/corev/executor_runner``.  It
is heavily based on the generic runtime driver in
``examples/portable/executor_runner/executor_runner.cpp``.

We start by bringing in some standard C++ headers to give us the fixed size
integer declarations (``cstdint``), Standard Template Library (:term:`STL`)
vectors (``vector``) and :term:`STL` smart pointers (``memory``).

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 25-27

We then bring in the headers for the functionality we need from the
:term:`ExecuTorch` runtime.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 29-37

.. index:: ExecuTorch runtime; driver printf support

There is one more header, and we include it separately.  This is to allow us
direct use of ``printf``, which will pass output to the console when we run
our target as a :term:`Verilator` model.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 39-40

.. index:: RUNNER_MODEL_PATH (environment variable)

At this point we create an array by including the model generated by the
:term:`ExecuTorch` :term:`AoT` (see :ref:`aot_running_aot`).  The actual path to
the model should have been defined in preprocessor variable
``RUNNER_MODEL_PATH``.  We check first that this actually exists, and if not
throw an error.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 42-48

.. _runtime_buffer_alignment_note:

.. note::

   We align this to the maximum alignment of the architecture.  This is not
   actually needed at this stage, but it will be useful later in this
   application note (see :ref:`why align the memory buffers
   <delegation_why_align_memory>`).

.. index:: ExecuTorch runtime; driver memory buffers
.. _allocator_pool_declaration:

Dynamically allocating memory in embedded systems is not a good idea in
general.  You really want to know that your program is going to fit.  To
address this, :term:`ExecuTorch` allocates two fixed buffers for use by models
at the start of execution.  One is for the various data structures declared
for the model (``method_allocator_pool``), the second is for temporary
intermediate data structures needed while evaluating individual operators
(``temp_allocator_pool``).  We are running on a 4MiB platform, so we keep
these small.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 50-52

.. index:: ! FLAGS_num_executions (variable)

.. _runtime_definition_FLAGS_num_executions:

The standard code we are basing this on allows a model to be run multiple
times, perhaps with different data.  The main loop is controlled by a count
``FLAGS_num_executions``, which in our implementation we just define as
constant 1, so we only ever execute a model once.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 54

For ease of reading there are then ``using`` declarations for common classes
from the :term:`ExecuTorch` runtime.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 57-66

.. index:: ExecuTorch runtime; driver main function

We now come to the main program.  It starts by calling the standard
:term:`ExecuTorch` runtime initialization.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 68-69

.. index:: ExecuTorch runtime; driver loader creation

We need to load the model created by the :term:`AoT`.  This is a two stage
process.  First we need to create a loader suitable for the representation of
the model.  In our case a byte vector in memory.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 71-76

.. index:: ET_LOG (macro)

.. note::

   The ``ET_LOG`` macro is defined in ``runtime/platform/log.h`` and provides
   the main interface to the logging infrastructure (documented in that file).
   For us one useful feature is that if logging is not enabled at
   configuration time, to code is generated at all.

.. index:: ExecuTorch runtime; driver model loading

Then we need to load the program, parsing it in the process.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 78-85

.. index:: Result (class)

.. important::

   We see here a standard programming paradigm in :term:`ExecuTorch`, in which
   we wrap results with an error code, using the ``Result`` class (defined in
   ``executorch/runtime/core/result.h``).  The class holds a union of the
   value (of the template type) and a result (of class ``Error``, defined in
   ``executorch/runtime/core/error.h``).  The various operators are
   overloaded, so that indirection on an instance will yield the underlying
   value.

   Thus a result of this single type can hold either a value or an error code.
   This makes for more compact coding when testing for success or failure of a
   method which returns a result.  In this case we expect ``Program::load`` to
   return the loaded program if it parsed successfully, or else an error code.

.. index:: ExecuTorch runtime; driver initial method identification

Having loaded the program, we can get its first method.  We still use a
``Result`` wrapper, but for convenience we also keep a copy of its contents,
since it is used several times later in the ``main`` function.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 87-94

.. index:: ET_CHECK_MSG (macro)

If the test fails, the supplied message is reported through the logging system
and then the :term:`PAL` abort method is called.

.. note::

   The ``ET_CHECK_MSG`` macro is one of several convenience macros in
   ``runtime/platform/assert.h`` for handling values wrapped in ``Result``.

.. index:: ExecuTorch runtime; driver method metadata

Having obtained the method, we need to look at its metadata to see the
resources it requires.  The metadata is kept in its own structure, so that it
can be inspected without the initialization cost of initializing the full
method.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 96-102

.. index:: ExecuTorch runtime; driver memory manager
.. index:: ExecuTorch runtime; driver memory allocator
.. index:: MemoryManager (class)
.. index:: method_allocator_pool (variable)
.. index:: temp_allocator_pool (variable)

We now need to allocate the memory for the method.  This will be used for all
the dynamic C++ metadata and objects used to represent the loaded method.  If
there is not enough memory, we will fail with an error at this point.

The :term:`ExecuTorch` runtime does not use ``malloc`` or ``new``.  It instead
uses a memory manager (class ``MemoryManager``) provided for the particular
target.  In our case, we use a memory allocator based on the statically
allocated pool, ``method_allocator_pool``, declared earlier (see
:ref:`declaration of memory pools <allocator_pool_declaration>`).

In addition we may need to allocate temporary memory, for use by the kernels
which implement the operators of the method.  In our case, we use a memory
allocator based on the statically allocated pool, ``temp_allocator_pool``,
declared earlier (see :ref:`declaration of memory pools
<allocator_pool_declaration>`).

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 120-126

.. note::

   At this point, we have not allocated any memory, just created the memory
   allocators we need for such allocation.

The actual memory allocation is more complex, because we can associate
arbitrarily complex memory hierarchies to the target.  These are captured in
the meta data of the model as a series of memory blocks of different sizes.
It is up to the memory manager for the target to determine how to do the
allocation.

.. index:: HierarchicalAllocator (class)

We will represent the sequence of memory areas for the method using the
:term:`ExecuTorch` ``HierarchicalAllocator`` class.  We extract the sequence
of buffer sizes from the method metadata, and for each buffer create a unique
pointer to an array of bytes of that size.  We can then pass the vector of
pointers and vector of corresponding sizes to the constructor for an instance
of ``HierarchicalAllocator``.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 132-148

.. note::

   At this point, we have still not allocated any memory, but we have a
   detailed description of the memory areas that are available to the runtime.

.. index:: ExecuTorch runtime; driver memory allocator

We can then create a memory manager for the method.  It needs the memory
allocator for the method memory, and its associated description (as a
``HierarchicalAllocator``), along with the allocator for temporary memory
(which has no detailed structure).

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 150-153

.. note::

   The memory manager has still not allocated any memory, but it is passed to
   the method when it is loaded, so that it can allocate memory as needed.

.. index:: ExecuTorch runtime; driver method load

A this point, we can load the actual method, passing it the memory manager to
allocate memory as needed.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 159-166

With the model loaded, we can now run it.  As noted earlier (see
:ref:`defintion of FLAGS_num_executions
<runtime_definition_FLAGS_num_executions>`, we could have multiple
invocations.  In our program we have just a ``for`` loop, controlled by a
constant value, ``FLAGS_num_executions``, whose value is 1.  So we will only
execute our model once.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 168-170

.. index:: ExecuTorch runtime; driver input data setup

At this point we need to set up the input data for this model run.  Our model
adds two 5-element vectors of type ``float32``.  We are not
concerned with specific data at this stage, so for now, we can just set all
the inputs to 1.

.. index:: prepare_input_tensors (method)

For this we can use the ``prepare_input_tensors`` method.  We pass in the
method, and from its metadata, it can see the number and :term:`shape` of the
inputs.  With default values of its remaining parameters, it will just set
each value to 1.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 171-184

.. hint::

   We will see how to set specific input values later in this application note
   (see :ref:`setting explicit input values
   <benchmarking_setting_input_data>`).

.. index:: ExecuTorch runtime; driver model execution

So now there is only one thing left to do, which is to execute the model and
check it completed successfully.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 186-191

Having completed our execution, we can put out a message to this effect.  The
correct way to do this is to use the logging infrastructure, but as we shall
see shortly (see :ref:`runtime_running_executorch_on_verilator`), this is
something we do not want to enable for a very small microcontroller.  Instead,
we cheat and use ``printf`` to print the output message.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 198-204

Having completed everything, we can return successfully.

.. literalinclude:: code-samples/runtime/corev_executor_runner.cpp
   :language: C++
   :lines: 206-207

In a real embedded system, returning from ``main`` is not useful.  However
with our :term:`Verilator` model this is detected and causes the model to
terminate, reporting execution statistics.

Building and running ExecuTorch runtime for CORE-V
==================================================

.. index:: ExecuTorch runtime; building

As we saw in :ref:`modifying_runtime_build_system`, there are a great many
parameters configuring how to build the :term:`ExecuTorch` runtime.  We use a
script for convenience.

.. _runtime_build_script:

The ExecuTorch runtime build script
-----------------------------------

.. index:: ExecuTorch runtime; build script

The script is ``build-executorch-runtime.sh`` in the main ``tooling`` repository.

We start with a helper shell function used to report errors or if the
``--help`` option is specified.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 20-31

To avoid embarrassment, we provide a simple check we are running in a Python
virtual environment.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 33-43

We then identify the top level directory---the one in the directory about the
one containing this script.  From this we can derive the other directories of
interest.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 45-51

Our target is specified using a standard :term:`GNU` triplet.  We put this in
a local variable.  It is the only target supported in this project, but in
principle we could set this using a command line option to generalize the
script for multiple architectures.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 53-54

We then set defaults for the various parameters that can be set from the
command line.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 56-62

We can now process the command line, to override any of the parameter
defaults. We'll process any unrecognized parameters with an error message.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 64-101

As described in :ref:`modifying_runtime_build_system`, we have a target
specific :term:`CMake` configuration file, whose name is derived from the
target triplet.  We check this is found.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 103-109

A number of the parameters configuring :term:`CMake` are set by environment
variables. We assign these and export them.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 111-125

We create the build and install directories if they do not exist.  If a clean
build has been requested, the contents of these directories is deleted.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 127-141

We create a unique log file for each run, timestamped when the run occurred.
This is placed in the ``log`` directory, which is created if necessary.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 143-146

Having done all the setup, we start the build process, with a time stamp to
the log file and switching to the build directory.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 148-149

The first step is to check we have a model file.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 151-157

We then convert it to a C array of bytes using `xxd` as described in
:ref:`including_the_aot_generated_model_in_the_runtime`.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 159-177

We can now configure the :term:`ExecuTorch` runtime ready to be built.  There
are a number of :term:`CMake` parameters on the command line which we have not
described, but which are defined. Most will be self-evident, but they are all
described within the :term:`ExecuTorch` online documentation. Some are
controlled with the parameters set within the script.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 179-203

Having configured the runtime, we can then use standard :term:`CMake` ``build``
and ``install`` commands to build the runtime and install it.  However at this
stage we install in the *build* directory, not the *install* directory.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 205-209

We then move the installed executable from the build directory to the install
directory.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 211-213

The reason for this, is that the install process puts the executable down in
the file hierarchy, and this way we can put it at the top of the install
directory.

.. _runtime_creating_hexdump_model:

The final step is to take the executable and put it into :term:`Verilog`
:term:`hexdump` format, suitable for use with the :term:`Verilator` model.  We
conclude with a timestamp in the log.

.. literalinclude:: code-samples/runtime/build-executorch-runtime.sh
   :language: bash
   :lines: 215-219

.. _runtime_running_executorch_on_verilator:

Running the ExecuTorch runtime on the Verilator model
-----------------------------------------------------

.. index:: ExecuTorch runtime; logging

We can use our script from the tooling repository to build the
:term:`ExecuTorch` runtime.  It is tempting to turn on logging, so we can see
detail as everything runs.

.. code-block:: bash

   ./build-executorch-runtime.sh --executorch-logging Info

The problem is that the :term:`ExecuTorch` runtime has a *lot* of logging, and
this adds a few MiB to the binary size.  Given we only have a 4MiB
micro-controller we end up with a program that will not fit.  So we build
without logging (the default), and use explicitly ``printf`` wherever we want
to produce some output.

.. code-block:: bash

   ./build-executorch-runtime.sh

.. index:: ExecuTorch runtime; running

Assuming this all builds correctly, we can now just pass the binary in hexdump
format to the :term:`Verilator` model.

.. code-block:: bash

   ../core-v-verif/cv32e40p/sim/core/cobj_dir/Vtb_top_verilator \
       +firmware=../install/executor_runner.hex

The output will be as follows:

.. literalinclude:: code-samples/runtime/baseline-run.txt
   :language: text
   :emphasize-lines: 1-2

.. index:: ExecuTorch runtime; performance

The initial output is some logging from the :term:`Verilator` model.  It is
the highlighted lines that matter.  The model executed successfully, taking
113,333,120 ps of simulated time and existed the :term:`Verilator` model.

The :term:`Verilog` of this core is defined with a nominal clock period of
10ps (i.e. 100GHz).  However this is just a simulation, and we can divide the
time by 10 to get the number of cycles.  A typical low power silicon
implementation of this :term:`MCU` would run at 400MHz, and from the cycle
count we can work out the real world execution time.

+----------------+----------------+
| Verilator time | 113,333,120 ps |
+----------------+----------------+
| Cycle count    |     11,333,312 |
+----------------+----------------+
| Time @ 400 MHz |       28.33 ms |
+----------------+----------------+

Now 28.33 ms does not seem impressive to add 5 single precision floating point
values, even without a floating point unit.  There is a lot of overhead in
:term:`ExecuTorch`.  We'll look at this in much more detail in
:ref:`executorch_benchmarking`.