.. Embecosm Application Note "Howto: Bringing up ExecuTorch on a Bare Metal Microcontroller". ExecuTorch overview Copyright (C) 2026 Embecosm Limited Contributor: Jeremy Bennett Contributor: Pietra Ferreira Contributor: William Jones Contributor: Shane Slattery SPDX-License-Identifier: CC-BY-SA-4.0 .. _executorch_overview: ******************* ExecuTorch Overview ******************* About PyTorch and ExecuTorch ============================ .. index:: ! PyTorch .. index:: ! ExecuTorch :term:`PyTorch` is an :term:`AI` Framework with a Python Interface, originally written by Meta Inc. It provides comprehensive training and inference of :term:`AI` models of any size. :term:`ExecuTorch` is an extension to :term:`PyTorch` for use with resource constrained devices, and provides only support for inference. Their relationship is summarized in the following diagram. .. image:: images/pytorch-executorch-overview.png :align: center .. index:: single: neural network All :term:`neural network`\ s have the same basic structure. The :term:`neural network` is represented as a graph, where the operations are nodes and connect to elements representing the data as :term:`tensor`\ s. A preprocessor may carry out some transformation of the graph to make it more efficient to evaluate. The :term:`neural network` is evaluated (i.e. executed) by walking over the graph, and for each node marshaling its :term:`tensor` arguments and then deciding where the operation should be executed. In almost all architectures, the choice is whether to execute the operation on a host processor (a general purpose :term:`CPU`) or on some form of accelerator (for example a GPU). This is summarized in the following diagram. .. index:: single: neural network; evaluation single: neural network; inference .. image:: images/ai-generic-high-level-lazy.png :align: center :term:`PyTorch` supports several thousand different operations as standard, and is easily extended with custom operations. The host could be a large multi processor server, and the accelerator a bank of high-end GPUs. Such systems are used to train and then infer from state-of-the-art Large Language Models (:term:`LLM`\ s). By contrast, :term:`ExecuTorch` supports around 150 standard operators. The host might be a smartphone, with a multi-core application processor and vector accelerator (for example Arm SVE2). But it can be something much smaller, such as a multi-core microcontroller with custom :term:`AI` accelerators for operations such as matrix multiplication or convolution. A smart phone can use modern :term:`AI` tools to run complex :term:`AI` analyses for applications, while a microcontroller might be carrying out low-resolution video analysis, or :term:`AI` enabled audio stream analysis. The ExecuTorch architecture =========================== .. index:: single: ExecuTorch; architecture single: ExecuTorch; ATen operators single: ExecuTorch; Edge operators :term:`ExecuTorch` will work out of the box with minimal work. It preprocesses the :term:`AI` model to simplify from the several thousand PyTorch operators (the :term:`ATen` operators) to the 150 or so :term:`ExecuTorch` operators (the :term:`Edge` subset of the :term:`ATen` operators). The resulting graph will then be executed, using a generic runtime with a C++ implementation of each operator, which will run on a single :term:`CPU`. This is summarized in the following diagram. .. image:: images/executorch-generic-architecture-no-delegation.png :align: center This will work quite happily, but is never going to be very efficient. It does not take advantage of other cores and accelerators which exist, it does not optimize for the memory hierarchy and it does not take advantage of any special accelerators or features of the target processor. .. index:: pair: ExecuTorch; delegation To solve this, :term:`ExecuTorch` introduces the idea of :term:`delegation`. :term:`ExecuTorch` can be customized for a specific architecture by delegating particular operators to run on multiple cores, to use custom accelerators, or to use algorithms that take advantage of the memory architecture. This is represented in the following diagram. .. image:: images/executorch-generic-architecture.png :align: center The delegate partitioner is central to customizing :term:`ExecuTorch` :term:`AoT`. It examines the supplied :term:`neural network` and identifies subgraphs for custom processing in two ways. .. index:: pair: ExecuTorch; quantization 1. **Quantization.** This involves adjusting the model to work with less precise parameters to reduce the amount of memory required. It is quite normal to train models using 32-bit floating point parameters. However for inference, 8-bit integers may suffice. Indeed in many cases 4 bits per parameter is enough. The quantization of parameters can be carried out ahead of time. .. index:: pair: ExecuTorch; delegation 2. **Delegation.** This is where we provide a customized implementation of an individual operator or a subgraph. This may be more than an individual operator, where we want to fuse operators. For example it is quite common for a convolution operator to be followed by a rectified linear update (:term:`ReLU`), and these two operators can be computed efficiently together. .. index:: single: ExecuTorch; runtime Then in the :term:`ExecuTorch` runtime, we must provide custom code implementing quantized and/or delegated operators. This will be picked up through delegate execution when they are encountered while evaluating the model. Structure of the ExecuTorch repository ====================================== .. index:: single: ExecuTorch; submodules :term:`ExecuTorch` is a large project, and pulls in third party projects as submodules of the repository (23 such sub-modules in :term:`ExecuTorch` 1.3.1). Using :term:`sloccount`, in total it is nearly 6M lines of code, but of this only around 317k lines of code is actually :term:`ExecuTorch` itself. Approximately 91% of this code is C/C++, 6% is Python, 2.5% is shell and other types of script, with tiny amounts of other languages. ExecuTorch directory hierarchy ------------------------------ .. index:: single: ExecuTorch; repository structure The key top level directories are as follows backends The implementation of a particular target backend lives here, both :term:`AoT` and runtime. In this application note, we will be creating a ``corev`` backend here. There is an ``example`` directory providing the basic templates for creating a new backend. Within each target directory there is typically a `runtime` subdirectory, containing the code for the target specific runtime, handling delegated operations. A number of backends include submodules bringing in third party packages. devtools These are tools for the developers of :term:`ExecuTorch` rather than the general user. docs The documentation of :term:`ExecuTorch`, written in :term:`Markdown`. examples Rather confusingly named, this is where targets place drivers for the code in the ``backends`` directory. There is no common way of implementing such code, so there is no ``example`` directory here. exir Python code to manage the :term:`neural network` graphs being processed during :term:`ExecuTorch` :term:`AoT` processing. export Python code to export the result of :term:`AoT` processing in :term:`FlatBuffer` format (typically as a ``.pte`` file). extension Assorted extensions required for various backends and types of neural network model. kernels The default implementation of each operator, written in C++ to run single threaded on the host processor. pip-out A generated staging directory used when building the various :term:`ExecuTorch` Python packages in :term:`wheel` format. profiler The code to support profiling of :term:`ExecuTorch` models. runtime C++ code providing the common core of the :term:`ExecuTorch` runtime. scripts Assorted useful scripts. src This is a symlink to other parts of the directory hierarchy. It exists to get round a limitation of the ``pip install`` command. There should be no need to ever touch this directory test The test infrastructure for :term:`ExecuTorch`. third-party The location of third party packages (as git submodules) used throughout the system. tools Support for various tools used by the :term:`ExecuTorch` build system. Notably the :term:`Buck` and :term:`CMake` build systems and the :term:`wheel` packaging system for Python. util Some useful scripts, largely for profiling. website The source of the :term:`ExecuTorch` website. Key third party packages ------------------------ .. index:: single: ExecuTorch; third party packages The main third-party packages used across all systems are .. index:: single: FlatBuffer FlatBuffer This is an efficient architecture neutral serialization format developed by Google. Alongside the ``flatbuffers`` package is the ``flatcc`` package, which provides an efficient implementation in C for compiling :term:`FlatBuffer` schemas. GoogleTest Google's test infrastructure, on which the :term:`ExecuTorch` tests are based. There are some backend specific packages of note. Vulkan The :term:`Vulkan` backend brings in three packages (``Vulkan-Headers``, ``VulkanMemoryAllocator`` and ``volk``) to allow it to reuse existing :term:`Vulkan` infrastructure. XNNPACK :term:`XNNPACK` is Google's efficient implementation of linear algebra operations for the Arm, x86, WebAssembly, :term:`RISC-V` and Hexagon architectures. It is provided as the ``xnnpack`` :term:`ExecuTorch` backend, as well as the :term:`XNNPACK` package, it also brings in the ``FP16``, ``FXdiv``, ``cpuinfo`` and ``pthreadpool`` packages. The only other third party package of note is the ``tokenizers`` package to provide support for :term:`LLM`\ s, in the ``extension/llm/tokenizers`` directory. The target: CV32E40Pv2 ====================== CV32E40P is a small and efficient, 32-bit, in-order :term:`RISC-V` core with a 4-stage pipeline that implements the RV32IMFC instruction set architecture, and eight :term:`PULP` custom extensions for achieving higher code density, performance, and energy efficiency. It started its life as a fork of the OR10N :term:`CPU` core that is based on the :term:`OpenRISC` :term:`ISA`. Then, under the name of RI5CY, it became a :term:`RISC-V` core (2016). It was maintained by the :term:`PULP` platform team until February 2020, when it was been contributed to the Open Hardware Foundation. .. image:: images/cv32e40p-block-diagram.png :align: center :alt: Block diagram of the CV32E40P processor. (Image courtesy of the Open Hardware Foundation, via the CV32E40P `git repository `_.) Version 2 of this processor is particularly useful for this application note, for the following reasons: * it can be simulated using :term:`Verilator`; * its verification repository provides a simple :term:`Verilator` test bench and a :term:`BSP`; * the :term:`BSP` enables running programs to print messages via ``printf``. * it supports all eight :term:`PULP` :term:`ISA` extensions. .. note:: While version 2 supports a single-precision hardware floating point unit (:term:`FPU`), this is known to have a bug, which can lead to incorrect computation. As a result we disable it for this application note and use software emulation of floating point operations throughout.