2. ExecuTorch Overview

2.1. About PyTorch and ExecuTorch

PyTorch is an AI Framework with a Python Interface, originally written by Meta Inc. It provides comprehensive training and inference of AI models of any size. ExecuTorch is an extension to PyTorch for use with resource constrained devices, and provides only support for inference. Their relationship is summarized in the following diagram.

All neural networks have the same basic structure. The neural network is represented as a graph, where the operations are nodes and connect to elements representing the data as tensors. A preprocessor may carry out some transformation of the graph to make it more efficient to evaluate.

The neural network is evaluated (i.e. executed) by walking over the graph, and for each node marshaling its tensor arguments and then deciding where the operation should be executed. In almost all architectures, the choice is whether to execute the operation on a host processor (a general purpose CPU) or on some form of accelerator (for example a GPU). This is summarized in the following diagram.

PyTorch supports several thousand different operations as standard, and is easily extended with custom operations. The host could be a large multi processor server, and the accelerator a bank of high-end GPUs. Such systems are used to train and then infer from state-of-the-art Large Language Models (LLMs).

By contrast, ExecuTorch supports around 150 standard operators. The host might be a smartphone, with a multi-core application processor and vector accelerator (for example Arm SVE2). But it can be something much smaller, such as a multi-core microcontroller with custom AI accelerators for operations such as matrix multiplication or convolution. A smart phone can use modern AI tools to run complex AI analyses for applications, while a microcontroller might be carrying out low-resolution video analysis, or AI enabled audio stream analysis.

2.2. The ExecuTorch architecture

ExecuTorch will work out of the box with minimal work. It preprocesses the AI model to simplify from the several thousand PyTorch operators (the ATen operators) to the 150 or so ExecuTorch operators (the Edge subset of the ATen operators). The resulting graph will then be executed, using a generic runtime with a C++ implementation of each operator, which will run on a single CPU. This is summarized in the following diagram.

This will work quite happily, but is never going to be very efficient. It does not take advantage of other cores and accelerators which exist, it does not optimize for the memory hierarchy and it does not take advantage of any special accelerators or features of the target processor.

To solve this, ExecuTorch introduces the idea of delegation. ExecuTorch can be customized for a specific architecture by delegating particular operators to run on multiple cores, to use custom accelerators, or to use algorithms that take advantage of the memory architecture. This is represented in the following diagram.

The delegate partitioner is central to customizing ExecuTorch AoT. It examines the supplied neural network and identifies subgraphs for custom processing in two ways.

1. Quantization. This involves adjusting the model to work with less precise parameters to reduce the amount of memory required. It is quite normal to train models using 32-bit floating point parameters. However for inference, 8-bit integers may suffice. Indeed in many cases 4 bits per parameter is enough. The quantization of parameters can be carried out ahead of time.

2. Delegation. This is where we provide a customized implementation of an individual operator or a subgraph. This may be more than an individual operator, where we want to fuse operators. For example it is quite common for a convolution operator to be followed by a rectified linear update (ReLU), and these two operators can be computed efficiently together.

Then in the ExecuTorch runtime, we must provide custom code implementing quantized and/or delegated operators. This will be picked up through delegate execution when they are encountered while evaluating the model.

2.3. Structure of the ExecuTorch repository

ExecuTorch is a large project, and pulls in third party projects as submodules of the repository (23 such sub-modules in ExecuTorch 1.3.1). Using sloccount, in total it is nearly 6M lines of code, but of this only around 317k lines of code is actually ExecuTorch itself. Approximately 91% of this code is C/C++, 6% is Python, 2.5% is shell and other types of script, with tiny amounts of other languages.

2.3.1. ExecuTorch directory hierarchy

The key top level directories are as follows

backends

The implementation of a particular target backend lives here, both AoT and runtime. In this application note, we will be creating a corev backend here. There is an example directory providing the basic templates for creating a new backend.

Within each target directory there is typically a runtime subdirectory, containing the code for the target specific runtime, handling delegated operations.

A number of backends include submodules bringing in third party packages.

devtools

These are tools for the developers of ExecuTorch rather than the general user.

docs

The documentation of ExecuTorch, written in Markdown.

examples

Rather confusingly named, this is where targets place drivers for the code in the backends directory. There is no common way of implementing such code, so there is no example directory here.

exir

Python code to manage the neural network graphs being processed during ExecuTorch AoT processing.

export

Python code to export the result of AoT processing in FlatBuffer format (typically as a .pte file).

extension

Assorted extensions required for various backends and types of neural network model.

kernels

The default implementation of each operator, written in C++ to run single threaded on the host processor.

pip-out

A generated staging directory used when building the various ExecuTorch Python packages in wheel format.

profiler

The code to support profiling of ExecuTorch models.

runtime

C++ code providing the common core of the ExecuTorch runtime.

scripts

Assorted useful scripts.

src

This is a symlink to other parts of the directory hierarchy. It exists to get round a limitation of the pip install command. There should be no need to ever touch this directory

test

The test infrastructure for ExecuTorch.

third-party

The location of third party packages (as git submodules) used throughout the system.

tools

Support for various tools used by the ExecuTorch build system. Notably the Buck and CMake build systems and the wheel packaging system for Python.

util

Some useful scripts, largely for profiling.

website

The source of the ExecuTorch website.

2.3.2. Key third party packages

The main third-party packages used across all systems are

FlatBuffer

This is an efficient architecture neutral serialization format developed by Google. Alongside the flatbuffers package is the flatcc package, which provides an efficient implementation in C for compiling FlatBuffer schemas.

GoogleTest

Google’s test infrastructure, on which the ExecuTorch tests are based.

There are some backend specific packages of note.

Vulkan

The Vulkan backend brings in three packages (Vulkan-Headers, VulkanMemoryAllocator and volk) to allow it to reuse existing Vulkan infrastructure.

XNNPACK

XNNPACK is Google’s efficient implementation of linear algebra operations for the Arm, x86, WebAssembly, RISC-V and Hexagon architectures. It is provided as the xnnpack ExecuTorch backend, as well as the XNNPACK package, it also brings in the FP16, FXdiv, cpuinfo and pthreadpool packages.

The only other third party package of note is the tokenizers package to provide support for LLMs, in the extension/llm/tokenizers directory.

2.4. The target: CV32E40Pv2

CV32E40P is a small and efficient, 32-bit, in-order RISC-V core with a 4-stage pipeline that implements the RV32IMFC instruction set architecture, and eight PULP custom extensions for achieving higher code density, performance, and energy efficiency.

It started its life as a fork of the OR10N CPU core that is based on the OpenRISC ISA. Then, under the name of RI5CY, it became a RISC-V core (2016). It was maintained by the PULP platform team until February 2020, when it was been contributed to the Open Hardware Foundation.

(Image courtesy of the Open Hardware Foundation, via the CV32E40P git repository.)

Version 2 of this processor is particularly useful for this application note, for the following reasons:

• it can be simulated using Verilator;

• its verification repository provides a simple Verilator test bench and a BSP;

• the BSP enables running programs to print messages via printf.

• it supports all eight PULP ISA extensions.

Note

While version 2 supports a single-precision hardware floating point unit (FPU), this is known to have a bug, which can lead to incorrect computation. As a result we disable it for this application note and use software emulation of floating point operations throughout.