info@embecosm.com

Over past months I have posted a great deal about our work bringing up the ExecuTorch AI framework on bare metal microcontrollers. Much of this work has been done in collaboration with our colleagues at Mosaic SoC. This is true Edge AI, where we run real world AI models on hardware that is small enough to be battery powered. The runtime for these AI models can be made very energy efficient. It can make use of specialist accelerators, multi-core processors and hierarchical memory, with small, fast caches.

While ExecuTorch is very popular, it is not the only game in town. For example, LiteRT (formerly TensorFlow Lite) has been around for longer, and is widely used for embedded AI applications. Both ExecuTorch and LiteRT offer an eager evaluation of AI models. The model is represented as a graph of operators working on data in the form of Tensors. The runtime walks over this graph invoking a library of compiled operators to process each node as it is encountered.

The alternative (newer) approach is AI compilation, which offers lazy evaluation of AI models. The model is again represented as a graph, but this whole graph is manipulated by the compiler to simplify and partition the work as appropriate for the target architecture. Almost all such compilers are based on the LLVM project infrastructure, and in particular using the Multi Level Intermediate Representation (MLIR) to represent the model. Much of the work has been carried out in-house by major players in the AI space (AMD, Nvidia, Google). However in the past two years, Google has donated its AI compiler project, IREE, to the Linux Foundation, and it has become a community project.

IREE is still officially an experimental project. But it is maturing to the level where the first stable product offerings using it are appearing. There are challenges with compiled AI. Existing AI framesworks need importers to use the IREE infrastructure and the lazy evaluation semantics differ from traditional eager evaluation. But it offers the potential to yield smaller and faster code.

I’ll be at the RISC-V European Summit from 8-11 June in Bologna if you would you like to know more.

At 5pm on Monday 8 June I am running a workshop on how to bring up ExecuTorch for bare metal RISC-V . I’ll cover the basics of getting this AI Framework running in the absence of an operating system. Then I will look at how to optimize your implementation, using RISC-V extensions, custom AI accelerators, multithreading across cores and caching within a hierarchical memory structure. I have prepared a short video giving an overview of the workshop.

Later in the Summit, on Tuesday 9 June, I will be presenting a poster comparing ExecuTorch and IREE when used with bare metal edge AI platforms. You can find the poster on Poster Island C.

I look forward to seeing you there.