edge-nix: Packaging ONNX Runtime for edge devices using Nix (Arduino UNO Q)

[Ali Sao (Sao-Ali)]

I've been working on edge-nix, a small project that uses Nix to build and package ONNX models with their inference runtime for edge devices.

The goal is to make it easier to move an ONNX model from a development machine to an edge device without having to manually set up the runtime and dependencies on the target.

Right now, the project supports two types of artifacts:

• Model-only archive: Packages an ONNX model and a manifest for devices that already have a compatible runtime.

• Arduino UNO Q bundle: Packages the model, an ARM64 CPU runner, and ONNX Runtime libraries into a single transferable archive.

The build process also supports optional dynamic INT8 quantization and validation against sample inputs.

I've tested the MNIST example in a clean Debian container and on a physical Arduino UNO Q. The bundle runs without requiring Nix or a separate ONNX Runtime installation on the board.

It's still an early project. I've only tested one model on the physical board, and inference currently runs on the CPU rather than the NPU.

I'd be interested in hearing from others using ONNX Runtime on embedded Linux or ARM64 devices:

• How are you currently packaging ONNX Runtime and model dependencies for deployment?

• Are there better approaches to building smaller or more portable runtime bundles?

• Are there specific models, execution providers, or devices worth targeting next?

I'm also curious whether something like this would be useful outside my own use case.

Repo: https://github.com/Sao-Ali/edge-nix