Profiling and Benchmarking

This document covers Kandelo's performance measurement tools: a runtime syscall profiler for detailed per-syscall analysis, and a benchmark suite for repeatable cross-host comparisons.

Syscall Profiler

The host runtime includes a built-in syscall profiler that measures every syscall handled by the kernel worker. It is zero-cost when disabled — no data structures are allocated and no timing code runs.

Enabling

Set the WASM_POSIX_PROFILE environment variable before starting the kernel:

scripts/dev-shell.sh env WASM_POSIX_PROFILE=1 \
  npx tsx examples/run-example.ts hello

Collecting Results

The profiler accumulates data in memory. Call dumpProfile() on the kernel worker instance to print results to stderr:

import { KernelWorker } from "kandelo/host";

const kernel = new KernelWorker(/* ... */);
// ... run workload ...
kernel.dumpProfile();

A common pattern is to dump on SIGINT so you can interrupt a long-running program:

process.on("SIGINT", () => {
  kernel.dumpProfile();
  process.exit();
});

Output Format

=== Syscall Profile ===
Syscall       Count     Time(ms)    Avg(ms)    Retries
----------------------------------------------------
4                50        12.34      0.247          0
3                30         8.21      0.274          5
63               20         3.45      0.173          2
----------------------------------------------------
TOTAL           100        23.00      0.230          7
Pending pipe readers: 0, writers: 0
=== End Profile ===

Columns:

Column	Meaning
Syscall	Linux syscall number (e.g., 3 = read, 4 = write, 63 = pread64). See `crates/shared/src/syscall_number.rs` for the full mapping.
Count	Total number of times this syscall was invoked.
Time(ms)	Cumulative wall-clock time spent handling this syscall, including kernel Wasm execution and host I/O.
Avg(ms)	Average time per call (`Time / Count`).
Retries	Number of times a blocking syscall (read/write on pipes/sockets, accept, or conflicting F_SETLKW) returned EAGAIN and was re-queued for retry. ENOLCK is a completed failure and is not counted as a retry. High retry counts indicate contention or slow producers/consumers.

Results are sorted by total time (highest first), so the most expensive syscalls appear at the top.

The footer shows pending pipe reader/writer counts — non-zero values at shutdown may indicate leaked pipes or unfinished I/O.

Limitations

Only available in Node.js (relies on process.env).
Measures host-side wall-clock time, which includes both kernel Wasm execution and any host I/O (filesystem, network). It does not isolate kernel computation from host latency.
Retry counts track how often a syscall was re-queued due to EAGAIN, not the total number of underlying host retries.

Benchmark Suite

The benchmark suite runs reproducible workloads on both Node.js and browser hosts, producing JSON results that can be compared across runs.

Prerequisites

Build the kernel and benchmark programs:

./run.sh setup
scripts/dev-shell.sh bash scripts/build-programs.sh

./run.sh setup provisions the musl sysroot for you on a fresh checkout. If a sysroot already exists and you changed libc/musl-overlay or libc/glue, setup only re-syncs headers — rebuild it explicitly first:

scripts/dev-shell.sh bash scripts/build-musl.sh

Some suites require additional binaries:

Suite	Requires
`syscall-io`	Base sysroot + benchmark programs
`process-lifecycle`	Base sysroot + benchmark programs
`spawn-scratch`	Benchmark programs; supplies both executables and uses an empty VFS
`erlang-ring`	Pre-built Erlang binary
`wordpress`	Pre-built PHP, nginx, WordPress
`mariadb`	Pre-built MariaDB

Application benchmark suites fail if their required binaries are not found. A missing artifact means the suite did not measure its workload; build or fetch the prerequisite before running it.

Running Benchmarks

# All suites on Node.js (3 rounds each, reports median)
scripts/dev-shell.sh npx tsx benchmarks/run.ts

# All suites in the browser (via Playwright)
scripts/dev-shell.sh npx tsx benchmarks/run.ts --host=browser

# Single suite
scripts/dev-shell.sh npx tsx benchmarks/run.ts --suite=syscall-io

# More rounds for stability
scripts/dev-shell.sh npx tsx benchmarks/run.ts --rounds=5

# Combine options
scripts/dev-shell.sh npx tsx benchmarks/run.ts --host=browser --suite=process-lifecycle --rounds=5

# Focused host-to-kernel spawn scratch evidence
scripts/dev-shell.sh npx tsx benchmarks/run.ts --suite=spawn-scratch --rounds=3
scripts/dev-shell.sh npx tsx benchmarks/run.ts --host=browser --suite=spawn-scratch --rounds=3

Results are saved as JSON in benchmarks/results/.

Each result records the selected benchmark artifact paths, sizes, and SHA-256 digests for that host's measurements. The application evidence covers the PHP/opcache inputs, Node WordPress source/config/router, browser WordPress and MariaDB VFS images, and each Node MariaDB architecture's server, client, and bootstrap SQL. The WordPress source-tree digest follows source symlinks but excludes runtime-owned wp-content/database state and wp-content/debug.log. Resolver-selected paths are retained alongside the logical artifact names; browser VFS evidence reflects the public asset that the benchmark page selects first. Kernel fingerprints use the same policy-aware binary resolver as each host. Node rootfs evidence records which of the runtime's rootfs.vfs.zst then programs/rootfs.vfs.zst fallback requests won, and is required for the syscall-io and established process-lifecycle suites that boot that default image. The Node spawn-scratch suite supplies both of its executables and explicitly uses an empty VFS, so it neither resolves nor records a rootfs. Browser benchmarks do not record or require the default rootfs because they boot generated empty or app images. Node static benchmark Wasm inputs are required only by the selected suite that consumes them. The browser benchmark page imports its seven micro Wasm URLs at module load, so every runnable browser suite requires all seven; exec-bench.wasm remains Node-only. After printing the artifact report, the runner stops before workloads when a required, selected input is missing.

Results also fingerprint host/src. Node results fingerprint the compiled worker bundle and reject a bundle whose recorded TypeScript input content does not match the current source. gitHead and gitRef do not record worktree dirtiness, so benchmark-relevant sources and rebuilt artifacts must match the exact committed final head. Preserve and report unrelated user-owned worktree or submodule changes rather than cleaning them to manufacture a globally clean status.

Available Suites

syscall-io

Measures raw I/O throughput, syscall round-trip overhead, and advisory-lock manager operations. The lock workload uses OFD locks so an independently opened descriptor can perform a real conflicting lookup without including a fork in the timed region. One shape holds one to three separated records on each of 64 files (127 total records); the other holds 256 separated ranges on one file. The 64-file shape retains 128 OFDs, below the browser SharedFS backend's 160-handle ceiling while still exercising file-index selection at scale. These counts fit both sides of a before/after comparison with the former fixed table. They are performance workloads, not substitutes for the 4096-record capacity tests.

Metric	Unit	What it measures
`pipe_mbps`	MB/s	Write 1 MB through a pipe (fork + read/write loop)
`file_write_mbps`	MB/s	Write 1 MB to a file
`file_read_mbps`	MB/s	Read 1 MB from a file
`syscall_latency_us`	microseconds	Average `getpid()` round-trip over 1000 calls
`lock_many_files_acquire_us_per_op`	microseconds/operation	Acquire separated write locks across 64 files with one to three records each
`lock_many_files_conflict_us_per_op`	microseconds/operation	Find each many-file lock from an independently opened OFD
`lock_many_files_replace_us_per_op`	microseconds/operation	Replace each many-file write lock with a read lock under the same OFD owner
`lock_many_files_unlock_us_per_op`	microseconds/operation	Unlock each many-file record
`lock_dense_file_acquire_us_per_op`	microseconds/operation	Acquire 256 separated write ranges on one file
`lock_dense_file_conflict_us_per_op`	microseconds/operation	Find each dense-file lock from an independently opened OFD
`lock_dense_file_replace_us_per_op`	microseconds/operation	Replace each dense-file write range with a read range under the same OFD owner
`lock_dense_file_unlock_us_per_op`	microseconds/operation	Unlock each dense-file range

For a focused advisory-lock comparison, run the same benchmark source and round count before and after the change on both hosts. Rebuild the guest artifact for each side's ABI epoch; an ABI 39 benchmark must not be reused with an ABI 40 kernel, or vice versa:

scripts/dev-shell.sh npx tsx benchmarks/run.ts --suite=syscall-io --rounds=3
scripts/dev-shell.sh npx tsx benchmarks/run.ts --host=browser --suite=syscall-io --rounds=3

The complete Node and browser suite is still required before making a broad performance or no-regression claim. Adding these cases does not itself establish that the lock-manager migration is faster, slower, or neutral.

process-lifecycle

Measures process management primitives against the default Node rootfs. The suite keeps that established environment so a focused scratch measurement cannot silently change the meaning of the hello/fork/exec/clone results.

Metric	Unit	What it measures
`hello_start_ms`	ms	Cold start: load Wasm + run hello world to exit
`fork_ms`	ms	Fork a child + wait for it to exit
`exec_ms`	ms	Exec a new program
`clone_ms`	ms	Create a thread via clone

spawn-scratch

Measures posix_spawn transport latency and the retained kernel-owned scratch high-water mark. The workload supplies spawn-bench.wasm and /bin/hello directly and uses an empty VFS, independently of the default-rootfs process-lifecycle suite. Its ordinary spawn uses the fixed environment LANG=C, PATH=/bin; the large cases use a deterministic 84,386-byte complete wire blob. Every sample is accepted only after waitpid reports that the child exited normally with status zero.

Metric	Unit	What it measures
`spawn_ms`	ms	One ordinary spawn plus successful child wait
`spawn_large_wire_bytes`	bytes	Complete deterministic large-spawn wire size asserted by both host wrappers
`spawn_large_first_ms`	ms	First spawn with an 84,386-byte complete wire blob
`spawn_large_repeat_ms`	ms	Mean of five subsequent 84,386-byte spawns in the same kernel
`spawn_scratch_retained_bytes`	bytes	Rust-owned spawn scratch capacity retained after the workload
`spawn_scratch_kernel_bytes`	bytes	Kernel WebAssembly memory size after the workload

For this workload, post-run kernel memory is also peak kernel memory only because WebAssembly memory grows monotonically and cannot shrink. Likewise, post-run Rust Vec<u8> capacity is the retained scratch high-water mark because the kernel keeps that reusable allocation and does not shrink it between spawns. These are properties of the measured implementation, not a general claim that a final sample can substitute for peak-memory instrumentation.

erlang-ring

Runs the Erlang/OTP BEAM VM, spawning 1000 lightweight processes in a ring topology and passing a token around 100 times.

Metric	Unit	What it measures
`total_ms`	ms	Wall-clock time for full ring completion
`messages_per_sec`	msg/s	Total messages (100,000) / elapsed seconds

Prerequisites:

Component	Path	Build command
BEAM VM	`packages/registry/erlang/bin/beam.wasm`	`scripts/dev-shell.sh bash packages/registry/erlang/build-erlang.sh`
OTP libraries	`packages/registry/erlang/erlang-install/`	(built by same script)
Ring program	`packages/registry/erlang/demo/ring.beam`	(included in repo)

Build requirements: Erlang/OTP 28 and the wasm32posix-cc SDK supplied through the declared development shell. If a required tool is absent there, report the block and update the declared environment rather than substituting an undeclared host binary for validation.

wordpress

Runs PHP 8.4 with a full WordPress 6.7 installation. Two measurements: cold CLI load time and HTTP server first-response time.

Metric	Unit	What it measures
`cli_require_ms`	ms	`php -r "require 'wp-load.php'"` — process start to exit
`http_first_response_ms`	ms	Start PHP built-in server, time to first HTTP response

Each suite round first copies the WordPress tree and the router script into a fresh scratch directory under the OS temp directory, following symlinks (see Checkout location). The measurements run against that copy; the checkout's tree is only the source. Each Node measurement starts from the WordPress setup state: the benchmark removes and recreates wp-content/database, removes wp-content/debug.log, and does the same cleanup after the measurement. When the OPcache side module is available and NO_OPCACHE is not 1, the measurement uses opcache.file_cache_only=1 with timestamp validation disabled. Each suite round creates a private cache root inside its scratch copy, and the CLI and HTTP measurements each receive a separate empty cache directory that is reset before and after timing. The scratch copy is removed when the round finishes, so compiled scripts cannot carry across metrics, rounds, concurrent benchmark processes, or worktrees. The HTTP measurement fails if the first response carries a PHP warning or fatal error, rather than timing an error page.

Prerequisites:

Component	Path	Build command
PHP CLI	`packages/registry/php/php-src/sapi/cli/php`	`scripts/dev-shell.sh bash packages/registry/php/build-php.sh`
WordPress	`packages/registry/wordpress/wordpress/wp-settings.php`	See below
Router script	`packages/registry/wordpress/demo/router.php`	(included in repo)

Build requirements: wasm32posix-cc SDK. The PHP build script automatically builds dependencies (SQLite, zlib, OpenSSL, libxml2). WordPress must be downloaded separately into packages/registry/wordpress/wordpress/.

mariadb

Runs MariaDB 10.5 with the Aria or InnoDB storage engine. Measures bootstrap (system table creation) and a sequence of SQL operations.

Four suite variants are registered so wasm32 and wasm64 builds can be compared side-by-side in one run.ts invocation:

Suite	Engine	Wasm ABI	Install dir
`mariadb-aria`	Aria	wasm32 (ILP32)	`mariadb-install/`
`mariadb-aria-64`	Aria	wasm64 (LP64)	`mariadb-install-64/`
`mariadb-innodb`	InnoDB	wasm32 (ILP32)	`mariadb-install/`
`mariadb-innodb-64`	InnoDB	wasm64 (LP64)	`mariadb-install-64/`

Metric	Unit	What it measures
`bootstrap_ms`	ms	System table initialization (`--bootstrap` mode)
`query_create_ms`	ms	CREATE TABLE (2 tables)
`query_insert_ms`	ms	Batch INSERT (100 rows into each table)
`query_select_ms`	ms	SELECT with WHERE clause
`query_join_ms`	ms	JOIN across two tables

Set MARIADB_BENCH_VERBOSE=1 to forward mariadbd stdout/stderr to the shell (useful for debugging hangs or slow bootstraps).

Each query metric times one mysqltest session: process start, connect, the SQL, and exit. mariadbd, mysqltest, and the bootstrap SQL are resolved and read once before any timing. Resolving a program runs the binary resolver, which checks the program index (about 0.9 s, growing with the number of packages); it used to run inside every timed query and made up about 90% of each query number. The data directory is a fresh scratch directory under the OS temp directory, removed after the run.

Prerequisites:

Component	Path	Build command
MariaDB server (wasm32)	`packages/registry/mariadb/mariadb-install/bin/mariadbd.wasm`	`scripts/dev-shell.sh bash packages/registry/mariadb/build-mariadb.sh`
MariaDB server (wasm64)	`packages/registry/mariadb/mariadb-install-64/bin/mariadbd.wasm`	`scripts/dev-shell.sh bash packages/registry/mariadb/build-mariadb.sh --wasm64`
mysqltest client	`<install-dir>/bin/mysqltest.wasm`	(built by same script)
System table SQL	`<install-dir>/share/mysql/mysql_system_tables*.sql`	(built by same script)

Build requirements: cmake and the wasm32posix-cc / wasm64posix-cc SDK supplied through the declared development shell. The build is a two-phase cross-compilation (host build for code generators, then wasm cross-compile). The wasm64 build uses -O1 instead of -O2 to avoid an LLVM 21 wasm64 backend miscompilation in table-lookup sign-extension. A missing declared tool is a reported environment block, not permission to use an undeclared host binary.

Checkout location does not change the numbers

On Node the application suites run guest programs directly on the host filesystem, so the paths a guest touches are real host paths, and work that walks or mirrors a path grows with the checkout's depth on disk. PHP's opcache file cache, for example, recreates each script's absolute path inside the cache one directory at a time: a checkout one directory deeper cost WordPress about 560 extra mkdir calls per run. The WordPress and MariaDB suites therefore stage their path-sensitive data under the OS temp directory, which has the same path for every checkout in one environment. Run before/after comparisons in the same environment (for example, both under scripts/dev-shell.sh, which sets its own TMPDIR).

Building All Suite Prerequisites

To run the complete benchmark suite, build all prerequisites in order:

# 1. SDK toolchain (required by all application suites)
scripts/dev-shell.sh bash -lc 'cd sdk && npm link'

# 2. Base benchmark programs (syscall-io, process-lifecycle, spawn-scratch)
scripts/dev-shell.sh bash scripts/build-programs.sh

# 3. Erlang/OTP
scripts/dev-shell.sh bash packages/registry/erlang/build-erlang.sh

# 4. PHP + WordPress (PHP build includes SQLite, zlib, OpenSSL, libxml2)
scripts/dev-shell.sh bash packages/registry/php/build-php.sh
# Download WordPress into packages/registry/wordpress/wordpress/

# 5. MariaDB
scripts/dev-shell.sh bash packages/registry/mariadb/build-mariadb.sh          # wasm32
scripts/dev-shell.sh bash packages/registry/mariadb/build-mariadb.sh --wasm64 # wasm64 (optional)

Running the Complete Suite for Performance Work

When measuring the performance impact of kernel changes, run every enabled suite on both hosts. Application-level suites (WordPress and the MariaDB variants) exercise different syscall patterns and threading models that micro-benchmarks miss.

# Full comparison workflow:
# 1. Run baseline on both hosts
scripts/dev-shell.sh npx tsx benchmarks/run.ts --rounds=3
scripts/dev-shell.sh npx tsx benchmarks/run.ts --host=browser --rounds=3

# 2. Apply changes

# 3. Run again on both hosts
scripts/dev-shell.sh npx tsx benchmarks/run.ts --rounds=3
scripts/dev-shell.sh npx tsx benchmarks/run.ts --host=browser --rounds=3

# 4. Compare
scripts/dev-shell.sh npx tsx benchmarks/compare.ts benchmarks/results/<before>.json benchmarks/results/<after>.json

When a required binary is missing, the runner prints the artifact report and fails before starting workloads. Build the missing prerequisites (see above) before drawing conclusions about performance impact.

Comparing Results

Use the comparison tool to diff two benchmark runs:

scripts/dev-shell.sh npx tsx benchmarks/compare.ts benchmarks/results/before.json benchmarks/results/after.json

Output is a markdown table with percentage change for each metric. Regressions (>5% worse) are bolded:

| Benchmark                      | Before | After | Change  |
|--------------------------------|--------|-------|---------|
| syscall-io/pipe_mbps           | 245.5  | 260.1 | +5.9%   |
| syscall-io/syscall_latency_us  | 12.3   | 15.1  | **+22.8%** |
| process-lifecycle/fork_ms      | 45.2   | 44.8  | -0.9%   |

For throughput metrics (*_mbps, messages_per_sec), a decrease is a regression. For latency/duration metrics, an increase is a regression.

Writing a Custom Suite

Create a TypeScript file in benchmarks/suites/ that exports a BenchmarkSuite:

import type { BenchmarkSuite } from "../types.js";

const suite: BenchmarkSuite = {
  name: "my-suite",
  async run(): Promise<Record<string, number>> {
    // Run workload, return metric_name → value
    return { ops_per_sec: 1234.5 };
  },
};

export default suite;

const SUITE_MODULES: Record<string, string> = {
  // ...existing suites...
  "my-suite": "./suites/my-suite.js",
};

Benchmark programs (C source) live in benchmarks/programs/ and output metrics as key=value lines on stdout, which suites parse with a simple regex.

FilesExpand file tree

profiling.md

Latest commit

History

profiling.md

File metadata and controls

Profiling and Benchmarking

Syscall Profiler

Enabling

Collecting Results

Output Format

Limitations

Benchmark Suite

Prerequisites

Running Benchmarks

Available Suites

syscall-io

process-lifecycle

spawn-scratch

erlang-ring

wordpress

mariadb

Checkout location does not change the numbers

Building All Suite Prerequisites

Running the Complete Suite for Performance Work

Comparing Results

Writing a Custom Suite