Skip to content

About

Archlinux's mkinitcpio hook to run Tailscale daemon during early userspace (aka initramfs)

Topics

Resources

Contributing

Stars

28 stars

Watchers

2 watching

Forks

Repository files navigation

mkinitcpio-tailscale

Reach an Arch Linux machine over Tailscale while it is still in early boot, before the root filesystem has been unlocked.

A server with an encrypted root cannot finish booting on its own: it stops in the initramfs and waits for a passphrase. When that machine is in another room, or another country, a kernel update or a power cut leaves it sitting at that prompt until somebody walks up to a keyboard.

This project provides a mkinitcpio hook that starts tailscaled inside the initramfs, so the machine joins your tailnet before it mounts its root filesystem. It registers as a device of its own, homeserver-initrd for a host called homeserver, separate from the Tailscale node the booted system runs, so you can SSH in, enter the passphrase, and let the boot carry on.

Because the path is Tailscale, nothing is exposed to the internet: no port forwarding, no firewall holes, no SSH server for the rest of the world to knock on. Who may reach the machine is decided by your tailnet ACLs and can be narrowed to the initrd device alone; see Security considerations, which also covers the trade-off of keeping a node key in an unencrypted initramfs.

Tailscale's built-in SSH server is enabled by default, so no separate dropbear or tinyssh is needed; see Tailscale SSH server. Both systemd- and busybox-based initramfs images are supported.

Unlocking the root filesystem is configured separately; the ArchWiki covers that half:

Entering the passphrase once the machine is waiting for it is covered in Unlocking.

Installation

You can install mkinitcpio-tailscale from the AUR:

yay -S mkinitcpio-tailscale

Configure

1. Register the initrd node

setup-initcpio-tailscale

The helper starts a throwaway tailscaled and prints a URL and QR code to authenticate with; it does not touch the Tailscale service your booted system runs. No root needed up front: it escalates through sudo or doas, whichever is present, in a single elevated step for the final writes under /etc/initcpio/tailscale/, so you are asked for a password at most once. Any extra arguments are passed straight through to tailscale up, so flags like --login-server= work as usual.

It registers a node named after your host with an -initrd suffix (a machine called homeserver appears as homeserver-initrd) with Tailscale SSH turned on, and leaves the node key and the SSH host keys in /etc/initcpio/tailscale/. Pass --no-ssh if you would rather run dropbear or tinyssh in the image instead.

Disable key expiry for that node on the Machines page of the Tailscale admin console. Node keys expire by default, and an expired initrd node cannot reach your tailnet, which you would discover while locked out of a machine that is waiting for its passphrase. A tagged node skips this step, since tagging disables expiry, and tagging is what you want anyway: see Security considerations.

2. Give the initramfs a network

Nothing in mkinitcpio's stock hooks brings up networking, and tailscaled cannot do anything without an address. Add one of these alongside this hook:

initramfs hook package configured by
systemd sd-network mkinitcpio-systemd-extras (AUR) .network files copied from /etc/systemd/network
busybox net mkinitcpio-nfs-utils (core) the ip= kernel parameter
busybox netconf mkinitcpio-extras (AUR) the ip= kernel parameter

Both busybox hooks take the same parameter, and either works with this hook; they differ in what they can be told to do and where they come from:

ip=192.168.1.50::192.168.1.1:255.255.255.0::eth0:none    # static, both hooks
ip=dhcp                                                  # netconf

net comes from the official repositories and needs no AUR build, which is reason enough to prefer it when the machine has a fixed address. Its DHCP client did not come up in testing, so reach for netconf if you need ip=dhcp. That is also what the ArchWiki's remote-unlock guide uses, and mkinitcpio-extras is the maintained replacement for the retired mkinitcpio-netconf, mkinitcpio-dropbear and mkinitcpio-tinyssh, so it is the one to pick if you also want a dropbear or tinyssh alongside.

Use the kernel's device name (eth0) rather than the name the booted system shows (enp1s0): the predictable names are not in effect this early. For sd-network, match on a glob (Name=en*) instead of one fixed name, for the same reason.

3. Add the hook

Edit /etc/mkinitcpio.conf and put tailscale after the network hook and before whatever blocks waiting for a passphrase, be it sd-encrypt, encrypt or encryptssh. If a preset in /etc/mkinitcpio.d/ names a configuration file of its own (ALL_config= or <preset>_config=), edit that file instead:

# systemd-based
HOOKS=(base systemd autodetect microcode modconf kms keyboard sd-vconsole block sd-network tailscale sd-encrypt filesystems fsck)

# busybox-based
HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block net tailscale encryptssh filesystems fsck)

On a busybox-based initramfs the passphrase hook has to be encryptssh (mkinitcpio-utils): the stock encrypt hook only reads the console, so nothing typed over SSH can answer it.

On a systemd-based initramfs tailscale must come after systemd. The hook decides which kind of image it is building by looking for helpers that the systemd hook defines, so listing it earlier silently produces the busybox layout, which a systemd initramfs never runs, leaving you with an image that builds cleanly and never connects.

4. Rebuild and check

sudo mkinitcpio -P

Before rebooting a machine you cannot walk up to, confirm the setup end to end:

setup-initcpio-tailscale --check

It verifies, without changing anything, that the configuration files exist, that tailscale sits correctly in HOOKS= (after systemd, after the network hook, before the encrypt hook) in every configuration your presets use, that the images they build, UKIs included, actually contain tailscaled, and (with jq installed and this machine on the tailnet) that the initrd node's key is not about to expire. Then confirm the node is live:

tailscale status | grep -- -initrd    # from any other node on your tailnet

The image carries only what boots: tailscaled, not the tailscale CLI, which nothing in early userspace executes and which costs about a quarter of the image. To have it available for debugging from the emergency shell, set CLI="yes" in /etc/initcpio/tailscale/default.env and rebuild.

From here on the image looks after itself: the package ships a pacman hook that reruns mkinitcpio -P whenever the tailscale package is upgraded, since the image carries its own copy of tailscaled that would otherwise stay at the old version until the next kernel update. The hook does nothing on machines where tailscale is not in HOOKS=.

5. Give yourself more than 90 seconds

On a systemd-based initramfs, whatever device root= names is a systemd device unit, and systemd gives up on a device after 90 seconds (DefaultDeviceTimeoutSec). That device only appears once the passphrase has been entered, so the clock is running while you are still reaching for a terminal. When it expires the mount jobs fail and the initramfs drops to emergency mode; the passphrase prompt is still there, but answering it no longer resumes the boot.

Whether it bites depends on how root= is written. systemd-cryptsetup disables the timeout for the device it unlocks itself, so root=/dev/mapper/root is safe. root=UUID=<filesystem uuid>, or an LVM volume on top of LUKS, is not: that device unit is a different one, and it keeps the default. Removing the limit costs nothing where it was not needed, so add it to your boot loader entry either way:

rootflags=x-systemd.device-timeout=0

This is a systemd-initramfs concern only; the busybox encrypt and encryptssh hooks have no such timeout. And even when the timeout fires you are not cut off: the boot test lets a systemd image run past it on purpose, and the node stays online and still answers SSH in emergency mode, so you can log in and finish the job by hand with systemctl start initrd-root-fs.target after unlocking the device. A busybox image is covered too, for a different reason: its init tears tailscaled down before deciding the boot failed, so this hook starts it again on the way into the emergency shell.

Unlocking

Log in to the initrd node as root:

ssh root@homeserver-initrd

On a systemd-based initramfs a login that finds the passphrase prompt waiting goes straight to it. It keeps answering prompts as they appear, one per encrypted volume and again after a typo, until the boot moves on and the connection closes. Press Ctrl-C to get a shell instead; type unlock there to come back to the prompt. To skip the shell altogether:

ssh -t root@homeserver-initrd unlock

The -t matters, since the prompt needs a terminal.

On a busybox-based initramfs encryptssh makes its own unlock prompt root's login shell, so a root login asks for the passphrase straight away, and unlock runs the same prompt from any other shell.

Tailscale SSH server

Tailscale includes a built-in SSH server, and setup-initcpio-tailscale turns it on unless told otherwise, so you need no dropbear, tinyssh, or other SSH server inside the initramfs. Logging in is then:

ssh root@homeserver-initrd

The server only accepts connections from within your tailnet. The node won't accept local connections unless the client is also part of your Tailscale network, which reduces exposure compared to a traditional SSH server reachable from everywhere.

The isolated tailscaled the setup helper runs also generates the SSH host keys its server will use, and they are stored alongside the node key, so the initramfs presents the same host key every time and your client does not warn about a changed identity.

Works on systemd- and busybox-based initramfs alike, though the second needs a hand: Tailscale's SSH server has to resolve the user you log in as, and of the hooks mkinitcpio ships only systemd writes a user database into the image. On a busybox-based initramfs this hook therefore writes a minimal one itself: root, with /bin/sh as the shell, since that is what such an image actually contains. Where a database already exists it is left untouched, so a systemd-based image keeps the richer one mkinitcpio built. The same database is what lets a dropbear or tinyssh in the image accept logins at all; without one they greet the connection and then refuse it with Permission denied (publickey).

Bringing your own SSH server

Turn Tailscale SSH off with:

setup-initcpio-tailscale --no-ssh

which also removes any host keys an earlier run left in /etc/initcpio/tailscale/, so they stop being copied into new images.

Run one SSH server, not two. When Tailscale SSH is enabled, tailscaled answers port 22 on the tailnet itself, so a dropbear or tinyssh in the same initramfs never sees those connections; it still answers on other interfaces, but not on the address you would actually reach it at. Either keep the default and use Tailscale SSH, or register with --no-ssh and use your own daemon.

Two things your daemon needs. Inbound tailnet connections reach it through a proxy that dials the same port on 127.0.0.1, so it must listen on a wildcard or loopback address (one bound to a specific interface address will never see a connection); this hook brings lo up itself, since nothing else in a busybox image would. And log in as root: everything in an initramfs is owned by root, and dropbear refuses an authorized_keys it does not consider owned by the user logging in, so root is the login that works. That is the same reason the retired dropbear hooks used a root_key. If your daemon really must bind an interface address, register with --tun: the initramfs then runs on a kernel TUN device (tailscale0) and inbound connections arrive through it the ordinary way.

The test suite covers all of this end to end: every scenario boots under QEMU and is logged into from a second node on a throwaway tailnet, Tailscale SSH and dropbear alike, over both network stacks, with the host key offered checked against the expected one.

Using headscale

Nothing here assumes Tailscale's hosted control plane. Register against your own headscale the way you would any node:

setup-initcpio-tailscale --login-server=https://headscale.example.net

or non-interactively with a pre-auth key:

setup-initcpio-tailscale --login-server=https://headscale.example.net \
    --authkey=file:node.key

One Tailscale-ism to translate: key expiry is headscale's to manage, not the Tailscale admin console's, so the console URL in the setup output and in --check's advice does not apply to you. headscale nodes list shows each node's expiry, and how it is set (and whether it applies at all) depends on your headscale version and configuration; the probe in --check reads the booted system's view of the tailnet and works against headscale all the same.

headscale only tags a node through its auth key, and refuses --advertise-tags at registration, so create the key with the tag instead (the tag still needs a tagOwners entry in the policy):

headscale preauthkeys create --user <id> --tags tag:initrd

Tailscale SSH needs an ssh policy in headscale's ACLs before the node accepts a session; the Security considerations example below is valid headscale policy once alice@example.com names one of your headscale users. This is not a theoretical combination: the test suite boots every scenario against a throwaway headscale, so the path you are on is the one CI exercises.

Security considerations

The initrd node's key sits unencrypted in the initramfs: it has to, since the image uses it before anything is unlocked. Anyone who can read your boot partition can copy it and join your tailnet as homeserver-initrd. So give that node as little as possible: it should accept your SSH session and never be able to open a connection of its own.

Tailscale's access rules make that possible, in two steps.

Tag the node. An untagged node belongs to the user who registered it and has all of that user's permissions. A tag replaces that identity, so only rules that name the tag apply to it, and a tagged node's key does not expire by default either. Allow your account to apply the tag by adding it to tagOwners (example below), then register with it:

setup-initcpio-tailscale --advertise-tags=tag:initrd

An auth key created with the tag works too, and is the only way with headscale. Without the tagOwners entry, registration fails with requested tags [tag:initrd] are invalid or not permitted. An existing node can also be tagged from the Machines page of the admin console.

Keep the tag out of every source. Access rules only allow what they list, one direction at a time: a rule with tag:initrd as its destination lets you connect to the node, and replies to that connection flow back, but the node itself can start a connection only where a rule lists it as a source. Leave tag:initrd out of every src, including through "*". Tailscale's default policy is a single allow-all rule with "src": ["*"], which covers tagged nodes too, so replace it if your tailnet still has it. Rules using autogroup:member are fine: tagged nodes are not members.

A policy that does both, where alice@example.com stands for your login:

{
  "groups": {
    "group:admins": ["alice@example.com"]
  },
  "tagOwners": {
    "tag:initrd": ["group:admins"]
  },
  "acls": [
    { "action": "accept", "src": ["group:admins"], "dst": ["*:*"] }
  ],
  "ssh": [
    {
      "action": "accept",
      "src": ["group:admins"],
      "dst": ["tag:initrd"],
      "users": ["root"]
    }
  ]
}

The acls rule lets your devices reach everything, the initrd node included, which Tailscale SSH also needs; the ssh rule lets you log in to the node as root. Nothing lists tag:initrd as a source, so a stolen key gets a node that can be reached but cannot reach anything. See Tailscale's access control documentation for the full syntax.

setup-initcpio-tailscale suggests tagging when the node comes out untagged, and setup-initcpio-tailscale --check warns when it is still untagged.

If you suspect the key has been copied:

  • Remove the initrd node from the Machines page of the admin console.
  • Re-run setup-initcpio-tailscale to register a fresh node, then rebuild your initramfs.

Development

The test suite runs in a throwaway Arch container and never touches your system:

make test        # lint, packaging, initramfs image contents
make test-all    # adds the QEMU boot matrix against a local headscale

Everything else a contributor needs -- what ships, the template PKGBUILD, the boot scenarios, how releases are cut -- is in CONTRIBUTING.md.

Prior work and big thanks

About

Archlinux's mkinitcpio hook to run Tailscale daemon during early userspace (aka initramfs)

Topics

Resources

Contributing

Stars

28 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages