Skip to main content
HetznerDigitalOceanLinodeVultr
Multi-node covers enrolling a machine you already have. This page covers the step before that: having Levelrail create the machine at a cloud provider. For the short command sequence, see the multi-cloud provisioning quickstart. Provisioning drives the same join-token enrollment that multi-node uses. The control plane mints a token, hands it to the new VM’s first-boot script, and the agent enrolls the way it always does.

Provider credentials

Store a credential once under Settings > Cloud node providers (/settings/node-providers) or with levelrail-cli nodes providers set-credential --provider <name>. Each provider card reads “Not connected” until you save one, and a saved credential is never shown again. It is envelope-encrypted at rest like every other integration credential. Levelrail Cloud node providers settings with Hetzner, DigitalOcean, AWS, and Azure cards, all not connected Levelrail Cloud node providers settings with Hetzner, DigitalOcean, AWS, and Azure cards, all not connected --provider-token is accepted for scripting, but its value lands in shell history and the process list. Pipe the credential instead, or omit the flag for a no-echo prompt:
A project API token with Read & Write access. Tokens are scoped to one project, and servers are created in that project.

Connection methods

Each provider offers only the methods the control plane can use today. Pick one in the connect dialog, or deep link to it: /settings/node-providers?connect=aws&method=role. Closing the dialog removes both parameters from the URL. GCP external account JSON and Azure managed identity are not wired into the provisioners yet, so they are not offered. Linode, Vultr and Scaleway are planned: each needs one provision.Provisioner implementation (regions, sizes, create, get, delete) plus its entry in nodeProviderNames.

Test connection and disconnect

Test connection makes one read-only call (list regions). With a credential typed into the dialog it tests that candidate and stores nothing. Test and connect runs the same test first and refuses to store the credential when it fails, with the missing permissions in plain language. Test stored connection re-tests what is saved and records the time. The credential value is never returned by any endpoint or written to the audit log.
connect tests before storing unless --verify=false. Every command takes --json. Access is split in two: seeing status, regions and sizes needs read; connecting, testing and disconnecting need write:sensitive; creating servers and join tokens stays root. Grant the middle tier to a team to let it manage provider connections without making it an admin. Connect, test and disconnect appear in the audit log with the actor. There is no automatic rotation reminder yet: the card shows who connected and when it was last tested, so rotate on your own schedule and re-test.

Custom data center

Choose Custom data center for bare metal, on-prem or any VM with no cloud API. Connecting records a name, a location label and notes. Add a node then mints the same one-time join token and install command as the manual enrollment flow. The agent dials out, so the firewall only needs outbound TCP to the control plane’s agent port (9443 by default) and outbound HTTPS for image pulls.

How it works

  1. POST /api/v1/nodes/provision (or nodes provision, or the Nodes page’s Add node wizard) mints a join token through the same code path as POST /api/v1/nodes/join-tokens, then asks the provider to create a server with a cloud-init script as user-data.
  2. The script installs Docker if the image lacks it (the same get.docker.com script install.sh uses) and starts the agent as a systemd-managed container. The join token and CA fingerprint go through a root-only environment file, not the container’s command line.
  3. GET /api/v1/node-provisions/{id} (or nodes provisions show <id>, or the wizard’s progress view) recomputes status on every call. It asks the provider whether the server is up and checks whether a node with the expected name has enrolled. Status moves through creating, booting, enrolling and ready, or failed with a reason.
  4. Once the node is found, its workload kinds are set from the provision’s role: general accepts apps, build accepts builds (see Build node routing). Enrollment itself always starts a node as a plain app node.
A name already used by an enrolled node, or by another provision that has not failed, is rejected with 409. The name is how step 3 matches a node to its provision.
Why there is no installing stageThe status vocabulary includes installing, but nothing emits it: the control plane cannot see inside the VM while cloud-init runs. Everything between “booted” and “dialed home” reports as enrolling. If a provision sits there for a long time, read /var/log/cloud-init-output.log from the provider’s console. APP_NODE_PROVISION_TIMEOUT (default 20 minutes) marks it failed on its own.

The agent runs from an image

The control plane binary that install.sh installs is checked against a published checksums.txt. The agent in provisioning is different: cloud-init pulls the container image ghcr.io/glincker/levelrail-agent, one tag per release and cosign-signed by the release pipeline. The pull is a plain docker pull over the registry’s TLS, and cloud-init does not verify the cosign signature before running it. Treat this like any unverified script run on first boot. Later releases also attach a raw agent binary (see Docker), but the provisioning script does not use it.

Token visibility on the node

The join token and CA fingerprint are container environment variables, so docker inspect levelrail-agent on the node shows them while the container exists. Treat a provisioned node as trusted infrastructure, as you would a manually enrolled one.

Cost

Every provider bills by the hour for as long as the server exists. Provisioning never deletes a server for you. If a provision fails partway, or you stop using a node, delete the server at the provider. levelrail-cli nodes delete <id> removes the node record but not the VM. Azure and GCP also bill for the networking resources created next to the VM: a public IP on Azure, and an external IPv4 address on GCP. These are small but not free. The size picker shows a live monthly price for Hetzner and DigitalOcean, because both return it from their APIs. For AWS, Azure and GCP it shows “pricing varies, see provider console”, since those catalogs need a separate pricing API that Levelrail does not call. A build role node benefits from more CPU and memory.

Known limitations

The server keeps provisioning. Use nodes provisions show <id> or GET /api/v1/node-provisions/{id} to follow it. There is no screen yet to reopen the progress view.
Azure requires a password when no SSH key is supplied, and there is no credential input for one, so a random throwaway password is generated and discarded. The agent enrolls by dialing out, so this does not block provisioning. To SSH into an Azure node, configure access in the Azure console.
The VM’s NIC, public IP and OS disk are created with deleteOption: "Delete", which Azure documents as cascading on VM delete. This was not verified against a real subscription. If it does not behave as documented, a deleted node can leave a NIC and public IP behind.
Only the service account JSON key is supported. See the GCP tab above.

Troubleshooting

Provision fails with “creating” status stuck

The cloud provider API call succeeded but the server has not started yet. Likely cause: the provider is slow to launch the server, or there is a quota or billing issue preventing VM creation. How to confirm: go to the provider’s console (Hetzner, DigitalOcean, AWS, etc.) and check whether a server with the expected name exists. Check levelrail-cli nodes provisions show <provision-id> to see the exact status and any error messages. Fix: wait a few more minutes and re-check. If the server never appears in the provider console, the API call failed even though the response looked successful; delete the provision in Levelrail and check your provider account for quota limits, billing issues, or API token permissions.

Provision fails with “enrolling” status stuck

The server was created and cloud-init started, but the agent never connected. Likely cause: Docker is taking a long time to install, cloud-init failed to start the agent, or the agent cannot reach the control plane. How to confirm: access the server’s console or SSH into it (if the provider gives you access) and check /var/log/cloud-init-output.log for errors. Also check the agent logs with docker ps | grep levelrail and docker logs if the container is running. Fix: if cloud-init failed, the log will name the error (permission denied, network failure, etc.). If Docker installation timed out, wait longer and re-check. If the agent started but could not connect, check that APP_AGENT_ADVERTISE_HOST is set correctly on the control plane (should be the control plane’s public hostname or IP, not 127.0.0.1). The provision will timeout after APP_NODE_PROVISION_TIMEOUT (default 20 minutes) and report failed with the last known status.

The provider credential was rejected

The control plane cannot authenticate with the cloud provider. Likely cause: the credential (API token, access key, or service account key) is invalid, expired, or has insufficient permissions. How to confirm: go to Settings > Cloud node providers and re-enter the credential to see if a validation error appears. Try to create a provision and check the error message. Fix: verify the credential matches what Provider credentials requires. For API tokens, confirm they have not expired. For IAM access keys, check that they have Compute permissions. Regenerate a new credential at the provider if the old one is invalid, and update it in Levelrail.

Provision succeeds but the node stays “pending”

The server was created and the agent started, but the control plane did not receive a heartbeat. Likely cause: APP_AGENT_ADVERTISE_HOST is wrong, or the new node cannot reach the control plane. How to confirm: SSH into the provisioned node (via the provider’s console) and check the agent logs with docker logs levelrail-agent or journalctl -u levelrail-agent. Look for TLS or connection errors. Fix: see the Multi-node troubleshooting section under “A node stays in pending status”. The root cause is usually APP_AGENT_ADVERTISE_HOST. Set it to the control plane’s public hostname or IP (the one the nodes can reach from outside), restart the control plane, and reenroll the node with a new join token. Still stuck? Check your provider’s VM console for cloud-init logs, or run levelrail-cli nodes provisions show <provision-id> and attach the output to an issue on GitHub.

What was not tested

This feature was built without real cloud accounts. The provider clients in internal/provision are tested against fake HTTP servers for Hetzner, DigitalOcean, Azure and GCP, and against a hand-written fake of the narrow EC2 interface for AWS. Azure and GCP were never run against a real subscription or project, so their full resource chains are unverified end to end. AWS default-VPC lookup, image resolution, security group handling, and the STS and ambient-credential paths are only as correct as their fakes. Before relying on this in production, provision one real node per provider and confirm it reaches ready.

See also

Multi-cloud quickstart

The three-command version.

Multi-node

Placement, drain, certificates and the mesh.
Last modified on October 19, 2026