Skip to main content
This page describes the harness. It does not publish results yet: the harness was written and unit tested on a machine with no Docker daemon, so every number in the results table below is marked not yet measured. Performance covers the idle footprint instead.

​
What it measures

levelrail-cli bench run drives an app URL at a constant arrival rate and reports, client side:
  • Requests per second achieved, and error rate (transport errors, timeouts and 5xx responses).
  • Latency p50, p95, p99 and max.
  • Ingress overhead: the same run against one replica directly (--direct-url), subtracted from the run through the ingress, per percentile.
  • Per failure scenario: dropped requests, the error window (first to last failed request after the event), and recovery time (event to last failed request).

​
Method

  • Open loop. Request i is scheduled at i / rate seconds and launched on its own goroutine whether or not earlier requests have returned. A closed-loop generator slows down when the server stalls and so hides the stall (coordinated omission).
  • Latency is measured from the intended send time, not from when the request left the client. A request that was due during a stall reports the queueing delay.
  • The client caps in-flight requests at 20000. Requests over the cap are counted as errors and reported as client_saturated. A non-zero value means the load generator was the bottleneck and the run is not valid evidence about the server.
  • Percentiles are nearest-rank over every request, failures included.
  • A run is recovered when its last failure is more than one second before the end of the run. Pick a duration long enough to leave a clean tail.
  • The generator should run on a different machine from the control plane, on the same network segment as a real client.

​
Reproduce

Build the fixture app, deploy it with the replica count you want to test, and put it behind the load balancer with a domain.
The fixture answers GET / with ok and exposes POST /__bench/die, which makes that one container exit so the reconciler has to replace it. Steady state, with the overhead comparison:
Replica failover (stops one replica’s container one third into the run):
Node drain on a multi-node placement:
Control plane restart (ADR 025: ingress should keep serving while the control plane is down):
The restart report includes ingress_kept_serving, true when no request failed after the restart was issued.

​
Hardware disclosure

Publish these with every result, via --hardware and the report’s client block:

​
Results

Not yet measured. No Docker daemon was available where the harness was written, so none of these have been run. Do not quote a number from this table until a row has a date and a hardware note.

​
What is tested without Docker

The load driver is unit tested against fake HTTP servers: arrival schedule, percentile math, error accounting, coordinated omission and failure-window analysis. Deploying the fixture and running the scenarios against real containers is not covered by those tests.
Last modified on October 17, 2026