Node scheduling: taints, tolerations, affinity and score mode

​

How a node is chosen

1. Drops nodes that are offline, cordoned, missing a node_selector label or too small for the workload’s requests.
2. Drops nodes with a NoSchedule or NoExecute taint the workload does not tolerate.
3. Drops nodes that fail a required affinity term (node_required, or a together or apart term with weight 0).
4. Scores the rest: the score mode picks the base order, preferred affinity terms add or subtract weight, and an untolerated PreferNoSchedule taint ranks the node lower.

​

Taints: reserve a node

# keep everything off this node unless it tolerates dedicated=gpu
levelrail-cli nodes taint <node-id> dedicated=gpu:NoSchedule
# a key with no value
levelrail-cli nodes taint <node-id> maintenance:PreferNoSchedule
# show the taints, then remove them all
levelrail-cli nodes taint <node-id>
levelrail-cli nodes taint <node-id> --clear

• A taint is key[=value]:Effect. Passing taints replaces the node’s whole set.
• NoSchedule and NoExecute keep new placements off the node. PreferNoSchedule only ranks it lower.
• Workloads already running on the node stay where they are. Use levelrail-cli nodes drain <node-id> to move them.
• API: GET and PUT /api/v1/nodes/{id}/taints with body {"taints":[{"key":"dedicated","value":"gpu","effect":"NoSchedule"}]}. Root only.

​

Tolerations: let an app onto a tainted node

# tolerate one exact taint
levelrail-cli nodes placement app web --toleration dedicated=gpu:NoSchedule
# tolerate any value of a key, for any effect
levelrail-cli nodes placement app web --toleration-exists dedicated
# remove the app's tolerations and affinity
levelrail-cli nodes placement app web --clear-placement-rules

​

Affinity: near, away, or on a kind of node

# affinity.yaml
node_required:            # every term must match the node's labels
  - {key: disk, op: In, values: [ssd]}
node_preferred:           # soft, weight 1 to 100
  - weight: 50
    term: {key: zone, op: In, values: [eu-1]}
together:                 # prefer nodes already running matching apps
  - match: [{key: name, op: In, values: [cache]}]
    weight: 20
apart:                    # weight 0 means required: never share a zone with db
  - match: [{key: name, op: In, values: [db]}]
    topology: zone

levelrail-cli nodes labels <node-id> disk=ssd zone=eu-1
levelrail-cli nodes placement app web --affinity-file affinity.yaml

• Operators are In, NotIn, Exists and DoesNotExist.
• Apps are matched by their labels and by name=<app name>.
• topology is node (the default) or a node label key such as zone or region.
• The term count per app is capped by APP_PLACEMENT_AFFINITY_MAX_TERMS (8 unless set). Invalid rules return 400 with the reason.

services:
  web:
    tolerations:
      - {key: dedicated, value: gpu, effect: NoSchedule}
    affinity:
      node_required:
        - {key: disk, op: In, values: [ssd]}

​

Score mode: spread or bin-pack

levelrail-cli settings runtime get placement.score_mode
levelrail-cli settings runtime set placement.score_mode binpack
levelrail-cli settings runtime set placement.score_mode spread

• spread (default) picks the node that keeps the most headroom.
• binpack fills the fullest node that still fits, which leaves whole nodes free.
• Failure-domain spread is applied before the score mode. When any node carries a zone label, a new replica goes to the zone with the fewest replicas first, and binpack only orders the nodes inside that zone. The decision output says so (“spread across failure domains”). Remove the zone labels to get pure bin-packing.
• The setting is read on every placement and needs no restart. APP_PLACEMENT_SCORE_MODE only seeds the default. The dashboard has it under Settings, Runtime.

​

See why a node was picked or skipped

levelrail-cli nodes placement app web

​

Not built yet

• One instance per matching node (workload: per-node in app.yaml) is accepted by validation but not scheduled yet.
• NoExecute does not evict running workloads by itself. A drain does.