SSH Host Keys
Every SSH server proves who it is with a host key. When a step opens an SSH session to a device, the worker checks that key against the keys the run's organization trusts for that host, before it sends a password or a private key. A host that presents a different key is refused, and the refusal is recorded in the audit log.
This stops a machine in the network path from posing as the device and collecting its credentials (a "man-in-the-middle" attack).
What Is Checked
- Shell steps (
shell.execute, and any handler that opens a session withservices.open_shell). - Network-CLI steps (
netcli.*, and any handler that opens a session withservices.connect), whichever transport the device uses: netmiko, asyncssh or scrapli.
For both, the device and, when the device has one, its jump host are checked the same way. A netmiko, asyncssh or scrapli transport older than step plugins 0.28.0 cannot check host keys, so the worker refuses to connect with it and the step fails with a message that names the package to upgrade.
Not checked:
- A device transport from another plugin that does not support the check connects without it. The worker logs a warning that names the transport.
- Git over SSH keeps its own rule: it trusts a host's key on first contact for the length of one operation (
accept-new). - The Ansible step has its own
host_key_checkingoption.
Weak Algorithms and Small Keys
Shell and network-CLI steps also refuse weak SSH settings, on the device and its jump host, before any credential is sent:
- key exchange with SHA-1, or group exchange, where the server picks a Diffie-Hellman group that could be smaller than 2048 bits (the netmiko transport keeps group exchange, because it refuses a group under 2048 bits itself);
- CBC, 64-bit block or RC4 ciphers, and SHA-1, MD5 or truncated MACs;
- DSA keys, SHA-1 signatures and RSA keys under 2048 bits, both host keys and login keys.
A host key that fails these rules is not recorded on first contact. The step setting Allow legacy SSH algorithms (allow_legacy_ssh) lifts them for one step, for an older host on a trusted network; host keys are still checked. It is a setting of shell.execute and of the network-CLI steps, and it is off unless a step turns it on. Legacy SSH algorithms lists exactly what it adds for each transport; the step editor links to that page from the setting. The rules are the step SDK's (hegemony_step_sdk.ssh); each SSH library's algorithm lists are built from them in hegemony_transport_support.ssh (paramiko and asyncssh), so the pinned step plugins release decides them for every transport and for shell steps.
A device can carry the same opt-in itself, so no step has to: Allow legacy SSH algorithms for this device on the device form (access_config.ssh.allow_legacy_ssh). It covers connections to that device only, in every step, and not its jump host, which has its own setting (access_config.jump_host.allow_legacy_ssh). The value must be true or false; text is refused when the device is saved or synced. A device with either setting shows a Legacy SSH allowed badge on its page.
A device that comes from an inventory provider gets the setting from its source, not from the device form:
- Git inventory:
allow_legacy_ssh: trueunderaccess_config.ssh(oraccess_config.jump_host) in the device file. - NetBox: a boolean custom field on the device,
hegemony_ssh_allow_legacy_sshby default. A device without the field, or with it empty or false, stays on secure SSH. A field of another type, such as text, fails that device's sync instead of being read as a flag.
Modes
Each organization chooses one mode. The default is accept_new.
| Mode | First connection to a host | Later connections |
|---|---|---|
accept_new | Accepts the key the host presents and stores it (trust on first use). | Accepts only the stored keys. |
strict | Refused, unless an admin pinned a key for the host. | Accepts only the stored keys. |
off | Accepts any key. Stores nothing. | Accepts any key. |
off switches the protection off. Use it only for a short time, for example while a lab rebuilds all its hosts. Every change of mode is audited (org.updated, with the old and new mode).
For single hosts whose key changes all the time, such as a lab's, an admin can exempt the host instead of switching checks off for every host.
Trust on First Use
In accept_new mode, the first step that connects to a host stores the key the host presents. It does this before it logs in:
- It fetches the host's key without logging in.
- It stores the key.
- It logs in, accepting only that key.
So no password or private key is sent until the key is stored. From then on, the host must present that key. If the host refuses the login (for example, a wrong password), the key stays stored, and the next connection is checked against it.
- Keys are stored per host and port, with the host as the device or jump host names it, in lower case. A device reached once by IP address and once by DNS name has two entries.
- A host can have several trusted keys, for example one per key type.
- Two steps that meet a new host at the same moment cannot store two different keys: the second one is refused before it sends any credential.
First contact still trusts whatever answers. For hosts that matter, an admin can pin the key before the first run (see below) and switch the organization to strict.
When a Host Key Changes
If a host presents a key that is not trusted, the step fails before any credential is sent. The error names the host, the trusted fingerprints and the key the host presented, for example:
The SSH host key of 192.0.2.10:22 does not match the key organization acme
trusts (SHA256:mxkz…). If the host's key was changed on purpose, an
organization admin checks the new key on the host, then forgets the old key
or pins the new one (/ssh-host-keys). The host presented ssh-ed25519
SHA256:Xk3v….The audit log records a run.host_key_rejected entry with the same details.
The run's Events tab shows the refusal too. When the step ends, it adds one entry for each host it refused, with that message and how many of the step's connections the host refused:
SSH host key check refused 192.0.2.10:22 (4 connections): The SSH host key
of 192.0.2.10:22 does not match the key organization acme trusts …The step's own error says it as well when the step reports failures per device. From step plugins 0.31.2, the collect-evidence step names each failed device with its reason, and lists devices that failed alike together: Failed on: r1, r2 (The SSH host key of 192.0.2.10:22 does not match …). The Events entry adds the host, the trusted and presented keys, and how many connections it refused, for any step that connects over SSH. A jump host that every device goes through is one entry, not one per device. Hosts refused in strict mode because no key is trusted for them get an entry too. A step gives its first 10 refused hosts an entry each, and names the rest in one more entry.
Accept the New Key
A key changes for good reasons too: the host was reinstalled, or its keys were rotated. Before you trust the new key, check it on the host itself, through a channel you trust (a console, or someone who can log in there):
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pubIf the fingerprint matches the one in the error, accept the new key. You can do it in three places:
- the refused host's entry on the run's Events tab (Accept New Key);
- Settings → SSH Host Keys, where Refused Keys lists the last key each host was refused for;
- a device's page, where the SSH Host Keys card shows the key the device or its jump host was refused for.
Accepting trusts the key the host presented to the refused step, in place of the keys trusted for that host and port. Nobody types or pastes a key, so accepting cannot bring in a key the host never presented. If the host was refused for another key since, nothing is accepted: run the step again, and accept the key it presents now. The key is listed with the source accepted and the name of the member who accepted it.
Operators may accept a key (ssh_host_key:accept), so the person who runs the flow does not need an admin. In strict mode, only a member who may also pin keys (an admin, by default) can accept one, because there only checked keys are trusted.
An admin can also forget the old key, so the next run stores the new one (accept_new), or pin the new key and then forget the old one (any mode).
Exempt a Host
Some hosts change their key all the time, for example a lab that is rebuilt for every demo. An admin can exempt such a host. Steps then accept whatever key it presents, in any mode but off. When its key has changed, the step stores the new key in place of the old one before it logs in, and the run's Events tab says so, with both fingerprints:
SSH host key of dind:2222 changed, and was accepted because the host is exempt
from the key check: now SHA256:AJ6N…, was SHA256:lY+F….To exempt a host, use Exempt on its row under Trusted Keys or Refused Keys, or Exempt Host on a refused host's run event. Exempt Hosts lists the exempt hosts. Remove checks the host again, against the key a step stored last. An exemption is for one host and port, like a key.
An exempt host has no protection: a machine posing as the host is accepted too. Use exemptions for lab and test hosts, never for production devices.
Managing Keys
Settings → SSH Host Keys lists the keys the organization trusts: the host and port, the key type and fingerprint, where the key came from, and when it was added. From there an admin can:
- pin a key checked on the host (paste the key line, see below);
- forget a key, after asking first. The page says what the next connection to the host then does in the current mode;
- accept the key a host was last refused for (operators too, see Accept the New Key);
- exempt a host, and remove an exemption (see Exempt a Host);
- choose the mode. Switching checks
offasks for a confirmation, and the page then shows that checks are off.
A device's page has an SSH Host Keys card. It shows the keys for the device and, when it has one, for its jump host, by the host and port the device uses. An admin can pin or forget a key for either hop there, for example right after a planned key change.
Viewers see the keys and the mode. Pinning, forgetting and changing the mode need the permissions in the table below; without them the buttons are locked.
The same is possible through the API, with the organization selected (X-Org-Id):
| Route | Permission | What it does |
|---|---|---|
GET /ssh-host-keys | ssh_host_key:view (viewer) | Lists the trusted keys. ?host= limits the list to one host. |
POST /ssh-host-keys | ssh_host_key:manage (admin) | Pins a key: {"host": "r1.example.net", "port": 22, "public_key": "ssh-ed25519 AAAA…", "comment": "checked on the console"}. |
DELETE /ssh-host-keys/{key_id} | ssh_host_key:manage (admin) | Forgets a key. |
GET /ssh-host-keys/refused | ssh_host_key:view (viewer) | Lists the last key each host was refused for. ?host= limits the list to one host. |
POST /ssh-host-keys/accept | ssh_host_key:accept (operator) | Accepts the key a host was last refused for: {"host": "r1.example.net", "port": 22, "fingerprint": "SHA256:…"}. |
GET /ssh-host-keys/exemptions | ssh_host_key:view (viewer) | Lists the exempt hosts. ?host= limits the list to one host. |
POST /ssh-host-keys/exemptions | ssh_host_key:policy (admin) | Exempts a host: {"host": "dind", "port": 2222, "comment": "lab bastion"}. |
DELETE /ssh-host-keys/exemptions/{exemption_id} | ssh_host_key:policy (admin) | Removes an exemption. |
GET /ssh-host-keys/policy | ssh_host_key:view (viewer) | Shows the mode. |
PUT /ssh-host-keys/policy | ssh_host_key:policy (admin) | Sets the mode: {"mode": "strict"}. |
public_key is the line of the host's /etc/ssh/ssh_host_*_key.pub file, or the last two fields of an ssh-keyscan line. The comment after the key is ignored. Accepted key types are ssh-ed25519, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521 and ssh-rsa.
Each key in the list shows its source: learned (stored on first contact, or by a step that met an exempt host's new key, with the run that stored it), pinned (added by an admin) or accepted (the key a host presented after a change, accepted by a member, with the run that was refused).
Audit
| Event | When |
|---|---|
entity.created (ssh_host_key) | A step stored a key on first contact (the actor is the run and step: username run:<run id>/<step id>, subject run:<run id>), or an admin pinned one. |
entity.deleted (ssh_host_key) | An admin forgot a key. |
entity.updated (ssh_host_key) | A member accepted a host's new key. The entry holds the keys it replaced. |
run.host_key_rejected (ssh_host_key, outcome denied) | A host presented a key that is not trusted. |
run.host_key_replaced (ssh_host_key) | A step stored the new key of an exempt host in place of its old keys. |
entity.created, entity.deleted (ssh_host_key_exemption) | An admin exempted a host, or removed the exemption. |
org.updated (organization) | An admin changed the mode. |
Algorithms in the Evidence
Each step that opens SSH sessions saves what every session agreed on, as a transport_meta artifact named ssh-algorithms on the run's Artifacts tab. This covers the same steps as the host key check: shell steps and network-CLI steps. The artifact has one entry per device and hop (device, or jump_host for its jump host):
| Field | Meaning |
|---|---|
kex | The key exchange, for example mlkem768x25519-sha256. |
post_quantum_kex | true when the key exchange includes a post-quantum algorithm (ML-KEM, Streamlined NTRU Prime or Classic McEliece). |
host_key | The host key's signature algorithm, for example ssh-ed25519. |
cipher, mac | The cipher and MAC from the worker to the device. mac is empty for a cipher with integrity built in. |
transport | The device transport: netmiko, asyncssh or scrapli, or shell for a shell step's session. |
legacy_algorithms | What the hop agreed on that its transport offers only with Allow legacy SSH algorithms, by kind (kex, host_key, cipher, mac); empty when nothing legacy was used. |
weak_host_key | true when the host presented a DSA key or an RSA key under 2048 bits, which only a hop that allows legacy SSH accepts; false for a strong key; null when the key could not be classified. |
legacy_allowed_by_inventory | true when the device (or, for a jump_host entry, the jump host) has the legacy setting in its access configuration. |
legacy_flag_unneeded | true when the inventory allows legacy SSH for the hop, but it agreed only on secure algorithms and presented a host key known to be strong. The setting can probably be removed. |
The artifact's own post_quantum_kex is true only when every entry's is. Its legacy_algorithms_used is true when any entry names a legacy algorithm, and the worker logs a warning for each such hop. Its legacy_flags_unneeded is true when any entry's legacy_flag_unneeded is, and the worker logs a warning for each such hop too. A small login key also needs the setting and is not part of this check. A transport that does not publish its algorithm lists never gets this mark, because its weak algorithms cannot be told from secure ones. A step keeps at most 1,000 entries. If it makes more, the artifact's truncated is true and its post_quantum_kex is false, because the entries left out are unknown.
The artifact is saved when the step succeeds, fails or is cancelled.
This matters because someone can record SSH traffic today and decrypt it once a large quantum computer exists ("harvest now, decrypt later"). A post-quantum key exchange protects recorded traffic against that. To move a device whose entries say false:
- Use the asyncssh or scrapli transport. Netmiko's SSH library (paramiko) has no post-quantum key exchange, so a device reached with netmiko always records
false, whatever the device supports. The asyncssh and scrapli transports, and shell steps, offermlkem768x25519-sha256first. - Give the device an SSH server that offers
mlkem768x25519-sha256, such as OpenSSH 9.9 or later. The worker cannot use Streamlined NTRU Prime (sntrup761x25519-sha512), the post-quantum key exchange of OpenSSH 9.0 to 9.8.
A netmiko, asyncssh or scrapli transport older than step plugins 0.29.0, or a transport from another plugin that does not report what it agreed on, adds no entries. A step whose sessions all add none saves no ssh-algorithms artifact.
Organizations and Exports
Trusted keys belong to one organization. A key one organization trusts means nothing to another, and the shared organization's keys are not shared.
Configuration exchange bundles do not carry trusted keys, refused keys or exempt hosts. Trust is something a deployment earns by connecting, or an admin grants after checking. Another deployment must do the same.
Limits
- Host certificates signed by an SSH certificate authority are not supported yet. Pin the host's key instead.
- Steps that run outside the platform's worker (for example inside a container step) make their own SSH connections and are not covered.