Dynamic Secrets Design Decisions

Design record for issued credentials: values a secret backend mints on demand (one-time SSH passwords, signed SSH certificates, X.509 certificates, expiring database users, TOTP codes) as opposed to the static key/value secrets Hegemony resolves today. Feature documentation for static secrets lives in docs/features/secrets.md; the bundled store is described in docs/features/openbao-internal.md.

Status note: design record written in September 2026 from a code review of hegemony, hegemony-secret-plugins and hegemony-step-plugins. Only the wave-0 items under "Delivery waves" are shipped. Follow-ups are tracked in TODO.md.

Problem

A secret backend is a static key/value store to Hegemony. Every value flows through one call shape: a secret() reference is parsed as scheme://path/key (split on the last slash), tenant-confined by validate_org_secret_path in packages/core/templates/resolver.py, read with backend.read(path), and collapsed to one string. The plugin contract (SecretBackend in hegemony-secret-plugins/packages/secret_sdk) matches that shape: read, write, delete, test, plus an optional list.

Vault and OpenBao mostly issue rather than store, and 1Password holds values that are not plain strings. None of it is reachable, for four structural reasons:

1. read(path) carries no parameters. A one-time SSH password needs the target address and username; a certificate needs a public key and principals.
2. The resolver returns one field. A certificate bundle is a certificate, a private key and a chain.
3. Nothing tracks a lifetime. The only lease-aware code is the Vault plugin's handling of its own authentication token.
4. Tenancy would break. validate_org_secret_path confines only paths that start with orgs/; engine paths such as ssh/sign/net-admin are global mounts, so a naive reference would let any organization mint from any role. The hole is latent today because the worker policy grants no engine paths.

Decisions Log

#	Decision	Rationale
1	Issuance is an optional protocol; SecretBackend itself never changes.	The host feature-detects list with hasattr today. Adding a method to SecretBackend breaks issubclass(VaultSecretsBackend, SecretBackend) in tests/core/test_secret_sdk_contract.py for every backend that exists.
2	No new register_backend_type keyword.	The registration signature is asserted equal between host and SDK, and packages/core/plugin_loader.py skips a whole plugin that raises TypeError on an unknown keyword. Capabilities are read from the backend instance.
3	SDK_ABI_VERSION stays 1.	It scopes the registration contract, which is untouched. Both directions keep working: a new host with an old plugin fails hasattr cleanly; an old host never calls the new methods.
4	Tenancy: the organization slug in the path first; credential profiles with the first parameterised kind.	A reference such as otp://orgs/SLUG/ROLE/IP/USER/key is confined by the guard that already protects every secret() call, with no host code. An org-scoped credential_profiles table (name, backend, kind, role, fixed and allowed parameters, maximum TTL) becomes the boundary when a kind needs parameters (SSH certificate signing) or a lease (database users). Both satisfy the invariant: no engine path reaches the backend without an organization-bound check, and that check lands before the worker policy is widened.
5	One-time passwords only where Hegemony dials once per step.	Every device transport (netmiko, asyncssh, scrapli wheels) opens a fresh SSH session inside each transport method and re-enters enable mode each time, so a one-time password is consumed by the first call. shell.execute opens one session per device per step and is the only consumer.
6	SSH CA certificates are the SSH story.	The bastion hop and the shell transport already load a private key; asyncssh takes (key, certificate) pairs and paramiko loads OpenSSH certificates. A certificate is reusable for its lifetime, so per-call re-login and the upgrade verify step's reconnect loop are harmless. No agent on the host, no network path from host to backend at login.
7	Short TTLs are the mechanism; revoke-on-step-exit is an optimisation, not a guarantee. No lease table, no sweeper.	The only deterministic per-step seam is the outer finally of execute_step in apps/worker/flow_activities.py; a crashed worker runs nothing, and the two run-end activities are best-effort and may land on another worker. Storing lease ids in Postgres and granting the API revoke rights is more blast radius than "a credential outlives its step by at most its TTL", which the backend enforces on its own.
8	Issued secret fields ride the existing redaction observer; short all-digit values are never registered.	set_secret_observer already feeds progress events, heartbeats, evidence artifacts and the live output sink. A 6-digit TOTP code clears the 4-character floor, and substring redaction would corrupt every counter in evidence to protect a value that is dead in 30 seconds.
9	Host-key trust is a prerequisite, split by library.	Every hop accepts any server key today. Certificates and one-time passwords prove the worker to the host; only host-key trust proves the host to the worker. asyncssh honours @cert-authority; paramiko 5.0 has no certificate-authority support, so the default netmiko wheel needs fingerprint pinning on both hops.
10	The API process never mints.	Git sync and file-repository credentials resolve in the API, which has no step lifecycle to revoke against. Minting paths belong to the worker's policy only; the API policy keeps key/value rights, so an API-side reference to an engine path fails closed.
11	Recorded as not reachable: one-time passwords for IOS-XE (including through TACACS+ and PAM), X.509v3 SSH certificates for IOS-XE, device-hop OpenSSH certificates for IOS-XE fleets.	See "Not reachable". Recording them prevents the same investigation and the same bug reports from recurring.

What the backends offer

Capability	Vault	OpenBao	1Password	Verdict for Hegemony
Key/value (static)	yes	yes	yes	shipped today
One-time SSH password, ssh/creds/:role	yes	yes	no	Linux hosts with vault-ssh-helper in PAM, via shell.execute only
SSH CA signing, ssh/sign/:role	yes	yes	no	bastions and Linux hosts; OpenSSH certificate format
SSH dynamic keys	removed in Vault 1.13	no	no	do not build
X.509 certificates, pki/issue/:role	yes	yes	no	client certificates for mTLS calls from containers; the only certificate form IOS-XE accepts for SSH, which no Python SSH library implements
Database users, database/creds/:role	yes	yes	no	the one leased kind; justifies revoke, a per-step ledger and a multi-field result
TOTP codes, totp/code/:name	yes	yes	one-time-password fields	30-second value; nominal second factor when the same worker identity holds the password
Transit (encrypt, sign)	yes	yes	no	later; a checksum plus an immutability guard on artifacts captures most of the value
AWS, Azure, GCP credentials	yes	not in the supported plugin set	no	Vault-only
Key/value version history, check-and-set, undelete	yes	yes	item history	read-side only; rollback of a bad rotation
SSH-key items, file attachments	no	no	yes	read-side enrichment

Where a credential can land

The host resolves every credential before a transport is built, so what matters is what each hop accepts and how often it authenticates.

Hop	Code	Authentication today	Dials per step	One-time password	OpenSSH certificate	Host-key check
Linux host via shell.execute	apps/worker/shell_transport.py	password or private key	1 per device	works	works	none today; CA or pinning possible
Bastion hop (three device wheels and shell)	hegemony-step-plugins/transports/*	password or private key	1 per transport call	no	works; one SDK field and three call sites	CA on asyncssh-based hops, pinning on netmiko
Device hop, netmiko (default, the only wheel with image staging)	transport_netmiko	password only	1 per transport call, plus enable each time	no	needs an SDK field; IOS-XE cannot consume it	fingerprint pinning only
Device hop, asyncssh or scrapli	opt-in wheels	password only	1 per transport call	no	needs an SDK field; OpenSSH-server devices only	CA or pinning possible
Git sync	apps/api/services/git_ops.py (API process)	token or private key	1 per operation	not applicable	small, but forces mint rights onto the API	accept-new only
container.run environment values	template-resolved env	any string	not applicable	as a string	as PEM strings	not applicable
probe.http, monitors, image staging	out-of-tree handlers	none, or a proxy literal	not applicable	no field	no field	not applicable

Use cases and verdicts

Twenty-five use cases from three personas were scored by a security lens (does it measurably reduce standing credentials, blast radius or man-in-the-middle exposure for Hegemony's real targets) and a delivery lens (effort, risk and sequencing across the three repositories). Scores are 1 to 5.

Use case	Security	Delivery	Verdict
SSH CA user certificates for shell.execute on a Linux fleet	5	5	must-have
Make the SSH private-key field work for shell.execute	4	5	must-have, shipped in wave 0
Short-lived bastion certificates from the SSH CA	5	4	must-have
Leaked worker AppRole: blast radius and an issuing-identity split	5	4	must-have; the hardening half shipped with the OpenBao overlay
Per-organization isolation of issuing roles	5	4	must-have before any policy widening
Host-key trust on every hop	5	3	must-have
Per-run database user for a container.run migration	4	4	worth it
Revocation on cancel or worker death	4	3	worth it, with the database kind
Audit trail of mints on both sides	4	3	worth it, as run events
One-time SSH passwords for shell.execute	3	4	worth it; the cheapest end-to-end proof of issuance
Key/value versions, check-and-set, rollback	3	4	worth it
1Password SSH-key items and one-time-password fields read properly	2	4	worth it
Credential TTL clamped to the step timeout	3	3	nice to have
X.509 client certificate for an mTLS call from a container	3	3	nice to have
Response wrapping for containers	3	2	not now
TOTP inside container.run	2	2	not now; the code is often dead by the time the image is pulled
Short-lived git SSH certificates	2	2	not now; forces mint rights onto the API
Transit-signed evidence artifacts	2	2	not now
One-time passwords for IOS-XE through TACACS+ and PAM	1	1	never
TOTP as a bastion second factor	1	1	never
X.509v3 SSH user certificates for IOS-XE	1	1	never

For Cisco IOS-XE devices no item removes a standing credential. The security payoff lands on bastions, Linux hosts, database containers and the worker's own backend identity.

Delivery waves

Each wave is independently shippable. Every change to hegemony-step-plugins is a lockstep release of all its packages, so SDK fields are batched.

1. Wave 0, hygiene and fixes (shipped with this record). Config-resolved secrets now reach the redaction registry: _run_blocking_resolution in apps/worker/template_resolver.py copies the caller's context onto the pool thread. Git-auth secret references are confined to the repository's owning organization in apps/api/services/git_ops.py, mirroring the file-repository resolver. access_config.ssh.private_key_ref is honoured by the shell transport, with a warning when a network-CLI device carries one. Already landed upstream before this record: plugin pin repair, the bundled store moved to OpenBao over TLS with an audit device and hardened AppRoles. Recorded as follow-ups: the never-register rule for short all-digit values, and the authentication failure that the Cisco install driver records as success.
2. Wave 1, one secret-plugins release, no ABI change. A vault_ssh_otp (and openbao_ssh_otp) backend type whose read() mints from ssh/creds/:role with the organization slug in the path, referenced from the device's password ref and consumed by shell.execute; the worker policy gains ssh/creds/+; a versioned-backend protocol; 1Password SSH-key items and one-time-password codes (never the seed).
3. Wave 2a, SSH CA on the shell path. An ephemeral key signed by ssh/sign/:role in open_shell_transport, presented as a (key, certificate) pair, minted off the event loop so the output sink keeps flushing. The first parameterised kind, so the tenancy mechanism is chosen here.
4. Wave 2b, step-plugins release. JumpHostSpec.certificate plus host-key trust fields on both specs, all defaulted and hidden from repr; the wheel call sites; then host pins and _ACCESS_CONFIG_PATHS entries with a literal carve-out.
5. Wave 3, the leased kind. Database credentials through an issue-style call, a per-step ledger captured at resolver construction, revoke under asyncio.shield in the outer finally of execute_step, a check that the backend token outlives the requested lease, and run events for mints and revocations. X.509 client certificates as the second kind. Credential profiles and their UI if wave 2a chose the path grammar.

Not reachable, and why

• One-time SSH passwords for IOS-XE, including through TACACS+ and PAM. The chain exists (tac_plus with PAM login, FreeRADIUS with rlm_pam, vault-ssh-helper from pam_exec.so), but every device transport re-authenticates on each command, so the second call presents a dead password; the helper matches the address the backend returns against its own interfaces or allowed_cidr_list, so on an AAA server the binding relaxes from one host to one subnet; the local fallback account and the enable secret stay static because the AAA method list needs local; and only cleartext-to-PAM methods (TACACS+ ASCII, RADIUS PAP) work.
• X.509v3 SSH user certificates for IOS-XE. IOS-XE implements RFC 6187 (x509v3-ssh-rsa); no Python SSH library does, and there is no RESTCONF or NETCONF handler to hand a client certificate to.
• Device-hop OpenSSH certificates. A coordinated three-repository change that excludes the largest device family. Possible for OpenSSH-server devices; revisit when a non-Cisco fleet needs it.
• TOTP as a second factor. Both factors become reachable by the same worker identity, so it is multi-factor in name only; a CA-signed principal with AuthenticationMethods publickey is the correct exemption.

Open decisions

1. Tenancy mechanism for parameterised kinds: path grammar (no host code, no catalogue) or the credential-profile table (catalogue, per-org TTL caps, a test button, five test-enforced registries). Recommendation: path grammar for waves 1 and 2a, profiles in wave 3.
2. Where a "can I mint from this role" test runs: the API policy has no engine paths by design, so an API-side test either widens it or dispatches through the worker as the notification test does.
3. Whether a consumer run may mint from a shared-organization profile (governed today by HEGEMONY_SHARED_ORG_SECRET_SCOPE for key/value paths).
4. Whether the device API should validate access_config at all; the protected-template rule exists only in the inventory-provider pipeline.

Sources

External facts this record relies on:

• OpenBao SSH secrets engine and API: https://openbao.org/docs/secrets/ssh/ and https://openbao.org/api-docs/secret/ssh/
• OpenBao supported plugins: https://openbao.org/community/policies/plugins/
• Vault leases: https://developer.hashicorp.com/vault/docs/concepts/lease
• vault-ssh-helper and its address matching: https://github.com/hashicorp/vault-ssh-helper
• IOS-XE X.509v3 SSH authentication: https://www.cisco.com/c/en/us/support/docs/security-vpn/secure-shell-ssh/223290-configuring-certificate-authentication.html
• paramiko certificate loading: https://docs.paramiko.org/en/stable/api/keys.html
• asyncssh client keys and certificates: https://asyncssh.readthedocs.io/en/latest/api.html
• 1Password secret references: https://developer.1password.com/docs/cli/secret-references/
• tac_plus PAM login: https://manpages.ubuntu.com/manpages/trusty/man5/tac_plus.conf.5.html