Secrets Management
Hegemony provides secure secrets management with pluggable backends. This document explains how secrets work, reference formats, and deployment options.
Secret and backend names follow the platform's object name rules. They are names, not reference syntax — see Reference Formats for how a secret is addressed from a flow.
Key Principles
Secret values are never stored in the database - Only metadata (name, description, backend, path) is stored in PostgreSQL.
Secret values are stored in configured backends - The actual secret data lives in the configured backend (e.g. OpenBao or HashiCorp Vault KV v2).
Secrets are resolved just-in-time - Workers fetch secret values right before they're needed, never caching or logging them.
References replace values in configs - Flow definitions and step parameters use Jinja template syntax like
{{ secret('vault://path/key') }}.The platform's own credentials stay the platform's -
{{ env() }}and{{ file() }}read the API or worker process's environment and secrets directory, so they work only in platform configuration. See Platform-Only References.
Reference Formats
Secret references use {{ }} Jinja template syntax.
Dynamic Backend References
{{ secret('vault://orgs/default/secrets/db/password') }}
{{ secret('vault_external://shared/api-keys/stripe/secret_key') }}Components:
backend_scheme: Reference scheme configured on the backend (for examplevault,vault_external)path: Path to the secret in the backendkey: Specific key within the secret data
Platform-Only References (env and file)
{{ env('VAR_NAME') }}
{{ env('VAR_NAME', 'fallback') }}
{{ file('FILENAME') }}env() reads an environment variable of the API or worker process, and file() reads a file in the secrets directory (HEGEMONY_SECRETS_DIR, default /run/secrets). Both hold platform credentials, such as the internal API token and the OpenBao login, so they work only in configuration that platform admins manage and the platform resolves itself:
- secret backend settings (for example a backend's bootstrap
token) - inventory provider settings (the provider
token_ref)
A fallback value is for settings that are not credentials, such as a backend's address. A secret backend's credential fields (token, role_id, secret_id, password and the fields a backend type marks as secret references) refuse {{ env('VAR', 'value') }} with a value: the fallback would store the credential itself in the backend settings. Use {{ env('VAR') }} and set the variable, or file().
Everything an organization writes or runs refuses them: flows, variables, devices, notification destinations and templates, git and file repositories, schedules and webhooks. Saving such content returns 422 naming the fields, imports report an item error, and a run fails with:
env() is only available in platform configuration (secret backend and inventory
provider settings). Store the value as a secret and use {{ secret('backend://path/key') }}.The run-time refusal is the real guard, so a name built at run time (for example {{ (env)(vars.NAME) }}) is refused as well.
Upgrading content that used env() or file()
Content saved before this rule keeps loading but fails when used. Move each value into a secret and reference it with {{ secret('scheme://path/key') }}; values that are not secret can be global variables ({{ vars.NAME }}) or plain text. Places to check:
- device
access_configrefs (SSH, enable, jump host) - notification destination fields (SMTP, Shoutrrr URL, webhook tokens) and notification templates
- git repository
auth_secret_ref/ssh_key_secret_ref - file repository credential refs
- variables, flow step config and params, schedule and webhook inputs
A flow's Validate check lists every field that still uses them.
Plaintext values saved earlier
Credential fields accept references only:
| Where | Fields | Accepted |
|---|---|---|
| Git repositories | auth_secret_ref, ssh_key_secret_ref | {{ secret('…') }} |
| Notification destinations | smtp_password_ref, url_secret, auth_token_ref, auth_password_ref, auth_header_value_ref, signing.secret_ref, credential-like webhook headers (Authorization, Cookie, X-Api-Key and names containing token, secret, password, auth or api-key) | {{ secret('…') }} or {{ vars.NAME }}; url_secret and headers may also surround them with text |
| Secret backends | credential fields (see above) | {{ env('VAR') }}, {{ file('NAME') }}, {{ secret('…') }} |
Each template block must be the bare reference and nothing more: a filter, a fallback or any other expression inside the braces (for example {{ vars.TOKEN | default('…') }}) could hold the value itself, so it is refused too. The API refuses anything else (422 for git repositories, 400 for notification destinations and secret backends), and an import reports the item as an error.
A value saved before this rule keeps working when it is used, but Hegemony no longer shows it: API responses, exports and audit snapshots leave git credentials out (the *_set flag stays true) and show other fields as ***REDACTED***, and the API logs a warning naming the object and field. Saving the object again with the redacted value is refused, so replace it with a reference. Treat such a value as exposed: store it as a secret, reference it, and rotate the credential, since earlier responses and exports returned it as stored.
The worker's HEGEMONY_SMTP_* email settings (port, TLS, STARTTLS and sender) are gone as well: they filled empty email fields from the worker's environment. Set the sender (from) on every email destination; an empty port, TLS or STARTTLS field now takes a fixed default (see Notifications).
Using Secrets in Flows
Secret references can be used anywhere in step config or params. They are resolved just-in-time by the worker right before handler execution.
How It Works
Store references, not values: Flow definitions store secret references (e.g.,
{{ secret('backend://path/key') }}) in the database.Just-in-time resolution: When a step executes, the worker's
execute_stepactivity resolves all secret references inconfigandparamsimmediately before calling the handler.No secrets in history: Resolved values exist only in worker memory. They never appear in Temporal workflow history or database.
Redaction for logging: The
redact_for_logging()helper creates safe-to-log versions of data that may contain secrets.Redaction of step output: A secret a step used is replaced by
[redacted]in everything the step leaves behind, including the output later steps read (see below).
Secret References in Templates
Use the secret() helper in Jinja templates:
nodes:
- id: api_call
type: step
name: "API Call"
handler: container.run
params:
image: "curlimages/curl:latest"
command: 'curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/refresh'
env:
API_TOKEN: "{{ secret('vault_internal://orgs/default/secrets/api/token') }}"Secret fields that use step values
Some handlers keep their secret fields as written and resolve each reference only when they need the value, so it is staged as a file instead of passing through the step's settings: container.run and container.buildsecrets, ansible.playbook secrets, group_secrets, vault_password and vault_ids, tf.plan and tf.apply secret_vars, secret_env and state_encryption_key. Handlers from later step-plugins releases that keep secret fields this way follow the same rules.
A reference in one of these fields can use the step's own values (inputs, step outputs, run.vars, loop, targets, device, run metadata) to choose the secret, written as a secret() call whose path joins quoted text, vars and step values with ~:
secrets:
clm_key_b64: "{{ secret('vault://orgs/acme/secrets/clm/certs/' ~ inputs.cert_name ~ '-current/key_b64') }}"The reference is rendered once, with the same values the step's other settings rendered with. The rules:
- Each step value fills part of one path segment. It must be text or a number made of letters, digits,
.,_and-, start with a letter, a digit or_, and be at most 128 characters. A value can therefore never add a segment or form.or... A missing value fails the step, also when filters follow it (| first | pprint); a fallback (| default(...)) is refused. - The backend and the namespace come from the text you write. The text before the first step value must start with
<backend>://orgs/<this run's organization>/, so such a reference reaches only the run organization's own secrets. The designated shared organization and the legacy paths outsideorgs/stay reachable through references written out in full. - Step values only build paths. The field takes
secret()calls,varsand plain text; a step value outside asecret()path, a{% %}statement, a nestedsecret()call or+instead of~is refused. A variable whose value is itself a template is rendered without the step's values and cannot callsecret(), so it cannot bring an unchecked value into the path. - Resolved values are data. A value that contains
{{ }}or looks like a reference is returned as it is; nothing is rendered or resolved a second time. Like everysecret()value, it is redacted from the step's output.
A reference with {{ }} inside its own text (vault://orgs/acme/secrets/{{ inputs.name }}/key) is not supported and is refused with the form above. A reference that uses no step value resolves as it always did. When a reference built from step values breaks these rules, the step fails with the reason, for example inputs.cert_name cannot be part of a secret path: it may hold only letters, digits, '.', '_' and '-'. When the secret itself cannot be read (it does not exist, or the backend fails), the step's error says so without the path, like other resolution failures, and the worker log has the detail. The log writes the path with each step value by name (orgs/acme/secrets/clm/certs/{inputs.cert_name}-current), never by value, and redacts the step's known secrets, since a step value can itself come from a secret.
These rules protect the handler's own secret fields. A secret() reference in the step's other settings (such as env above) is rendered by the platform before the handler runs, and keeps its existing rules.
Choosing the secret from a value does not decide whether this run may use that secret: within the organization, any name that passes the rules is resolved. A flow that serves more than one subject should take the value from a source it trusts (a record the run fetched and checked) rather than straight from a launch form, and say so in its own checks.
Redaction of step output
While a step runs, the worker keeps a list of the secret values it resolved for that step: the device, enable and jump-host passwords and keys it logs in with, every secret() value in the step's configuration, and the token a step with managed Terraform state uses. Wherever one of these values appears in what the step leaves behind, it is replaced by [redacted]:
- the live output on the run's Logs tab,
- evidence artifacts,
- the step's result: its summary, error, metrics and output,
- the worker's own log, including what the device libraries log while the step runs (scrapli and asyncssh write each command they send at INFO), also from the threads they hand blocking work to (netmiko), and a value the log shows escaped, as repr() or JSON quote it.
The step's result is what later steps, branch conditions and approval templates read as steps.NODE_ID.output, so a secret never passes from one step to the next. A Terraform output that embeds a password, or a command that prints a token, reaches the next step as [redacted]. A step that needs a secret takes it with its own secret() reference.
When redaction changed the result, the step's output shows one line naming the fields it changed (never the values), for example Secret values this step used were replaced by [redacted] in its result: output.stdout.
Limits to keep in mind:
- Only values the platform resolved are covered. A password typed straight into a command is ordinary text to the platform.
- Values shorter than 4 characters are never redacted: replacing them would garble ordinary output. The worker logs a warning when it meets one.
- A value that spans lines (a private key, a JSON credentials file) is also replaced line by line, each line without its surrounding spaces, so it stays hidden when output holds only some of its lines (a step that cuts long output at a line end) or holds it reformatted (other line endings or indentation). Lines shorter than 8 characters (
{,},, a key's short last line) are not replaced on their own. - Usernames are not treated as secrets, even when they are stored as
secret()references: they appear in prompts and command output and stay as they are. - Redaction replaces text wherever it matches. A secret value that is also an ordinary word or number (
true,8080) is replaced in every place it appears in the output, so use values that do not occur in normal output.
Secrets written by steps
A step can also store a secret: the secret.write step (from the hegemony-steps-secret wheel, installed with the platform) copies files a container step wrote, or values from other secrets, into the organization's secret store, swaps or deletes them. A certificate pack uses it to keep a new private key without the key ever reaching a run variable, an artifact or a log. Only handlers that declare it (writes_secrets in the step SDK) can use these services, and only where the organization allows it.
Off until an organization admin turns it on. On the Secrets page (or PUT /api/secrets/step-writes), an admin of the organization switches secret writes by steps on and names the path prefixes steps may use, for example clm/certs and clm/acme. A step may then read, write and delete secrets at a prefix or below it (clm/certs/web-current), and nothing else: not clm/certs-old, not clm. Such a step can replace or destroy any secret under those prefixes, so keep them to folders that only flows manage. The designated shared organization can never turn it on: every organization reads its secrets.
Where a written secret lives. A path is relative to the organization's own root and the platform picks the place, never the flow: clm/certs/web is stored at orgs/<organization>/secrets/clm/certs/web on the default secrets backend, and a later step reads one key of it as {{ secret('vault://orgs/<organization>/secrets/clm/certs/web/key_b64') }}. GET /api/secrets/step-writes shows that root (reference_root). A path has at most three segments of letters, digits, _, . and - (200 characters in all); a key starts with a letter or _ and uses letters, digits, _, . and -. A secret holds at most 100 keys, 64 KiB per value and 256 KiB in all; for a merge, the keys already stored count towards the 100 keys and 256 KiB.
What the platform guarantees for each call:
- Only a running step. The call names its step run; the platform refuses a step that is not running, a run that has ended and a handler that does not declare
writes_secrets. - Never a read-only backend for writes and deletes (reads are allowed), and no write without a default backend.
- Redacted from the start. The worker adds every value a step writes to the step's redaction list before it sends it, and every value a step reads before handing it over, so neither appears in the step's output, artifacts or logs.
- No old value left in the store's history. A write that replaces a secret, and a delete, leave none of the earlier values readable through the backend's version history. A backend that can destroy earlier versions after a write does so; otherwise (the Vault/OpenBao KV v2 backend today) the path is deleted, which destroys every version, and then written. In that case the secret is absent for a moment, and when the write fails after the delete the platform puts the previous values back where it can and says so in the step's error.
- One call on a path at a time. Reads, writes and deletes of one path wait for each other, and for changes to that path on the Secrets page, also across API processes. No other write lands between a write's delete and write, and a step never reads the path in that moment.
- Listed and audited. A written secret gets a row on the Secrets page (named after its path, with
Written by a flow stepas description), and a delete removes it. Every read, write and delete is in the audit log asentity.read,entity.created/entity.updatedorentity.deletedon the secret, withrun:<run id>/<step id>as the actor and the path and key names in the details, never a value.
A merge (merge: true) reads the secret, adds the keys and writes it back; as calls on one path wait for each other, two steps merging into the same path both keep their keys. A call that fails may still have been applied (the backend stored the value and then timed out), so a step that needs certainty reads the secret back.
API Endpoints
Secret metadata responses
Secret metadata responses expose backend_scheme, which is the canonical reference prefix used in secret refs.
When generating references, always use Jinja syntax:
{{ secret('<backend_scheme>://<path>/<key>') }}List Secrets (metadata only)
GET /api/secretsReturns secret metadata without values.
Create Secret
POST /api/secrets
Content-Type: application/json
{
"name": "database-password",
"description": "Production database password",
"backend_id": "uuid-of-backend",
"folder": "orgs/default/secrets/database-password",
"values": {
"password": "super-secret-value"
}
}The values map is stored in the configured backend; only metadata is saved to the database.
Discover and Claim Existing Backend Secrets
GET /api/secrets/discoveredLists paths that exist in enabled, listable backends but that no Hegemony secret manages. Only names are listed; values are never read. Paths under another organization's orgs/<slug>/ prefix are hidden, and so are paths that any organization already manages.
To register one of these paths as a managed secret, claim it:
POST /api/secrets
Content-Type: application/json
{
"name": "legacy-radius",
"backend_id": "uuid-of-backend",
"folder": "shared/legacy-radius",
"claim_existing": true
}A claim writes nothing to the backend, so it also works on read-only backends. It returns 404 when the path holds no data.
Each backend path is managed by at most one secret across all organizations. Creating or claiming a secret at a path that another secret already manages, in any organization, returns 400.
Where steps may write secrets
GET /api/secrets/step-writes
PUT /api/secrets/step-writes
Content-Type: application/json
{
"enabled": true,
"prefixes": ["clm/certs", "clm/acme"]
}Reads the active organization's setting for secrets written by steps, or replaces it (organization admins, action secret:step_writes). Prefixes are normalized (surrounding slashes dropped, duplicates removed); at most 20. The answer adds root (where the paths live in the default backend), reference_root (how a later step references a written secret, or null without a default backend) and shared_organization. A change is audited as org.updated with the old and new setting.
Update/Rotate Secret
PATCH /api/secrets/{id}
Content-Type: application/json
{
"values": {
"password": "new-rotated-value"
}
}Updates last_rotated_at and merges the provided keys in the backend.
Every call is written to the audit log as entity.updated on the secret, naming the keys written and the keys deleted. A request that changed nothing is recorded too, carrying no_changes: true, because reaching for a secret is worth a trace whether or not it altered anything. Values never appear in the entry.
A secret's page and a secret backend's page each carry a History card listing the most recent audit log entries about that object: how long ago, the action, the fields that changed, and who. A row opens the same entry dialog the audit log uses, and the card links into the log filtered to the object. Values are never shown; only that they were written. The card appears only for callers who may read the audit log.
Import Secrets with Configuration Exchange
Configuration Exchange has a secrets section in the single_yaml layout. An export lists each secret's backend, folder and key names, never the values. An import can also carry values, which is useful for instance bootstrap data and controlled migrations where operators intentionally provide secret values inbound:
schema_version: 3
secrets:
- name: database-password
description: Production database password
backend_ref: vault_internal
folder: orgs/default/secrets/database-password
values:
username: app_user
password: super-secret-valueAn exported secret looks like this. Importing it creates the listed keys with empty values where the backend does not have them yet, so you fill them in afterwards; keys that already exist keep their values:
secrets:
- name: database-password
backend_ref: vault_internal
folder: orgs/default/secrets/database-password
claimed: false
keys: [password, username]The backend_ref must name a backend that exists on the importing instance. If it does not, the preview lists the secret as an error with the reason, and the import refuses it.
Secret values are written to the selected/default backend and are never returned by Configuration Exchange exports, plan/apply summaries, audit diffs, or normal secret metadata responses. Exports may still include safe secret references such as {{ secret('vault://orgs/default/secrets/database-password/password') }} in credentials, notification destinations, or flow configuration.
A secret's backend is part of what it is: an import never moves a secret to another backend. When an imported secret already exists, it stays in its backend, and the folder is kept unless the payload sets another one. A backend_ref naming a different backend than the existing secret's is an error, because secret names are unique: rename the secret in the bundle to create a new one in that backend. Provided values are merged into the backend object: matching keys are overwritten, omitted keys are preserved, and last_rotated_at advances only when the backend write changes stored data.
Delete Secret
DELETE /api/secrets/{id}Removes from both database and the configured backend. Deleting a claimed secret removes only its metadata; the backend data stays.
Runtime Cache Behavior
Hegemony caches backend clients/configuration, not secret values:
- API process caches backend clients for metadata operations.
- Cache entries are invalidated when a backend is created, updated, or deleted.
- Cache entries are also invalidated automatically when backend config fingerprint changes.
- Worker process caches backend configuration/resolver instances with a short TTL (currently 60s) to pick up backend changes without restart.
Secret values remain just-in-time resolved and are never persisted in these caches.
Propagation Timing (What to Expect)
When backend configuration changes:
- API metadata operations reflect changes immediately after backend create/update/delete because the API backend client cache is explicitly invalidated.
- Worker runtime resolution converges within the worker cache TTL window (currently up to ~60 seconds) without worker restart.
For emergency rollout (for example broken credentials), restart worker processes to force immediate cache refresh.
Operational Runbook
Rotate backend credentials (AppRole / token)
The bundled backend rotates its own AppRole secret IDs and revokes the superseded ones; this runbook is for backends you manage yourself.
- Rotate credentials in the backend first.
- Update backend config in Settings → Secret Backends.
- Validate with a known secret reference in a non-production run.
- Confirm worker logs show successful secret resolution after change.
- If resolution errors persist beyond the TTL window, restart workers and re-test.
Backend outage response
If backend is unreachable or auth fails:
- Check worker logs for resolution failures (scheme/source context is logged).
- Validate backend health/auth externally (
bao status/vault status, then a login). - Pause new runs or route critical flows away from affected references.
- Restore backend availability/auth and run a smoke flow.
- Resume paused runs.
Internal OpenBao Backend (Automatic, Overlay-Driven)
Hegemony ships with an optional bundled secret store provided by the openbao-internal compose overlay. It runs OpenBao, the Linux Foundation fork of HashiCorp Vault. When enabled, the API auto-provisions the vault_internal backend entry at startup using AppRole credentials generated by openbao-init.
The tag and the reference scheme are deliberately unchanged: references stay {{ secret('vault://...') }}, because that is the address every stored reference already resolves through. OpenBao keeps Vault's HTTP API, so the same plugin and client serve both. See Internal OpenBao Lifecycle.
Default semantics are bootstrap-only:
- The bundled OpenBao becomes the default only when its overlay is enabled and no default backend exists yet.
- If an admin later sets an external backend as default, restarts do not override that choice.
Operational toggles:
HEGEMONY_OPENBAO_INTERNAL_ENABLED(defaultfalse; theopenbao-internalcompose overlay sets it totrue) — enable/disable bundled backend provisioning. The former nameHEGEMONY_VAULT_INTERNAL_ENABLEDis still read, so an existing.envneeds no change.HEGEMONY_S3_INTERNAL_ENABLED(defaulttrue) — enable/disable bundled object store provisioning (bucket bootstrap and the managed repository).
You can still use the Settings UI to add additional backends (external Vault, cloud secret managers, etc.) and choose which backend is default.
Deployment Options
Option 1: Bundled OpenBao (opt-in via the openbao-internal overlay)
Available as a compose overlay and auto-bootstraps when enabled. It generates its own auto-unseal key and TLS certificate on first boot, then:
- Initializes OpenBao and stores the recovery keys
- Enables KV v2 at the
hegemonymount - Creates AppRole roles for the API and worker
- Writes AppRole credentials to an internal secrets volume, and rotates them
- Revokes the initial root token
No additional configuration is required. See Internal OpenBao Lifecycle for the full lifecycle, including what is already hardened and what is left to the operator.
Option 2: An external OpenBao or Vault (optional)
Configure your existing server (bao for OpenBao, vault for Vault — the commands are otherwise identical):
Enable KV v2:
bashbao secrets enable -path=hegemony kv-v2Create policies (see
deploy/openbao/policies/*.hcl)Enable AppRole and create roles:
bashbao auth enable approle bao write auth/approle/role/hegemony-api \ token_policies=hegemony-api \ token_ttl=20m \ token_max_ttl=1h \ secret_id_ttl=24hConfigure Hegemony via Settings → Secret Backends to add the backend and set it as default if desired. The page is open to every signed-in user, like its tile on the Settings hub. Adding, editing, testing and deleting a backend need the platform-wide
adminrole: secret backends are platform-scoped, so an organization's admin role does not count, and those controls appear locked (with a tooltip saying so) for everyone else. Prefer the*_filecredential fields over inline values if you rotate secret IDs: they are re-read on every authentication.
The address must be https://. A plain http:// address is refused, because the AppRole secret ID and every secret value would cross the network in cleartext. If you genuinely need it — a server -dev instance, or a network you already trust to carry that traffic — set allow_insecure_http: true in the backend config; the opt-in is logged on every backend build. This is enforced from hegemony-secret-vault 0.2.0, so a backend left on an http:// address stops working when you upgrade until you fix the address or set the flag.
Redirects are never followed, because the token travels in a header that HTTP clients replay across hosts. A Vault or OpenBao standby node with request forwarding disabled answers with a redirect; point the backend at the active node.
Security Notes
What's Protected
- ✅ Secret values never appear in database
- ✅ Secret values never appear in Temporal workflow history
- ✅ Secret values never logged (redacted automatically)
- ✅ API responses never include secret values
- ✅ Backend tokens have limited TTL and scoped permissions (when supported)
Best Practices
- Use separate credentials for Worker — Worker requires read-only access
- Rotate secrets via API - Never access backends directly
- Monitor
last_rotated_at- Track secret freshness - Use TLS in production - Never disable
verifyfor external backends, and never setallow_insecure_http - Limit path_prefix scope - Use org-based paths for isolation
Troubleshooting
Quick diagnosis matrix
| Symptom | Likely cause | First action |
|---|---|---|
| Secret ref resolves to empty | Missing key/path or wrong reference | Verify <scheme>://path/key and backend data |
| Secret resolution fails with auth errors | Expired/invalid AppRole or token | Rotate credentials and update backend config |
| API sees backend change, worker still uses old config | Worker cache TTL not elapsed yet | Wait up to ~60s or restart workers |
Notifications fail for url_secret refs | Invalid secret ref or missing URL secret | Resolve the reference manually and verify destination secret |
"Secrets backend not configured"
Check that backends are configured in Settings (navigate to Settings in the sidebar). At least one backend must be enabled and set as default.
"Authentication failed" (OpenBao / Vault)
Verify AppRole credentials are correct and haven't expired:
# Check role exists
bao read auth/approle/role/hegemony-api
# Generate new secret_id if expired
bao write -f auth/approle/role/hegemony-api/secret-idFor the bundled backend, docker compose ... restart openbao-init re-runs the bootstrap and re-mints both roles' credentials. Note that its secret IDs also carry a CIDR binding: if the OpenBao network's subnet changed, authentication fails even with correct credentials.
"Secret not found"
Verify the path in your reference matches the backend path. For the bundled backend (use vault instead of bao against a HashiCorp Vault):
# List secrets
bao kv list hegemony/orgs/default/secrets
# Read specific secret
bao kv get hegemony/orgs/default/secrets/db