Secrets Management

Hegemony provides secure secrets management with pluggable backends. This document explains how secrets work, reference formats, and deployment options.

Secret and backend names follow the platform's object name rules. They are names, not reference syntax — see Reference Formats for how a secret is addressed from a flow.

Key Principles

1. Secret values are never stored in the database - Only metadata (name, description, backend, path) is stored in PostgreSQL.

2. Secret values are stored in configured backends - The actual secret data lives in the configured backend (e.g. OpenBao or HashiCorp Vault KV v2).

3. Secrets are resolved just-in-time - Workers fetch secret values right before they're needed, never caching or logging them.

4. References replace values in configs - Flow definitions and step parameters use Jinja template syntax like {{ secret('vault://path/key') }}.

5. The platform's own credentials stay the platform's - {{ env() }} and {{ file() }} read the API or worker process's environment and secrets directory, so they work only in platform configuration. See Platform-Only References.

Reference Formats

Secret references use {{ }} Jinja template syntax.

Dynamic Backend References

{{ secret('vault://orgs/default/secrets/db/password') }}
{{ secret('vault_external://shared/api-keys/stripe/secret_key') }}

Components:

• backend_scheme: Reference scheme configured on the backend (for example vault, vault_external)
• path: Path to the secret in the backend
• key: Specific key within the secret data

Platform-Only References (env and file)

{{ env('VAR_NAME') }}
{{ env('VAR_NAME', 'fallback') }}
{{ file('FILENAME') }}

env() reads an environment variable of the API or worker process, and file() reads a file in the secrets directory (HEGEMONY_SECRETS_DIR, default /run/secrets). Both hold platform credentials, such as the internal API token and the OpenBao login, so they work only in configuration that platform admins manage and the platform resolves itself:

• secret backend settings (for example a backend's bootstrap token)
• inventory provider settings (the provider token_ref)

Everything an organization writes or runs refuses them: flows, variables, devices, notification destinations and templates, git and file repositories, schedules and webhooks. Saving such content returns 422 naming the fields, imports report an item error, and a run fails with:

env() is only available in platform configuration (secret backend and inventory
provider settings). Store the value as a secret and use {{ secret('backend://path/key') }}.

The run-time refusal is the real guard, so a name built at run time (for example {{ (env)(vars.NAME) }}) is refused as well.

Upgrading content that used env() or file()

Content saved before this rule keeps loading but fails when used. Move each value into a secret and reference it with {{ secret('scheme://path/key') }}; values that are not secret can be global variables ({{ vars.NAME }}) or plain text. Places to check:

• device access_config refs (SSH, enable, jump host)
• notification destination fields (SMTP, Shoutrrr URL, webhook tokens) and notification templates
• git repository auth_secret_ref / ssh_key_secret_ref
• file repository credential refs
• variables, flow step config and params, schedule and webhook inputs

A flow's Validate check lists every field that still uses them.

The worker's HEGEMONY_SMTP_* email settings (port, TLS, STARTTLS and sender) are gone as well: they filled empty email fields from the worker's environment. Set the sender (from) on every email destination; an empty port, TLS or STARTTLS field now takes a fixed default (see Notifications).

Using Secrets in Flows

Secret references can be used anywhere in step config or params. They are resolved just-in-time by the worker right before handler execution.

How It Works

1. Store references, not values: Flow definitions store secret references (e.g., {{ secret('backend://path/key') }}) in the database.

2. Just-in-time resolution: When a step executes, the worker's execute_step activity resolves all secret references in config and params immediately before calling the handler.

3. No secrets in history: Resolved values exist only in worker memory. They never appear in Temporal workflow history or database.

4. Redaction for logging: The redact_for_logging() helper creates safe-to-log versions of data that may contain secrets.

5. Redaction of step output: A secret a step used is replaced by [redacted] in everything the step leaves behind, including the output later steps read (see below).

Secret References in Templates

Use the secret() helper in Jinja templates:

nodes:
  - id: api_call
    type: step
    name: "API Call"
    handler: container.run
    params:
      image: "curlimages/curl:latest"
      command: 'curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/refresh'
      env:
        API_TOKEN: "{{ secret('vault_internal://orgs/default/secrets/api/token') }}"

Redaction of step output

While a step runs, the worker keeps a list of the secret values it resolved for that step: the device, enable and jump-host passwords and keys it logs in with, every secret() value in the step's configuration, and the token a step with managed Terraform state uses. Wherever one of these values appears in what the step leaves behind, it is replaced by [redacted]:

• the live output on the run's Logs tab,
• evidence artifacts,
• the step's result: its summary, error, metrics and output.

The step's result is what later steps, branch conditions and approval templates read as steps.NODE_ID.output, so a secret never passes from one step to the next. A Terraform output that embeds a password, or a command that prints a token, reaches the next step as [redacted]. A step that needs a secret takes it with its own secret() reference.

When redaction changed the result, the step's output shows one line naming the fields it changed (never the values), for example Secret values this step used were replaced by [redacted] in its result: output.stdout.

Limits to keep in mind:

• Only values the platform resolved are covered. A password typed straight into a command is ordinary text to the platform.
• Values shorter than 4 characters are never redacted: replacing them would garble ordinary output. The worker logs a warning when it meets one.
• Usernames are not treated as secrets, even when they are stored as secret() references: they appear in prompts and command output and stay as they are.
• Redaction replaces text wherever it matches. A secret value that is also an ordinary word or number (true, 8080) is replaced in every place it appears in the output, so use values that do not occur in normal output.

API Endpoints

Secret metadata responses

Secret metadata responses expose backend_scheme, which is the canonical reference prefix used in secret refs.

When generating references, always use Jinja syntax:

{{ secret('<backend_scheme>://<path>/<key>') }}

List Secrets (metadata only)

GET /api/secrets

Returns secret metadata without values.

Create Secret

POST /api/secrets
Content-Type: application/json

{
  "name": "database-password",
  "description": "Production database password",
  "backend_id": "uuid-of-backend",
  "folder": "orgs/default/secrets/database-password",
  "values": {
    "password": "super-secret-value"
  }
}

The values map is stored in the configured backend; only metadata is saved to the database.

Discover and Claim Existing Backend Secrets

GET /api/secrets/discovered

Lists paths that exist in enabled, listable backends but that no Hegemony secret manages. Only names are listed; values are never read. Paths under another organization's orgs/<slug>/ prefix are hidden, and so are paths that any organization already manages.

To register one of these paths as a managed secret, claim it:

POST /api/secrets
Content-Type: application/json

{
  "name": "legacy-radius",
  "backend_id": "uuid-of-backend",
  "folder": "shared/legacy-radius",
  "claim_existing": true
}

A claim writes nothing to the backend, so it also works on read-only backends. It returns 404 when the path holds no data.

Each backend path is managed by at most one secret across all organizations. Creating or claiming a secret at a path that another secret already manages, in any organization, returns 400.

Update/Rotate Secret

PATCH /api/secrets/{id}
Content-Type: application/json

{
  "values": {
    "password": "new-rotated-value"
  }
}

Updates last_rotated_at and merges the provided keys in the backend.

Every call is written to the audit log as entity.updated on the secret, naming the keys written and the keys deleted. A request that changed nothing is recorded too, carrying no_changes: true, because reaching for a secret is worth a trace whether or not it altered anything. Values never appear in the entry.

A secret's page and a secret backend's page each carry a History card listing the most recent audit log entries about that object: how long ago, the action, the fields that changed, and who. A row opens the same entry dialog the audit log uses, and the card links into the log filtered to the object. Values are never shown; only that they were written. The card appears only for callers who may read the audit log.

Import Secrets with Configuration Exchange

Configuration Exchange supports an import-only secrets section in the single_yaml layout. This is useful for instance bootstrap data and controlled migrations where operators intentionally provide secret values inbound:

schema_version: 2

secrets:
  - name: database-password
    description: Production database password
    backend_ref: vault_internal
    folder: orgs/default/secrets/database-password
    values:
      username: app_user
      password: super-secret-value

Secret values are written to the selected/default backend and are never returned by Configuration Exchange exports, plan/apply summaries, audit diffs, or normal secret metadata responses. Exports may still include safe secret references such as {{ secret('vault://orgs/default/secrets/database-password/password') }} in credentials, notification destinations, or flow configuration.

When an imported secret already exists, Hegemony selects the existing backend and folder unless the payload overrides them. Provided values are merged into the backend object: matching keys are overwritten, omitted keys are preserved, and last_rotated_at advances only when the backend write changes stored data.

Delete Secret

DELETE /api/secrets/{id}

Removes from both database and the configured backend. Deleting a claimed secret removes only its metadata; the backend data stays.

Runtime Cache Behavior

Hegemony caches backend clients/configuration, not secret values:

• API process caches backend clients for metadata operations.
  • Cache entries are invalidated when a backend is created, updated, or deleted.
  • Cache entries are also invalidated automatically when backend config fingerprint changes.
• Worker process caches backend configuration/resolver instances with a short TTL (currently 60s) to pick up backend changes without restart.

Secret values remain just-in-time resolved and are never persisted in these caches.

Propagation Timing (What to Expect)

When backend configuration changes:

• API metadata operations reflect changes immediately after backend create/update/delete because the API backend client cache is explicitly invalidated.
• Worker runtime resolution converges within the worker cache TTL window (currently up to ~60 seconds) without worker restart.

For emergency rollout (for example broken credentials), restart worker processes to force immediate cache refresh.

Operational Runbook

Rotate backend credentials (AppRole / token)

The bundled backend rotates its own AppRole secret IDs and revokes the superseded ones; this runbook is for backends you manage yourself.

1. Rotate credentials in the backend first.
2. Update backend config in Settings → Secret Backends.
3. Validate with a known secret reference in a non-production run.
4. Confirm worker logs show successful secret resolution after change.
5. If resolution errors persist beyond the TTL window, restart workers and re-test.

Backend outage response

If backend is unreachable or auth fails:

1. Check worker logs for resolution failures (scheme/source context is logged).
2. Validate backend health/auth externally (bao status / vault status, then a login).
3. Pause new runs or route critical flows away from affected references.
4. Restore backend availability/auth and run a smoke flow.
5. Resume paused runs.

Internal OpenBao Backend (Automatic, Overlay-Driven)

Hegemony ships with an optional bundled secret store provided by the openbao-internal compose overlay. It runs OpenBao, the Linux Foundation fork of HashiCorp Vault. When enabled, the API auto-provisions the vault_internal backend entry at startup using AppRole credentials generated by openbao-init.

The tag and the reference scheme are deliberately unchanged: references stay {{ secret('vault://...') }}, because that is the address every stored reference already resolves through. OpenBao keeps Vault's HTTP API, so the same plugin and client serve both. See Internal OpenBao Lifecycle.

Default semantics are bootstrap-only:

• The bundled OpenBao becomes the default only when its overlay is enabled and no default backend exists yet.
• If an admin later sets an external backend as default, restarts do not override that choice.

Operational toggles:

• HEGEMONY_OPENBAO_INTERNAL_ENABLED (default false; the openbao-internal compose overlay sets it to true) — enable/disable bundled backend provisioning. The former name HEGEMONY_VAULT_INTERNAL_ENABLED is still read, so an existing .env needs no change.
• HEGEMONY_S3_INTERNAL_ENABLED (default true) — enable/disable bundled object store provisioning (bucket bootstrap and the managed repository).

You can still use the Settings UI to add additional backends (external Vault, cloud secret managers, etc.) and choose which backend is default.

Deployment Options

Option 1: Bundled OpenBao (opt-in via the openbao-internal overlay)

Available as a compose overlay and auto-bootstraps when enabled. It generates its own auto-unseal key and TLS certificate on first boot, then:

• Initializes OpenBao and stores the recovery keys
• Enables KV v2 at the hegemony mount
• Creates AppRole roles for the API and worker
• Writes AppRole credentials to an internal secrets volume, and rotates them
• Revokes the initial root token

No additional configuration is required. See Internal OpenBao Lifecycle for the full lifecycle, including what is already hardened and what is left to the operator.

Option 2: An external OpenBao or Vault (optional)

Configure your existing server (bao for OpenBao, vault for Vault — the commands are otherwise identical):

1. Enable KV v2:

   bao secrets enable -path=hegemony kv-v2

2. Create policies (see deploy/openbao/policies/*.hcl)

3. Enable AppRole and create roles:

   bao auth enable approle
   bao write auth/approle/role/hegemony-api \
     token_policies=hegemony-api \
     token_ttl=20m \
     token_max_ttl=1h \
     secret_id_ttl=24h

4. Configure Hegemony via Settings → Secret Backends to add the backend and set it as default if desired. The page is open to every signed-in user, like its tile on the Settings hub. Adding, editing, testing and deleting a backend need the platform-wide admin role: secret backends are platform-scoped, so an organization's admin role does not count, and those controls appear locked (with a tooltip saying so) for everyone else. Prefer the *_file credential fields over inline values if you rotate secret IDs: they are re-read on every authentication.

The address must be https://. A plain http:// address is refused, because the AppRole secret ID and every secret value would cross the network in cleartext. If you genuinely need it — a server -dev instance, or a network you already trust to carry that traffic — set allow_insecure_http: true in the backend config; the opt-in is logged on every backend build. This is enforced from hegemony-secret-vault 0.2.0, so a backend left on an http:// address stops working when you upgrade until you fix the address or set the flag.

Redirects are never followed, because the token travels in a header that HTTP clients replay across hosts. A Vault or OpenBao standby node with request forwarding disabled answers with a redirect; point the backend at the active node.

Security Notes

What's Protected

• ✅ Secret values never appear in database
• ✅ Secret values never appear in Temporal workflow history
• ✅ Secret values never logged (redacted automatically)
• ✅ API responses never include secret values
• ✅ Backend tokens have limited TTL and scoped permissions (when supported)

Best Practices

1. Use separate credentials for Worker — Worker requires read-only access
2. Rotate secrets via API - Never access backends directly
3. Monitor last_rotated_at - Track secret freshness
4. Use TLS in production - Never disable verify for external backends, and never set allow_insecure_http
5. Limit path_prefix scope - Use org-based paths for isolation

Troubleshooting

Quick diagnosis matrix

Symptom	Likely cause	First action
Secret ref resolves to empty	Missing key/path or wrong reference	Verify <scheme>://path/key and backend data
Secret resolution fails with auth errors	Expired/invalid AppRole or token	Rotate credentials and update backend config
API sees backend change, worker still uses old config	Worker cache TTL not elapsed yet	Wait up to ~60s or restart workers
Notifications fail for url_secret refs	Invalid secret ref or missing URL secret	Resolve the reference manually and verify destination secret

"Secrets backend not configured"

Check that backends are configured in Settings (navigate to Settings in the sidebar). At least one backend must be enabled and set as default.

"Authentication failed" (OpenBao / Vault)

Verify AppRole credentials are correct and haven't expired:

# Check role exists
bao read auth/approle/role/hegemony-api

# Generate new secret_id if expired
bao write -f auth/approle/role/hegemony-api/secret-id

For the bundled backend, docker compose ... restart openbao-init re-runs the bootstrap and re-mints both roles' credentials. Note that its secret IDs also carry a CIDR binding: if the OpenBao network's subnet changed, authentication fails even with correct credentials.

"Secret not found"

Verify the path in your reference matches the backend path. For the bundled backend (use vault instead of bao against a HashiCorp Vault):

# List secrets
bao kv list hegemony/orgs/default/secrets

# Read specific secret
bao kv get hegemony/orgs/default/secrets/db