Production Hardening
This guide collects every consciously-accepted development convenience in the Hegemony stack that must be revisited before a production deployment. Each section explains the risk, where it lives, and what to do about it.
The API refuses to start in production (HEGEMONY_ENVIRONMENT=prod or production) with the most dangerous misconfigurations: disabled authentication, a missing or placeholder internal API token, a missing PAT pepper, or wildcard CORS. The items below cannot be enforced automatically and require operator attention.
Credentials
Demo and example credentials are public knowledge
The demo environment file is maintained in the separate demo repository. Every value in it (objectstoreadmin, demo-internal-token-not-for-production, ...) is public. The same applies to any value that ships in .env.dev.example.
For production:
- Start from
deploy/compose/.env.prod.example, which leaves all credentials empty, and fill in every value. - Generate secrets with
openssl rand -base64 32. This applies toHEGEMONY_INTERNAL_API_TOKEN,HEGEMONY_PAT_PEPPER, and all database, Keycloak, and object store credentials. The pepper can be replaced later without breaking every token: see Rotating the pepper. - Set
HEGEMONY_TF_STATE_ENCRYPTION_KEY(32 random bytes, base64) unless the internal OpenBao or another Transit-capable secret backend wraps the keys of managed Terraform/OpenTofu state. The production default runs no internal OpenBao, so without the key everytf.planwith managed state fails. Back the key up with the database and never change it: the versions it wrapped become unreadable. - Never reuse
.env.demoor.env.dev.examplevalues: the API rejects the known placeholder internal tokens in production, but it cannot detect reused database or object store credentials.
Internal API token
HEGEMONY_INTERNAL_API_TOKEN is a shared secret between the worker and the API that protects the internal (worker-callback) endpoints. It is required in production and should be injected at deploy time (compose secret, orchestrator secret store), not stored in a long-lived .env file.
Flow-container sandbox
Every container a flow starts — container.run steps, lab nodes, images a step builds — runs on a dedicated privileged docker:dind daemon, the dind service from deploy/compose/docker-compose.dind.yml, and never on the host's Docker daemon. That file is not an overlay you choose: dc.sh and the compose tasks apply it to every stack, SERVICES=core included. There is no host-socket mode to fall back to, so a stack that cannot run the sandbox does not run container steps at all.
The sandbox needs two values from the env file, and compose refuses to render the stack without either:
HEGEMONY_SHARED_WORKSPACE_VOLUME, the volume name your base file declares (hegemony-dev-…/hegemony-prod-…; both.envexamples set it). It is required rather than defaulted because one sandbox file serves both base files and the two must agree.HEGEMONY_SANDBOX_TOKEN, which.env.prod.exampleleaves empty for you to generate (openssl rand -hex 32).dind-initstamps it on the sandbox daemon'shegemony-sandbox-markervolume, and the worker and the image-prune job refuse a daemon that does not carry it, so an endpoint pointed at the wrong daemon is caught rather than used. It is a marker, not a credential, but give each stack its own: a token two stacks share would let either one attest the other's sandbox. Never reuse the dev example's placeholder.
The worker checks the sandbox before it connects to Temporal, and the two ways it can fail are handled differently because only one of them can fix itself:
- Misconfigured: the worker exits. An empty
HEGEMONY_CONTAINER_DOCKER_HOSTor sandbox token, or a daemon that answers but is not the attested sandbox (the marker volume is missing or carries a different token), makes it logWorker refused to startand exit non-zero; its restart policy keeps retrying until the configuration is fixed. Waiting cannot cure any of these, and a worker that started anyway would take container steps it could only run somewhere else. The shipped compose files fill intcp://dind:2375for an empty endpoint and refuse an empty token, so in practice this means a worker started some other way, an endpoint pointed at the wrong daemon, or a token changed after the sandbox was stamped (see Rotating the sandbox token). - Late: the worker starts degraded. A sandbox that does not answer gets a bounded wait (
HEGEMONY_CONTAINER_DOCKER_WAIT_SECONDS, 120 seconds by default), after which the worker starts anyway and keeps probing the sandbox in the background. It logs why the sandbox is not attested yet (sandbox_attestation_probe_failed) when a reason first appears or changes, and every five minutes while it lasts. Until the sandbox answers and passes the same attestation, every container step is refused (it fails at once and is not retried, whatever its step policy'sretries, and its error quotes the last failure), and every other activity keeps working. The worker's heartbeat meanwhile advertises nocontainerscapability (a restarting worker withdraws the one its previous process advertised before it waits), so new runs withsharedorexplicitaffinity are not pinned to it, and when no other worker advertises it either, such a run fails at worker assignment instead of waiting for the sandbox. A run pinned to the worker before it restarted keeps its pin, so its container steps are refused there. Once the sandbox passes, the worker sweeps it for orphaned flow containers and egress networks, then enables container steps and advertisescontainers, without a restart; it sweeps no daemon that has not passed. One that answers but fails attestation stops the worker: it logsWorker stoppedand exits non-zero, and from then on its restarts are refused at startup like any other misconfiguration. A daemon that answers but then cannot report its marker counts as late, not misconfigured. At host boot the sandbox can simply be late, because the daemon starts containers without compose's ordering. Setting the wait to0only stops the worker from blocking at startup, not from probing: it asks the sandbox once, so a daemon that answers is attested, or refused, before the worker connects to Temporal, and one that does not is attested in the background before the worker runs any container step.
Ordinary Docker-API effects of a flow are confined to the sandbox: attach_docker_socket hands out the dind daemon's own socket, and privileged/pid_mode: host/network_mode: host grant power over the dind container only. This is containment, not a hard security boundary: the dind service itself runs --privileged, so kernel, container-runtime, or mismounted-host-path vulnerabilities can still reach the outer host. Keep the host kernel and Docker engine patched, minimize what else runs on the machine, and treat a dedicated host (below) as defense in depth rather than an alternative. The dind daemon's TCP endpoint is TLS-off by design and is served on the stack network only: the dind service's entrypoint drops the API port on every other interface, so flow containers reach the daemon only through an attach_docker_socket bind. It is never published to the host and must stay that way (a unix-socket-volume transport is the documented zero-listener hardening alternative, see Container Sandbox).
Host requirements
Nothing falls back when the host cannot run the sandbox, so check these before deploying or upgrading:
- Privileged containers. The
dindservice runsprivileged: true. A daemon or policy layer that forbids privileged containers cannot run the stack. NET_ADMINfor the NFLOG sidecar.egress-nflogshares the sandbox's network namespace and needsCAP_NET_ADMINto bind the NFLOG group that egress policies sample denials into. If the sidecar cannot run, only the egress report loses its per-destination detail; enforcement is unaffected.- IPv6 and netfilter in the host kernel. The stack's default network is dual-stack, and the sandbox daemon runs with
--ipv6 --ip6tablesbecause egress policy networks are dual-stack. The host kernel must have IPv6 enabled (a host booted with IPv6 disabled cannot create the network, so the stack does not start at all) and must provide what the sandbox programs inside its own namespace: iptables and ip6tables, with NAT, and the NFLOG target. Thedindservice installs its API-port rules with both before the daemon starts, and does not start if either refuses them. Docker masquerades the stack's ULA addresses out through the host's IPv6 uplink, so outbound IPv6 also needs the host daemon to manage ip6tables. IPv6 is not switchable. - MTU. The sandbox's inner networks use
HEGEMONY_DIND_MTU(1500). On a VPN or overlay uplink with a smaller effective MTU, lower it to match, or flow containers' large transfers stall. - Images. Every stack runs
docker:28-dind,docker:28-cliand theegress-nflogimage (ghcr.io/hegemony-sh/hegemony/egress-nflog, pulled for the release tag or built fromdeploy/compose/Dockerfile.egress-nflog), anddind-initpulls the egress helper image (docker:28-dindby default) into the sandbox. An air-gapped mirror or a registry allow-list must carry them. - Disk. The sandbox keeps its own image store, containers, volumes and networks in the
dind-datavolume, and starts empty: the first run of a flow pulls its step images again. Thesandbox_image_pruningmaintenance job reclaims dangling images and build cache; tagged images and inner volumes stay until someone removes them (docker execinto thedindcontainer and prune there), and nothing caps the volume's size.down -v,task compose:dev:resetandtask compose:prod:resetdelete it with everything flows ever created.
Rotating the sandbox token
dind-init stamps HEGEMONY_SANDBOX_TOKEN on the sandbox's hegemony-sandbox-marker volume with docker volume create, which never relabels a volume that already exists, and the marker persists in dind-data. So after you change the token in the env file and recreate the stack, the worker finds the old token on the marker and stops with sandbox marker token mismatch, at startup or as soon as a late sandbox answers. Its restarts keep failing the same way, and the image-prune job fails with the same reason, until the marker carries the new token. From deploy/compose, with the ENV (and SERVICES) you deploy with:
# 1. Remove the stale marker inside the sandbox.
ENV=prod ./dc.sh exec dind docker volume rm hegemony-sandbox-marker
# 2. Re-run dind-init, which recreates the marker with the new token.
ENV=prod ./dc.sh up -d --no-deps --force-recreate dind-init
# 3. Restart the worker instead of waiting out its restart back-off.
ENV=prod ./dc.sh restart workerIf step 1 reports the volume is in use, a stopped egress helper container still mounts it; list it with ENV=prod ./dc.sh exec dind docker ps -a --filter volume=hegemony-sandbox-marker, remove it with docker rm the same way, and repeat step 1.
Docker socket exposure
The worker still mounts the host Docker socket (/var/run/docker.sock, see deploy/compose/docker-compose.prod.yml) and joins the host's docker group (DOCKER_GID), for one purpose only: deriving its worker id from its own Compose container name, which only the daemon hosting the worker can answer. A stable id keeps the worker's per-host task queue (hegemony-host-<id>) valid across container recreation, so runs pinned to that worker are not stranded. No flow container reaches the host socket: container steps run only on the sandbox, and a step's attach_docker_socket mounts the sandbox daemon's socket.
Access to the Docker socket is still equivalent to root on the host. What remains is a worker-level privilege: a compromised worker process, not a flow definition, can escalate to the host. Mitigations, in order of preference:
- Dedicated hosts. Run the stack on machines that hold nothing else of value, so socket access compromises only that host. The sandbox's own
--privilegedasks for the same. - Socket proxy. Front the socket with a filtering proxy (e.g., docker-socket-proxy) that allows only container inspection, the one call the worker makes on the host daemon (
docker inspectof its own container), plus the ping and version calls the Docker CLI uses to negotiate an API version. - Rootless Docker. Run the host daemon rootless; socket access then maps to an unprivileged user, not root. The shipped stack is not tested this way: the sandbox needs privileged containers, dual-stack networking and NFLOG from the daemon it runs on, so verify container steps and egress policies end to end before relying on it.
- Restrict flow-editing permissions. Flow authors are sandbox administrators:
attach_docker_socket,privileged,pid_mode: hostandnetwork_mode: hostcontrol the whole sandbox, including other runs' containers in it. Treat flow definitions that use container steps as code: limit who can author them via RBAC (theflow:manageaction inapps/api/auth/actions.yaml, held by theeditorrole by default and reallocatable under Settings → Permissions → Actions) and review changes through the git sync integration rather than ad-hoc UI edits.
Keep the socket mount even if no flow uses container steps. Removing it takes nothing away from flows, which never reach it; it only makes each worker fall back to its container ID as its worker id, which changes whenever the container is recreated and strands pinned runs on a queue nobody polls. Retiring the mount is tracked in TODO.md.
Temporal console
The Temporal Web UI is an administrative console for the workflow engine. Two properties make it unlike anything else in the stack:
- It authenticates nobody. The
temporalio/uiimage ships no login, so anything that can reach its port is an administrator. - It is scoped to no organization. The platform runs a single Temporal namespace and
temporalio/auto-setupships no authorizer plugin, so whoever reaches the console reads every organization's workflow inputs and outputs and can terminate, signal or reset their runs. There is no per-tenant view of it.
It is therefore closed by default, at layers that stop traffic rather than hide a link: the temporal-ui service sits behind a Compose profile so a plain up never creates it, its port (when opened) is pinned to the literal 127.0.0.1 with no variable that can widen it, and it declares restart: "no" so the daemon never resurrects one that was left open. deploy/compose/https/Caddyfile and deploy/compose/docker-compose.https.yml carry no vhost for it. Do not proxy it from your own reverse proxy either: it has no authentication.
A platform admin opens it on demand from the host and closes it again; most questions are answered without it, by tctl inside the container. See Temporal Admin Access.
Two things to watch on a production host:
COMPOSE_PROFILESmust stay unset in.env.prod. Compose readsCOMPOSE_*from--env-file, so a singleCOMPOSE_PROFILES=temporal-uiline re-enables the console on everyup, permanently and with no other signal.- The application does not link to it. If a response or a page in your deployment still offers a Temporal URL, something local is re-adding it.
Secret stores
One secret-store overlay ships with the compose stack: docker-compose.openbao-internal.yml, the supported bundled store, running OpenBao. TLS, static-key auto-unseal, AppRole auth with rotated and revoked secret IDs, an audit device, no published port, and a revoked root token. Suitable for single-node production-like deployments. See Internal OpenBao Lifecycle.
The demo stack, which lives in the separate demo repository, adds sample services of its own with well-known credentials. None of them belongs in a deployment.
For real external integration, point a Secret Backend at your own OpenBao or Vault cluster (TLS, properly initialized, least-privilege AppRole or token) via Settings → Secret Backends.
What the bundled store already does
These are the defaults:
- It auto-unseals, so it comes back serving from every restart without an operator and without any container holding an unseal key.
- Traffic is encrypted. The API and worker verify a certificate generated at first boot.
- API and worker credentials expire and are revoked. Their secret IDs carry a TTL, are re-minted on a timer, and superseded ones are destroyed after a grace period. All credentials are bound to the OpenBao network's CIDR.
- The bootstrap credential deliberately does not expire.
hegemony-bootstrapis what re-mints the others, so an expiry it could sleep through would strand rotation with no way back. It is re-minted on every run and its superseded accessors are destroyed immediately, so exactly one is valid at a time, and it is CIDR-bound like the rest. - The root token is revoked once that scoped bootstrap AppRole can take over. Its policy has no
sys/*wildcard, so it cannot mount an engine or add an audit device, and it cannot rewrite its own policy or its own role.
Do not read the bootstrap AppRole as a barrier against reading secrets.
openbao-initissues the api and worker secret IDs and writes them to the shared credentials volume, so it necessarily holds credentials that read the wholehegemony/tree — before policy comes into it. Whoever can read theopenbao-bootstrap-credsvolume is effectively an administrator of the bundled store, and restricting that mount is the control that matters. What the AppRole buys over a stored root token is a credential that is short-lived, rotated, CIDR-bound and revocable without re-initialising the server.
- The container is hardened: every capability dropped, no new privileges, a read-only root filesystem, and no published port.
What you still have to decide
- The seal key lives on the same host as the data. Static-key auto-unseal is what buys unattended restarts, and the trade is explicit: anyone with host access to both volumes has both halves. That is the right call for an evaluation or a single-node deployment that must survive reboots unattended, and the wrong one where seal hygiene actually matters. There, move to a
transitor cloud KMS seal, where the key lives somewhere else entirely. - The certificate is self-signed. It encrypts the hop and exercises the verification path; it is not identity. Replace it with one from your own CA.
- Storage is a single Raft node. Integrated storage rather than the
filebackend, which OpenBao removes in 2.7.0 — so snapshots are available (bao operator raft snapshot save) and should be scheduled. What is missing is quorum: one node means a lost volume is lost data. Production wants three nodes. A snapshot is encrypted with the seal key, so it does not replace backing upopenbao-seal. - The audit device writes to stdout. The right sink for a container, but not retention: ship those logs somewhere durable and access-controlled.
- Recovery keys need custody. They are written to their own volume with a loud notice, as three shares any two of which make the quorum. Move them off the host and give each share to a different person; they are the only way back to admin access once the root token is revoked.
- Losing the seal key loses the data. Recovery keys do not decrypt storage. Back up the
openbao-sealvolume, or supply your own key. - Swap has to be handled on the host. OpenBao has removed mlock support --
disable_mlockis now a startup error rather than a setting -- so the root key can be paged to disk. Turn swap off (swapoff -a, and drop the/etc/fstabentry) or encrypt the swap device. This is the one place the bundled backend is weaker than the Vault it replaces; see Internal OpenBao.
Restart policy and reboot survival
Every long-running service declares restart: unless-stopped, so the Docker daemon brings the stack back at boot; every one-shot declares restart: "no" so it is never re-run in a loop; and on-demand services (today just the Temporal console) declare restart: "no" alongside a profile, so a reboot re-closes them rather than reopening what an admin left running. Three consequences worth planning around:
unless-stopped, notalways— a stack an operator stopped stays stopped across a reboot. That is deliberate. If you want a stack to come back even after a deliberate stop,alwaysis the policy you want, and you should know you are giving up the ability to park it.- No boot-time ordering.
depends_onis adocker compose upfeature and is not honoured by the daemon's boot-time auto-start, which starts every container carrying a policy at once. Convergence is therefore a property of the services, not a guarantee of the restart policy: it works because each dependent fails fast when a dependency is missing, so the policy restarts it within Docker's exponential backoff until the dependency is up. A service that hung instead of exiting, or that did its setup only once at startup, would not recover this way. The bundled OpenBao does not depend on that ordering at all: it auto-unseals, so it is serving as soon as it starts. If your deployment needs deterministic ordered bring-up at boot — or fullcompose upsemantics, which also re-runs the init one-shots — put a systemd unit in front ofdocker compose up -d. That is deliberately out of scope for the shipped compose files. - The daemon itself must start at boot. Restart policies are worth nothing if Docker is not enabled: check
systemctl is-enabled docker.
Multi-replica deployments
BFF session tickets (used to authenticate SSE/EventSource connections) are stored in a pluggable backend (apps/api/services/ticket_store.py):
HEGEMONY_BFF_TICKET_STORE=memory(default) — per-process storage. Correct only with a single API replica; tickets created on one instance are invisible to others, causing intermittent SSE auth failures behind a load balancer. The API logs a startup warning when this backend runs in production.HEGEMONY_BFF_TICKET_STORE=redis— shared storage for load-balanced deployments. SetHEGEMONY_BFF_TICKET_REDIS_URL(e.g.,redis://redis:6379/0; requires Redis 6.2+). Redis keeps each ticket only under a name derived from it, with what the ticket grants encrypted under a key derived from it, and names each user's index of tickets by a hash of the user's ID rather than the ID. Reading Redis yields no usable ticket, but writing to it could still plant one, so protect the Redis instance like a session store: network-isolate it and enable authentication. The API uses fail-fast Redis timeouts by default (HEGEMONY_BFF_TICKET_REDIS_SOCKET_CONNECT_TIMEOUT_SECONDS=2.0,HEGEMONY_BFF_TICKET_REDIS_SOCKET_TIMEOUT_SECONDS=5.0); tune them only if your network path legitimately needs longer before declaring Redis unhealthy.
Extensions follow their switch in every replica. A switch notifies every replica through PostgreSQL when it commits, and each one also checks every HEGEMONY_EXTENSION_RECONCILE_SECONDS (default 10). Each replica then runs the extension's own startup or shutdown, so its background work starts and stops everywhere, with no restart.
Public webhook endpoint
POST /hooks/{path_token} is unauthenticated by design — anyone who can reach the API can send it traffic, and authentication happens inside the handler. Two budgets bound what that traffic can cost before it is authenticated:
HEGEMONY_WEBHOOK_GLOBAL_INGRESS_PER_MINUTE(default6000) — an instance-wide ceiling charged before the endpoint lookup, so a flood aimed at the public path does not cost a database query per request. One bucket shared by every endpoint, so it is an aggregate ceiling, not a per-endpoint one:6000/minute is 100 requests/second for the whole instance. Size it above your expected aggregate rate and its bursts — set too close to normal traffic, one busy endpoint sheds another's.HEGEMONY_WEBHOOK_INGRESS_LIMIT_MULTIPLIER(default10) — a per-endpoint budget, a multiple of that endpoint'srate_limit_per_minute, charged before the request body is read and before its secret is resolved. Keep it above 1: at the floor of1the ingress budget is the same size as the delivery limit, which converts ordinary duplicate retries from409into429.
Both are process-local, like the per-endpoint delivery limiter. Behind a load balancer each replica sheds independently, so a multi-replica deployment wants an external rate limiter in front of the API as well — see Multi-replica deployments for the same caveat applied to session tickets.
Health probes
GET /health and GET /ready are unauthenticated by design, because orchestrators call them without a credential. Through the bundled UI proxy they are reachable as /api/health and /api/ready by anyone who can reach the UI port.
/healthdoes no work: it answers as long as the process runs./readychecks the database and Temporal on every call and answers onlyready(200) ornot_ready(503). Which dependency failed, and why, goes to the API log; the admin-onlyGET /platform/healthreports each subsystem. Each dependency gets 2 seconds, so a hung one makes the probe answernot_readywithin the 5-second budget of the compose healthchecks rather than hold the request open.
Neither probe is rate-limited, and the bundled nginx does not limit them either. The compose healthchecks and the scheduler call the API directly on the internal network, not through the proxy. If your orchestrator or monitoring does not need the proxied paths, deny /api/health and /api/ready at your own proxy or load balancer, or limit them to your monitoring network. When you rate-limit them instead, remember that a Kubernetes probe comes from the node's address, so a per-address limit can mark healthy pods as not ready.
Step handlers in the images
The api and worker images install every first-party step-handler wheel, so every instance offers these steps. Some of them reach outside the worker, and they are gated in different ways:
| Wheel | Steps | What it can do | What limits it |
|---|---|---|---|
hegemony-steps-shell | shell.execute | Run commands on Linux/Unix hosts over SSH | Only devices whose access config holds SSH credentials; the organization's trusted host keys; unquoted template values in a command are refused |
hegemony-steps-container | container.run, container.build | Run any image in the sandbox | The flow-container sandbox and the flow's Egress Policy |
hegemony-steps-ansible, hegemony-steps-tf | ansible.playbook, tf.plan, tf.apply | Run Ansible or OpenTofu/Terraform in a container | As container.run (they use its runner) |
hegemony-steps-http | http.request | Send HTTP requests from the worker | The flow's Egress Policy. A flow without one can reach any address the worker can, except loopback, link-local (cloud metadata included), unspecified and multicast addresses |
hegemony-steps-netbox | netbox.* | Read and write NetBox objects | As http.request (it uses its client) |
hegemony-steps-secret | secret.write | Write, swap or delete secrets of the run's organization | Nothing until an organization admin turns on secret writes by steps, and then only under the path prefixes they name (Secrets) |
hegemony-steps-inventory | inventory.sync | Start an inventory provider sync | Nothing until an administrator ticks "Flows may sync this provider" on a provider |
Whoever can edit flows decides which of these steps run, so gate flow editing with RBAC (see Trusted input boundaries). Give flows that call http.request or netbox.* an Egress Policy that names only the hosts they need.
There is no runtime switch that hides an installed step. To withhold one, build the api and worker images without its wheel. Remove it from both images, or the editor offers a step that the worker cannot run:
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
USER root
RUN uv pip uninstall --python /opt/venv/bin/python hegemony-steps-shell
USER appuserRemove hegemony-steps-netbox together with hegemony-steps-http, which it needs. A flow that uses a removed step can no longer run.
Trusted input boundaries
These inputs are trusted by design; treat write access to them as equivalent to editor access:
- Git inventory repositories. Inventory YAML loaded from git providers (the
hegemony-inventory-pluginswheelinventory_git; only the local provider is in-tree) is parsed with a SafeLoader-derived parser and size-limited, but the content (device addresses, credentials references, platform data) is trusted. Only point inventory providers at repositories with controlled write access. - Flow templates. Template expressions in flow definitions are rendered with Jinja2
autoescape=False(packages/core/templates/resolver.py). This is intentional — the output is device/CLI configuration, not HTML — and the UI re-escapes everything it renders. The consequence: whoever can edit flows can fully control rendered step payloads. Gate flow editing with RBAC accordingly.
Platform environment and secret files
The API and worker processes hold platform credentials in their environment (for example HEGEMONY_INTERNAL_API_TOKEN) and in the secrets directory HEGEMONY_SECRETS_DIR (the bundled store's AppRole files are mounted at /run/secrets). The {{ env() }} and {{ file() }} template helpers read exactly those, so they work only in platform configuration that platform admins manage: secret backend and inventory provider settings. Every organization template, including flows, variables, devices, notifications and git and file repositories, is refused at save time and again at run time. See Platform-only references.
Keep it that way when you add configuration: a value an organization needs belongs in a secret backend, not in the worker's environment.
For the same reason the worker no longer reads the HEGEMONY_SMTP_* email defaults. Set the sender, port and encryption on each email destination, and delete those lines from an existing .env.prod: they have no effect.
In the compose stack, only the services that run Hegemony's Python code (api, worker, scheduler and the migrate job) load the whole env file. Every other service names the settings it reads in its environment: block, and compose fills them in from the same --env-file the launchers pass: the databases get their POSTGRES_* login, Temporal its database login, and the production ui the settings listed under Security response headers. A value you add to the env file for one of those services reaches it only once its compose environment: block names it.
A setting the env file holds comes from the env file, even when the shell or CI job that starts the stack exports a variable of the same name. Compose itself prefers the shell when it fills in ${VAR}, so an exported POSTGRES_PASSWORD would give postgres-app one password while the API, which loads the file, used another. dc.sh and the compose tasks therefore run compose through deploy/compose/with-env-file.sh, which first removes every variable the env file sets from compose's environment. To change a setting, edit the env file. A variable the file does not set still comes from the shell, and so does one the file names without a value (a line holding only NAME). Run docker compose yourself and the shell wins again.
Shared organization secrets
When a shared organization is designated, workers admit its secret namespace (orgs/<shared-slug>/…) on the strength of a slug the API stamps into each run's Temporal payload. The default HEGEMONY_SHARED_ORG_SECRET_SCOPE=shared_resources stamps it only for runs that execute a shared-org flow, execute against a shared-org device, or are subscribed to notify a shared-org destination, and for notifications of such runs or sent to a shared-org destination. The admission is run-wide: one shared-org device among the targets lets every step of that run resolve {{ secret('orgs/<shared-slug>/…') }}. Per-repository credential refs of a tenant's file repositories never reach the shared namespace under this setting. HEGEMONY_SHARED_ORG_SECRET_SCOPE=always restores the behaviour from before the setting existed: every run in every organization is stamped, so any flow author on the platform can resolve any shared-org secret value whether or not their flow uses shared content. Either way, treat every secret stored in the shared organization as readable by each tenant that can use it.
Registry credentials
Logins for external container registries (Container Registries) are stored as secret references and resolved by the worker in the run organization's namespace, so the same secret-store hygiene applies as for every other {{ secret('…') }} reference: the password lives in the secret store, the database holds its path. The worker writes the resolved logins into a per-step directory readable only by the worker process and deletes it when the step ends; the values are redacted from run output. The Test login button makes the API itself call the registry, so it is refused for hosts that resolve to private or loopback addresses — a registry on an internal network is still used by steps, only the test is unavailable for it.
Platform registry
The registry overlay (Container Registries) runs zot with bearer authentication: every request carries a token the API signed, and zot checks it against the repository and action asked for. The boundaries are these:
- The signing key is the API's alone.
HEGEMONY_REGISTRY_TOKEN_KEY(openssl rand -hex 32, or a PEM-encoded Ed25519 private key) goes in the env file with the other secrets. The services built from this repository load that file, so the overlay overrides the key with an empty value for each one exceptapi: the worker (which runs step-handler plugin code), the scheduler, the migration job and the dev UI server never hold it. The databases and the production UI do not load the file at all (see Platform environment and secret files). Settings → Platform Configuration lists it as a secret: the API reports only whether it is set, never its value. zot holds only the public key, which the API writes into the sharedregistry-token-keyvolume at every start while the registry extension is on. A deployment that does not use these compose files must give the key to the API process alone. Keep it out of git and rotate it like the internal API token: change it in.env, restartapi, thenregistry(zot reads the key once, when it starts; tokens signed with the old key stop working at that restart). The registry extension cannot be turned on without it. - zot takes tokens and nothing else. There is no proxy, no password file and no web page; a request without a valid token for its repository gets a 401 with the challenge. zot publishes no host port and serves on the compose network under the registry's host alias. Keep it that way: never add
ports:to theregistryservice, and never add another authentication method to its configuration. - Steps hold short-lived run tokens, minted per step and revoked when the step ends; the run's end bounds them too (
HEGEMONY_RUN_TOKEN_END_GRACE_SECONDSapplies to every run token). The bearer tokens the API signs for them last five minutes and carry only what the step is entitled to: pulls from its organization's paths, the shared organization's andglobal, pushes into its own cache paths, never a delete, never the catalog. Scopes left out of a token are logged with the run and step. The token endpoint (GET /registry/token) takes only run tokens, and the web image's nginx answers 404 for its public path. - A build step may push only its organization's own image names. A
container.buildstep's token (scoperegistry:build:<slug>) adds push onorgs/<slug>/<image>whose first segment does not look like a host, and nothing else: not another organization, notglobal, never a delete or the catalog. Within that, it pushes whatever its flow says, so whoever may edit and run the organization's flows can replace any tag of its images (latestincluded). Reference images that matter by digest, and keep an eye on who holdsflow:manageandrun:trigger. - Builds run on the shared sandbox, isolated by what they are given. The build's
RUNinstructions get no network unless the step's egress policy is open; base images andADDURLs are fetched by the sandbox daemon outside the policy, as every step's image is. Each build pulls its base images with its own logins and never reads the sandbox's build cache, which every organization shares, and only the official Dockerfile frontend runs. Built images and the build cache grow the sandbox'sdind-datavolume like pulled images do; the daemon's builder garbage collection and an occasionaldocker builder pruneinsidedindkeep it in check. - Plain HTTP on the compose network. The overlay lists the registry's host in the sandbox daemon's
insecure-registries; the traffic never leaves the Docker host. A multi-host deployment (remote workers) must put TLS in front instead: a reverse proxy of yours terminating TLS for the registry and for the token endpoint, the CA distributed to each sandbox daemon under/etc/docker/certs.d/<host>/ca.crt, the insecure-registry entry dropped,HEGEMONY_REGISTRY_URLset tohttps://…andHEGEMONY_REGISTRY_TOKEN_REALMto the URL the remote daemons fetch tokens from. - The operator's token is an admin's.
python -m hegemony_registry.services.registry_tokensin theapicontainer signs a token for whatever scopes it is given, the catalog and deletes in any namespace included; it lasts five minutes by default. Use it from a terminal on the Docker host and never store it. - Image import and export run in the API. They reach zot with tokens the API signs for itself, for the one repository an import writes or a download reads; the routes decide who may (organization admins in their namespace, platform admins in
global), and an import never writes a cache-shaped path. Imported images are what an admin chose to load: the import checks that each blob matches its digest, not what the image does. Uploaded archives sit inHEGEMONY_REGISTRY_TRANSFER_DIRon the API's disk until their image is pushed, and are removed afterwards and at startup; give that directory room for the uploads in flight (each up toHEGEMONY_REGISTRY_TRANSFER_MAX_UPLOAD_BYTES), on a volume when the container's own disk is small. The web image takes uploads up to 2 GiB on the upload route only and streams them to the API. Downloads are streamed from zot and never stored. - A daily cleanup deletes images; platform admins set what. The registry extension's maintenance job (
registry:image_retention) deletes cached copies nobody used for a while and old build tags, by the rules on Artifacts → Container Images → Retention, which only platform admins change, preview and run. It deletes tags with tokens the API signs for itself, keeps any image a run that has not finished names by digest or that was pulled within a day, and records one audit entry per repository it deleted from. zot itself deletes no tag; its garbage collection removes the images left without one. After an upgrade the first daily run applies the defaults until the page is saved: build tags older than 30 days go, except each image's newest 10; open Preview first if that matters to you.HEGEMONY_REGISTRY_CACHE_RETENTIONonly seeds the page's cached-copy window. - One S3 identity. zot stores under the platform's S3 key, in its own bucket. On the bundled Versity gateway a second identity would not be a boundary until a release carries the path-traversal fix from its pull request 2457; until then the bucket is a tidy place, not a wall. Back it up with the object store; zot's local volume is only a rebuildable index.
- Its tables are its own. The registry keeps image imports in
registry_image_transfersand the cleanup's rules inregistry_retention_policies, which its own migrations create. Themigratejob applies them with the platform's (task db:migrate:prod); its first run after this change renames the platform'simage_transferstable, rows included. Until the job has run, the API refuses the registry extension and says why on Settings → Extensions. The job needs no registry secret.
File repositories and the platform bucket
A file repository without per-repository credential refs runs on the deployment's HEGEMONY_S3_* identity. Organization admins may only point such repositories at the platform bucket, and their key prefix is confined to the organization's namespace (<s3_prefix>/orgs/<slug>/… - a prefix outside it is scoped into it). Only platform admins can address other buckets or prefixes with platform credentials, which is how a shared golden-image bucket is provisioned for the shared organization. To reach a bucket the platform identity should not hold keys for, set per-repository credential refs instead. The location rules apply when a repository is created, when its bucket or prefix is edited, and when per-repository credential refs are removed (the repository then returns to platform credentials); other edits leave a stored location alone, so a repository a platform admin pointed outside the platform bucket stays there until a platform admin moves it. Default-organization admins keep the bare platform prefix but cannot address the orgs/ subtree under it, where every other organization's namespace lives.
Object store data and backups
The bundled object store (Versity S3 Gateway, started by the s3 overlay) keeps one file per object under <data dir>/<bucket>/<key> and stores each object's metadata (ETag, Content-Type) in user.* extended attributes next to the file. Three consequences:
- Put the data on a dedicated local disk. Set
HEGEMONY_S3_DATA_HOST_DIRin.env.prodto an absolute path (for example/srv/hegemony/objectstore) and create the directory before the firstup; unset, the store uses theobjectstore-dataDocker volume. The filesystem must support user extended attributes — ext4, xfs, btrfs and zfs do; NFS, CIFS and tmpfs generally do not, and objects written there lose their metadata silently. - Back it up with the bundled task, which stops the writers, dumps the matching catalog tables and archives the data directory with GNU tar and
--xattrs:ACTION=backup ENV=prod task compose:objectstorefor the default production stack,SERVICES=auth,s3,otel; for any other, add theSERVICES,BUILD=localandEXTRA_FILESit was started with (see Give the task the stack you run below). A plaincp/rsyncwithout-X, or a busyboxtar, drops the attributes. The task works for both the named volume and a host directory. - Restore with the same task, and the same stack variables as the backup:
ACTION=restore ENV=prod BACKUP=/path/to/hegemony-storage-prod-<stamp>.tar.gz CONFIRM=restore task compose:objectstore(withoutBACKUPthe newest bundle inobjectstore-backups/is used). It refuses a bundle taken from another environment or in another format, stops the writers and the store, makes sure the data location exists before touching anything, snapshots the database tables it is about to replace, restores the tables and the data directory, rolls both back if either step fails (atarthat cannot set extended attributes counts as a failure, so a target filesystem without user xattrs is caught rather than silently losing metadata), and returns once the store and the writers report healthy again; if they do not within three minutes the data is in place but the task exits non-zero, so an automated restore never reads an unhealthy stack as success. Backup bundles taken before the object store changed (minio-data.tar.gz) hold a different on-disk format and cannot be restored. - Give the task the stack you run. Both actions stop and restart the API, worker and scheduler through
dc.sh, and compose recreates them from the files that run applies, so pass theSERVICES,BUILD=localandEXTRA_FILESthe stack was started with. Unset,SERVICESis the default forENV, as fortask compose:prod:up:auth,s3,otelin production. A production stack that also runs the bundled OpenBao, for example, is backed up withACTION=backup ENV=prod SERVICES=auth,s3,otel,openbao-internal task compose:objectstore.EXTRA_FILESpaths are relative todeploy/compose. A stack withouts3inSERVICEShas no bundled store, and the task refuses it before stopping anything. - Credentials are the store's root account.
HEGEMONY_S3_ACCESS_KEY_IDandHEGEMONY_S3_SECRET_ACCESS_KEYare both the API's identity and the gateway's root credentials; rotate them together and restart both services. - Serve downloads over HTTPS. Presigned browser download URLs are signed for
HEGEMONY_S3_EXTERNAL_URLand carry the object bytes, so point it at anhttps://base served by a TLS-terminating proxy that forwards to the gateway on port 9000 with theHostheader unchanged. Plainhttp://belongs to loopback and lab use; the gateway's own port and web UI stay bound to127.0.0.1unless you put TLS in front of them. - Devices and servers download through the API, not the store. A step that stages a file on a target hands it a download link under
HEGEMONY_API_EXTERNAL_URL, and the API streams the file. Targets therefore need to reach that address only, and the store's port can stay on127.0.0.1. The token in a link's path is its only credential: it opens one file, for minutes, but anyone who reads it in that time can download the file. Prefer anhttps://address; where devices cannot validate the certificate, keep thehttp://address on the management network. The API masks the token in its own access log and records no trace span for these requests (paths you exclude from traces yourself, withOTEL_PYTHON_FASTAPI_EXCLUDED_URLS, are added to that list, they do not replace it), and the bundled UI proxy writes them to neither its access log nor its error log. A reverse proxy of your own in front logs the request line unless told otherwise: in its access log for every request, and in its error log when the upstream fails or times out. Keep paths under the download path out of both, or redact the token segment. The path is whatever follows the address inHEGEMONY_API_EXTERNAL_URL:/api/dl/through the UI proxy,/dl/when the address is the API itself.
The gateway's per-request access log is off (VGW_QUIET), because every presigned download would otherwise land in docker compose logs with its signed query string. Startup, health and error output still appear there.
To use an S3 service you already run instead of the bundled store, leave s3 out of SERVICES and point the HEGEMONY_S3_* variables at it. Leave HEGEMONY_S3_INTERNAL_ENABLED at its default of true there too: the flag controls the whole managed-repository mechanism, not only bucket creation. With it on, the API creates its buckets where the identity may (a store that refuses CreateBucket is logged and skipped, so create the bucket yourself in that case) and provisions every organization's Internal object storage repository. Set it to false only when no organization should get a repository until an administrator creates one by hand.
UI port exposure
UI_PORT sets the host port the UI's nginx container is published on; UI_BIND_ADDRESS sets the address it binds, and defaults to 0.0.0.0. The UI is meant to be reachable -- it authenticates its own users through Keycloak -- so all-interfaces is the right default, unlike the loopback-only consoles above.
State the address rather than leaving it out. A ports entry written as "8080:80", with no host IP, is published on both address families: 0.0.0.0 and [::]. The IPv6 half only carries traffic if the daemon and the network agree about IPv6, and where they do not, the v6 listener accepts the connection and immediately resets it. That is worse than a refusal: a client resolving a name to an IPv6 address first has already established the connection, so it never falls back to IPv4, and the UI looks hung rather than absent.
To publish on IPv6 as well, once the host is known to route it, add a second ports entry bound to :: beside this one; setting UI_BIND_ADDRESS=:: on its own moves the port to IPv6 only. If you terminate TLS at a reverse proxy on the same host, bind the UI to 127.0.0.1 so the only way in is through the proxy.
Security response headers
The production UI image's nginx sends these headers on every response, error pages included:
| Header | Value | What it stops |
|---|---|---|
X-Content-Type-Options | nosniff | The browser guessing a file's type, for example running a download as a script |
X-Frame-Options | SAMEORIGIN | Another site showing the UI in a frame to trick clicks (clickjacking) |
Referrer-Policy | same-origin | UI addresses, which name runs and devices, reaching other sites in the Referer header |
Permissions-Policy | camera, microphone, geolocation, payment, USB, serial, HID and MIDI all off | Browser features the UI never uses |
Content-Security-Policy | frame-ancestors 'self'; base-uri 'self'; object-src 'none' | Framing, an injected <base> tag, and plugins |
Content-Security-Policy-Report-Only | The full policy, below | Nothing yet: the browser only reports what the policy would block |
The container writes them when it starts (apps/ui/security-headers.sh), from its environment. The compose file passes the ui service these settings from the env file by name, and nothing else from it:
- The full Content Security Policy (CSP) lets the browser load scripts, styles, images and fonts from the UI itself only, and talk to the UI and to the Keycloak realm: the one in
HEGEMONY_KEYCLOAK_PUBLIC_ISSUER, else inHEGEMONY_KEYCLOAK_ISSUER, the same issuer the API gives the UI. An issuer that is not a plainhttp(s)://address is left out, with a warning in the container log. HEGEMONY_UI_CSP_ENFORCE=trueenforces the full policy in place of the limited one.HEGEMONY_UI_HSTS_MAX_AGEturns onStrict-Transport-Security(HTTP Strict Transport Security, HSTS) for that many seconds;31536000is one year. It is off by default.
A container that cannot write the file, because its file system is read-only or it runs as another user, logs a warning and keeps the defaults. Give it a writable /etc/nginx/snippets to use the settings above.
Turn HSTS on only when the UI is served over HTTPS with a certificate that browsers trust. A browser that has seen the header uses HTTPS for this host until the time runs out, and no longer lets anyone click through a certificate warning. The bundled HTTPS overlay's self-signed certificate always raises that warning, so the overlay never sends HSTS. If your own reverse proxy terminates TLS and already sends the header, leave this setting empty, so the header is not sent twice.
The HTTPS overlay's Caddy adds the five enforced headers to a UI response that carries none of them. That covers the Vite dev server; the production nginx sends its own, and Caddy leaves them as they are.
Enforcing the full Content Security Policy
The full policy starts in report-only mode, so a page it would break still works. To move it to enforcing:
- Open the UI with the browser's developer console showing. Sign in, stay signed in past a token renewal, and use the pages your users use, editors and diagrams included. Each violation shows as a console message naming the blocked address and the directive.
- If a message names something the UI needs, report it as an issue: the policy, not your deployment, needs the fix.
- When the console stays clean, set
HEGEMONY_UI_CSP_ENFORCE=truein.env.prodand recreate theuicontainer (task compose:prod:updoes). Check the console once more: a violation now blocks the address instead of only reporting it.
CORS
HEGEMONY_CORS_ORIGINS must be an explicit, comma-separated origin list in production; the wildcard * is rejected at startup because the API sends credentialed CORS responses. List only the origins that serve the Hegemony UI.
Time synchronization
Every host that runs part of the stack must keep accurate time from NTP (Network Time Protocol): chrony, systemd-timesyncd or ntpd, pointed at time servers you trust. Containers read the host's clock, so syncing the hosts covers every container, and nothing in the stack sets the time itself.
The stack depends on the clock in several places:
- Audit log and run evidence. Every audit entry, run event and step result carries a timestamp. When hosts drift, the order of events across the API, the worker and Keycloak no longer matches what happened, and the record is harder to defend.
- Login tokens. Keycloak stamps a token's issue and expiry times, and the API checks them with 30 seconds of tolerance. When the two hosts' clocks differ by more than that, the API rejects tokens that are still valid or keeps accepting expired ones.
- One-time login codes, if you enable them in Keycloak, are time-based and fail when the Keycloak host's clock is off.
- Certificates are checked against the clock: a host that is far off rejects valid TLS certificates or accepts expired ones.
- Scheduled work (maintenance jobs, image cleanup, schedules) runs by the clock.
Check each host with timedatectl (System clock synchronized: yes) or chronyc tracking, and alert when synchronization is lost.
Checklist
Before going to production, confirm:
- [ ]
HEGEMONY_ENVIRONMENT=prodorHEGEMONY_ENVIRONMENT=production(enables all startup safety rails) - [ ] All credentials generated fresh; nothing copied from
.env.demoor.env.dev.example - [ ]
HEGEMONY_INTERNAL_API_TOKENandHEGEMONY_PAT_PEPPERinjected as secrets - [ ] Managed state has a key source:
HEGEMONY_TF_STATE_ENCRYPTION_KEYinjected as a secret and backed up with the database, or a Transit backend (internal OpenBao, or one named inHEGEMONY_TF_STATE_KEY_BACKEND) whose storage and seal key are backed up with the database - [ ] Host meets the sandbox requirements: privileged containers, IPv6, ip6tables and NFLOG
- [ ]
HEGEMONY_SANDBOX_TOKENgenerated for this stack, and the stack's subnet distinct from every other stack and route on the host - [ ] Worker's host Docker socket exposure mitigated (the mount stays: it derives the worker id)
- [ ] Nothing from the demo repository (its overlays,
.env.demoor sample identities) is part of the deployment - [ ] Single API replica, or
HEGEMONY_BFF_TICKET_STORE=rediswith a secured Redis instance - [ ] Git inventory repositories have controlled write access
- [ ] Step handlers in the images reviewed: flows that call
http.requestornetbox.*have an Egress Policy, and any step your deployment must not offer is left out of both images - [ ]
HEGEMONY_CORS_ORIGINSlists only your UI origins - [ ] Security response headers reviewed: HSTS set once browsers trust the certificate, and the Content Security Policy enforced once the browser console shows no reports
- [ ] The
hegemonyrealm's password policy kept, or replaced by one your organization requires - [ ] Users with the
adminrole, of the realm or of thehegemony-apiclient, sign in with a one-time code (Multi-factor authentication):HEGEMONY_ADMIN_MFA_REQUIREDnotfalse, or your identity provider requires multi-factor authentication - [ ] Step-up authentication left on (
HEGEMONY_STEP_UP_MAX_AGE_SECONDSnot0), andHEGEMONY_STEP_UP_ALLOW_TOKENSset only if a pipeline must manage secret backends, permissions, members or tokens - [ ]
UI_BIND_ADDRESSreviewed:127.0.0.1when a reverse proxy on the same host is the only intended way in, and IPv6 published only if the host routes it - [ ] Public webhook ingress budgets reviewed, and an external rate limiter in front of the API if running more than one replica
- [ ] TLS terminated in front of the UI/API (reverse proxy), and nothing in front of the Temporal console
- [ ] Every host keeps its time from NTP, and loss of synchronization is alerted on (Time synchronization)
- [ ] Temporal console closed:
docker compose ps -alists notemporal-ui, and nothing is listening on 8080/8088/5445 - [ ]
COMPOSE_PROFILESunset in.env.prod - [ ] Reboot tested end to end (
systemctl restart docker, then confirm the stack converges andbao statusreportsSealed: false) - [ ]
systemctl is-enabled dockerreturnsenabled - [ ] Seal-key-on-host trade-off accepted, or a transit/KMS seal configured
- [ ] Recovery key shares moved off the host and given to different people, and the
openbao-sealvolume backed up - [ ] An administrative recovery path decided — the recovery keys alone no longer mint a root token (OpenBao 2.6 made
generate-rootan authenticated endpoint); see Internal OpenBao - [ ] Self-signed certificate replaced, or accepted for this deployment
- [ ] Audit log shipped off the host
- [ ] The personal data the platform keeps entered in your records of processing, and retention chosen where a setting exists (
HEGEMONY_AUDIT_RETENTION_DAYSand run event pruning are off by default)
Initial identity provisioning
The bundled production realm contains application clients and roles, and imports no sample users or organization groups. Use the Keycloak administration account configured by KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD to create or federate your users in the hegemony realm. Assign the admin realm role to the intended initial Hegemony administrator, then sign in through the Hegemony UI. The Keycloak administration account belongs to the management realm; it is not a Hegemony user. Passwords in the realm follow its password policy: at least 12 characters. At their first sign-in, users with the admin role also set up a one-time code from an authenticator app, and give one at every sign-in after that.
The realm also contains the hegemony-automation client, a service account for CI/CD and other automation that signs in with the client credentials flow. The realm file ships it without a secret. When the updated keycloak-init step runs, it gives the client a secret Keycloak generates if it has none or still has the default from an older realm file, and leaves a secret you set alone. Read or rotate the secret in the Keycloak administration console under Clients → hegemony-automation → Credentials, and keep it in your secret store.
Demo accounts and groups live only in the hegemony-demo repository and are mounted by its demo overlay. KEYCLOAK_IMPORTED_USERS (space-separated usernames) names accounts a realm import created that keycloak-init.sh grants the realm's default role; the shipped env files leave it empty, and only the demo overlay sets it. Existing Keycloak databases retain their current users: importing the revised realm file does not delete accounts or reset realms.