Production Hardening

This guide collects every consciously-accepted development convenience in the Hegemony stack that must be revisited before a production deployment. Each section explains the risk, where it lives, and what to do about it.

The API refuses to start in production (HEGEMONY_ENVIRONMENT=prod or production) with the most dangerous misconfigurations: disabled authentication, a missing or placeholder internal API token, a missing PAT pepper, or wildcard CORS. The items below cannot be enforced automatically and require operator attention.

Credentials

Demo and example credentials are public knowledge

The demo environment file is maintained in the separate demo-data repository. Every value in it (hegemony, objectstoreadmin, demo-internal-token-not-for-production, ...) is public. The same applies to any value that ships in .env.dev.example.

For production:

• Start from deploy/compose/.env.prod.example, which leaves all credentials empty, and fill in every value.
• Generate secrets with openssl rand -base64 32. This applies to HEGEMONY_INTERNAL_API_TOKEN, HEGEMONY_PAT_PEPPER, and all database, Keycloak, and object store credentials.
• Set HEGEMONY_TF_STATE_ENCRYPTION_KEY (32 random bytes, base64) unless the internal OpenBao or another Transit-capable secret backend wraps the keys of managed Terraform/OpenTofu state. The production default runs no internal OpenBao, so without the key every tf.plan with managed state fails. Back the key up with the database and never change it: the versions it wrapped become unreadable.
• Never reuse .env.demo or .env.dev.example values: the API rejects the known placeholder internal tokens in production, but it cannot detect reused database or object store credentials.

Internal API token

HEGEMONY_INTERNAL_API_TOKEN is a shared secret between the worker and the API that protects the internal (worker-callback) endpoints. It is required in production and should be injected at deploy time (compose secret, orchestrator secret store), not stored in a long-lived .env file.

Flow-container sandbox

Every container a flow starts — container.run steps, lab nodes, images a step builds — runs on a dedicated privileged docker:dind daemon, the dind service from deploy/compose/docker-compose.dind.yml, and never on the host's Docker daemon. That file is not an overlay you choose: dc.sh and the compose tasks apply it to every stack, SERVICES=core included. There is no host-socket mode to fall back to, so a stack that cannot run the sandbox does not run container steps at all.

The sandbox needs two values from the env file, and compose refuses to render the stack without either:

• HEGEMONY_SHARED_WORKSPACE_VOLUME, the volume name your base file declares (hegemony-dev-… / hegemony-prod-…; both .env examples set it). It is required rather than defaulted because one sandbox file serves both base files and the two must agree.
• HEGEMONY_SANDBOX_TOKEN, which .env.prod.example leaves empty for you to generate (openssl rand -hex 32). dind-init stamps it on the sandbox daemon's hegemony-sandbox-marker volume, and the worker and the image-prune job refuse a daemon that does not carry it, so an endpoint pointed at the wrong daemon is caught rather than used. It is a marker, not a credential, but give each stack its own: a token two stacks share would let either one attest the other's sandbox. Never reuse the dev example's placeholder.

The worker checks the sandbox before it connects to Temporal, and the two ways it can fail are handled differently because only one of them can fix itself:

• Misconfigured: the worker exits. An empty HEGEMONY_CONTAINER_DOCKER_HOST or sandbox token, or a daemon that answers but is not the attested sandbox (the marker volume is missing or carries a different token), makes it log Worker refused to start and exit non-zero; its restart policy keeps retrying until the configuration is fixed. Waiting cannot cure any of these, and a worker that started anyway would take container steps it could only run somewhere else. The shipped compose files fill in tcp://dind:2375 for an empty endpoint and refuse an empty token, so in practice this means a worker started some other way, an endpoint pointed at the wrong daemon, or a token changed after the sandbox was stamped (see Rotating the sandbox token).
• Late: the worker starts degraded. A sandbox that does not answer gets a bounded wait (HEGEMONY_CONTAINER_DOCKER_WAIT_SECONDS, 120 seconds by default), after which the worker starts anyway and keeps probing the sandbox in the background. It logs why the sandbox is not attested yet (sandbox_attestation_probe_failed) when a reason first appears or changes, and every five minutes while it lasts. Until the sandbox answers and passes the same attestation, every container step is refused (it fails at once and is not retried, whatever its step policy's retries, and its error quotes the last failure), and every other activity keeps working. The worker's heartbeat meanwhile advertises no containers capability (a restarting worker withdraws the one its previous process advertised before it waits), so new runs with shared or explicit affinity are not pinned to it, and when no other worker advertises it either, such a run fails at worker assignment instead of waiting for the sandbox. A run pinned to the worker before it restarted keeps its pin, so its container steps are refused there. Once the sandbox passes, the worker sweeps it for orphaned flow containers and egress networks, then enables container steps and advertises containers, without a restart; it sweeps no daemon that has not passed. One that answers but fails attestation stops the worker: it logs Worker stopped and exits non-zero, and from then on its restarts are refused at startup like any other misconfiguration. A daemon that answers but then cannot report its marker counts as late, not misconfigured. At host boot the sandbox can simply be late, because the daemon starts containers without compose's ordering. Setting the wait to 0 only stops the worker from blocking at startup, not from probing: it asks the sandbox once, so a daemon that answers is attested, or refused, before the worker connects to Temporal, and one that does not is attested in the background before the worker runs any container step.

Ordinary Docker-API effects of a flow are confined to the sandbox: attach_docker_socket hands out the dind daemon's own socket, and privileged/pid_mode: host/network_mode: host grant power over the dind container only. This is containment, not a hard security boundary: the dind service itself runs --privileged, so kernel, container-runtime, or mismounted-host-path vulnerabilities can still reach the outer host. Keep the host kernel and Docker engine patched, minimize what else runs on the machine, and treat a dedicated host (below) as defense in depth rather than an alternative. The dind daemon's TCP endpoint is TLS-off by design; it is never published to the host and must stay that way (a unix-socket-volume transport is the documented zero-listener hardening alternative, see docs/development/design-dind-sandbox.md §6.3).

Host requirements

Nothing falls back when the host cannot run the sandbox, so check these before deploying or upgrading:

• Privileged containers. The dind service runs privileged: true. A daemon or policy layer that forbids privileged containers cannot run the stack.
• NET_ADMIN for the NFLOG sidecar. egress-nflog shares the sandbox's network namespace and needs CAP_NET_ADMIN to bind the NFLOG group that egress policies sample denials into. If the sidecar cannot run, only the egress report loses its per-destination detail; enforcement is unaffected.
• IPv6 and netfilter in the host kernel. The stack's default network is dual-stack, and the sandbox daemon runs with --ipv6 --ip6tables because egress policy networks are dual-stack. The host kernel must have IPv6 enabled (a host booted with IPv6 disabled cannot create the network, so the stack does not start at all) and must provide what the sandbox programs inside its own namespace: iptables and ip6tables, with NAT, and the NFLOG target. Docker masquerades the stack's ULA addresses out through the host's IPv6 uplink, so outbound IPv6 also needs the host daemon to manage ip6tables. IPv6 is not switchable.
• MTU. The sandbox's inner networks use HEGEMONY_DIND_MTU (1500). On a VPN or overlay uplink with a smaller effective MTU, lower it to match, or flow containers' large transfers stall.
• Images. Every stack runs docker:28-dind, docker:28-cli and the egress-nflog image (ghcr.io/hegemony-sh/hegemony/egress-nflog, pulled for the release tag or built from deploy/compose/Dockerfile.egress-nflog), and dind-init pulls the egress helper image (docker:28-dind by default) into the sandbox. An air-gapped mirror or a registry allow-list must carry them.
• Disk. The sandbox keeps its own image store, containers, volumes and networks in the dind-data volume, and starts empty: the first run of a flow pulls its step images again. The sandbox_image_pruning maintenance job reclaims dangling images and build cache; tagged images and inner volumes stay until someone removes them (docker exec into the dind container and prune there), and nothing caps the volume's size. down -v, task compose:dev:reset and task compose:prod:reset delete it with everything flows ever created.

Rotating the sandbox token

dind-init stamps HEGEMONY_SANDBOX_TOKEN on the sandbox's hegemony-sandbox-marker volume with docker volume create, which never relabels a volume that already exists, and the marker persists in dind-data. So after you change the token in the env file and recreate the stack, the worker finds the old token on the marker and stops with sandbox marker token mismatch, at startup or as soon as a late sandbox answers. Its restarts keep failing the same way, and the image-prune job fails with the same reason, until the marker carries the new token. From deploy/compose, with the ENV (and SERVICES) you deploy with:

# 1. Remove the stale marker inside the sandbox.
ENV=prod ./dc.sh exec dind docker volume rm hegemony-sandbox-marker
# 2. Re-run dind-init, which recreates the marker with the new token.
ENV=prod ./dc.sh up -d --no-deps --force-recreate dind-init
# 3. Restart the worker instead of waiting out its restart back-off.
ENV=prod ./dc.sh restart worker

If step 1 reports the volume is in use, a stopped egress helper container still mounts it; list it with ENV=prod ./dc.sh exec dind docker ps -a --filter volume=hegemony-sandbox-marker, remove it with docker rm the same way, and repeat step 1.

Docker socket exposure

The worker still mounts the host Docker socket (/var/run/docker.sock, see deploy/compose/docker-compose.prod.yml) and joins the host's docker group (DOCKER_GID), for one purpose only: deriving its worker id from its own Compose container name, which only the daemon hosting the worker can answer. A stable id keeps the worker's per-host task queue (hegemony-host-<id>) valid across container recreation, so runs pinned to that worker are not stranded. No flow container reaches the host socket: container steps run only on the sandbox, and a step's attach_docker_socket mounts the sandbox daemon's socket.

Access to the Docker socket is still equivalent to root on the host. What remains is a worker-level privilege: a compromised worker process, not a flow definition, can escalate to the host. Mitigations, in order of preference:

1. Dedicated hosts. Run the stack on machines that hold nothing else of value, so socket access compromises only that host. The sandbox's own --privileged asks for the same.
2. Socket proxy. Front the socket with a filtering proxy (e.g., docker-socket-proxy) that allows only container inspection, the one call the worker makes on the host daemon (docker inspect of its own container), plus the ping and version calls the Docker CLI uses to negotiate an API version.
3. Rootless Docker. Run the host daemon rootless; socket access then maps to an unprivileged user, not root. The shipped stack is not tested this way: the sandbox needs privileged containers, dual-stack networking and NFLOG from the daemon it runs on, so verify container steps and egress policies end to end before relying on it.
4. Restrict flow-editing permissions. Flow authors are sandbox administrators: attach_docker_socket, privileged, pid_mode: host and network_mode: host control the whole sandbox, including other runs' containers in it. Treat flow definitions that use container steps as code: limit who can author them via RBAC (the flow:manage action in apps/api/auth/actions.yaml, reallocatable under Settings → Permissions → Actions) and review changes through the git sync integration rather than ad-hoc UI edits.

Keep the socket mount even if no flow uses container steps. Removing it takes nothing away from flows, which never reach it; it only makes each worker fall back to its container ID as its worker id, which changes whenever the container is recreated and strands pinned runs on a queue nobody polls. Retiring the mount is tracked in TODO.md.

Upgrading to the mandatory sandbox

Releases before the sandbox became mandatory applied docker-compose.dind.yml only when SERVICES named dind; every other stack ran flow containers on the host daemon. Upgrading such a stack takes the steps below, in order. A stack that already named dind has the token and the dual-stack network, and its flow containers were in the sandbox all along: it skips steps 1, 5 and 6, and step 4 unless step 2 moves its subnet. SERVICES=...,dind still works; the name is now a deprecated no-op that prints a warning.

1. Add the sandbox token to the env file. Existing .env.dev and .env.prod files have no HEGEMONY_SANDBOX_TOKEN, and until they do every compose command (up, run migrate, the db:* tasks) stops with required variable HEGEMONY_SANDBOX_TOKEN is missing a value. Generate one per stack:

   echo "HEGEMONY_SANDBOX_TOKEN=$(openssl rand -hex 32)" >> deploy/compose/.env.prod

   An env file older than HEGEMONY_SHARED_WORKSPACE_VOLUME stops the same way; copy that line from the example too.

2. Give the stack its own subnet. The sandbox pins the stack's default network, and the compose defaults are the dev stack's addresses. The convention is one /24 per stack: 172.28.100.0/24 for dev, 172.28.101.0/24 for prod and 172.28.102.0/24 for the demo, with ULA counterparts (fd00:28:100::/64 and so on). .env.prod.example now carries the prod values as live lines; copy its HEGEMONY_COMPOSE_* and HEGEMONY_DIND_* address block into an existing .env.prod, or a prod stack lands on the dev addresses and collides with a dev stack on the same host. Whatever you pick must not overlap a route the host already has, 172.17.0.0/16 (the sandbox's inner default bridge), or the bundled OpenBao network (172.29.240.0/24 unless you moved it).

3. Check the host against the host requirements.

4. Recreate the network: down, then up. The default network changes from an implicit IPv4 network to a pinned dual-stack one, and compose does not change an existing network in place. down without -v keeps every volume. Use the SERVICES you deploy with:

   task compose:prod:down
   task compose:prod:up

5. Look for flow containers in the sandbox. docker ps on the host no longer shows them; they live inside the sandbox daemon:

   docker exec hegemony-prod-dind-1 docker ps -a --filter label=hegemony.run_id

6. Sweep what the old release left on the host daemon. The janitor now watches the sandbox only, so containers, networks, volumes and images flows created on the host daemon stay there until you remove them. Run this once every worker of the stack is on the new release, with the stack up. List first, delete second.

   Containers: flow containers are the only ones labeled hegemony.run_id, so this filter cannot select the stack's own services. The third column is the stack that created each one; an empty value means it predates stack labels.

   docker ps -a --filter label=hegemony.run_id \
     --format 'table {{.ID}}\t{{.Names}}\t{{.Label "hegemony.stack"}}\t{{.Status}}\t{{.CreatedAt}}'

   Remove this stack's, with their anonymous volumes (use your HEGEMONY_STACK_NAME if you changed it). Any still running belong to steps whose worker is gone; those steps were retried in the sandbox.

   docker ps -aq --filter label=hegemony.run_id --filter label=hegemony.stack=hegemony-prod \
     | xargs -r docker rm -f -v

   Remove unlabeled ones by ID from the listing once no other stack on this host still runs flows on the host daemon. Containers a step started itself through attach_docker_socket (a containerlab lab, for example) carry whatever labels that step gave them rather than these; find them by name.

   Networks: a network a step created on the host daemon (a containerlab lab's management bridge, for example) outlives its containers and carries no platform label either. Once the containers above are gone, list the networks no container uses and remove, by name, only those you recognize as a flow's. Do not run docker network prune: a network counts as unused as soon as its containers stop, so a stack stopped on this host would lose its own network.

   docker network ls --filter dangling=true \
     --format 'table {{.Name}}\t{{.Driver}}\t{{.CreatedAt}}'
   docker network rm NETWORK_NAME

   Volumes: flow-created named volumes carry no platform label, so list the ones no container uses and remove, by name, only those you recognize as a flow's. Do not prune them wholesale: a stack stopped on this host has dangling volumes too, its databases among them.

   docker volume ls --filter dangling=true
   docker volume rm VOLUME_NAME

   Images: images flow steps pulled or built on the host carry no platform label either. Docker refuses to remove an image a container still uses, so the stack's own images are safe while it runs; remove the ones your flows named, then the build cache their docker build steps left:

   docker image ls --format 'table {{.Repository}}:{{.Tag}}\t{{.ID}}\t{{.CreatedSince}}\t{{.Size}}'
   docker image rm IMAGE:TAG
   docker builder prune

   docker builder prune asks before it deletes, and the cache it removes includes your own local builds' (BUILD=local), which only costs rebuild time.

Flows may notice these changes:

• attach_docker_socket hands a step the sandbox daemon's socket, not the host's, and network_mode: host shares the sandbox's network namespace.
• Host paths in a step's extra_mounts resolve on the dind container's filesystem, not the host's.
• The sandbox starts with an empty image store, so first runs pull again.
• Allow-list and deny-rule egress policies, which the host daemon could not enforce and so failed closed at run time, are now enforced: steps carrying them run instead of failing.
• down -v and the reset tasks now wipe dind-data with the rest.

There is no switch back to the host daemon. An empty HEGEMONY_CONTAINER_DOCKER_HOST resolves to the sandbox like an unset one, and the worker refuses any daemon that is not the attested sandbox. To roll back, take the stack down with this release first, while it is still checked out and with the SERVICES you deploy with. Its file list names the sandbox, so its down also removes dind, dind-init, egress-nflog and the dual-stack network. Then switch to the previous release (its compose files and images) and up:

task compose:prod:down    # while this release is still checked out
# switch to the previous release, then:
task compose:prod:up

The previous release's own down does not know those three services unless its SERVICES names dind, so it leaves them behind as orphans, the privileged sandbox still running among them. They keep the network in use, so it cannot change back, and that release's up then fails to recreate it. If you have already switched, run that down with --remove-orphans before up, or remove the three containers by hand first:

task compose:prod:down -- --remove-orphans
task compose:prod:up

The network returns to its old single-stack form only once nothing is attached to it. Unless its SERVICES names dind, the previous release runs flows on the host daemon again and does not see what is in dind-data, which stays until you remove it; the new env lines do no harm there.

Temporal console

The Temporal Web UI is an administrative console for the workflow engine. Two properties make it unlike anything else in the stack:

• It authenticates nobody. The temporalio/ui image ships no login, so anything that can reach its port is an administrator.
• It is scoped to no organization. The platform runs a single Temporal namespace and temporalio/auto-setup ships no authorizer plugin, so whoever reaches the console reads every organization's workflow inputs and outputs and can terminate, signal or reset their runs. There is no per-tenant view of it.

It is therefore closed by default, at layers that stop traffic rather than hide a link: the temporal-ui service sits behind a Compose profile so a plain up never creates it, its port (when opened) is pinned to the literal 127.0.0.1 with no variable that can widen it, and it declares restart: "no" so the daemon never resurrects one that was left open. deploy/compose/https/Caddyfile and deploy/compose/docker-compose.https.yml no longer carry the :5445 vhost that used to proxy it on 0.0.0.0 with no authentication of any kind — if you are upgrading, confirm nothing in your own reverse proxy still does.

A platform admin opens it on demand from the host and closes it again; most questions are answered without it, by tctl inside the container. See Temporal Admin Access.

Two things to watch on a production host:

• COMPOSE_PROFILES must stay unset in .env.prod. Compose reads COMPOSE_* from --env-file, so a single COMPOSE_PROFILES=temporal-ui line re-enables the console on every up, permanently and with no other signal.
• The application does not link to it. If a response or a page in your deployment still offers a Temporal URL, something local is re-adding it.

Secret stores

One secret-store overlay ships with the compose stack: docker-compose.openbao-internal.yml, the supported bundled store, running OpenBao. TLS, static-key auto-unseal, AppRole auth with rotated and revoked secret IDs, an audit device, no published port, and a revoked root token. Suitable for single-node production-like deployments. See Internal OpenBao Lifecycle.

The demo stack, which lives in the separate demo-data repository, adds sample services of its own with well-known credentials. None of them belongs in a deployment.

For real external integration, point a Secret Backend at your own OpenBao or Vault cluster (TLS, properly initialized, least-privilege AppRole or token) via Settings → Secret Backends.

What the bundled store already does

Most of what used to be a checklist here is now the default:

• It auto-unseals, so it comes back serving from every restart without an operator and without any container holding an unseal key.
• Traffic is encrypted. The API and worker verify a certificate generated at first boot.
• API and worker credentials expire and are revoked. Their secret IDs carry a TTL, are re-minted on a timer, and superseded ones are destroyed after a grace period. All credentials are bound to the OpenBao network's CIDR.
• The bootstrap credential deliberately does not expire. hegemony-bootstrap is what re-mints the others, so an expiry it could sleep through would strand rotation with no way back. It is re-minted on every run and its superseded accessors are destroyed immediately, so exactly one is valid at a time, and it is CIDR-bound like the rest.
• The root token is revoked once that scoped bootstrap AppRole can take over. Its policy has no sys/* wildcard, so it cannot mount an engine or add an audit device, and it cannot rewrite its own policy or its own role.

Do not read the bootstrap AppRole as a barrier against reading secrets.openbao-init issues the api and worker secret IDs and writes them to the shared credentials volume, so it necessarily holds credentials that read the whole hegemony/ tree — before policy comes into it. Whoever can read the openbao-bootstrap-creds volume is effectively an administrator of the bundled store, and restricting that mount is the control that matters. What the AppRole buys over a stored root token is a credential that is short-lived, rotated, CIDR-bound and revocable without re-initialising the server.

• The container is hardened: every capability dropped, no new privileges, a read-only root filesystem, and no published port.

What you still have to decide

• The seal key lives on the same host as the data. Static-key auto-unseal is what buys unattended restarts, and the trade is explicit: anyone with host access to both volumes has both halves. That is the right call for an evaluation or a single-node deployment that must survive reboots unattended, and the wrong one where seal hygiene actually matters. There, move to a transit or cloud KMS seal, where the key lives somewhere else entirely.
• The certificate is self-signed. It encrypts the hop and exercises the verification path; it is not identity. Replace it with one from your own CA.
• Storage is a single Raft node. Integrated storage rather than the file backend, which OpenBao removes in 2.7.0 — so snapshots are available (bao operator raft snapshot save) and should be scheduled. What is missing is quorum: one node means a lost volume is lost data. Production wants three nodes. A snapshot is encrypted with the seal key, so it does not replace backing up openbao-seal.
• The audit device writes to stdout. The right sink for a container, but not retention: ship those logs somewhere durable and access-controlled.
• Recovery keys need custody. They are written to their own volume with a loud notice. Move them off the host; they are the only way back to admin access once the root token is revoked.
• Losing the seal key loses the data. Recovery keys do not decrypt storage. Back up the openbao-seal volume, or supply your own key.
• Swap has to be handled on the host. OpenBao has removed mlock support -- disable_mlock is now a startup error rather than a setting -- so the root key can be paged to disk. Turn swap off (swapoff -a, and drop the /etc/fstab entry) or encrypt the swap device. This is the one place the bundled backend is weaker than the Vault it replaces; see Internal OpenBao.

Restart policy and reboot survival

Every long-running service declares restart: unless-stopped, so the Docker daemon brings the stack back at boot; every one-shot declares restart: "no" so it is never re-run in a loop; and on-demand services (today just the Temporal console) declare restart: "no" alongside a profile, so a reboot re-closes them rather than reopening what an admin left running. Three consequences worth planning around:

• unless-stopped, not always — a stack an operator stopped stays stopped across a reboot. That is deliberate. If you want a stack to come back even after a deliberate stop, always is the policy you want, and you should know you are giving up the ability to park it.
• No boot-time ordering. depends_on is a docker compose up feature and is not honoured by the daemon's boot-time auto-start, which starts every container carrying a policy at once. Convergence is therefore a property of the services, not a guarantee of the restart policy: it works because each dependent fails fast when a dependency is missing, so the policy restarts it within Docker's exponential backoff until the dependency is up. A service that hung instead of exiting, or that did its setup only once at startup, would not recover this way. The bundled OpenBao does not depend on that ordering at all: it auto-unseals, so it is serving as soon as it starts. If your deployment needs deterministic ordered bring-up at boot — or full compose up semantics, which also re-runs the init one-shots — put a systemd unit in front of docker compose up -d. That is deliberately out of scope for the shipped compose files.
• The daemon itself must start at boot. Restart policies are worth nothing if Docker is not enabled: check systemctl is-enabled docker.

Multi-replica deployments

BFF session tickets (used to authenticate SSE/EventSource connections) are stored in a pluggable backend (apps/api/services/ticket_store.py):

• HEGEMONY_BFF_TICKET_STORE=memory (default) — per-process storage. Correct only with a single API replica; tickets created on one instance are invisible to others, causing intermittent SSE auth failures behind a load balancer. The API logs a startup warning when this backend runs in production.
• HEGEMONY_BFF_TICKET_STORE=redis — shared storage for load-balanced deployments. Set HEGEMONY_BFF_TICKET_REDIS_URL (e.g., redis://redis:6379/0; requires Redis 6.2+). Protect the Redis instance like a session store: network-isolate it and enable authentication — tickets grant authenticated API access for their (5-minute) lifetime. The API uses fail-fast Redis timeouts by default (HEGEMONY_BFF_TICKET_REDIS_SOCKET_CONNECT_TIMEOUT_SECONDS=2.0, HEGEMONY_BFF_TICKET_REDIS_SOCKET_TIMEOUT_SECONDS=5.0); tune them only if your network path legitimately needs longer before declaring Redis unhealthy.

Public webhook endpoint

POST /hooks/{path_token} is unauthenticated by design — anyone who can reach the API can send it traffic, and authentication happens inside the handler. Two budgets bound what that traffic can cost before it is authenticated:

• HEGEMONY_WEBHOOK_GLOBAL_INGRESS_PER_MINUTE (default 6000) — an instance-wide ceiling charged before the endpoint lookup, so a flood aimed at the public path does not cost a database query per request. One bucket shared by every endpoint, so it is an aggregate ceiling, not a per-endpoint one: 6000/minute is 100 requests/second for the whole instance. Size it above your expected aggregate rate and its bursts — set too close to normal traffic, one busy endpoint sheds another's.
• HEGEMONY_WEBHOOK_INGRESS_LIMIT_MULTIPLIER (default 10) — a per-endpoint budget, a multiple of that endpoint's rate_limit_per_minute, charged before the request body is read and before its secret is resolved. Keep it above 1: at the floor of 1 the ingress budget is the same size as the delivery limit, which converts ordinary duplicate retries from 409 into 429.

Both are process-local, like the per-endpoint delivery limiter. Behind a load balancer each replica sheds independently, so a multi-replica deployment wants an external rate limiter in front of the API as well — see Multi-replica deployments for the same caveat applied to session tickets.

Trusted input boundaries

These inputs are trusted by design; treat write access to them as equivalent to operator access:

• Git inventory repositories. Inventory YAML loaded from git providers (the hegemony-inventory-plugins wheel inventory_git; only the local provider is in-tree) is parsed with a SafeLoader-derived parser and size-limited, but the content (device addresses, credentials references, platform data) is trusted. Only point inventory providers at repositories with controlled write access.
• Flow templates. Template expressions in flow definitions are rendered with Jinja2 autoescape=False (packages/core/templates/resolver.py). This is intentional — the output is device/CLI configuration, not HTML — and the UI re-escapes everything it renders. The consequence: whoever can edit flows can fully control rendered step payloads. Gate flow editing with RBAC accordingly.

Platform environment and secret files

The API and worker processes hold platform credentials in their environment (for example HEGEMONY_INTERNAL_API_TOKEN) and in the secrets directory HEGEMONY_SECRETS_DIR (the bundled store's AppRole files are mounted at /run/secrets). The {{ env() }} and {{ file() }} template helpers read exactly those, so they work only in platform configuration that platform admins manage: secret backend and inventory provider settings. Every organization template, including flows, variables, devices, notifications and git and file repositories, is refused at save time and again at run time. See Platform-only references.

Keep it that way when you add configuration: a value an organization needs belongs in a secret backend, not in the worker's environment.

For the same reason the worker no longer reads the HEGEMONY_SMTP_* email defaults. Set the sender, port and encryption on each email destination, and delete those lines from an existing .env.prod: they have no effect.

Shared organization secrets

When a shared organization is designated, workers admit its secret namespace (orgs/<shared-slug>/…) on the strength of a slug the API stamps into each run's Temporal payload. The default HEGEMONY_SHARED_ORG_SECRET_SCOPE=shared_resources stamps it only for runs that execute a shared-org flow, execute against a shared-org device, or are subscribed to notify a shared-org destination, and for notifications of such runs or sent to a shared-org destination. The admission is run-wide: one shared-org device among the targets lets every step of that run resolve {{ secret('orgs/<shared-slug>/…') }}. Per-repository credential refs of a tenant's file repositories never reach the shared namespace under this setting. HEGEMONY_SHARED_ORG_SECRET_SCOPE=always restores the behaviour from before the setting existed: every run in every organization is stamped, so any flow author on the platform can resolve any shared-org secret value whether or not their flow uses shared content. Either way, treat every secret stored in the shared organization as readable by each tenant that can use it.

Registry credentials

Logins for external container registries (Container Registries) are stored as secret references and resolved by the worker in the run organization's namespace, so the same secret-store hygiene applies as for every other {{ secret('…') }} reference: the password lives in the secret store, the database holds its path. The worker writes the resolved logins into a per-step directory readable only by the worker process and deletes it when the step ends; the values are redacted from run output. The Test login button makes the API itself call the registry, so it is refused for hosts that resolve to private or loopback addresses — a registry on an internal network is still used by steps, only the test is unavailable for it.

Platform registry

The registry overlay (Container Registries) runs zot with no authentication of its own. Everything rests on topology and on the proxy in front of it:

• zot is unreachable except through the proxy. It serves on one fixed address (HEGEMONY_REGISTRY_ADDRESS) on an internal-only compose network (HEGEMONY_REGISTRY_NETWORK_SUBNET) where only the proxy, the API and the on-demand console sit, and publishes no port. It also has an interface on the default compose network, which is its way to the object store (or an external S3 endpoint), but it binds nothing there: the sandbox daemon's containers get a refused connection. Keep it that way: never add ports: to the registry service, and never widen the bind address in its configuration to 0.0.0.0.
• The proxy asks the API about every request with the shared secret in HEGEMONY_REGISTRY_PROXY_SECRET. Generate the secret (openssl rand -hex 32), keep it out of git and rotate it like the internal API token (change it in .env, restart api and registry-proxy). Only the API and the proxy hold it: the API refuses to start without it, and a worker runs without it. Without the right secret the authorize route answers 404, and the UI's nginx answers 404 for its public path, so the route does not exist outside the compose network.
• Steps hold short-lived tokens, minted per step and revoked when the step ends; the run's end bounds them too (HEGEMONY_TF_STATE_RUN_END_GRACE_SECONDS applies to every run token). A token reaches its organization's paths, the shared organization's and global, pushes only into its own cache paths and never deletes. Refusals are logged with the run and step.
• Plain HTTP on the compose network. The overlay lists the registry's host in the sandbox daemon's insecure-registries; the traffic never leaves the Docker host. A multi-host deployment (remote workers) must put TLS in front instead: terminate it on registry-proxy (Caddy's tls internal, or a certificate of yours), distribute the CA to each sandbox daemon under /etc/docker/certs.d/<host>/ca.crt, drop the insecure-registry entry and set HEGEMONY_REGISTRY_URL to https://….
• The console is an admin console. registry-ui is profile-gated and bound to loopback like the Temporal console, and it talks to zot past the proxy: whoever reaches it browses and deletes every organization's images. Open it from a terminal, close it when done, never publish its port.
• One S3 identity. zot stores under the platform's S3 key, in its own bucket. On the bundled Versity gateway a second identity would not be a boundary until a release carries the path-traversal fix from its pull request 2457; until then the bucket is a tidy place, not a wall. Back it up with the object store; zot's local volume is only a rebuildable index.

File repositories and the platform bucket

A file repository without per-repository credential refs runs on the deployment's HEGEMONY_S3_* identity. Organization admins may only point such repositories at the platform bucket, and their key prefix is confined to the organization's namespace (<s3_prefix>/orgs/<slug>/… - a prefix outside it is scoped into it). Only platform admins can address other buckets or prefixes with platform credentials, which is how a shared golden-image bucket is provisioned for the shared organization. To reach a bucket the platform identity should not hold keys for, set per-repository credential refs instead. The location rules apply when a repository is created, when its bucket or prefix is edited, and when per-repository credential refs are removed (the repository then returns to platform credentials); other edits leave a stored location alone, so a repository a platform admin pointed outside the platform bucket stays there until a platform admin moves it. Default-organization admins keep the bare platform prefix but cannot address the orgs/ subtree under it, where every other organization's namespace lives.

Object store data and backups

The bundled object store (Versity S3 Gateway, started by the s3 overlay) keeps one file per object under <data dir>/<bucket>/<key> and stores each object's metadata (ETag, Content-Type) in user.* extended attributes next to the file. Three consequences:

• Put the data on a dedicated local disk. Set HEGEMONY_S3_DATA_HOST_DIR in .env.prod to an absolute path (for example /srv/hegemony/objectstore) and create the directory before the first up; unset, the store uses the objectstore-data Docker volume. The filesystem must support user extended attributes — ext4, xfs, btrfs and zfs do; NFS, CIFS and tmpfs generally do not, and objects written there lose their metadata silently.
• Back it up with the bundled task, which stops the writers, dumps the matching catalog tables and archives the data directory with GNU tar and --xattrs: ACTION=backup ENV=prod task compose:objectstore for the default production stack, SERVICES=auth,s3,otel; for any other, add the SERVICES, BUILD=local and EXTRA_FILES it was started with (see Give the task the stack you run below). A plain cp/rsync without -X, or a busybox tar, drops the attributes. The task works for both the named volume and a host directory.
• Restore with the same task, and the same stack variables as the backup: ACTION=restore ENV=prod BACKUP=/path/to/hegemony-storage-prod-<stamp>.tar.gz CONFIRM=restore task compose:objectstore (without BACKUP the newest bundle in objectstore-backups/ is used). It refuses a bundle taken from another environment or in another format, stops the writers and the store, makes sure the data location exists before touching anything, snapshots the database tables it is about to replace, restores the tables and the data directory, rolls both back if either step fails (a tar that cannot set extended attributes counts as a failure, so a target filesystem without user xattrs is caught rather than silently losing metadata), and returns once the store and the writers report healthy again; if they do not within three minutes the data is in place but the task exits non-zero, so an automated restore never reads an unhealthy stack as success. Backup bundles taken before the object store changed (minio-data.tar.gz) hold a different on-disk format and cannot be restored.
• Give the task the stack you run. Both actions stop and restart the API, worker and scheduler through dc.sh, and compose recreates them from the files that run applies, so pass the SERVICES, BUILD=local and EXTRA_FILES the stack was started with. Unset, SERVICES is the default for ENV, as for task compose:prod:up: auth,s3,otel in production. A production stack that also runs the bundled OpenBao, for example, is backed up with ACTION=backup ENV=prod SERVICES=auth,s3,otel,openbao-internal task compose:objectstore. EXTRA_FILES paths are relative to deploy/compose. A stack without s3 in SERVICES has no bundled store, and the task refuses it before stopping anything.
• Credentials are the store's root account. HEGEMONY_S3_ACCESS_KEY_ID and HEGEMONY_S3_SECRET_ACCESS_KEY are both the API's identity and the gateway's root credentials; rotate them together and restart both services.
• Serve downloads over HTTPS. Presigned browser and device download URLs are signed for HEGEMONY_S3_EXTERNAL_URL and carry the object bytes, so point it at an https:// base served by a TLS-terminating proxy that forwards to the gateway on port 9000 with the Host header unchanged. Plain http:// belongs to loopback and lab use; the gateway's own port and web UI stay bound to 127.0.0.1 unless you put TLS in front of them. A device that cannot validate the proxy's certificate can fetch through the API's /files/{id}/{filename} stream instead of the presigned URL.

The gateway's per-request access log is off (VGW_QUIET), because every presigned download would otherwise land in docker compose logs with its signed query string. Startup, health and error output still appear there.

To use an S3 service you already run instead of the bundled store, leave s3 out of SERVICES and point the HEGEMONY_S3_* variables at it. Leave HEGEMONY_S3_INTERNAL_ENABLED at its default of true there too: the flag controls the whole managed-repository mechanism, not only bucket creation. With it on, the API creates its buckets where the identity may (a store that refuses CreateBucket is logged and skipped, so create the bucket yourself in that case) and provisions every organization's Internal object storage repository. Set it to false only when no organization should get a repository until an administrator creates one by hand.

UI port exposure

UI_PORT sets the host port the UI's nginx container is published on; UI_BIND_ADDRESS sets the address it binds, and defaults to 0.0.0.0. The UI is meant to be reachable -- it authenticates its own users through Keycloak -- so all-interfaces is the right default, unlike the loopback-only consoles above.

State the address rather than leaving it out. A ports entry written as "8080:80", with no host IP, is published on both address families: 0.0.0.0 and [::]. The IPv6 half only carries traffic if the daemon and the network agree about IPv6, and where they do not, the v6 listener accepts the connection and immediately resets it. That is worse than a refusal: a client resolving a name to an IPv6 address first has already established the connection, so it never falls back to IPv4, and the UI looks hung rather than absent.

To publish on IPv6 as well, once the host is known to route it, add a second ports entry bound to :: beside this one; setting UI_BIND_ADDRESS=:: on its own moves the port to IPv6 only. If you terminate TLS at a reverse proxy on the same host, bind the UI to 127.0.0.1 so the only way in is through the proxy.

CORS

HEGEMONY_CORS_ORIGINS must be an explicit, comma-separated origin list in production; the wildcard * is rejected at startup because the API sends credentialed CORS responses. List only the origins that serve the Hegemony UI.

Checklist

Before going to production, confirm:

• [ ] HEGEMONY_ENVIRONMENT=prod or HEGEMONY_ENVIRONMENT=production (enables all startup safety rails)
• [ ] All credentials generated fresh; nothing copied from .env.demo or .env.dev.example
• [ ] HEGEMONY_INTERNAL_API_TOKEN and HEGEMONY_PAT_PEPPER injected as secrets
• [ ] Managed state has a key source: HEGEMONY_TF_STATE_ENCRYPTION_KEY injected as a secret and backed up with the database, or a Transit backend (internal OpenBao, or one named in HEGEMONY_TF_STATE_KEY_BACKEND) whose storage and seal key are backed up with the database
• [ ] Host meets the sandbox requirements: privileged containers, IPv6, ip6tables and NFLOG
• [ ] HEGEMONY_SANDBOX_TOKEN generated for this stack, and the stack's subnet distinct from every other stack and route on the host
• [ ] Worker's host Docker socket exposure mitigated (the mount stays: it derives the worker id)
• [ ] Nothing from the demo-data repository (its overlays, .env.demo or sample identities) is part of the deployment
• [ ] Single API replica, or HEGEMONY_BFF_TICKET_STORE=redis with a secured Redis instance
• [ ] Git inventory repositories have controlled write access
• [ ] HEGEMONY_CORS_ORIGINS lists only your UI origins
• [ ] UI_BIND_ADDRESS reviewed: 127.0.0.1 when a reverse proxy on the same host is the only intended way in, and IPv6 published only if the host routes it
• [ ] Public webhook ingress budgets reviewed, and an external rate limiter in front of the API if running more than one replica
• [ ] TLS terminated in front of the UI/API (reverse proxy), and nothing in front of the Temporal console
• [ ] Temporal console closed: docker compose ps -a lists no temporal-ui, and nothing is listening on 8080/8088/5445
• [ ] COMPOSE_PROFILES unset in .env.prod
• [ ] Reboot tested end to end (systemctl restart docker, then confirm the stack converges and bao status reports Sealed: false)
• [ ] systemctl is-enabled docker returns enabled
• [ ] Seal-key-on-host trade-off accepted, or a transit/KMS seal configured
• [ ] Recovery keys moved off the host, and the openbao-seal volume backed up
• [ ] An administrative recovery path decided — the recovery keys alone no longer mint a root token (OpenBao 2.6 made generate-root an authenticated endpoint); see Internal OpenBao
• [ ] Self-signed certificate replaced, or accepted for this deployment
• [ ] Audit log shipped off the host

Initial identity provisioning

The bundled production realm contains application clients and roles, and imports no sample users or organization groups. Use the Keycloak administration account configured by KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD to create or federate your users in the hegemony realm. Assign the admin realm role to the intended initial Hegemony administrator, then sign in through the Hegemony UI. The Keycloak administration account belongs to the management realm; it is not a Hegemony user.

Demo accounts and groups live exclusively in the demo-data repository and are mounted by its demo overlay. KEYCLOAK_IMPORTED_USERS (space-separated usernames) names accounts a realm import created that keycloak-init.sh grants the realm's default role; the shipped env files leave it empty, and only the demo overlay sets it. Existing Keycloak databases retain their current users: importing the revised realm file does not delete accounts or reset realms.