Skip to content

Keycloak OIDC Authentication ​

This document describes the authentication and authorization architecture for Hegemony, implemented using Keycloak as the OIDC identity provider.


Table of Contents ​

  1. Overview
  2. High-Level Architecture
  3. Keycloak's Role
  4. Authentication Flows
  5. Role-Based Access Control
  6. Operating Modes
  7. Security Features
  8. Failure Scenarios & Error Handling
  9. BFF Auth Gateway
  10. Internal API Protection
  11. External Identity Provider Integration
  12. RBAC Database Schema
  13. Configuration Reference
  14. Quick Reference
  15. Glossary

Overview ​

Hegemony uses a modern enterprise authentication architecture:

  • Frontend: React SPA with PKCE Authorization Code flow via react-oidc-context
  • Backend: FastAPI with JWT validation via JWKS
  • Worker: Shared secret token (X-Internal-Token) for internal API calls
  • Identity Provider: Keycloak as the OIDC/OAuth2 identity provider

Design Principles ​

PrincipleImplementation
Zero-trustAll API requests require authentication except explicitly public endpoints: health checks, webhook triggers (POST /hooks/{path_token} authenticates by signed payload instead) and file download links (GET /dl/{token}/{filename}, where the short-lived token in the path is the credential)
Fail-fastProduction startup fails if security config is missing
Defense in depthMultiple validation layers (JWT + RBAC + internal tokens)
Separation of dutiesDistinct roles keep building, running and approving apart (editor builds flows, operator runs them, approver approves; neither editor nor operator can approve)
Secure by defaultAuth enabled by default; dev opt-out is explicit and blocked in prod

High-Level Architecture ​

Complete System View ​

Component Responsibilities ​


Keycloak's Role ​

Keycloak serves as the central identity provider for Hegemony, handling all aspects of identity management:

Core Responsibilities ​

Keycloak Configuration Topology ​

Token Claims Structure ​

Access tokens issued by Keycloak include:

json
{
  "sub": "user-uuid",
  "preferred_username": "admin",
  "email": "[email protected]",
  "realm_access": {
    "roles": ["admin", "editor", "operator", "approver"]
  },
  "resource_access": {
    "hegemony-api": {
      "roles": ["admin"]
    }
  },
  "groups": ["/Admins"],
    "aud": ["hegemony-api"],
    "iss": "http://localhost:8081/realms/hegemony",
  "exp": 1706918400,
  "iat": 1706914800,
  "auth_time": 1706914790
}

auth_time is when the user last signed in at Keycloak, not when this token was issued: a token renewed in the background keeps the sign-in's time. Step-up authentication reads it.


Authentication Flows ​

User Authentication (Frontend → API) ​

Worker Internal API Calls ​

Terraform/OpenTofu State Backend ​

OpenTofu and Terraform reach managed state with their http backend, which can only send HTTP Basic credentials. So the two backend paths, /tf-states/{name}/state and /tf-states/{name}/lock, also accept Authorization: Basic, with a token as the password (the username is ignored). Every other route refuses Basic, including POST /tf-states/{name}/import, which uploads a state file as a new state: a script sends its token there as Authorization: Bearer.

  • A personal access token works as on any route: its scopes are its roles in its organization.
  • A run step's token (hgm_run_…) is minted by the worker through POST /internal/runs/{run_id}/tf-state-access for one tf.* step. It acts for the run's organization with the editor role, only on the backend paths, and only for the one state it names. Only a digest is stored; it expires, is revoked when the step finishes, and stops working at most HEGEMONY_RUN_TOKEN_END_GRACE_SECONDS (600 by default) after the run ends, so a step still stopping can save its state. Audit entries record it as the run_token authentication method, naming the run and step.

Platform Registry ​

A container step pulls from, and caches into, the platform registry with the Docker client's own HTTP Basic login, turned into a bearer token on the way: the registry (zot) takes only tokens the API signs, and sends a client without one to GET /registry/token (policy custom, a route of the registry extension). That endpoint answers 404 while the extension is off (the web image's nginx refuses its public path as well); 503 while its signing key is missing; 401 with a Basic challenge when the credential is missing or is not a live run token with a registry scope; 400 for another registry's service name; and otherwise a signed token carrying the part of the requested scopes the step is entitled to. What a token leaves out, zot refuses with a fresh challenge. Only run tokens are accepted there, never a personal access token or a session.

  • The run token is minted when the worker asks for one container-backed step's access (POST /internal/runs/{run_id}/step-access), with the scope registry:step:<org slug>. The step may pull from its organization's namespace, the shared organization's and global, and may push only into its organization's own cache (orgs/<slug>/<host>/…). A step that builds an image (container.build) is minted the scope registry:build:<org slug> instead: the same, plus push on the organization's own image names (orgs/<slug>/<image>, a first segment that does not look like a host). Deletes and the catalogue are never granted whatever the scope.
  • The bearer tokens are Ed25519-signed JWTs (the Docker token specification's access claim) with a five-minute life, signed with the key behind HEGEMONY_REGISTRY_TOKEN_KEY; zot verifies them with the public key the API writes for it when the registry extension starts. The extension's Container Images pages sign themselves tokens for the catalogue and for deletes, the two scopes a step never gets, and image import and download sign theirs for pushing to or pulling from the one repository they write or read. Who may do that is decided on the API's routes (image:manage, image:manage_global), not by the grant policy above.
  • As with the state backend, only a digest of the run token is stored; it expires, is revoked when the step finishes, and stops working at most HEGEMONY_RUN_TOKEN_END_GRACE_SECONDS after the run ends. Pulls are reads and leave no audit entry; the step's run events record which image was served and from where.

Secrets Written by Steps ​

A step whose handler declares writes_secrets (secret.write) reads, writes and deletes secrets of its run's organization through the worker: POST /internal/runs/{run_id}/secrets/read, …/write and …/delete, with the internal token and the calling step run's id. No run token is minted and the step's container gets no credential. The API refuses the call unless the step run is running, its run has not ended, its handler declares writes_secrets, and the organization's setting allows the path: an organization admin turns secret writes by steps on and names the path prefixes (PUT /secrets/step-writes, action secret:step_writes, default policy admin); the shared organization never can. The API writes with its own backend credentials, so the worker's backend access stays read-only, and records each call as run:<run id>/<step id>.

Inventory Syncs Started by Steps ​

A step whose handler declares syncs_inventory (inventory.sync) asks the API to sync one of its run organization's inventory providers: POST /internal/runs/{run_id}/inventory/sync, with the internal token and the calling step run's id. Syncing a provider is otherwise inventory_provider:sync (default policy admin), and any editor can add a step to a flow, so the API refuses the call unless the step run is running, its run has not ended, its handler declares syncs_inventory, the provider belongs to the run's organization and is enabled, and an admin has turned on Flows may sync this provider for it (step_sync_enabled, set through PATCH /inventory/providers/{provider_id}, which is inventory_provider:manage). The sync is recorded with trigger flow and audited as run:<run id>/<step id>.

A step that stages a file on a target (netcli.file_download, shell.file_download) asks the API for a download link to one file-repository file: POST /internal/runs/{run_id}/file-links, with the internal token and the calling step run's id. The API refuses the call unless the step run is running and its run has not ended, and it finds only files the run's organization may read: its own, and those the shared organization publishes. A file anywhere else is answered like a file that does not exist. Reading a file is otherwise file:view (default policy viewer), which every member who can write a flow already holds, so a link gives a flow nothing its author could not download.

The link is GET /dl/{token}/{filename} under HEGEMONY_API_EXTERNAL_URL (action file:link_download, policy public): the token is the only credential, because the HTTP client of a network device sends no header and often handles neither a query string nor a redirect. The API streams the file, so a target never talks to the object store.

  • The token is a run token (hgm_run_…) with the scope file:<file id>. It opens that one file under its stored name, for at most an hour (ten minutes unless the step asks otherwise). A download that started in time runs to its end.
  • The worker revokes it when the step is done with it (DELETE /internal/runs/{run_id}/file-links/{token_id}), and the run's end ends it like any run token.
  • Run tokens are accepted as a bearer credential on the state backend and the registry token endpoint only, so the token of a link opens nothing but the link; a run token of another scope opens no link.
  • Only a digest of the token is stored. The worker adds the token to the step's redaction registry before the handler sees the link. The API masks it wherever it writes a request path down: its access log, the warning for a path no route matches, and the audit entry of a refused request (a link asked for with the wrong method is such a request). It records no trace span for a link, and answers with Cache-Control: private, no-store, so no cache answers in the token's place after the link has ended.
  • Issuing a link is audited as run:<run id>/<step id> reading the file. Every failure of the public route (an unknown, expired or revoked token, a wrong file name, a file deleted since, an object missing from the store, a repository that cannot be reached) answers the same 404; the reason is in the API's log.

SSH Host Keys Checked by Steps ​

Before a shell or network-CLI step logs in to a device or its jump host, the worker asks the API which SSH host keys the run's organization trusts for that host and port: POST /internal/runs/{run_id}/ssh-host-keys/trust, with the internal token and the calling step run's id. On first contact it fetches the host's key without logging in, stores it (…/learn), and then logs in accepting only that key. It also reports a host whose key did not match (…/mismatch). The API refuses all three unless the step run is running and its run has not ended, and takes the organization from the run. learn stores a key only while the host has none, unless an admin exempted the host: then it replaces the host's keys. mismatch keeps the key the host presented, one per host and port. Admins manage the keys under /ssh-host-keys (ssh_host_key:view, default policy viewer; ssh_host_key:manage, admin) and choose the mode with PUT /ssh-host-keys/policy (ssh_host_key:policy, admin), a separate action because one of the modes switches the check off. The same action exempts hosts (/ssh-host-keys/exemptions), which switches the check off for them. Operators accept the key a host was last refused for with POST /ssh-host-keys/accept (ssh_host_key:accept, operator). The route takes the key from that refusal, never from the caller, and in strict mode it also requires the caller to pass ssh_host_key:manage, resolved through the registry.

Trusted CAs Applied to Steps ​

Steps that verify TLS trust the worker's system store plus the trusted CAs of the run's organization. The worker asks for them with GET /internal/runs/{run_id}/trusted-cas and the internal token; the API takes the organization from the run, never from the caller, so a flow shared from the shared organization trusts the running organization's CAs and two organizations' lists are never combined. Admins manage the list under /trusted-cas (trusted_ca:view, default policy viewer; trusted_ca:manage, admin). Adding a CA lets servers whose certificates chain to it pass the organization's TLS checks, which is why changing the list is an admin action.

Role-Based Access Control ​

Hegemony uses a centralized permission registry for API authorization, declared in two files:

  • Action registry: apps/api/auth/actions.yaml — every action is a resource:verb key (flow:view, run:trigger, secret:manage) with a default policy and a description. Resources that hold a credential reference (git repositories, file repositories, registry credentials, secret backends, inventory providers, notification destinations) also carry a :test action for their connection or login probe, so the right to see a configuration and the right to make the API use its credential stay separate
  • Route map: apps/api/auth/routes.yaml — every API route belongs to exactly one action and inherits its policy. Every route is listed by its own path, including a fixed sub-path that a sibling's {param} wildcard would also match (GET /inventory/objects/{object_type}/resolve beside .../{object_id}), so the registry, its snapshot test and the generated reference name the route rather than relying on the wildcard
  • Loader and lookup: apps/api/auth/permissions.py flattens both files into one rule table (lookup_permission). Installed extensions add their own actions and routes to it at import, in the same shape, and may own organization-scoped prefixes (see multi-tenancy); an extension whose actions collide with existing ones, whose routes name actions it did not declare, or whose prefixes would change the scope of a route already declared, is refused and the platform's files load unchanged. The platform's own files are validated on their own first, so a mistake there is never blamed on an extension. An extension's web pages may call only routes it declared, so the web app gates their menu entries and buttons with the same GET /auth/capabilities answers as the platform's own pages (extension:list lets every signed-in user read which extensions are enabled, with their pages and menu)
  • Runtime enforcement: apps/api/auth/rbac_middleware.py
  • Runtime overrides: the permission_action_overrides and permission_overrides tables, edited under Settings → Permissions
  • OpenAPI auth metadata: enriched from the same registry in apps/api/main.py

This keeps runtime behavior and documentation aligned without duplicating auth rules in route decorators. The generated permissions reference lists every action with its default policy and routes.

Actions and Overrides ​

An action is the unit administrators allocate to roles. Its default policy comes from actions.yaml; at runtime two override layers can change it, both persisted in the database, audited, and cached in-process:

LayerTableSettings → Permissions tabScope
Action overridepermission_action_overridesActionsReallocates a whole action to a policy; every route of the action follows
Route overridepermission_overridesRoutesPins a single method + path to a policy; wins over the action (escape hatch)

The policy in effect for a request is resolved as route override → action override → action default. public, internal and custom policies are structural and cannot be overridden at either layer, permission:manage is pinned to its default so no override can hand the permission matrix to a lower role, and a target-org policy (org_member/org_admin) can only be assigned where every affected route carries an {org_ref} parameter. Both override layers round-trip through configuration exchange as the permission_action_overrides and permission_overrides bundle sections.

The disabled policy ​

One override target is not a role. Setting an action or a route to disabled switches it off, and the middleware then refuses every caller — platform admins included, and in dev mode (HEGEMONY_AUTH_DISABLED=true) too. It is override-only: no action may declare it as its default.

The way back is protected at both layers, so an administrator can always reach the editor and assign another policy:

  • permission:manage is pinned, so the editor's own routes accept no override at all.
  • auth:me — the identity and capability lookups the console loads before it renders anything — may be reallocated, but never disabled.

Like every override it is enforced from the in-process snapshot, so an instance that starts while the database is unreachable serves the registry defaults until its first successful refresh; a disabled endpoint answers normally for that window.

Every refusal by the kill switch is recorded (see Recording Refusals), so a route that stopped answering can be told apart from one that was switched off.

Handler-level narrowing ​

The middleware decides whether a request may reach a handler at all. A handler may still refuse a specific option within a route its caller legitimately holds, when that one option is more dangerous than the rest of the route:

RouteAction (policy)Narrowed to admin in the handler
POST/PUT /flows… (including the YAML tab's POST /flows/{id}/yaml/check and PUT /flows/{id}/yaml)flow:manage (editor)container.run host-access options and egress widening beyond the flow's network policy
/flows/{id}/attachments…, POST /flows/{id}/commitflow_attachment:manage, flow:commit (editor)Changing or publishing attachments a step with host-access options mounts
POST /runsrun:trigger (operator)?force=true, which starts a run of a flow that refuses manual runs

These are deliberately not separate actions: splitting the route would mean an editor who may create flows could not create ordinary ones, and the registry would grow an action per dangerous field. They check principal.has_role("admin") and raise 403 naming the restriction, so the refusal reads differently from the middleware's "you may not call this route".

The narrowing is a property of the handler, not of the registry, so it is not reallocatable through an override. An administrator who needs to hand one of these to a lower role changes the code, not the permission matrix.

Permission Matrix ​

RoleDescriptionInventory (CRUD)Build flowsRun flowsApprovalsView AllAudit Log (read)
adminFull administrative access✅✅✅✅✅✅
editorBuilds flows and runs them❌✅✅❌✅✅
operatorRuns flows and controls runs❌❌✅❌✅✅
approverReviews and approves requests❌❌❌✅✅❌
auditorRead-only with audit visibility❌❌❌❌✅✅
viewerBasic read-only access❌❌❌❌✅❌

Build flows is everything that changes what a flow does or when it starts: editing and committing flows and their attachments and notification subscriptions, git repositories and git sync, schedules and webhooks, listing the secrets a backend holds, and reading and writing managed Terraform state. Run flows is starting runs, running a schedule now, and cancelling, pausing and resuming runs, including moving the maintenance window of a waiting run and starting it before its window. The RBAC reference lists every action with its default policy.

The Audit Log column is the audit_log:view action (auditor policy, so operator, editor and admin hold it through the role hierarchy). It is read-only and organization-scoped: a member reads the entries of the active organization - listing and filtering them, opening one, its statistics, and an object's History card - and never another organization's or the platform's own. Only a platform admin reads every organization's entries, and only a platform admin runs the retention cleanup (audit_log:cleanup). There is no separate audit-log export. See Audit Log.

Centralized Enforcement Flow ​

Request-Scoped Audit Context ​

Once a request is admitted, the middleware binds a request-scoped audit context around the handler: the principal's subject and username, how they authenticated (jwt, pat, internal, dev_bypass, public), the client address, the user agent, and the route. Audit entries read their actor and client fields from it, so a handler records who acted without threading the principal through every call, and every entry written during the request agrees on the answer. The context is reset when the request finishes.

A route whose policy is public records a masked client address, because nothing authenticated the caller. The address is taken from the connection peer; set HEGEMONY_TRUST_PROXY_HEADERS when a trusted proxy appends to X-Forwarded-For, and the header is read from the right instead, because a caller can only add entries on the left: the right-most entry outside HEGEMONY_TRUSTED_PROXY_PEERS when that lists the proxies, otherwise the entry HEGEMONY_TRUSTED_PROXY_HOPS places from the right (Audit Log).

Recording Refusals ​

A refused request never reaches a handler, so the middleware is the only place that can record it. Three kinds of refusal get a row in the audit table:

RefusalEntryActor
Authorization failure (403): a known caller lacked the role, the org membership, or the platform standingauthz.deniedThe caller's principal
Authentication failure that still names an identity (401 on a revoked or expired personal access token)auth.failedThe token's owner
A route switched off by the disabled policy (403)authz.deniedThe caller's address, because the check runs before authentication

The audited object is the method and the matched route pattern (POST /orgs/{org_ref}/members), not the concrete path, so one route's history lists every refusal it produced. The recorded policy is the effective one, so a denial on an overridden route names the policy the override put there. An authentication failure that names nobody is log-only — there is no principal to attribute it to and any client can produce one at any rate — while the disabled-policy refusal above is recorded against the caller's address because the route itself is the object. Bursts are throttled — see Access decisions.

Policy Mapping Reference ​

The branch labels in the diagram are the runtime policy types (roles, target_org_roles, internal_token, …). Policy names in actions.yaml (and in overrides) map to them as follows:

PolicyRuntime behavior
publicNo authentication required
authenticatedAny valid JWT
viewer/auditor/operator/editor/approver/adminRole-based JWT checks (each role includes those below it: admin > editor > operator > auditor > viewer; approver is a separate branch under admin)
org_member/org_adminMembership in the organization named by the route's {org_ref} parameter (any role / the admin role); platform admins always pass
internalX-Internal-Token validation
customEndpoint handles auth explicitly
disabled403 for every caller; assignable only as an override

The API's own documentation routes are public (the system:docs action): Swagger UI at /docs, ReDoc at /redoc, the contract at /openapi.json, and /openapi.public.json, the same contract without the operations that only the platform's worker and UI call. They describe the API and return no data of the instance.

The release the instance runs is not public. GET /system/version (the system:version action, authenticated) returns the version, the commit and the build date of the image, which the UI shows in the user menu: a version number tells an anonymous visitor which known vulnerabilities apply. The liveness and readiness probes return no version. The contract at /openapi.json still carries the release number in info.version.

Route Scope: Org vs Platform ​

Every route classifies as org-scoped or platform-scoped (apps/api/auth/org_scope.py), and the classification changes what a role name means:

ClassificationHow it is decidedWho satisfies an admin policy
orgPath is under an ORG_PREFIXES entry, or an ORG_EXCEPTIONS entry inside a platform prefix (/inventory/objects and /inventory/object-types)An organization admin of the active org, or a platform admin
platformEverything elseOnly a platform admin (realm admin); an org-local admin PAT scope is stripped

Org-scoped routes defer the role decision until the active organization is resolved, then evaluate the caller's effective org roles. Platform-scoped routes evaluate realm roles directly.

This is a security boundary, not a formality: putting a route that reads across tenants under an org prefix would let any organization admin read every other tenant's data. Deployment-wide endpoints therefore live outside the org prefixes deliberately - GET /platform/health is separate from /dashboard/stats for exactly this reason, and the access review export (GET /platform/access-review), which lists every user, membership and token, is a platform route so that only a platform admin can download it. The same goes for the platform configuration (GET /settings/platform-configuration, action platform_configuration:view), which shows every deployment setting: /settings is a platform namespace, so an organization admin cannot read it.

The audit log is the one org-scoped surface whose rows the ORM tenancy guard does not filter (its org_id is nullable, for platform-level entries), so its router enforces the boundary itself: every GET /audit-logs… handler confines a caller who is not a platform admin to the active organization's entries (an entry of another organization answers 404, as if absent). Its retention cleanup, POST /audit-logs/cleanup, deletes across every organization and is therefore classified platform-scoped (PLATFORM_PATH_PATTERNS), so an organization admin cannot satisfy its admin policy.

A platform-scoped request still resolves the organization it names, but the middleware also records it in platform_request_org_id, and audit entries written on a platform route do not take their organization from it (see Multi-tenancy): a platform-wide operation stays a platform-level entry that only a platform admin reads.

Separation of Duties ​

Building flows, running them and approving them are three separate roles: editor builds flows (and may run them), operator runs them, and approver decides approval requests. Neither editor nor operator includes approver:

This ensures:

  • Editors and operators cannot self-approve their own workflow runs
  • Operators cannot change the flows they run, nor their schedules, webhooks or git sync
  • Approvers cannot create runs they then approve
  • Only admins have full access (for emergency situations)

Operating Modes ​

Hegemony supports different operating modes for development and production environments.

Mode Comparison ​

Development Mode Features ​

When HEGEMONY_AUTH_DISABLED=true:

  1. Mock Principal: All requests receive a mock admin principal with full roles
  2. No Keycloak Required: Application runs without Keycloak container
  3. BFF Tickets Disabled: Returns 400 error (tickets require real auth)
  4. Internal Token Optional: Worker calls allowed without X-Internal-Token
python
# Mock principal returned when auth disabled
Principal(
    sub="dev-user",
    username="dev",
    email="dev@localhost",
    realm_roles=["admin", "editor", "operator", "auditor"],
    client_roles=["admin"],
    groups=[],
    is_service_account=False,
)

Production Mode Enforcements ​

CheckFailure Behavior
HEGEMONY_AUTH_DISABLED=trueStartup fails with ValueError
HEGEMONY_INTERNAL_API_TOKEN not setStartup fails with ValueError
HEGEMONY_KEYCLOAK_ISSUER not setStartup fails with ValueError
Keycloak unreachable503 Service Unavailable on auth attempts

Dual Issuer Configuration (Docker Environments) ​

In Docker environments, internal and external URLs may differ:

Configuration:

bash
# Internal URL for JWKS fetch (API → Keycloak)
HEGEMONY_KEYCLOAK_ISSUER=http://keycloak:8081/realms/hegemony

# External URL for browser redirects (Browser → Keycloak)
HEGEMONY_KEYCLOAK_PUBLIC_ISSUER=http://localhost:8081/realms/hegemony

Security Features ​

Token Storage Strategy ​

Hegemony uses a hybrid approach to token storage:

Storage TypeUsed ForXSS RiskPersistence
sessionStorageJWT tokens for API calls⚠️ Readable by JSCleared on tab close
BFF ticketsSSE connections✅ Server-side onlySingle-use, 5-min TTL

XSS Mitigation:

  • sessionStorage clears tokens when browser/tab closes (vs localStorage which persists)
  • BFF tickets for SSE are already immune to XSS (server-side storage)
  • Token refresh handled server-side via OIDC silent renewal

Because sessionStorage belongs to one tab, a link opened in a new tab or window (a bookmark, NetBox's run links) starts without tokens and signs in again; Keycloak's SSO session makes that silent. The redirect URI is the UI's root, so the page the link named travels in the OIDC state of the sign-in request, and the callback returns to it through the router (apps/ui/src/auth/returnTo.ts). Only a path on the UI's own origin is followed: an absolute URL, a protocol-relative //host, a backslash or a control character falls back to /, so the state cannot be turned into an open redirect.

JWT Validation Security ​

The JWT validator (jwt_validator.py) implements:

Security FeatureImplementation
Algorithm restrictionOnly RS256 accepted
Signature verificationVia JWKS public keys
Issuer validationMust match HEGEMONY_KEYCLOAK_ISSUER or HEGEMONY_KEYCLOAK_PUBLIC_ISSUER
Audience validationMust include hegemony-api
Expiration checkWith 30-second clock skew tolerance
Required claimsexp, iat, sub must be present
JWKS cachingConfigurable TTL (default 1 hour)

Internal Token Security ​

Worker-to-API authentication uses constant-time comparison to prevent timing attacks:

python
import hmac
if not hmac.compare_digest(provided_token, settings.internal_api_token):
    raise HTTPException(status_code=401, detail="Invalid internal token")

Keycloak Security Features ​

The realm export and keycloak-init give the realm:

FeatureConfiguration
Brute force protectionfailureFactor: 5, lockout after 5 failures
Account lockoutMax 15-minute wait, not permanent
SSL requirementsslRequired: external
Registration disabledNo self-registration
Token lifespansAccess: 5 min, Session idle: 30 min, Session max: 10 hr
Password policy12 to 128 characters, not the username or email, none of the last 5
Multi-factor authenticationA one-time code at every sign-in for the admin realm role and hegemony-api client role

Password policy ​

The realm sets length(12) and maxLength(128) and notUsername and notEmail and passwordHistory(5). It follows current guidance (the US National Institute of Standards and Technology's SP 800-63B, and OWASP ASVS 2.1): length counts, so there are no rules about character classes and no forced expiry, and long passphrases fit.

  • Existing deployments. A realm import never changes a realm that already exists, so keycloak-init gives a realm with no password policy this one. A policy set in the Keycloak administration console (Authentication → Policies → Password policy) is kept as it is.
  • Existing passwords. Keycloak checks the policy when a password is set, so a password set before keeps working until it is next changed.
  • Breached passwords. Keycloak can also refuse passwords on a list of known breached ones (the Password blacklist policy, which reads a file from its data directory). The bundled realm ships no such list.

Multi-factor authentication ​

Users with the admin realm role, or the admin role of the hegemony-api client (the API takes either for an administrator), give a one-time code from an authenticator app (TOTP, a time-based one-time password, as FreeOTP, Google Authenticator or Microsoft Authenticator make) every time they sign in. At their first sign-in, Keycloak asks them to set the app up. Other users can set one up in the Account Console (Account security → Signing in), and from then on they give a code too.

keycloak-init builds a browser flow for this, hegemony browser, and makes it the realm's browser flow. It does so for new and existing realms alike: a realm import never changes a realm that already exists. The flow is Keycloak's built-in browser flow without Kerberos (off by default) and Organizations (the realm does not use them), with the one-time code in three parts:

text
hegemony browser
├── Cookie                                                    ALTERNATIVE
├── Identity Provider Redirector                              ALTERNATIVE
└── hegemony browser forms                                    ALTERNATIVE
    ├── Username Password Form                                REQUIRED
    ├── hegemony browser admin one-time code                  CONDITIONAL
    │   ├── Condition - user role: admin                      REQUIRED
    │   └── OTP Form                                          REQUIRED
    ├── hegemony browser hegemony-api admin one-time code     CONDITIONAL
    │   ├── Condition - user role: hegemony-api.admin         REQUIRED
    │   ├── Condition - user role: not admin                  REQUIRED
    │   └── OTP Form                                          REQUIRED
    └── hegemony browser optional one-time code               CONDITIONAL
        ├── Condition - user configured                       REQUIRED
        ├── Condition - user role: not admin                  REQUIRED
        ├── Condition - user role: not hegemony-api.admin     REQUIRED
        └── OTP Form                                          REQUIRED

The conditions in each part must all hold. Each part skips the roles an earlier part asked, so a user with both admin roles is asked for the code once. The hegemony-api part, and its condition in the last part, are there only when the realm has that client role, as the bundled realm does: a condition on a role the realm lacks never holds. The client is the one HEGEMONY_KEYCLOAK_AUDIENCE names.

  • When it starts. At each user's next sign-in. A session that is already open keeps going until it ends.
  • A lost authenticator. In the Keycloak administration console, open Users → the user → Credentials and delete the otp credential. At the next sign-in the user sets up a new one.
  • Your own changes. keycloak-init builds the flow once and then leaves it as it is, so changes you make under Authentication → Flows → hegemony browser stay. A realm whose browser flow is neither the built-in browser nor hegemony browser is left alone entirely.
  • An interrupted build. keycloak-init builds the flow as hegemony browser incomplete and names it hegemony browser only once it is whole. If it stops part-way, its next run deletes the incomplete flow and builds it again.
  • A code from everyone. In each Condition - user role step on the admin realm role, change the role to default-roles-hegemony, which every user of the realm has. The first part then asks everyone for a code, and the others, which skip users with that role, never run.
  • Security keys and passkeys. Keycloak's WebAuthn steps can ask for these instead of a code, or as well; see "WebAuthn" in Keycloak's Server Administration Guide.
  • An outside identity provider. A user who signs in through an identity provider (the Identity Provider Redirector) never reaches the Keycloak forms, so require multi-factor authentication at that provider.
  • Turning it off. HEGEMONY_ADMIN_MFA_REQUIRED=false makes keycloak-init set the built-in browser flow back at its next run. It keeps hegemony browser, and true uses it again. Do this only where a code from administrators is not wanted, such as a throwaway demo. Setting the built-in flow back in the administration console does not last: keycloak-init sets hegemony browser again at its next run.

Step-up Authentication ​

Some actions need more than a signed-in user with the right role. The user must also have signed in recently, so the action cannot be done from a session left open, or with a token taken from the browser. This is step-up authentication (the OAuth 2.0 Step Up Authentication Challenge Protocol, RFC 9470).

These routes need it:

ActionRoutes
secret_backend:manageCreate, change and delete a secret backend
permission:manageSet and clear an action or route override
org_member:manageAdd, change and remove an organization member
api_token:manageIssue a personal access token
platform_configuration:manageChange or reset a platform setting under Settings → Platform Configuration

The generated permissions reference marks each route. Reading, such as listing backends, tokens or the permission matrix, needs no fresh sign-in.

How it works ​

  1. The middleware checks the role first, so only a caller who may do the action is asked to sign in again.

  2. It then compares the token's auth_time with the clock. A sign-in older than HEGEMONY_STEP_UP_MAX_AGE_SECONDS (300 by default), a token with no auth_time, or one more than a minute in the future, is answered:

    text
    HTTP/1.1 401 Unauthorized
    WWW-Authenticate: Bearer error="insufficient_user_authentication",
        error_description="A more recent authentication is required", max_age="300"
    
    {"detail": "This action needs a fresh sign-in", "code": "step_up_required", "max_age": 300}
  3. The web app shows a Confirm it is you dialog. Its button opens Keycloak in a pop-up window, asking for a sign-in no older than max_age. The page stays as it was, so nothing typed is lost.

  4. Keycloak runs its browser flow again. With the bundled realm an administrator gives the password and the one-time code (see Multi-factor authentication).

  5. The web app repeats the request once with the new token. If that is refused too, the page shows the error; there is no second prompt.

The web app treats this 401 differently from an expired session: it never starts the ordinary sign-in redirect, and cancelling the dialog leaves the user signed in on the same page.

Tokens and service accounts ​

A personal access token or a service account (the hegemony-automation client) cannot sign in again. On these routes it is refused with 403, and the refusal is recorded in the audit log as authz.denied with the reason step_up_token. Set HEGEMONY_STEP_UP_ALLOW_TOKENS=true to let them through, for example when a pipeline manages secret backends. A stolen token then passes too, which is what the check is there to stop.

Choosing the actions ​

The routes above are the default, declared in apps/api/auth/actions.yaml (the step_up field). An administrator changes it per action under Settings → Permissions → Actions, in the Fresh sign-in column:

ChoiceEffect
DefaultWhat the registry declares for the action
RequiredEvery route of the action that changes something (any method but GET and HEAD) asks, and so do the routes the registry marks
Not requiredNo route of the action asks
  • The choice is stored in permission_step_up_overrides, audited (permission.step_up_override.created, .updated, .deleted), applied at once on the API instance that saved it and within 30 seconds on the others.
  • Changing it is itself a permission:manage route that asks for a fresh sign-in.
  • permission:manage cannot be switched: it guards this switch, and turning its check off would let a session left open turn off every other one. An action whose policy signs nobody in (public, internal, custom) cannot be switched either.
  • Required also refuses personal access tokens and service accounts on those routes (see below). Check what calls an action before switching it on: a pipeline, or a run step's token on managed state.
  • The choices travel in a configuration exchange bundle, as the permission_step_up_overrides section. An import applies an entry only if the caller could make the same change in the editor (the role for permission:manage), and rejects an action that may not be switched. The import asks for the fresh sign-in the editor asks for (see below).

What to know ​

  • What it proves. That the user signed in again within the limit. It does not prove a one-time code was used: that depends on the realm's browser flow. With HEGEMONY_ADMIN_MFA_REQUIRED=false, or for a user who is not a realm administrator and has set up no authenticator app (an organization admin can be one), the fresh sign-in is a password only.
  • The auth_time claim. The access token must carry it. The bundled realm's hegemony-ui client has a mapper for it, and keycloak-init adds the mapper to a realm that was imported before it existed (it runs each time the stack starts). A realm you manage yourself needs the same: a User Session Note mapper for the note AUTH_TIME, or Keycloak's built-in basic client scope. Without the claim every step-up action is refused, however recently the user signed in.
  • An outside identity provider. A user who signs in through one signs in again there. Whether it asks for a password or a code again is that provider's decision.
  • Overrides. Giving an action to another role under Settings → Permissions keeps the check. Which actions ask is a separate switch; see Choosing the actions.
  • Configuration exchange. A configuration exchange import or apply that changes organizations, members, IdP mappings, permission overrides, which actions ask for a fresh sign-in, platform settings, or that adds or changes a secret backend asks for the fresh sign-in the matching route asks for, platform admins included. The web app asks the user to sign in again and repeats the import, as on the page; a personal access token or a service account is refused (403) unless HEGEMONY_STEP_UP_ALLOW_TOKENS is set. A preview does not ask. Other sections do not ask.
  • Not in the audit log. The 401 is the ordinary first half of every step-up, so it is written to the API log only. The action itself is audited as before, once it goes ahead.
  • Pop-ups. The sign-in window opens from a click, which browsers allow. A browser set to block every pop-up needs an exception for the site.
  • Turning it off. HEGEMONY_STEP_UP_MAX_AGE_SECONDS=0. A very short limit (under a minute) can expire while the user is still signing in.
  • Dev mode. With HEGEMONY_AUTH_DISABLED=true there is no sign-in, and the check passes.

Failure Scenarios & Error Handling ​

Authentication Failure Flow ​

Error Response Catalog ​

HTTP StatusScenarioResponse BodyRecovery Action
401Missing token{"detail": "Missing authentication token"}Login via Keycloak
401Expired token{"detail": "Token expired"}Refresh token or re-login
401Invalid token{"detail": "Invalid or expired token"}Re-login
401Invalid internal token{"detail": "Invalid internal authentication token"}Check HEGEMONY_INTERNAL_API_TOKEN
401Sign-in too old for the action{"detail": "This action needs a fresh sign-in", "code": "step_up_required", "max_age": 300}Sign in again (step-up)
403Insufficient roles{"detail": "Insufficient permissions. Required role: X"}Request role assignment
403Token on a step-up route{"detail": "This action needs a fresh sign-in, which an access token or a service account cannot give"}Do it signed in, or set HEGEMONY_STEP_UP_ALLOW_TOKENS
503Keycloak unreachable{"detail": "Authentication service temporarily unavailable"}Retry after 30 seconds
503Auth not configured{"detail": "Authentication service not configured"}Check Keycloak config

Silent Token Renewal ​

The UI handles token refresh automatically:

API 401 Recovery ​

When the API returns 401, the UI handles it gracefully:


BFF Auth Gateway ​

The BFF (Backend-for-Frontend) auth gateway handles SSE/EventSource connections where browsers cannot use Authorization headers.

The gateway issues single-use, server-side tickets for SSE/EventSource connections (implementation: routers/bff.py). Regular API calls authenticate with a Bearer token.

BFF Ticket Flow ​

Ticket Security Properties ​

  • Cryptographically random: 256-bit tokens via secrets.token_urlsafe(32)
  • Short-lived: 5-minute TTL (TICKET_TTL_SECONDS in routers/bff.py; not a setting)
  • Single-use: Tickets are consumed on first validation
  • Rate-limited: Max 10 active tickets per user

Rate Limit Behavior ​

When a user already holds 10 active (unexpired, unused) tickets:

  • POST /auth/bff/ticket answers HTTP 429 (Too many active tickets. Maximum: 10) and issues no ticket
  • No existing ticket is evicted; the ones the user holds stay valid
  • A slot frees up when a ticket is used or expires (5 minutes at most), so the client can retry

The check and the store happen in one step, so concurrent requests cannot push a user past the limit.

Endpoints ​

EndpointMethodDescription
/auth/bff/ticketPOSTExchange JWT for session ticket
/auth/bff/validateGETValidate and consume ticket

Internal API Protection ​

Secure-by-Default Model ​

Internal endpoints are protected by X-Internal-Token header validation:

Router Separation ​

Internal and public monitor endpoints are separated into distinct routers:

RouterPath PrefixAuthPurpose
monitors.py/runs/{id}/monitorsviewer policy via its action in apps/api/auth/routes.yaml (enforced by RBACMiddleware)UI queries
monitors_internal.py/api/v1/monitorsinternal policy via its action in apps/api/auth/routes.yaml (enforced by RBACMiddleware)Worker ingestion

External Identity Provider Integration ​

Keycloak supports federation with external identity providers, enabling enterprise SSO scenarios.

Supported Federation Methods ​

LDAP/Active Directory Integration ​

For enterprise environments using LDAP or Active Directory:

Configuration (Keycloak Admin Console):

  1. Navigate to User Federation → Add Provider → LDAP
  2. Configure connection settings (URL, bind DN, bind credential)
  3. Set up Mappers for:
    • Username (e.g., sAMAccountName)
    • Email
    • Groups → Realm roles mapping

SAML 2.0 Identity Provider ​

For SAML-based SSO with enterprise identity providers:

Configuration:

  1. Navigate to Identity Providers → Add Provider → SAML v2.0
  2. Import metadata from enterprise IdP
  3. Configure Mappers for attribute-to-role mapping
  4. Set up First Login Flow for user provisioning

Social Login Integration ​

For development or external collaborator access:

ProviderConfigurationUse Case
GitHubOAuth2 App credentialsDeveloper access
GoogleOAuth2 client ID/secretCorporate Google Workspace
MicrosoftAzure AD app registrationMicrosoft 365 organizations

Role Mapping from External IdPs ​

Keycloak can map external IdP attributes to Hegemony roles:


Configuration Reference ​

Complete Environment Variables ​

VariableRequiredDefaultDescription
Core Auth
HEGEMONY_KEYCLOAK_ISSUERYes*-Keycloak realm URL for API (e.g., http://keycloak:8081/realms/hegemony)
HEGEMONY_KEYCLOAK_PUBLIC_ISSUERNo-Keycloak realm URL for browser redirects (if different from internal)
HEGEMONY_KEYCLOAK_AUDIENCENohegemony-apiExpected audience claim in access tokens
HEGEMONY_KEYCLOAK_UI_CLIENT_IDNohegemony-uiOIDC client ID for the UI
HEGEMONY_KEYCLOAK_JWKS_CACHE_TTLNo3600JWKS cache TTL in seconds
Auth Control
HEGEMONY_AUTH_DISABLEDNofalseDisable auth for dev (blocked in prod)
HEGEMONY_ENVIRONMENTNodevEnvironment (dev, prod, production)
HEGEMONY_STEP_UP_MAX_AGE_SECONDSNo300Oldest sign-in accepted on step-up routes; 0 turns the check off
HEGEMONY_STEP_UP_ALLOW_TOKENSNofalseLet personal access tokens and service accounts call step-up routes
Internal API
HEGEMONY_INTERNAL_API_TOKENProd: Yes-Shared secret for worker→API calls
HEGEMONY_INTERNAL_API_TOKEN_OPTIONALNofalseAllow missing token in dev

*Required when HEGEMONY_AUTH_DISABLED=false

keycloak-init Variables ​

The API does not call Keycloak's admin API; there is no user sync job. The bundled keycloak-init container (deploy/compose/keycloak/keycloak-init.sh) configures the realm after import and reads:

VariableRequiredDescription
KEYCLOAK_URLNoInternal Keycloak base URL (default http://keycloak:8081)
KEYCLOAK_ADMINYesKeycloak admin username
KEYCLOAK_ADMIN_PASSWORDYesKeycloak admin password
KC_REALM_NAMENoRealm name (default hegemony)
HEGEMONY_UI_PUBLIC_URLYesPublic UI URL, for the hegemony-ui redirect URIs and web origins
KEYCLOAK_IMPORTED_USERSNoSpace-separated usernames from the realm import that get the realm's default roles
HEGEMONY_ADMIN_MFA_REQUIREDNofalse stops requiring a one-time code from the admin role (Multi-factor authentication); default true
HEGEMONY_KEYCLOAK_AUDIENCENoThe API's client, whose admin role also needs the one-time code; default hegemony-api, as for the API

Client Configuration Summary ​

Client IDTypeFlowPurpose
hegemony-uiPublicAuthorization Code + PKCEReact SPA authentication
hegemony-apiBearer-only-Access token audience target
hegemony-automationConfidentialClient CredentialsCI/CD and automation service accounts

RBAC Database Schema ​

Hegemony stores RBAC data in the following tables:

users ​

Local user profiles, taken from the token's claims: created on first login, with username and email updated on later logins:

ColumnTypeDescription
idUUIDPrimary key
subVARCHAR(255)Keycloak subject ID (unique, indexed)
usernameVARCHAR(255)Username (indexed)
emailVARCHAR(255)Email address
display_nameVARCHAR(255)Display name
is_activeBOOLEANAccount active status
created_atTIMESTAMPTZCreation timestamp
updated_atTIMESTAMPTZLast update timestamp
last_login_atTIMESTAMPTZLast login timestamp

organizations ​

Multi-tenant organization boundaries. These tables are actively enforced — see Multi-Tenancy for the full model (header contract, per-org RBAC, scope matrix, and integrity rules).

ColumnTypeDescription
idUUIDPrimary key
slugVARCHAR(100)URL-safe identifier (unique, indexed, immutable)
nameVARCHAR(255)Organization name
descriptionTEXTDescription
is_activeBOOLEANOrganization active status
created_atTIMESTAMPTZCreation timestamp
updated_atTIMESTAMPTZLast update timestamp
settings_jsonJSONBOrganization-specific settings

org_memberships ​

User-organization role bindings:

ColumnTypeDescription
idUUIDPrimary key
user_idUUIDFK to users.id (CASCADE delete)
org_idUUIDFK to organizations.id (CASCADE delete)
roleVARCHAR(50)Role within organization (default: viewer)
created_atTIMESTAMPTZCreation timestamp
updated_atTIMESTAMPTZLast update timestamp

Constraints: Unique constraint on (user_id, org_id) - one membership per user per org.

audit_logs ​

Security event logging:

ColumnTypeDescription
idUUIDPrimary key
org_idUUIDFK to organizations.id (nullable; NULL for platform-level events). Stamped from the active-org context
tsTIMESTAMPTZEvent timestamp (indexed)
event_typeVARCHAR(100)Event type (indexed)
actor_subVARCHAR(255)Actor's Keycloak subject ID (indexed)
actor_usernameVARCHAR(255)Actor's username
resource_typeVARCHAR(100)Type of resource affected
resource_idVARCHAR(255)ID of resource affected
actionVARCHAR(100)Action performed
outcomeVARCHAR(50)Success/failure outcome
details_jsonJSONBAdditional event details
client_ipVARCHAR(45)Client IP address
user_agentTEXTClient user agent

User Provisioning ​

There is no sync job and nothing to configure for one. The current implementation uses token-based RBAC. Users are provisioned on first authenticated request; new users with no membership auto-join the default org when HEGEMONY_ORG_AUTO_JOIN=true (the default). See Multi-Tenancy for org resolution and per-org role semantics.


Quick Reference ​

Auth Flow Cheat Sheet ​

ScenarioAuthentication MethodToken/Credential
User → UI → APIBearer TokenKeycloak JWT
Worker → APIX-Internal-TokenShared secret
SSE/EventSourceQuery param ticketBFF session ticket
CI/CD → APIBearer Tokenhegemony-automation client credentials
Dev modeMock PrincipalNo token required

Common Troubleshooting ​

SymptomLikely CauseSolution
401 on all requestsToken expired or invalidRe-login, check token expiry
503 "Auth service unavailable"Keycloak unreachableCheck Keycloak container, network
403 on specific endpointMissing required roleCheck user's Keycloak roles
"Confirm it is you" appears again after signing in, or the action fails with "needs a fresh sign-in"The access token carries no auth_time, or the identity provider did not ask the user to sign in againCheck the token's claims; see Step-up authentication
403 "needs a fresh sign-in" for a scriptA token or service account called a step-up routeDo it signed in, or set HEGEMONY_STEP_UP_ALLOW_TOKENS=true
Startup fails in prodMissing HEGEMONY_INTERNAL_API_TOKENSet the environment variable
SSE connection failsInvalid ticketGet fresh ticket from /auth/bff/ticket
Token issuer mismatchDocker network vs browser URLConfigure HEGEMONY_KEYCLOAK_PUBLIC_ISSUER

Glossary ​

TermFull NameDescription
ABACAttribute-Based Access ControlAuthorization model that evaluates attributes (user, resource, environment) to make access decisions. More granular than RBAC.
ADActive DirectoryMicrosoft's directory service for Windows domain networks, commonly used for enterprise user/group management.
APIApplication Programming InterfaceSet of protocols and tools for building software applications. In this context, the FastAPI backend.
ASVSApplication Security Verification StandardOWASP's list of security requirements for web applications, used here for the password policy.
BFFBackend-for-FrontendArchitecture pattern where a dedicated backend handles auth for a specific frontend, keeping tokens server-side.
CI/CDContinuous Integration / Continuous DeploymentAutomation practices for building, testing, and deploying code changes.
CORSCross-Origin Resource SharingHTTP mechanism that allows a server to indicate allowed origins for cross-origin requests.
CRUDCreate, Read, Update, DeleteThe four basic operations of persistent storage.
IdPIdentity ProviderService that creates, maintains, and manages identity information (e.g., Keycloak, Okta, Azure AD).
JWTJSON Web TokenCompact, URL-safe token format for securely transmitting claims between parties. Signed with RS256 in this system.
JWKSJSON Web Key SetA set of public keys used to verify JWT signatures. Fetched from Keycloak's /protocol/openid-connect/certs endpoint.
LDAPLightweight Directory Access ProtocolProtocol for accessing and maintaining distributed directory information services (user directories).
OAuth2Open Authorization 2.0Industry-standard authorization framework for token-based access delegation.
OIDCOpenID ConnectIdentity layer built on top of OAuth2, adding authentication and user identity claims.
OWASPOpen Worldwide Application Security ProjectNon-profit that publishes web security guidance, among it the ASVS.
PKCEProof Key for Code ExchangeOAuth2 extension for public clients (SPAs) that prevents authorization code interception attacks. Pronounced "pixy".
RBACRole-Based Access ControlAuthorization model where permissions are assigned to roles, and users are assigned roles.
RS256RSA Signature with SHA-256Asymmetric signing algorithm using RSA keys. Keycloak signs JWTs with private key; API verifies with public key.
SAMLSecurity Assertion Markup LanguageXML-based standard for exchanging authentication and authorization data between IdPs and service providers.
SPASingle-Page ApplicationWeb app that loads a single HTML page and dynamically updates content. The React UI is an SPA.
SSEServer-Sent EventsHTTP-based protocol for servers to push real-time updates to clients. Used for live monitor streaming.
SSOSingle Sign-OnAuthentication scheme allowing users to log in once and access multiple applications.
TLSTransport Layer SecurityCryptographic protocol for secure communication over networks (HTTPS).
TTLTime To LiveDuration for which a token, ticket, or cached item remains valid before expiration.
UUIDUniversally Unique Identifier128-bit identifier standard (e.g., 550e8400-e29b-41d4-a716-446655440000). Used for database primary keys.
XSSCross-Site ScriptingSecurity vulnerability where attackers inject malicious scripts into web pages viewed by other users.

Keycloak-Specific Terms ​

TermDescription
RealmA Keycloak namespace that manages a set of users, credentials, roles, and groups. Hegemony uses the hegemony realm.
ClientAn application or service that can request authentication. Hegemony has three: hegemony-ui, hegemony-api, hegemony-automation.
Realm RolesRoles defined at the realm level, shared across all clients. Primary authorization mechanism in Hegemony.
Client RolesRoles scoped to a specific client. Reserved for future fine-grained API permissions.
Protocol MapperConfigures how user attributes and roles are mapped into token claims.
Identity BrokeringKeycloak feature that delegates authentication to external identity providers (SAML, OIDC, social logins).
User FederationKeycloak feature that syncs users from external directories (LDAP, Active Directory).
Bearer-only ClientA client that only validates tokens and never initiates authentication flows. The hegemony-api client is bearer-only.
Public ClientA client that cannot securely store secrets (e.g., SPAs). Uses PKCE for security. The hegemony-ui client is public.
Confidential ClientA client that can securely store secrets (e.g., backend services). The hegemony-automation client is confidential.

Released as open source under the AGPL-3.0-or-later license. Development is sponsored by Rexonix s.r.o. Contact — [email protected].