Architecture Overview
Table of Contents
- Overview
- System Architecture
- Component Details
- Data Flow
- Database Schema
- Flow Engine
- Schedule Automation
- Git Repository Integration
- API Endpoints
- Deployment Architecture
- Security Considerations
Overview
Hegemony is a proof-of-execution platform for deterministic, long-running infrastructure workflows. Built on Temporal.io with a declarative Flow Engine, it provides:
- Declarative Workflows: YAML/JSON-defined flows with pluggable handlers
- Deterministic Execution: Reliable, resumable workflow execution
- Real-time Updates: Server-Sent Events (SSE) for live progress tracking
- Audit Trail: Persistent event logging for compliance and debugging
- Clean Separation: Workflow orchestration decoupled from execution logic
Core Principles
System Architecture
High-Level Architecture
Component Interaction Sequence
Component Details
Frontend (UI)
| Aspect | Details |
|---|---|
| Framework | React 18 + TypeScript |
| Build Tool | Vite |
| Routing | React Router DOM |
| State Management | React hooks + fetch API |
UI Pages
Runtime env-key registry: the worker self-registers the names (never values) of HEGEMONY_* environment variables it can see by POSTing to /internal/runtime/env-keys on startup and on every down -> up reconnect, and clears them via DELETE on graceful shutdown. Operators can view the merged API + worker view at Settings -> Environment Variables (admin-only, gated centrally by the RBAC registry in apps/api/auth/actions.yaml + routes.yaml). No operator action is required to register keys — they are discovered automatically from process environment.
Backend (API)
| Aspect | Details |
|---|---|
| Framework | FastAPI (async) |
| ORM | SQLAlchemy 2.0 (async) |
| Database | PostgreSQL 15 |
| Migrations | Alembic |
API Structure
Worker
| Aspect | Details |
|---|---|
| Framework | Temporal Python SDK |
| Transport | SSH (Netmiko/Paramiko) |
| Pattern | Handler-based execution |
Activity Types
Container Handler (Docker-in-Docker)
The container.run handler enables running arbitrary Docker containers as flow steps. This supports use cases like firmware image preparation, config generation scripts, and custom tooling that cannot run directly on the worker.
Architecture:
- Flow containers run on a dedicated privileged
docker:dindsandbox service (deploy/compose/docker-compose.dind.yml), never on the host Docker daemon.HEGEMONY_CONTAINER_DOCKER_HOSTnames it (tcp://dind:2375, set by that file), and every docker invocation the handler and the startup janitor spawn carries it as a per-invocationDOCKER_HOST, so the whole flow-container estate (step containers, demo lab, Gitea, flow-built images) lives inside the sandbox. There is no host-socket mode: the worker refuses to start when the setting or the sandbox token is empty, or when the daemon that answers is not the attested sandbox, and each step refuses an empty setting on its own. A sandbox that is merely slow to come up gets a bounded wait (HEGEMONY_CONTAINER_DOCKER_WAIT_SECONDS), after which the worker starts degraded and keeps probing in the background: every container step is refused until the sandbox answers and is attested, so no step runs on a daemon the worker has not checked, and the startup janitor sweeps only a daemon that has passed (after a degraded start, once it passes and before container steps are enabled). Until then the worker's heartbeat does not advertise thecontainerscapability (a starting worker withdraws the one its previous process advertised before it waits), so shared-worker assignment pins no newsharedorexplicitrun to it. A run already pinned to it keeps that pin across a restart, and its container steps are refused there until the sandbox is attested. A late sandbox that fails attestation stops the worker with a non-zero exit, as it would have at startup; a token changed afterdind-initstamped the marker fails this way, and Production Hardening gives the recovery. A wait of0only skips the blocking wait: the worker still probes once, and attests or refuses a daemon that answers before it connects to Temporal. See Container Sandbox. - The worker still mounts the host socket (
/var/run/docker.sock), for one query only: deriving its stable worker id from its Compose container name is a question only the daemon hosting the worker can answer, so that query never followsHEGEMONY_CONTAINER_DOCKER_HOST. No flow container can reach the host socket. - Container images are pulled on-demand; a configurable allow-list restricts which images may be used
- Flow attachments are mounted into the container at a configurable path (default:
/workspace) - An optional
entrypointoverride maps to Docker--entrypoint; when omitted, the image entrypoint/CMD behavior is preserved, andcommandremains shell-script lines joined underset -e - Container stdout/stderr is captured as run artifacts; exit code determines step success/failure
- Containers are cleaned up after execution (removed); orphan cleanup runs on worker startup
Container marking: every container a flow creates is attributable to its run. Names are prefixed with the run id (hegemony-<run_id[:8]>-<step_run_id[:8]>-a<attempt>), and labels carry the full ids plus the deployment stack: hegemony.run_id, hegemony.step_run_id, hegemony.handler_id, and hegemony.stack (the compose project name — hegemony-dev/hegemony-prod/hegemony-demo — from the worker's HEGEMONY_STACK_NAME env var). Anything that needs to find flow containers filters on hegemony.run_id; cleanup also scopes by hegemony.stack, so a janitor never touches another stack's containers even on a daemon they share:
- The worker's startup janitor removes its own stack's orphaned non-running containers older than an hour (younger ones may belong to a step another worker replica is still provisioning or harvesting; other stacks' containers are never touched; unlabeled legacy ones are still cleaned).
- The demo's
task resetremoves its containers and volumes, the sandbox's volume with them, so the flow containers inside it go too.
# Flow containers live inside the stack's sandbox daemon; <stack> is the
# compose project name (hegemony-dev, hegemony-prod, hegemony-demo).
docker exec <stack>-dind-1 docker ps -a --filter "label=hegemony.run_id"
docker exec <stack>-dind-1 docker ps -a --filter "label=hegemony.stack=hegemony-prod"
# The same from deploy/compose, without naming the container:
ENV=prod ./dc.sh exec dind docker ps -a --filter "label=hegemony.run_id"docker ps on the host daemon shows no flow containers.
Deployment:
- The sandbox is part of every stack:
dc.shand the compose tasks applydeploy/compose/docker-compose.dind.ymlwhateverSERVICESnames,coreincluded. It adds thedind,dind-initandegress-nflogservices, anddown -vremoves the entire flow estate with itsdind-datavolume - The host must allow privileged containers and IPv6 bridge networks; see production hardening
- Requires
DOCKER_GIDset to the host docker group GID, for the worker's host-socket mount (worker-id derivation only) - The worker Dockerfile installs Docker CLI (pinned version) with its buildx plugin (for
container.build), which reaches the sandbox over TCP and the host daemon only to derive the worker id
Data Flow
Flow Execution Flow
Event Emission Flow
Database Schema
Entity Relationship Diagram
Core Enumerations
Flow Engine
The Flow Engine is the core execution system for Hegemony. It provides declarative workflow execution.
Architecture Overview
Step Execution Lifecycle
Handler Registry Pattern
Handlers are provided by out-of-tree plugin wheels (repo hegemony-step-plugins, one claimed namespace prefix per wheel) that register handler classes under the hegemony.step_handlers entry-point group. The handler contract lives in the hegemony-step-sdk package; the host injects a HandlerServices implementation (ctx.services) that provides transports, secrets, notifications, and artifact operations. Every step handler ships as a plugin wheel, monitor.* included (hegemony-steps-monitor).
Flow Definition Structure
# Example flow definition (V2 graph format)
start_node_id: start
nodes:
- id: start
type: start
name: "Start"
- id: precheck
type: step
name: "Pre-upgrade validation"
tags: {phase: PREPARE, kind: CHECK}
handler: netcli.collect_evidence
params:
commands:
- "show version"
- "show inventory"
- id: backup_config
type: step
name: "Backup running config"
tags: {phase: PREPARE, kind: ACTION}
handler: netcli.execute
params:
commands:
- "copy running-config startup-config"
- id: end
type: terminal
name: "Done"
edges:
- source: start
source_outcome: start
target: precheck
- source: precheck
source_outcome: success
target: backup_config
- source: backup_config
source_outcome: success
target: endEdges carry no id — an edge is identified by its (source, source_outcome, target, type) combination. Compact exports merge source and source_outcome into source: "node:outcome".
A step's tags — phase and kind among them — are entries of its tags map and nothing else: FlowGraphNode has no phase, kind or labels field and rejects a node that carries one. Only the bundle/git importer (deserialize_flow_definition in apps/api/services/flow_serializer.py) still reads that retired shape, folding it into tags (explicit tags win, then labels, then phase/kind) alongside its other upgrades of older files, and git pull compares an incoming file in that upgraded shape (normalize_for_comparison) so an old-shape file is not a phantom change; migration 053_step_run_tags rewrote the definitions already stored in flow_definitions and flow_definition_versions the same way, once. Templates read them as {{ tags.phase }} / {{ tags.kind }}. Handlers receive the map as ctx.tags and nothing else: HandlerContext has no phase or kind field (hegemony-step-plugins#20), so a handler reads ctx.tags.get("phase", ""). The worker stamps run_events.phase / run_artifacts.phase from the same tag.
An edge may also carry an optional waypoints list (up to 20 {x, y} points in absolute canvas coordinates) that pins where the editor draws it:
edges:
- source: "backup_config:success"
target: end
waypoints:
- x: 840
y: 420
- x: 150
y: 420Waypoints are purely visual — routing behavior never depends on them. In the editor, drag anywhere on a selected edge to drop a routing point (snapped to the canvas grid), drag a handle to move it, double-click a handle to remove it, or use the edge's reset button to return to automatic routing. Edges without waypoints route automatically, and manual routes render identically in the editor, the run view, and version previews. Absent and null both mean automatic routing; unset waypoints are omitted from exports. Dragging both ends of an edge together (a group or loop with its contents, or a multi-node selection) shifts its waypoints by the same amount; dragging one end alone leaves them where they are.
A node may also move its ports, its input and its outputs, to any of the 32 compass points on its outline (N, NbE, NNE, NEbN, NE, NEbE, … NbW, the PortAnchor enum). port_anchors maps each moved port to its point: in for the input, termination for a monitor's termination input, and the outcome name for an output (success, failure, case:<id>, …):
nodes:
- id: deploy
type: step
handler: cli.run
port_anchors:
failure: S # the failure port in the middle of the bottom side
in: NW # the input at the top-left cornerNE, SE, SW and NW are the node's corners, where edges leave the port diagonally; the seven points between two corners are spread evenly along that side, and edges leave them straight out of it. The points follow the node's real outline: rounded corners and the round ends of start, end and loop-end pills bend them onto the curve, and on the fork and join diamonds N, E, S and W are the tips and the four corner points sit in the middle of their edges. Every edge on a moved port attaches there, and a port that is not listed stays in its usual place. A loop's iterate port stays inside the loop frame: a point stored for it is ignored, as is a name the node has no port for. In the editor, select a node and drag the grip next to a port to snap the port to the nearest free point; drop it back on its usual place, or double-click the grip, to put it back. Moved ports are purely visual, absent and null mean every port in its usual place, and unset port_anchors are omitted from exports.
The graph may also carry annotations: canvas notes with Markdown text, a color, a position and size, and an optional scope_id naming the group or loop they sit in.
graph:
annotations:
- id: note
kind: text
text: Runs **only** in the maintenance window.
color: yellow
position_x: 400
position_y: 100
width: 240
height: 120Notes are purely visual too. They are versioned and exported with the graph, but create_run_from_request sends the engine the graph without them, so they never reach the workflow input or the Temporal history. A flow without notes exports no annotations key.
Schedule Automation
Overview
Hegemony supports automated workflow execution through the Schedules feature. Schedules enable users to trigger flow runs at specific times or on recurring intervals, providing a powerful automation layer for routine infrastructure tasks.
Schedule Types
One-Time Schedules
- Execute a flow once at a specified date and time
- Status automatically transitions to
COMPLETEDafter execution - Use cases: maintenance windows, one-off upgrades, planned changes
Recurring Schedules
- Execute a flow repeatedly based on a cadence
- Two cadence options:
- Interval: Run every N seconds (e.g., every 3600 seconds = hourly)
- Cron: Run based on cron expression (e.g.,
0 2 * * *= daily at 2 AM)
- Remain
ACTIVEuntil explicitly disabled - Use cases: daily health checks, weekly backups, continuous monitoring
Schedule States
Scheduler Architecture
Key Components
Scheduler Service (
apps/scheduler/run.py+apps/scheduler/scheduler.py)- Dedicated service running independently from the Temporal worker
- Polls for due schedules every
HEGEMONY_SCHEDULE_POLL_INTERVAL_SECONDS(default: 15) - Triggers schedules via internal API endpoints
- Continues on errors to ensure fault tolerance
- Deployed as separate container in docker-compose (dedicated scheduler image)
Schedule Service (
apps/api/services/schedules.py)compute_next_run_at(): Calculates next execution time based on type/cadencetrigger_schedule_run(): Creates a new run from schedule parameters- Handles timezone conversions for cron expressions
Schedule API (
apps/api/routers/schedules.py)- Public CRUD endpoints:
/schedules - Trigger endpoint:
/schedules/{id}/run(manual trigger) - Validates schedule parameters (e.g., one-time must be in future)
- Public CRUD endpoints:
Internal API (
apps/api/routers/internal.py)/internal/schedules/due: Lists schedules ready to execute/internal/schedules/{id}/trigger: Triggers schedule (worker-only)- Prevents duplicate triggers with status checks
Schedule Parameters
Schedules support full flow parameterization:
- Flow + Version: Pin to specific flow version or use latest committed
- Inputs: Flow input parameters (e.g.,
target_version: "17.9.4") - Targets: Device role mappings for multi-device flows
- Timezone: Affects cron expression evaluation (e.g.,
America/New_York)
UI Integration
The Schedules page (/schedules) provides:
- Overview cards: One-time vs recurring schedule concepts
- Tabs: Upcoming schedules (active only) and all schedules
- Create form: Flow selection, schedule type, cadence, timezone, params
- Actions: Run now, pause/resume, delete
- Status badges: Visual indicators for active/disabled/completed
Git Repository Integration
Git Integration connects flows to external Git repositories, enabling version control, code review, and CI-driven workflows for flow definitions and attachments.
Architecture
All Git operations run inside the API process — workers never interact with Git directly. This keeps the trust boundary tight and avoids distributing credentials to the execution layer.
UI ──► API ──► local git CLI ──► Remote Git (HTTPS / SSH)
│
└──► PostgreSQL (git_repositories, git_sync_history,
flow git linkage columns)Key Components
| Component | Location | Responsibility |
|---|---|---|
| Git Repositories CRUD | apps/api/routers/git_repositories.py | Registry of remote repos, test connection |
| Git Operations | apps/api/services/git_ops.py | Clone, fetch, commit, push via subprocess; semaphore(4) concurrency limit |
| Exploded Format | apps/api/services/sync/serialization/layouts/exploded.py | Read/write flows as directory tree (flow.yaml + attachments/). Symlink & traversal guards |
| Pull Sync | apps/api/services/sync/flow_git/pull.py | Pull flow from Git into DB draft; dirty-draft conflict detection |
| Push Sync | apps/api/services/sync/flow_git/push.py | Push committed flow to Git; stale-remote detection with fast-forward |
| Async Dispatcher | apps/api/services/sync/flow_git/queue.py | Background loop that processes queued push jobs from git_sync_history |
| URL Validation | apps/api/services/git_ops.py | SSRF prevention — rejects private/loopback addresses |
Sync Modes
Flows are linked to a Git path via sync_mode:
synced— bidirectional: auto-push on commit, pull available.readonly— pull-only: flow is read from Git, local edits blocked.detached— linkage preserved but no sync occurs.
Data Model
git_repositories— registered remotes (name, URL, branch, credential ref).flow_definitionsgit columns —git_repo_id,git_path,git_branch,sync_mode,last_synced_sha,last_synced_at.flow_attachmentsgit columns —git_repo_id,git_file_path,last_synced_sha,last_synced_at.git_sync_history— audit log of every sync operation (also serves as the async push queue).
Configuration Exchange migration notes
Configuration Exchange and flow Git sync are opt-in additions to the existing application database. Operators should apply the normal Alembic migration chain before enabling scheduled sync or API-driven push jobs; migrations are never run implicitly at process startup.
The platform-sync migrations add Git linkage and operation tracking state:
flow_definitions:git_repo_id,git_path,git_branch,sync_mode,auto_commit,last_synced_sha,last_synced_at.flow_attachments:git_repo_id,git_file_path,last_synced_sha,last_synced_at.git_repositories: connection/health fields used by sync checks, includingsync_stale_secondsandlast_connected_at.git_sync_history: durable operation history and queued async push work.config_exchange_locksplus the platform-sync profile and run tables: layout ownership, leases, and export history. Platform sync only exports; migration080_platform_sync_export_onlydropped the plan, resource-state and destructive-authorization tables that Git-to-platform apply used.
Backfills normalize existing flow Git paths as file paths (no trailing slash) and profile base paths as directories (with trailing slash). Existing rows that have no Git linkage remain detached; there is no automatic remote bootstrap. After migration, create or update Git repositories and platform-sync profiles through the admin API/UI, then trigger a manual export before enabling scheduled operation.
Relevant toggles are standard HEGEMONY_* environment variables. Keep HEGEMONY_MAINTENANCE_ENABLED and the platform-sync maintenance interval settings aligned with operational intent: scheduled platform exports are driven by the maintenance subsystem, while manual Configuration Exchange API calls work as soon as the database migration and Git repository configuration are present.
Async Push Flow
1. User commits flow (sync_mode=synced)
2. API enqueues git_sync_history row (status=queued, direction=push)
3. Dispatcher loop claims row → runs git push
4. Row updated to success/failedAPI Endpoints
REST API Overview
SSE Event Format
Deployment Architecture
Docker Compose Stack
Service Dependencies
Container Configuration
| Service | Image | Ports | Health Check |
|---|---|---|---|
| postgres-app | postgres:15-alpine | 5432 | pg_isready |
| postgres-temporal | postgres:15-alpine | 5433 | pg_isready |
| temporal | temporalio/auto-setup:1.24.2 | 7233 | tctl cluster health |
| temporal-ui | temporalio/ui:2.26.2 | none (opt-in, 127.0.0.1 when opened) | - |
| api | Dockerfile.api | 8000 | /health |
| worker | Dockerfile.worker | - | - |
| scheduler | Dockerfile.scheduler | - | - |
| ui | Dockerfile.ui | 5173 | - |
| dind | docker:28-dind (privileged) | none (2375 on the stack network only) | docker info |
| dind-init | docker:28-cli | - (one-shot) | - |
| egress-nflog | Dockerfile.egress-nflog | - (shares dind's network namespace) | - |
Security Considerations
The Threat Model and Risk Assessment covers the whole platform: its assets and actors, the trust boundaries between its components, the threats at each boundary, the controls that handle them and the risk that is left. The sections below describe individual controls.
Orchestration surface
Temporal is internal. Its gRPC frontend is reachable only inside the Docker network in production (loopback in dev), and the Temporal Web UI is not started by a normal deployment: it authenticates nobody and is scoped to no organization, so a platform admin opens it deliberately from the host and closes it again. Nothing in the browser reaches Temporal directly — the UI reads workflow state through the API. See Temporal Admin Access.
Secret Reference Management
Hegemony stores secret references (not secret values) in the database. Workers resolve references at runtime using {{ }} Jinja template syntax: secret() reads a dynamic backend (for example OpenBao or Vault). The env() and file() helpers read the process's own environment and secrets directory, so only platform configuration (secret backend and inventory provider settings) may use them; organization templates are refused at save time and at run time.
Secret Reference Pattern
| Field | Description | Example |
|---|---|---|
| ssh_username_ref | Secret reference for SSH username | {{ secret('vault://orgs/default/secrets/ssh/username') }} |
| ssh_password_ref | Secret reference for SSH password | {{ secret('vault://orgs/default/secrets/ssh/password') }} |
| ssh_private_key_ref | Secret reference for SSH private key (shell steps only; network-CLI transports use the password) | {{ secret('vault://orgs/default/secrets/ssh/private_key') }} |
| enable_password_ref | Secret reference for enable password | {{ secret('vault://orgs/default/secrets/ssh/enable_password') }} |
Authentication Flow
Technology Stack Summary
| Layer | Technology | Purpose |
|---|---|---|
| Frontend | React + TypeScript + Vite | User interface |
| API | FastAPI + SQLAlchemy | REST API & control plane |
| Workflows | Temporal.io | Workflow orchestration |
| Worker | Python + Netmiko | Handler execution |
| Database | PostgreSQL 15 | Persistent storage |
| Deployment | Docker Compose | Container orchestration |
Appendix
Quick Reference URLs
| Service | URL | Description |
|---|---|---|
| Hegemony UI | http://localhost:5173 | Main application |
| Hegemony API | http://localhost:8000 | REST API |
| API Docs | http://localhost:8000/docs | OpenAPI/Swagger |
Environment Variables
| Variable | Default | Description |
|---|---|---|
| HEGEMONY_DATABASE_URL | postgresql+asyncpg://... | App database connection |
| HEGEMONY_TEMPORAL_HOST | temporal:7233 | Temporal server address |
| HEGEMONY_TEMPORAL_NAMESPACE | default | Temporal namespace |
| HEGEMONY_SCHEDULE_POLL_INTERVAL_SECONDS | 15 | Scheduler poll interval in seconds (Scheduler Service: apps/scheduler/run.py, apps/scheduler/scheduler.py) |
| HEGEMONY_DOCKER_BIN | docker | Path to Docker CLI binary |
| DOCKER_GID | - | Host docker group GID (REQUIRED for the worker's host-socket mount, used only to derive its worker id) |
| HEGEMONY_CONTAINER_DOCKER_HOST | tcp://dind:2375 (set by docker-compose.dind.yml) | Sandbox daemon for flow containers; the worker refuses an empty value |
| HEGEMONY_SANDBOX_TOKEN | - | REQUIRED sandbox attestation token; dind-init stamps it, the worker and the image-prune job check it |