Architecture Overview

Table of Contents

1. Overview
2. System Architecture
3. Component Details
4. Data Flow
5. Database Schema
6. Flow Engine
7. Schedule Automation
8. Git Repository Integration
9. API Endpoints
10. Deployment Architecture
11. Security Considerations

---

Overview

Hegemony is a proof-of-execution platform for deterministic, long-running infrastructure workflows. Built on Temporal.io with a declarative Flow Engine, it provides:

• Declarative Workflows: YAML/JSON-defined flows with pluggable handlers
• Deterministic Execution: Reliable, resumable workflow execution
• Real-time Updates: Server-Sent Events (SSE) for live progress tracking
• Audit Trail: Persistent event logging for compliance and debugging
• Clean Separation: Workflow orchestration decoupled from execution logic

Core Principles

---

System Architecture

High-Level Architecture

Component Interaction Sequence

---

Component Details

Frontend (UI)

Aspect	Details
Framework	React 18 + TypeScript
Build Tool	Vite
Routing	React Router DOM
State Management	React hooks + fetch API

UI Pages

Runtime env-key registry: the worker self-registers the names (never values) of HEGEMONY_* environment variables it can see by POSTing to /internal/runtime/env-keys on startup and on every down -> up reconnect, and clears them via DELETE on graceful shutdown. Operators can view the merged API + worker view at Settings -> Environment Variables (admin-only, gated centrally by the RBAC registry in apps/api/auth/actions.yaml + routes.yaml). No operator action is required to register keys — they are discovered automatically from process environment.

Backend (API)

Aspect	Details
Framework	FastAPI (async)
ORM	SQLAlchemy 2.0 (async)
Database	PostgreSQL 15
Migrations	Alembic

API Structure

Worker

Aspect	Details
Framework	Temporal Python SDK
Transport	SSH (Netmiko/Paramiko)
Pattern	Handler-based execution

Activity Types

Container Handler (Docker-in-Docker)

The container.run handler enables running arbitrary Docker containers as flow steps. This supports use cases like firmware image preparation, config generation scripts, and custom tooling that cannot run directly on the worker.

Architecture:

• Flow containers run on a dedicated privileged docker:dind sandbox service (deploy/compose/docker-compose.dind.yml), never on the host Docker daemon. HEGEMONY_CONTAINER_DOCKER_HOST names it (tcp://dind:2375, set by that file), and every docker invocation the handler and the startup janitor spawn carries it as a per-invocation DOCKER_HOST, so the whole flow-container estate (step containers, demo lab, Gitea, flow-built images) lives inside the sandbox. There is no host-socket mode: the worker refuses to start when the setting or the sandbox token is empty, or when the daemon that answers is not the attested sandbox, and each step refuses an empty setting on its own. A sandbox that is merely slow to come up gets a bounded wait (HEGEMONY_CONTAINER_DOCKER_WAIT_SECONDS), after which the worker starts degraded and keeps probing in the background: every container step is refused until the sandbox answers and is attested, so no step runs on a daemon the worker has not checked, and the startup janitor sweeps only a daemon that has passed (after a degraded start, once it passes and before container steps are enabled). Until then the worker's heartbeat does not advertise the containers capability (a starting worker withdraws the one its previous process advertised before it waits), so shared-worker assignment pins no new shared or explicit run to it. A run already pinned to it keeps that pin across a restart, and its container steps are refused there until the sandbox is attested. A late sandbox that fails attestation stops the worker with a non-zero exit, as it would have at startup; a token changed after dind-init stamped the marker fails this way, and Production Hardening gives the recovery. A wait of 0 only skips the blocking wait: the worker still probes once, and attests or refuses a daemon that answers before it connects to Temporal. See docs/development/design-dind-sandbox.md.
• The worker still mounts the host socket (/var/run/docker.sock), for one query only: deriving its stable worker id from its Compose container name is a question only the daemon hosting the worker can answer, so that query never follows HEGEMONY_CONTAINER_DOCKER_HOST. No flow container can reach the host socket.
• Container images are pulled on-demand; a configurable allow-list restricts which images may be used
• Flow attachments are mounted into the container at a configurable path (default: /workspace)
• An optional entrypoint override maps to Docker --entrypoint; when omitted, the image entrypoint/CMD behavior is preserved, and command remains shell-script lines joined under set -e
• Container stdout/stderr is captured as run artifacts; exit code determines step success/failure
• Containers are cleaned up after execution (removed); orphan cleanup runs on worker startup

Container marking: every container a flow creates is attributable to its run. Names are prefixed with the run id (hegemony-<run_id[:8]>-<step_run_id[:8]>-a<attempt>), and labels carry the full ids plus the deployment stack: hegemony.run_id, hegemony.step_run_id, hegemony.handler_id, and hegemony.stack (the compose project name — hegemony-dev/hegemony-prod/hegemony-demo — from the worker's HEGEMONY_STACK_NAME env var). Anything that needs to find flow containers filters on hegemony.run_id; cleanup also scopes by hegemony.stack, so a janitor never touches another stack's containers even on a daemon they share:

• The worker's startup janitor removes its own stack's orphaned non-running containers older than an hour (younger ones may belong to a step another worker replica is still provisioning or harvesting; other stacks' containers are never touched; unlabeled legacy ones are still cleaned).
• task compose:demo:reset force-removes everything labeled hegemony.stack=hegemony-demo — running included — plus stopped unlabeled legacy containers, along with the demo volumes.

# Flow containers live inside the stack's sandbox daemon; <stack> is the
# compose project name (hegemony-dev, hegemony-prod, hegemony-demo).
docker exec <stack>-dind-1 docker ps -a --filter "label=hegemony.run_id"
docker exec <stack>-dind-1 docker ps -a --filter "label=hegemony.stack=hegemony-prod"

# The same from deploy/compose, without naming the container:
ENV=prod ./dc.sh exec dind docker ps -a --filter "label=hegemony.run_id"

docker ps on the host daemon shows no flow containers. Any it lists with a hegemony.run_id label were left there by a release that ran flows on the host daemon; the upgrade notes show how to sweep them.

Deployment:

• The sandbox is part of every stack: dc.sh and the compose tasks apply deploy/compose/docker-compose.dind.yml whatever SERVICES names, core included. It adds the dind, dind-init and egress-nflog services, and down -v removes the entire flow estate with its dind-data volume
• The host must allow privileged containers and IPv6 bridge networks; see production hardening
• Requires DOCKER_GID set to the host docker group GID, for the worker's host-socket mount (worker-id derivation only)
• The worker Dockerfile installs Docker CLI (pinned version), which reaches the sandbox over TCP and the host daemon only to derive the worker id

---

Data Flow

Flow Execution Flow

Event Emission Flow

---

Database Schema

Entity Relationship Diagram

Core Enumerations

---

Flow Engine

The Flow Engine is the core execution system for Hegemony. It provides declarative workflow execution.

Architecture Overview

Step Execution Lifecycle

Handler Registry Pattern

Handlers are provided by out-of-tree plugin wheels (repo hegemony-step-plugins, one claimed namespace prefix per wheel) that register handler classes under the hegemony.step_handlers entry-point group. The handler contract lives in the hegemony-step-sdk package; the host injects a HandlerServices implementation (ctx.services) that provides transports, secrets, notifications, and artifact operations. Every step handler ships as a plugin wheel, monitor.* included (hegemony-steps-monitor).

Flow Definition Structure

# Example flow definition (V2 graph format)
start_node_id: start
nodes:
  - id: start
    type: start
    name: "Start"
  - id: precheck
    type: step
    name: "Pre-upgrade validation"
    tags: {phase: PREPARE, kind: CHECK}
    handler: netcli.collect_evidence
    params:
      commands:
        - "show version"
        - "show inventory"
  - id: backup_config
    type: step
    name: "Backup running config"
    tags: {phase: PREPARE, kind: ACTION}
    handler: netcli.execute
    params:
      commands:
        - "copy running-config startup-config"
  - id: end
    type: terminal
    name: "Done"
edges:
  - source: start
    source_outcome: start
    target: precheck
  - source: precheck
    source_outcome: success
    target: backup_config
  - source: backup_config
    source_outcome: success
    target: end

Edges carry no id — an edge is identified by its (source, source_outcome, target, type) combination. Compact exports merge source and source_outcome into source: "node:outcome".

A step's tags — phase and kind among them — are entries of its tags map and nothing else: FlowGraphNode has no phase, kind or labels field and rejects a node that carries one. Only the bundle/git importer (deserialize_flow_definition in apps/api/services/flow_serializer.py) still reads that retired shape, folding it into tags (explicit tags win, then labels, then phase/kind) alongside its other upgrades of older files, and git pull and platform-sync plans compare an incoming file in that upgraded shape (normalize_for_comparison) so an old-shape file is not a phantom change; migration 053_step_run_tags rewrote the definitions already stored in flow_definitions and flow_definition_versions the same way, once. Templates read them as {{ tags.phase }} / {{ tags.kind }}. Handlers receive the map as ctx.tags and nothing else: HandlerContext has no phase or kind field (hegemony-step-plugins#20), so a handler reads ctx.tags.get("phase", ""). The worker stamps run_events.phase / run_artifacts.phase from the same tag.

An edge may also carry an optional waypoints list (up to 20 {x, y} points in absolute canvas coordinates) that pins where the editor draws it:

edges:
  - source: "backup_config:success"
    target: end
    waypoints:
      - x: 840
        y: 420
      - x: 150
        y: 420

Waypoints are purely visual — routing behavior never depends on them. In the editor, drag anywhere on a selected edge to drop a routing point (snapped to the canvas grid), drag a handle to move it, double-click a handle to remove it, or use the edge's reset button to return to automatic routing. Edges without waypoints route automatically, and manual routes render identically in the editor, the run view, and version previews. Absent and null both mean automatic routing; unset waypoints are omitted from exports.

---

Schedule Automation

Overview

Hegemony supports automated workflow execution through the Schedules feature. Schedules enable users to trigger flow runs at specific times or on recurring intervals, providing a powerful automation layer for routine infrastructure tasks.

Schedule Types

One-Time Schedules

• Execute a flow once at a specified date and time
• Status automatically transitions to COMPLETED after execution
• Use cases: maintenance windows, one-off upgrades, planned changes

Recurring Schedules

• Execute a flow repeatedly based on a cadence
• Two cadence options:
  • Interval: Run every N seconds (e.g., every 3600 seconds = hourly)
  • Cron: Run based on cron expression (e.g., 0 2 * * * = daily at 2 AM)
• Remain ACTIVE until explicitly disabled
• Use cases: daily health checks, weekly backups, continuous monitoring

Schedule States

Scheduler Architecture

Key Components

1. Scheduler Service (apps/scheduler/run.py + apps/scheduler/scheduler.py)

   • Dedicated service running independently from the Temporal worker
   • Polls for due schedules every HEGEMONY_SCHEDULE_POLL_INTERVAL_SECONDS (default: 15)
   • Triggers schedules via internal API endpoints
   • Continues on errors to ensure fault tolerance
   • Deployed as separate container in docker-compose (dedicated scheduler image)

2. Schedule Service (apps/api/services/schedules.py)

   • compute_next_run_at(): Calculates next execution time based on type/cadence
   • trigger_schedule_run(): Creates a new run from schedule parameters
   • Handles timezone conversions for cron expressions

3. Schedule API (apps/api/routers/schedules.py)

   • Public CRUD endpoints: /schedules
   • Trigger endpoint: /schedules/{id}/run (manual trigger)
   • Validates schedule parameters (e.g., one-time must be in future)

4. Internal API (apps/api/routers/internal.py)

   • /internal/schedules/due: Lists schedules ready to execute
   • /internal/schedules/{id}/trigger: Triggers schedule (worker-only)
   • Prevents duplicate triggers with status checks

Schedule Parameters

Schedules support full flow parameterization:

• Flow + Version: Pin to specific flow version or use latest committed
• Inputs: Flow input parameters (e.g., target_version: "17.9.4")
• Targets: Device role mappings for multi-device flows
• Timezone: Affects cron expression evaluation (e.g., America/New_York)

UI Integration

The Schedules page (/schedules) provides:

• Overview cards: One-time vs recurring schedule concepts
• Tabs: Upcoming schedules (active only) and all schedules
• Create form: Flow selection, schedule type, cadence, timezone, params
• Actions: Run now, pause/resume, delete
• Status badges: Visual indicators for active/disabled/completed

---

Git Repository Integration

Git Integration connects flows to external Git repositories, enabling version control, code review, and CI-driven workflows for flow definitions and attachments.

Architecture

All Git operations run inside the API process — workers never interact with Git directly. This keeps the trust boundary tight and avoids distributing credentials to the execution layer.

UI ──► API ──► local git CLI ──► Remote Git (HTTPS / SSH)
         │
         └──► PostgreSQL (git_repositories, git_sync_history,
              flow git linkage columns)

Key Components

Component	Location	Responsibility
Git Repositories CRUD	apps/api/routers/git_repositories.py	Registry of remote repos, test connection
Git Operations	apps/api/services/git_ops.py	Clone, fetch, commit, push via subprocess; semaphore(4) concurrency limit
Exploded Format	apps/api/services/sync/serialization/layouts/exploded.py	Read/write flows as directory tree (flow.yaml + attachments/). Symlink & traversal guards
Pull Sync	apps/api/services/sync/flow_git/pull.py	Pull flow from Git into DB draft; dirty-draft conflict detection
Push Sync	apps/api/services/sync/flow_git/push.py	Push committed flow to Git; stale-remote detection with fast-forward
Async Dispatcher	apps/api/services/sync/flow_git/queue.py	Background loop that processes queued push jobs from git_sync_history
URL Validation	apps/api/services/git_ops.py	SSRF prevention — rejects private/loopback addresses

Sync Modes

Flows are linked to a Git path via sync_mode:

• synced — bidirectional: auto-push on commit, pull available.
• readonly — pull-only: flow is read from Git, local edits blocked.
• detached — linkage preserved but no sync occurs.

Data Model

• git_repositories — registered remotes (name, URL, branch, credential ref).
• flow_definitions git columns — git_repo_id, git_path, git_branch, sync_mode, last_synced_sha, last_synced_at.
• flow_attachments git columns — git_repo_id, git_file_path, last_synced_sha, last_synced_at.
• git_sync_history — audit log of every sync operation (also serves as the async push queue).

Configuration Exchange migration notes

Configuration Exchange and flow Git sync are opt-in additions to the existing application database. Operators should apply the normal Alembic migration chain before enabling scheduled sync or API-driven push jobs; migrations are never run implicitly at process startup.

The platform-sync migrations add Git linkage and operation tracking state:

• flow_definitions: git_repo_id, git_path, git_branch, sync_mode, auto_commit, last_synced_sha, last_synced_at.
• flow_attachments: git_repo_id, git_file_path, last_synced_sha, last_synced_at.
• git_repositories: connection/health fields used by sync checks, including sync_stale_seconds and last_connected_at.
• git_sync_history: durable operation history and queued async push work.
• config_exchange_locks plus platform-sync profile/run/plan/state tables: layout ownership, leases, reviews, and resource-state hashing.

Backfills normalize existing flow Git paths as file paths (no trailing slash) and profile base paths as directories (with trailing slash). Existing rows that have no Git linkage remain detached; there is no automatic remote bootstrap. After migration, create or update Git repositories and platform-sync profiles through the admin API/UI, then trigger a manual plan/export before enabling scheduled operation.

Relevant toggles are standard HEGEMONY_* environment variables. Keep HEGEMONY_MAINTENANCE_ENABLED and the platform-sync maintenance interval settings aligned with operational intent: scheduled platform exports are driven by the maintenance subsystem, while manual Configuration Exchange API calls work as soon as the database migration and Git repository configuration are present.

Async Push Flow

1. User commits flow (sync_mode=synced)
2. API enqueues git_sync_history row (status=queued, direction=push)
3. Dispatcher loop claims row → runs git push
4. Row updated to success/failed

---

API Endpoints

REST API Overview

SSE Event Format

---

Deployment Architecture

Docker Compose Stack

Service Dependencies

Container Configuration

Service	Image	Ports	Health Check
postgres-app	postgres:15-alpine	5432	pg_isready
postgres-temporal	postgres:15-alpine	5433	pg_isready
temporal	temporalio/auto-setup:1.24.2	7233	tctl cluster health
temporal-ui	temporalio/ui:2.26.2	none (opt-in, 127.0.0.1 when opened)	-
api	Dockerfile.api	8000	/health
worker	Dockerfile.worker	-	-
scheduler	Dockerfile.scheduler	-	-
ui	Dockerfile.ui	5173	-
dind	docker:28-dind (privileged)	none (2375 on the stack network only)	docker info
dind-init	docker:28-cli	- (one-shot)	-
egress-nflog	Dockerfile.egress-nflog	- (shares dind's network namespace)	-

---

Security Considerations

Orchestration surface

Temporal is internal. Its gRPC frontend is reachable only inside the Docker network in production (loopback in dev), and the Temporal Web UI is not started by a normal deployment: it authenticates nobody and is scoped to no organization, so a platform admin opens it deliberately from the host and closes it again. Nothing in the browser reaches Temporal directly — the UI reads workflow state through the API. See Temporal Admin Access.

Secret Reference Management

Hegemony stores secret references (not secret values) in the database. Workers resolve references at runtime using {{ }} Jinja template syntax: secret() reads a dynamic backend (for example OpenBao or Vault). The env() and file() helpers read the process's own environment and secrets directory, so only platform configuration (secret backend and inventory provider settings) may use them; organization templates are refused at save time and at run time.

Secret Reference Pattern

Field	Description	Example
ssh_username_ref	Secret reference for SSH username	{{ secret('vault://orgs/default/secrets/ssh/username') }}
ssh_password_ref	Secret reference for SSH password	{{ secret('vault://orgs/default/secrets/ssh/password') }}
ssh_private_key_ref	Secret reference for SSH private key (shell steps only; network-CLI transports use the password)	{{ secret('vault://orgs/default/secrets/ssh/private_key') }}
enable_password_ref	Secret reference for enable password	{{ secret('vault://orgs/default/secrets/ssh/enable_password') }}

Authentication Flow

---

Technology Stack Summary

Layer	Technology	Purpose
Frontend	React + TypeScript + Vite	User interface
API	FastAPI + SQLAlchemy	REST API & control plane
Workflows	Temporal.io	Workflow orchestration
Worker	Python + Netmiko	Handler execution
Database	PostgreSQL 15	Persistent storage
Deployment	Docker Compose	Container orchestration

---

Appendix

Quick Reference URLs

Service	URL	Description
Hegemony UI	http://localhost:5173	Main application
Hegemony API	http://localhost:8000	REST API
API Docs	http://localhost:8000/docs	OpenAPI/Swagger

Environment Variables

Variable	Default	Description
HEGEMONY_DATABASE_URL	postgresql+asyncpg://...	App database connection
HEGEMONY_TEMPORAL_HOST	temporal:7233	Temporal server address
HEGEMONY_TEMPORAL_NAMESPACE	default	Temporal namespace
HEGEMONY_SCHEDULE_POLL_INTERVAL_SECONDS	15	Scheduler poll interval in seconds (Scheduler Service: apps/scheduler/run.py, apps/scheduler/scheduler.py)
HEGEMONY_DOCKER_BIN	docker	Path to Docker CLI binary
DOCKER_GID	-	Host docker group GID (REQUIRED for the worker's host-socket mount, used only to derive its worker id)
HEGEMONY_CONTAINER_DOCKER_HOST	tcp://dind:2375 (set by docker-compose.dind.yml)	Sandbox daemon for flow containers; the worker refuses an empty value
HEGEMONY_SANDBOX_TOKEN	-	REQUIRED sandbox attestation token; dind-init stamps it, the worker and the image-prune job check it