Skip to content

Step Output Streaming ​

How a step's live output reaches the Run Detail Logs tab: container steps, shell steps and network-CLI steps all use the same pipeline. This page is for contributors and plugin authors; the user-facing view is the Logs tab described in Runs.

The pipeline ​

text
producer (container logs, a handler, or a transport's output sink)
  └─ ctx.emit_progress(text, attrs={"milestone": "output", "stream": "stdout"|"stderr", ...})
       │
       ▼
worker: the step's emit_progress callback          apps/worker/flow_activities.py
  ├─ redacts every secret the step resolved
  ├─ activity.heartbeat(message)                    (Temporal liveness)
  └─ POST /internal/runs/{run_id}/events  kind="progress"  (best effort)
       │
       ▼
API: next seq under a row lock, INSERT run_events   apps/api/routers/internal.py
       │
       ▼
GET /bff/runs/{run_id}/events/stream (SSE)          apps/api/routers/bff_runs.py
  └─ polls the table (default every 1 s, 0.5–30 s), up to 200 events a page
       │
       ▼
useRunEventStream → Logs tab terminal               apps/ui/src/lib/useRunEventStream.ts

An output event is an ordinary progress run event whose attrs.milestone is output and whose attrs.stream is stdout or stderr. Device steps add attrs.device_name and attrs.command; device ids are UUIDs only. The Logs tab keys on that shape, not on the handler, and the Events tab leaves output events out. Log data goes through PostgreSQL only: no pub/sub, no Redis.

The browser keeps the last 50,000 lines (HEGEMONY_UI_MAX_TERMINAL_LINES at UI build time changes it). The durable record of a command's output is its cli_output evidence artifact, not the stream.

Producers ​

  • container.run streams stdout and stderr line by line from the attached container.
  • Handlers (shell.execute, netcli.execute, collect_evidence) emit each command's completed output, split near 8 KB on line boundaries and capped near 64 KB per command with a truncation marker.
  • Transports stream mid-command through an output sink: the netmiko, scrapli and asyncssh timing-mode read loops, and the host's shell transport, which then reads the process incrementally. A transport that streams says so with streams_timing_output (CLI timing mode) or streams_exec_output (shell), and the handler then skips its completed-output emission, so nothing is posted twice. The transport, not the handler, writes the $ command marker line.

Not streamed mid-command: netmiko's plain send_command (it is prompt-based and returns no partial output), config-mode sessions (send_config_set), and the asyncssh exec path (execute_commands), whose command lists can carry credentials. Their completed output still reaches the Logs tab through the handler, redacted the same way.

The output sink ​

apps/worker/output_sink.py builds one sink per connect() or open_shell() call. The step SDK's TransportOutputSink protocol (push, push_from_thread, flush, flush_from_thread) is additive: a transport wheel takes an optional output_sink constructor argument and sets the class attribute supports_output_sink = True, and the host passes a sink only to transports that do. The sink:

  • strips ANSI escapes and normalizes CRLF;
  • redacts the connection's credentials, keeping back the end of each batch so a secret split across two batches is still redacted whole;
  • flushes at 1 KB (FLUSH_BYTES) or 0.5 s after the first buffered chunk (FLUSH_INTERVAL_SECONDS), and hard-splits at 16 KB (HARD_SPLIT_BYTES);
  • emits one stream at a time, so events never reorder;
  • never raises into the transport: a failed emit is dropped.

push_from_thread hands chunks from netmiko's executor thread to the event loop. Each flush is one HTTP request and one row-locked insert, which is why the sink batches instead of sending every chunk.

Redaction ​

Every secret the platform resolves while a step runs is registered in the step's secret registry (apps/worker/secret_registry.py): device, enable and bastion credentials when the connection opens, and every secret() and file() template value (env() is configuration and is not registered). The registry removes those values from:

  • every output message and attribute, before the heartbeat and before the run event is posted;
  • evidence artifacts, when they are stored;
  • the step result (summary, error, metrics, output), on the error path too. When that changes anything, the worker logs the field paths, never the values.

Values shorter than 4 characters (REDACT_MIN_CHARS) are not redacted, and a warning says so. A value that spans lines, such as a private key, is also redacted line by line for lines of 8 characters or more (REDACT_LINE_MIN_CHARS), because devices echo it with their own line endings and indentation. Only resolved values are covered: a literal secret typed into a command is ordinary text to the platform.

Several devices at once ​

Multi-device SSH handlers take max_parallel_devices (default 1, at most 32). The SDK's for_each_target_device bounds the concurrency and returns results in device order. Output from parallel devices interleaves in the Logs tab; each line carries its device name, and the terminal filters by device per step and across the run.

Released as open source under the AGPL-3.0-or-later license. Development is sponsored by Rexonix s.r.o. Contact — [email protected].