Skip to content

Connectivity Monitors ​

Connectivity monitors run health probes concurrently with a flow's steps - optionally in the background, so the flow keeps working while the probes watch. A monitor targets:

  • one or more device roles from the flow, and/or
  • one or more explicit IPv4/IPv6 addresses

Every measurement is stored as a time series, surfaced in the run's Monitors tab as a table plus graphs, and streamed live to the UI while the run is in progress.


0) Terms ​

  • Monitor: a configured probe running on an interval.
  • TargetSelector: how user selects targets (role/ip).
  • TargetInstance: resolved concrete endpoint (device+address or ip).
  • Sample: one measurement row (per tick, per target instance).

1) Flow step type ​

1.1 monitor.connectivity ​

Continuously probes targets in the background during a flow run. The step starts the monitor and returns immediately, so the flow keeps executing while the probes tick. Targets come from the step's target selectors (device roles and/or explicit IP addresses - see section 2); everything else is step config:

fieldtypedefaultnotes
check_typeenumtcp_connecticmp_ping, tcp_connect, dns_resolve, or http_health
portintegeremptyport for the TCP-based checks; empty means 80 for http_health and 22 for tcp_connect; ignored by icmp_ping and dns_resolve
interval_msinteger5000tick interval in milliseconds
schedule_modeenumcountcount, duration, or until_join
countinteger10ticks to run when schedule_mode: count
duration_secondsinteger60run time in seconds when schedule_mode: duration
attempt_timeout_secondsinteger10per-probe timeout in seconds
url_pathstring""request path for http_health (defaults to /health when omitted)
socks_proxystring""optional socks5[h]://[user:pass@]host:port proxy for the TCP-based checks (tcp_connect, http_health)
ping_countinteger3echo requests per icmp_ping probe, 1 to 10

The field names above are the step handler's own configuration keys, which the handler maps onto the platform's monitor API (for example duration_seconds becomes the API's schedule.duration_s). Steps saved with the older names duration_sec and timeout_sec keep working. The check types listed are those the bundled probe wheel registers; the platform's check vocabulary also defines tls_handshake and ssh_banner for probe plugins that implement them.


2) Targeting model ​

2.1 TargetSelector (input) ​

A monitor step can define multiple selectors. Each selector can expand to multiple concrete TargetInstances.

fieldtyperequirednotes
typeenumyesrole or ip
rolestringif type=roleflow device role name
addresseslist of stringif type=ipIPv4/IPv6 literals

2.2 TargetInstance (resolved at runtime) ​

fieldtypenotes
target_instance_idstringunique stable key
selector_indexintindex in target_selectors[]
selector_typestringrole or ip
rolestring?if role-based
device_idstring?if role-based
addressstringconcrete IPv4/IPv6

2.3 Resolution rules ​

  • type=role: resolves to the devices bound to that role in the run, using each device's management address; devices without a management address are skipped
  • type=ip: creates one TargetInstance per address in addresses
  • If no selector resolves to a usable address, the step fails.

3) Scheduling / lifetime (no fail/recover semantics) ​

Scheduling is part of the step config (section 1): interval_ms sets the tick interval, and schedule_mode (with count / duration_seconds) decides when sampling stops.

Rules:

  • count: Runs exactly N ticks in the background, then completes.

  • duration: Runs for the specified duration in the background, then completes.

    For both, the step succeeds as soon as the monitor starts and routes its token then, like any step that succeeds: the flow moves on while the monitor samples, and does not wait for it. When the monitor ends, the step records how it ended, so a monitor whose sampling loop failed leaves a failed step (section 9), but its failure outcome is not routed. A monitor that cannot start fails its step and routes the failure outcome.

  • until_join: Fire-and-forget - stopped when termination edge target (join/terminal) fires. The flow waits only for the monitor to start; a started monitor does NOT route token, and is cancelled when the linked join completes. A monitor that cannot start (no target with a usable address, an invalid configuration, the start itself failing) fails its step with the reason and routes the step's failure outcome, like any failed step; no join then records it as a success.

Every execution of a monitor step starts its own monitor. A monitor step inside a loop starts one per pass, and the passes run side by side when an earlier one is still sampling; they share the step's monitor_id (the node id unless the step sets one) and are told apart by the step run that started each, which the monitor records as step_run_id.


4) Check execution contract ​

4.1 Common configuration ​

Every check type shares the step config described in section 1: interval_ms, schedule_mode (with count / duration_seconds), and a per-probe attempt_timeout_seconds. port applies to tcp_connect and http_health, url_path to http_health, and socks_proxy to the TCP-based checks (tcp_connect, http_health).

4.2 Sample (stored + streamed) — one per tick per TargetInstance ​

fieldtypenotes
tsRFC3339 stringsample timestamp
run_idstringcorrelation
step_idstring?step that started monitor (optional)
monitor_idstring
check_idstring
target_instance_idstring
selector_typestringrole/ip
rolestring?
device_idstring?
addressstringresolved IPv4/v6
okboolprobe succeeded
metrics_jsonmapkeep numeric keys stable for graphs
error_kindstring?timeout, conn_refused, dns_nxdomain, ...
error_detailstring?short message
raw_textstring?optional raw probe output

5) Storage model (table + graph) ​

Persist every Sample row as timeseries.

5.1 The monitor_samples table ​

columntypenotes
iduuidpk
run_iduuid
step_idstring?index (optional)
monitor_idstringindex
check_idstring
tsdatetime
target_instance_idstringindex
selector_indexintposition in target_selectors[]
selector_typestring
rolestring?
device_iduuid?index
addressstringindex
okbool
metrics_jsonjsonflat-ish map
error_kindstring?
error_detailstring?
raw_texttext?

The indexed columns are seq, step_id, monitor_id, target_instance_id, device_id, and address.

5.2 Database Migration ​

The monitor_samples table is created by the standard database migrations; apply with task db:migrate.


6) API + streaming ​

6.1 REST endpoints ​

  • GET /runs/{run_id}/monitors
    • list a run's monitors (status, config, resolved targets, tick/sample counts, and the step run that started each)
  • GET /runs/{run_id}/monitors/{monitor_id}
    • fetch a single monitor (the most recently started one when the passes of a loop share the id)
  • GET /runs/{run_id}/monitors/{monitor_id}/samples?limit=...&offset=...&since=...&target_instance_id=...
    • fetch samples for table/graphs, newest first; since filters by timestamp, target_instance_id narrows to one target

6.2 Live streaming ​

Server-sent events (SSE):

  • GET /runs/{run_id}/monitors/stream

Event: sample Payload: the stored Sample row (plus seq for client-side ordering)

When a monitor reaches a terminal state, the stream also emits a monitor_status event with the monitor's id, its monitor_id, final status and total sample count - once per monitor, so each pass of a loop reports its own end.

Event: stream_error Payload: message, plus detail naming the exception class (for example OperationalError). The stream recovers on its own and keeps polling, so this is advisory rather than terminal. detail is deliberately the class name and not the driver's message, which would carry the database host and user; the full text goes to the server log.

UI behavior:

  • Monitor list: every monitor in the run with its status; selecting one shows its detail. Each pass of a monitor step in a loop is listed on its own, named after the step run that started it; the graphs and table show the samples of every pass of that step
  • Graphs: per-target charts of the numeric metrics over time, with toggleable series and summary statistics
  • Table: samples grouped per target, with each sample's ok state, metrics, and any error
  • Auto-refresh: while monitors are running, the Monitors tab falls back to polling whenever the live stream is not connected

7) Supported checks ​

check_typeNameCheck-specific configMetrics
icmp_pingICMP Echo—rtt_ms, jitter_ms, packet_loss_pct
tcp_connectTCP Connectport, socks_proxyconnect_ms
dns_resolveDNS Resolve—resolve_ms, addresses
http_healthHTTP(S) Healthport, url_path, socks_proxyhttp_status, connect_ms

8) Flow YAML examples ​

8.1 Background ping monitor until join (role + explicit IP) ​

yaml
- id: mon_ping_core
  type: step
  name: "Ping core devices"
  tags: {kind: CHECK}
  handler: monitor.connectivity
  target_selectors:
    - type: role
      role: upgraded_device
    - type: ip
      addresses: ["2001:4860:4860::8888"]
  params:
    check_type: icmp_ping
    interval_ms: 2000
    schedule_mode: until_join
    attempt_timeout_seconds: 2

8.2 TCP connect for duration (all upgraded devices) ​

yaml
- id: mon_tcp_ssh
  type: step
  name: "Watch SSH reachability"
  tags: {kind: CHECK}
  handler: monitor.connectivity
  target_selectors:
    - type: role
      role: upgraded_device
  params:
    check_type: tcp_connect
    port: 22
    interval_ms: 1000
    schedule_mode: duration
    duration_seconds: 30

8.3 Stopping a monitor ​

An until_join monitor is stopped by the flow engine: wire a termination edge from the monitor step to a join node, and the monitor is cancelled when that join completes. The join stops every monitor attached to it, each by the step run that started it. count and duration monitors complete on their own after their configured ticks or run time.


9) Runtime behavior ​

Targets are resolved once when the monitor starts. On every tick the monitor probes each target and stores one sample per target, publishing it to the live stream as soon as it is stored. The UI derives up/down status from the most recent samples.

A monitor runs on the worker that started it, but whatever stops, pauses or resumes it - a join firing, the run pausing for an approval or a pause step, the run ending - may be handled by any worker. That worker stops or pauses the monitors it runs itself at once and updates the monitor records of the rest; the worker running a monitor reads its record about every five seconds, also between the ticks of a long interval, and follows it. A stop or pause handled elsewhere therefore takes effect within about five seconds. A stop always wins: once a monitor's record says it has ended, no later pause or resume moves it back. Pauses and resumes are ordered by when the run asked for them: when the API was unreachable, the worker running a monitor writes the pause or resume it was given once the API answers, unless a later one has been recorded meanwhile.

A monitor's sampling lives in the memory of the worker running it, so it ends when that worker shuts down. On a graceful shutdown (a restart, a rolling deploy) the monitor is not reported as completed: its record is marked failed, its step fails with the reason - the worker running it shut down before the monitor finished - and an error event says how many samples it took. A join that fires later leaves that failure in place. A stop that reaches the monitor's record before the failure is written still counts as a stop. A worker that is killed outright records nothing: its monitors look running until their join or the run's end stops them.

The same holds when the monitor's own sampling loop fails, an error in the worker rather than a failed probe (a probe that fails is a failed sample): its record is marked failed, and its step fails with the error. The flow has already moved on from the step by then, so the step's failure outcome is not routed; the step and an error event show what went wrong.

A monitor can end within moments of starting (its loop fails, its worker shuts down, or a short count finishes) before the flow records the start on the step. The start is recorded only while the step is still running, so the end the monitor recorded stands.

When a monitor ends, the worker records the end on the monitor's record and on its step. If the API does not answer (for example while it restarts), the worker tries each again for a few seconds. A monitor record that still cannot be written does not keep the step open: the step is still closed with how the monitor ended.

The monitor's own run events (its egress report and its completion line) carry the phase tag of the step that started it, and none when that step has no such tag - phase is an ordinary, optional tag on a step.

Released as open source under the AGPL-3.0-or-later license. Development is sponsored by Rexonix s.r.o. Contact — [email protected].