ADR-018: Plugin Lifecycle, Watchers, and Developer Experience

Status: Accepted Date: 2026-05-03 Deciders: QNTX Core Team Related: ADR-003 (Plugin Communication), ADR-004 (Plugin-Pulse Integration)

Context

Plugins are separate processes that communicate with QNTX over gRPC. The lifecycle — boot, initialization, health monitoring, restart, shutdown — has implicit contracts that caused real bugs: double initialization, stale connections, silent watcher failures. This ADR documents the full plugin lifecycle and the watcher subsystem as canonical references.

Problems observed

Double initialization. ForceInitialize bypassed initOnce without consuming it. The HTTP routing lazy-init then called Initialize again, causing plugins to start their work twice and tear down what they just built.
WebSocket ping-pong is undocumented. Plugins implementing HandleWebSocket must read the incoming gRPC stream and respond to PING messages with PONG. If they ignore the stream (natural first instinct), QNTX logs "WebSocket pong timeout" and the connection dies.
Watcher lifecycle is implicit. The full path — InitializeResponse.watchers → DB → engine → ExecuteJob — is spread across four files. A plugin developer sees only the proto field and ExecuteJob.
Predicate matching rules are undocumented. Matching semantics (exact, OR, rate limiting) are only discoverable by reading engine.go.

Decision

Document the plugin lifecycle and watcher system as first-class concepts.

Plugin Lifecycle

Binary launch          gRPC connect          Initialize RPC         Health poll (10s)
     |                      |                      |                      |
  process starts        Metadata()           Initialize()            Health()
  binds port            validates name       plugin starts work      monitors liveness
  prints QNTX_PLUGIN_PORT                    returns watchers,       restarts on 2
                                             routes, handlers        consecutive failures

Boot sequence

QNTX launches the plugin binary and waits for it to bind a port
gRPC connection established, Metadata() called to validate plugin identity
Initialize(InitializeRequest) sent with config, ATS endpoint, auth token
Plugin returns InitializeResponse with watchers, routes, handlers, schedules
QNTX registers watchers in DB, sets up HTTP proxy routes, registers async handlers
Health polling begins (every 10s)

Initialize contract

Called once per proxy lifetime. A fresh process gets one Initialize call. The initOnce guard ensures HTTP routing lazy-init doesn't trigger a second call.
Re-initialization (ReinitializePlugin / ForceInitialize) is only for config updates on an existing proxy — not for new processes after restart.
Plugins must handle re-init gracefully. Stop previous state (nodes, connections, goroutines) before starting new ones. The Initialize handler may be called on a proxy that already has running state from a previous call.
Config comes from am.toml. The config map in InitializeRequest contains key-value pairs from the plugin's config section.

Restart

POST /api/plugins/{name}/restart or make install from a plugin directory.

QNTX unregisters the plugin from the registry (state → Restarting). Browser requests get 503 Plugin still loading during restart.
Mux cache invalidated so the next request after restart builds a fresh HTTP proxy.
Active ATS streams cancelled to release the database mutex.
Old process killed (SIGKILL to process group), old gRPC connection closed.
New binary discovered, launched, gRPC connected — new proxy with fresh initOnce.
registerRestarted: re-registers in registry, calls Initialize, re-registers handlers/schedules/providers, marks ready.
Banner emitted with health status after async health check.

A restart always produces a new process, new gRPC connection, new proxy. The single PluginManager is reused — LoadPluginsFromConfig receives the existing manager, no second manager is created.

Health polling

Every 10 seconds, QNTX calls Health() on each running plugin
2 consecutive failures trigger automatic restart with exponential backoff (10s, 20s, 40s, ... capped at 640s)
A single transient failure is tolerated
Health resets on success

Shutdown

QNTX sends Shutdown() RPC (best-effort, 5s timeout)
Plugin process is killed
Watchers pruned from DB, handlers unregistered

Watcher Lifecycle

Plugin                          QNTX Core                        Watcher Engine
  |                                |                                  |
  |-- InitializeResponse -------->|                                  |
  |   (watchers: [...])           |                                  |
  |                               |-- SetupPluginWatchers() -------->|
  |                               |   (write to DB, idempotent)      |
  |                               |                                  |
  |                               |-- ReloadWatchers() ------------>|
  |                               |   (load from DB into memory)     |
  |                               |                                  |
  |                               |   attestation arrives            |
  |                               |                                  |
  |                               |   <-- predicate match --------  |
  |                               |                                  |
  |<-- ExecuteJob(attestation) ---|                                  |
  |   (handler_name routes it)    |                                  |
  |                                                                  |

WatcherRegistration fields

Field	Required	Description
`id`	yes	Unique suffix. Core prefixes with plugin name: `{plugin}-{id}`
`handler_name`	yes	Which `ExecuteJob` handler receives the match
`predicates`	yes	Attestation predicates to match (exact match)
`contexts`	no	Additional context filters
`max_fires_per_second`	no	Rate limit. 0 = zero fires allowed (QNTX LAW: zero means zero). Default: 0

Validation

A watcher must declare at least one filter dimension. The storage layer (validateWatcher) rejects watchers with no structural filters (subjects, predicates, contexts, actors), no temporal filters (time_start, time_end), no attribute filters, no ax_query, and no semantic_query. Empty-filter watchers — which would match every attestation — cannot be created.

Predicate matching

A watcher fires when an attestation's predicates contain any of the watcher's predicates (OR semantics)
Match is exact string equality, not prefix
Multiple filter dimensions are ANDed: if a watcher sets both predicates and subjects, both must match
Structural matching runs through Rust WASM (batch, one call per attestation for all watchers) with Go fallback
Rate limiting is per-watcher, not per-predicate

UI signal: ⏿

The watcher symbol ⏿ appears inline next to predicates that have active watchers. One eye per watcher. Hover shows watcher names. Eye color follows dilation — bright spice-blue when strained, deep sea blue when relaxed, faded white when the watcher has never fired.

Hot-swap behavior

Watchers survive plugin restart. On every Initialize:

SetupPluginWatchers calls CreateOrReplace for each declared watcher
Stale watchers (previously declared, no longer in the list) are pruned
ReloadWatchers refreshes the engine's in-memory state
This works identically on cold start, hot-swap enable, and crash recovery

WebSocket keepalive contract

QNTX sends PING messages on the gRPC stream and expects PONG responses. This tells the plugin whether a browser client is still connected.

Plugins must read the incoming gRPC stream and reply to PING with PONG (echo the timestamp). Spawn a reader task or thread that checks the message type and responds accordingly.

Failure to respond causes QNTX to log WebSocket pong timeout. The keepalive interval and timeout are configurable in am.toml under [plugin.websocket.keepalive]:

[plugin.websocket.keepalive]
ping_interval_secs = 30
pong_timeout_secs = 60

Error flow

When things go wrong, QNTX emits:

Log message	Meaning	Plugin action
`Failed to parse AX query for watcher`	Watcher predicate is malformed	Fix the predicate string in `WatcherRegistration`
`gRPC ExecuteJob failed`	Plugin returned an error from `ExecuteJob`	Check plugin-side handler logic
`Max retries exceeded, giving up`	`ExecuteJob` failed repeatedly	Check plugin health, logs
`WebSocket pong timeout`	Plugin ignores incoming WebSocket stream	Read the incoming stream and reply to PING with PONG
`Failed to setup plugin watchers`	DB write failed during `Initialize`	Check DB connectivity

Consequences

Positive

Plugin developers have a single reference for watcher behavior
WebSocket keepalive requirement is discoverable before hitting the error
Error messages can be cross-referenced against this document

Negative

Another document to maintain as the watcher system evolves

ADR-003: Plugin Communication — watchers are the reactive complement to attestation polling
ADR-004: Plugin-Pulse Integration — Phase 5 introduced watchers
Plugin Hot-Swap — watcher registration during enable/disable
gRPC Plugin API — WatcherRegistration proto docs
WebSocket API — WebSocket message types