Secondary Channel (commander ↔ worker)

Version: 1.0.0 Status: SOURCE OF TRUTH Last Updated: 2026-07-10

Overview

The multi-worker model has two rails:

  • Primary channel: browser → server → commander → worker. The request path. Plain HTTP: the commander forwards with an httpx client (WorkerForwardClient) and relays the response untouched.

  • Secondary channel: commander ↔ worker. The pool socket (executors/pool/: wsx envelopes over UDS or TCP, FIFO per connection, EOF = death, tytx typing available). It carries the WHOLE commander/worker conversation: page lifecycle, datachanges that leave the worker, the global store, occupancy metrics, user moves.

The single-worker deployment has no channel: SpaSingleWorkerApplication is its own commander in-process, same events, no transport.

Ascending rail (worker → commander)

The worker pushes spontaneously, fire-and-forget. What goes up is only what concerns a page that might live elsewhere, or the global view only the commander has. Everything page-local stays on the worker.

Path

Payload

Purpose

/events

batch of shaped events (op taxonomy: LIFECYCLE_OPS, STORE_OPS, POST_OPS, EXCHANGE_OPS)

lifecycle (new_*/drop_*/change_connection_user with xgroup), subscribeTable, store_set/del, notifyDbEvents, datachanges for pages owned by other workers, the user package of a move

/occupancy

occupancy_report() — RAW sensor readings, fixed shape (see 11-elastic-pool.md)

periodic timer push; the commander archives it per worker

/announce_http

host/port

routable = REGISTER + announce

Events are born in executor threads; the Outbox buffers thread→loop and a single sender task on the worker loop ships batches immediately (FIFO: one sender, one connection).

Descending rail (commander → worker)

Path

Purpose

/update_global

push a global-store write to every replica

/store_snapshot

seed a late worker’s replica atomically at announce

/datachange_in

deliver forwarded datachanges/dbevents to the owner worker (deposited in the target page’s pending list)

/evict_user

commanded move: the source snapshots the user, drops the slice, ships the package up

/install_user

move destination: install the package, confirm

The switch model (datachanges)

The commander is an event switch, not a warehouse: it receives, resolves the owner from the surface, forwards immediately. The only queue is the pending list inside the page item on the owner worker (local deposits are the common case; commander forwards are the rest). At render the page drains its pending and merges it with its local_datachanges, ordered by change_ts (the producer’s stamp).

DB events are two-tier: the committing worker fans out LOCALLY to its own subscribed pages (zero hops), then sends notifyDbEvents up; the commander fans out to the OTHER workers’ subscribed pages only (never back to the sender). Table subscriptions live in the page item — they die with the page and travel with the move package.

Login and user move

The move is a message flow on the channel, with the user package inside:

  1. At login the source worker compares the avatar’s xgroup with its own group: different → it deletes the user from its registers and ships the package (user + connections + pages + pending) on the channel; same or absent → it keeps the user.

  2. The response carries one lean login header (scalars only: user, xgroup). It is a synchronization barrier, not an event transport: the commander picks the destination BEFORE releasing the response and writes user → destination with a not-ready flag. The browser cannot send the next request before receiving this response — by then the flag is set.

  3. The package travels source → commander (held only in the in-flight per-user state machine) → destination.

  4. The destination installs and confirms; the flag drops.

  5. A request for a not-ready user WAITS on the flag (await_ready), then goes to the destination. No locks: the gate is data.

  6. A commanded move (rebalance, tomorrow consolidation) is the same flow with a different trigger: the commander sets not-ready and sends /evict_user.

The sticky cookie (sticky_cid) is minted by the WORKER in its own response; the commander is a pure relay and learns the connection from the channel event moments later. An unknown cid lands on the welcome worker — coherent, guests live there.

What is NOT here

  • No piggyback event transport, no ping-drain, no acks: on the channel, loss ⟺ death (EOF → sweep). Liveness is the channel’s native ws ping/pong.

  • No reconciliation (anti-entropy): fail-loud. full_state survives as a diagnostic endpoint only.

  • No commander-side datachange store: the pull RPC and the mailbox do not exist; a page reads its pending locally on its worker.

  • HTTP survives only as: the primary channel (forward), /_commander/* service endpoints (ping, population).

Design history

The full design discussion (decisions, dissolved mechanisms and why each could go) is preserved in temp/secondary_channel_design.md (ratified 2026-07-10).