# Elastic Pool — occupancy, admission, scale-up, compaction **Version**: 1.1.0 **Status**: SOURCE OF TRUTH (v1.1 ratified 2026-07-11) **Last Updated**: 2026-07-11 --- ## Overview The commander places logged users on pool workers and grows each group's pool on **measured pressure**, not head counts. There are no per-user caps: an idle user costs ~1 MB and ~0 cpu (measured on a real GenroPy worker), so counting heads answers the wrong question. The architecture is four stages: ``` 1. MEASURE 2. TRANSPORT 3. INTERPRET 4. DECIDE (worker, (/occupancy (commander: (commander: every 5s) on the channel) window → number) policies) ``` - **The worker is a sensor**: it reads raw quantities and pushes them. It computes no percentage and passes no judgement. - **The commander is archive and judge**: per worker it keeps a window of the last readings; the window absorbs anomalies, and time rules read directly from it — anti-flapping IS the window, not an extra mechanism. ## The sensor report (fixed shape) Every `occupancy_interval` (5s) the worker pushes `occupancy_report()` on `/occupancy`: | Field | Source | Role | |---|---|---| | `users` / `pages` / `connections` | register counts | drift sentinels only — decisions use the commander's surface, which is fresher | | `active_users` | users with a non-system RPC within the window | activity (the touch happens at route resolution; `@route(meta_sysrpc=True)` routes never count) | | `executor` `{busy, total, queue_depth}` | dispatch executor metrics | saturation signal | | `cpu` | `getrusage` delta between ticks | process cpu fraction (None on first tick) | | `rss` | `/proc/self/status` (Linux; None elsewhere) | memory, bytes | | `backlog` | pending datachanges, outbox size | drain sentinels | | `load` | `getloadavg` | machine-level, diagnostic only (not attributable to one worker) | The commander archives each report in `worker_metrics` — a `deque(maxlen=METRICS_WINDOW=60)` of rows `{ts, report, forward}`, where `forward` snapshots the cumulative forward counters (requests, errors, seconds) the commander clocks around every relay. ## The evaluator `OccupancyEvaluator` (owned by the commander) turns a worker's window into an occupancy in [0, 1]: - per-row components: `memory` = rss / configured limit (present only when the reading has an rss AND `memory_limit_mb` is set), `cpu` = min(cpu, 1.0) (the measured wall is one core — the GIL), `executor` = busy/total; - each component is FIRST averaged over the last `SMOOTHING_ROWS = 6` rows (~30s — transient spikes dominate a naive read), THEN the occupancy is the MAX of the averaged components: the bottleneck answers "can I take another user?", an average across incommensurable resources does not; - a worker with no rows (just born) reads 0.0 — it admits. It also serves the monitor: `history_of` (per-row occupancy across the whole window, the histogram), `rates_of` (rps and latency from forward-counter deltas), `components_of`. Measured baseline (real GenroPy worker, 2026-07-10): RSS ~34 MB at boot, ~100 MB after the first page (lazy imports + ORM), +~1 MB per connection, under pressure cpu ≈ one core (GIL) at ~35 rps with the executor saturated — cpu and executor are the true saturation signals; scale-out by process is the correct answer to the GIL. ## Policies 1. **Placement (reception-first)**: every user is born on the group's reception (its FIRST routable worker) — guests live there, and a login happens where the guest already is. On login the reception KEEPS the user while its own occupancy is under `reception_threshold` (default 0.5): keeping is the null action. Over the threshold it PASSES: the user is assigned among the OTHER workers (rule 2). The sole worker of a group always keeps (a group of one cannot refuse a login). Users the reception kept stay kept — re-placing them later is compaction's business, not placement's. *(Replaces the v1.0 binary rule that excluded the reception whenever the group had more than one worker: `reception_threshold=0` reproduces it, `=admission_threshold` makes the reception a plain worker.)* 2. **Admission (the pass)**: among the routable workers other than the reception, pick the one with the MINIMUM occupancy under `admission_threshold` (default 0.8). None under threshold → place on the last one with a warning, and let check_capacity grow the group. 3. **Scale-up**: `check_capacity` spawns a worker when the group cannot place well: no non-reception worker under `admission_threshold` — or, in a group of one, the reception over its keep-threshold (the moment it first needs to pass and has nobody to pass to). Guards: a tracked child not yet routable is a spawn in flight (wait for it); `max_workers` caps the group (None = unbounded). 4. **Scale-down = compaction**: governed by the capacity ledger (next section). Drain the least-occupied non-reception worker onto the survivors with commanded moves (see "Moving a live user"), then retire it. `min_workers` is the floor (default 1: the reception). The reception is never compacted. Placement and admission evaluate **on demand**, on the request path (a login arriving); the capacity ledger is checked when an `/occupancy` report arrives — the periodic signal that already exists. No commander timer either way. ## Compaction — the capacity ledger Each worker contributes its threshold as useful capacity: the reception `reception_threshold` (0.5), every other worker `admission_threshold` (0.8). For a group of N workers: ```text C = 0.5 + (N−1) × 0.8 total useful capacity O = Σ occupancy(worker) measured, from the evaluator H = C − O headroom compact when H > 1.5 × 0.8 margin: one and a half workers' worth ``` Example (thresholds read as points): 4 workers → C = 290. At O = 230, H = 60: nothing to do. At O = 150, H = 140 > 120: compact — drain the least-occupied outer worker, retire it → C = 210, H = 60: stable. Two properties fall out of the 1.5 margin: - **Hysteresis built in**: retiring a worker costs 0.8 of headroom, so post-compaction H stays over 0.4 — never near the scale-up trigger (H ≈ 0). The two policies cannot chase each other. - **Moves always fit**: free space on the survivors after the retirement is `H − 0.8 + o(drained)`, i.e. at least `0.4 + o(drained)`: the drained worker's users always fit, with half a worker to spare. A worker already at zero users is trivially the minimum and its drain is an empty move-set — the passive case needs no rule of its own. The check repeats while H stays over the margin. `reception_threshold`, the margin and `min_workers` are commander parameters (constructor today, per-group config when that lands — the destiny already noted for `SMOOTHING_ROWS`). ## Moving a live user The move of user U from worker A to worker B is a quiesce–snapshot–switch, orchestrated by the commander over the pool channel (`send_to`): ```text 1. FLAG U marked "moving" on the surface: new requests for U are HELD on the commander (await with a hard timeout), not forwarded. 2. QUIESCE wait until U has no live request. The forward path keeps a dict of OPEN requests (id → {user, worker, ts}); quiesce = no entry for U. A dict, not a counter: a drifted counter never heals, a stale entry is visible and swept by its timestamp. 3. SNAPSHOT ask A for U's package: the user's register slice, its connection and page entries, plus U's undelivered mailbox deposits — one pickled blob (safe: compaction stays within a group, one runtime). 4. INSTALL send the package to B; B answers ok once installed. 5. SWITCH the surface re-points U → B, the flag drops, held requests release toward B. ``` Failure semantics: B not answering within the timeout → abort, the flag drops, U stays on A (the next ledger check retries). A dying mid-move is a normal worker death: U's state dies with the process and the surface sweeps. Guests never move — they live on the reception, which is never compacted. ## Per-user consumption and the monitor The commander attributes every forward it clocks to the user resolved from the sticky cookie: `user_consumption[user] = {requests, seconds}`, cumulative, forgotten when the user leaves the surface. Exposed by `/_commander/population` — an async route that fans out to every routable worker's `/_monitor/state` with a DEDICATED httpx client (monitor traffic never mixes with the forward pool: httpx connections are loop-affine) and fuses consumption into the user rows. A worker that does not answer degrades to an `error` row, never an exception. `monitor_state()` adds `metrics` (occupancy, components, history, rates, forward counters, all from the evaluator); the server monitor page renders per-app tabs — state, metrics (occupancy bars + SVG histogram), population. ## Group = runtime version Groups exist to serve different app versions (green/blue/canary) on different runtimes; elasticity always scales WITHIN a group. Vehicles: a venv per group (`python=` on the group config node) is supported today; container spawn (podman/docker) is a door kept open — the channel already does TCP and the announce carries a reachable host as data. ## Design history The ratified contract (requirements, measured numbers, decision log) is `temp/worker_occupancy_elastic_pool.md` (v1.0, 2026-07-10; the temp draft was removed once distilled here). v1.1 (2026-07-11, decided in session): reception-first placement with a keep threshold (replacing the binary reception exclusion), scale-down as compaction driven by the capacity ledger with the 1.5-worker margin, and the live-user move protocol (flag → quiesce on the open-requests dict → pickled package → install → switch).