Incidents & Retros
Production maturity is easier to claim than to demonstrate. These are real incidents from the system, written tight: what failed, how it was caught, what caused it, what fixed it, and what changed structurally so the same class of bug doesn't recur.
The public set is a curated selection, not every incident — listed newest first, across trading, ML modeling, data contracts, evaluation, security, and client reliability. The private interview kit holds deeper retros with full code paths and naive first attempts.
"Configured" is not "working" — the PWA that never told open tabs to reload
2026-07-08 · P1 · Client reliability
A PWA plugin was set to auto-update from day one, but the app never wired up the piece that actually tells an open tab to reload. Every deploy shipped correctly to the server; installed clients could keep running the old bundle indefinitely.
Direction argmax flip — the book flipped on zero new information
2026-07-06 · P1 · Model logic
Predicted direction was derived from a probability threshold that didn't match the label's true base rate, so ordinary day-to-day noise could flip almost the entire book. The fix was a one-line change to a single function.
The false-positive trio — three RED flags, three broken estimators
2026-06-07 · P2 · Evaluation harness
A single week produced three simultaneous critical RED flags on the system's own report card. All three turned out to be artifacts of how the metrics were estimated, not real problems — so the fix made the broken estimator classes impossible to construct at all.
avg_volume_20d units mismatch — 901 of 903 tickers silently failed a liquidity gate
2026-05-25 · P1 · Data contracts
A feature column was written as a normalized ratio and read as raw share volume. Nothing crashed for months — the scanner just quietly rejected almost the entire universe. This is the incident that started the units-suffix naming contract.
Stored prompt injection — a memory system that flagged itself as malware
2026-05-17 · P1 · Security
A memory server captured its own host's control markup as if it were ordinary conversation, replayed it into a later session, and watched a model correctly — and alarmingly — treat it as an active attack. The fix rewrote the system's entire trust model for recalled memory.
EOD pipeline recovery — 50 minutes to 4.5 minutes
2026-05-01 · P2 · Infrastructure
Three independent issues stacked into a 10× runtime regression. Closing them the same day cut runtime by an order of magnitude and propagated a defensive pattern to other parts of the data layer.
Predictor meta-model collapse — 27 UP / 0 DOWN
2026-04-28 · P1 · ML model
A weekly retrain produced a degenerate output distribution. Root cause was placeholder constants in production training data; the fix moved validation IC from 0.053 to 0.132.
PFE short-sell — accidental short on a long-only system
2026-04-22 · P1 · Trade execution
The executor opened a short position on a stock the system was only supposed to be exiting. Defense-in-depth fix shipped same day.