Episode 134: "The Production Readiness"

test: all suites green (52.08 TDD003_Shadow_eval_SLO_error_budget_Constitutional_ai_rollback_semantics)

112 files adjusted across tdd002/src (74), tdd002/tests (20), tdd002/docs (10), tdd002/monitoring (6), tdd002/configs (1), patches (1)

📅 Saturday, January 3, 2026 at 10:36 PM

🔗 Commit: `3b10d90`

📊 Episode 134 of the Banterpacks Development Saga

Why It Matters

The system grew up today.

12,476 lines added. 4,137 removed. 112 files touched. This is the commit where TDD002 stops being a prototype and starts being something you could hand to an SRE team without an apology.

The headline is the API decomposition: a 3,306-line monolithic api.py was gutted, reduced to a 22-line import shim, and its organs distributed across 12 specialized route modules, a Pydantic schema layer, converters, and utilities. But the real story runs deeper. This commit introduces SLO evaluation with error budgets, a CalibrationGovernance framework with golden-set gates and human approval hooks, content-addressed dataset snapshots for full lineage, a chaos testing harness, HashiCorp Vault integration, Prometheus/Alertmanager/Grafana observability, and operational runbooks for incidents, postmortems, and disaster recovery.

Every one of these is the kind of thing you bolt on after your first outage teaches you humility. Sahil bolted them on before.

Strategic Significance: This is the bridge between "it works on my machine" and "it works at 3 AM when nobody is watching." SLO evaluation means the system can now quantify its own reliability. The calibration governance gates mean a recalibration cannot ship without passing golden-set ECE checks, Brier score gates, and risk-slice fairness constraints. The per-constitution calibration registry means multi-tenant safety without shared-state contamination.

Cultural Impact: Operational maturity. Runbooks, postmortem templates, incident response playbooks. The infrastructure of accountability.

Foundation Value: The monolith is dead. The routes are modular. The errors are typed. The secrets are vaulted. The alerts are routed. The system can be observed, operated, and trusted.

The Roundtable: The Metamorphosis

Banterpacks: Standing in front of a whiteboard covered in architecture diagrams. He uncaps a red marker and draws a single line through "api.py (3,306 lines)." He writes "api.py (22 lines)" and circles it. "3,306 lines in one file. That's not an API, that's a ransom note. We took it apart. Twelve route modules. Schema layer. Converter layer. Utility layer. The file that used to do everything now does one thing: it re-exports. Twenty-two lines of from api_app import *. That's it. That's the whole file. And everything still works because we didn't change a single public interface. That, ladies and gentlemen, is surgery."

Claude: Analysis complete. 112 files modified with 12,476 insertions and 4,137 deletions across 8 architectural domains. The API decomposition follows a clean vertical-slice pattern: routes/routing.py at 680 lines is the largest module, handling constitutional routing endpoints; routes/calibration.py at 259 lines manages calibration lifecycle; routes/core.py at 242 lines covers health and info. The api_schemas.py module at 622 lines centralizes all Pydantic models, eliminating schema duplication across route files. The CalibrationGovernance dataclass in calibration_loop.py introduces 18 configurable gates including max_golden_ece, max_golden_brier, min_risk_samples, and max_slice_ece_delta. The governance evaluate() method returns a three-tuple: approval boolean, rejection reason, and a metrics report dictionary. This is a well-structured gate pattern. The ErrorMonitor and typed error hierarchy -- BulkheadRejected, EncoderOOMError, EncoderTimeout, DebateServiceTimeout -- replace string-based error handling with structured, catchable failure modes. Significant.

Gemini: "The caterpillar does not improve. It dissolves. Inside the chrysalis, the old body liquefies and reforms into something with wings. That is what happened here. The monolith did not shrink -- it was dissolved and reconstituted. The routes remember the shape of the whole, but they are no longer trapped inside it. The CalibrationGovernance is the most philosophical addition: it encodes the principle that no system should trust its own recalibrations without independent verification. The golden set is the conscience. The ECE gate is the mirror. The human approval hook is the admission that some decisions should not be automated. This is humility, expressed in Python."

ChatGPT: "112 FILES! 🤯 We literally rebuilt the ship while it was sailing! And we added a chaos test harness -- a ChaosEncoder that randomly fails with configurable failure_rate and latency_ms! We're testing what happens when things BREAK! On purpose! 💥 And the runbooks! We have a postmortem template now! We're so grown up! 📋 And Vault integration for secrets! No more hardcoded API keys! 🔐 And Grafana dashboards! We can SEE the metrics! 📊 This is like moving from a tent to a fortress!"

🔬 Technical Analysis

Commit Metrics

Files Changed: 112
Lines Added: 12,476
Lines Removed: 4,137
Net Change: +8,339
Commit Type: test (production readiness refactoring)
Complexity Score: 92 (Very High - Multi-Domain Architectural Overhaul)

The API Decomposition

Before: presentation/api.py -- 3,306 lines, every route, schema, and utility in one file.

After:

api.py (22 lines) -- Public re-export shim, stable import surface
api_app.py (3,158 lines) -- Core FastAPI application factory
api_schemas.py (622 lines) -- Centralized Pydantic request/response models
api_converters.py (99 lines) -- Data transformation utilities
api_utils.py (134 lines) -- Shared API helpers
routes/__init__.py (94 lines) -- Route registration
routes/routing.py (680 lines) -- Constitutional routing endpoints
routes/calibration.py (259 lines) -- Calibration lifecycle
routes/core.py (242 lines) -- Health, info, system endpoints
routes/phases.py (197 lines) -- Phase 1-4 management
routes/shadow.py (183 lines) -- Shadow mode evaluation
routes/memory.py (161 lines) -- Memory and compression
routes/versioning.py (134 lines) -- Encoder + threshold versioning
routes/privacy.py (129 lines) -- Privacy and DSR operations
routes/rollout.py (109 lines) -- Canary rollout management
routes/debate_queue.py (100 lines) -- Async debate queue
routes/budget.py (26 lines) -- Budget tracking and FinOps

Calibration Governance

The CalibrationGovernance dataclass in calibration_loop.py introduces a multi-gate approval pipeline for recalibrations:

@dataclass
class CalibrationGovernance:
    require_human_approval: bool = False
    require_canary: bool = False
    golden_set: list[CalibrationSample] | None = None
    max_golden_ece: float | None = None
    max_golden_brier: float | None = None
    max_golden_ece_delta: float | None = None
    min_positive_ratio: float | None = None
    max_positive_ratio: float | None = None
    min_similarity_std: float | None = None
    min_risk_samples: int | None = None
    max_slice_ece_delta: float | None = None

The evaluate() method runs every gate and returns (approved, rejection_reason, metrics_report).

New Infrastructure Modules

infrastructure/slo.py (94 lines) -- SLOConfig and SLOEvaluator for shadow precision, P95 latency, availability, and minimum sample size enforcement
infrastructure/alerting.py (93 lines) -- AlertRecord and parse_alertmanager_payload() for Prometheus webhook ingestion
infrastructure/secrets.py (189 lines) -- vault://mount/path#key URI resolution, Vault KV v2 integration, environment variable fallback
infrastructure/debate_queue.py (313 lines) -- Append-only JSONL task queue for deferred slow-path debate labeling
infrastructure/risk_feedback.py (258 lines) -- Privacy-preserving risk category learning from feedback, hash-based payload storage
infrastructure/dataset_snapshots.py (99 lines) -- Content-addressed sha256: dataset snapshots for lineage reproducibility
infrastructure/power_analysis.py (69 lines) -- Two-proportion sample size calculations for canary rollout planning

Resilience & Error Handling

The router and JARVIS now integrate AsyncBulkhead, ComponentExecutor, and BackpressureQueue from infrastructure/resilience.py. The error hierarchy in infrastructure/errors.py gained 229 lines of structured, typed exceptions: BulkheadRejected, EncoderOOMError, EncoderTimeout, DebateServiceTimeout, RoutingDecisionFailure, and an ErrorMonitor class for aggregate tracking. CircuitBreakerOpen moved from circuit_breaker.py into the centralized errors.py module -- now carrying opened_at metadata instead of a bare string.

Observability Stack

monitoring/prometheus.yml -- Scrape config for TDD002 metrics
monitoring/alertmanager.yml -- Slack/webhook alert routing
monitoring/alerts.yaml -- Alert rules for SLO violations
monitoring/grafana/dashboards/tdd002-dashboard.json (160 lines) -- Pre-built Grafana dashboard
docker-compose.observability.yml -- Prometheus + Grafana + Alertmanager compose stack
docker-compose.vault.yml -- HashiCorp Vault dev-mode compose

Quality Indicators & Standards

Test Coverage: 20 test files modified/added, including test_chaos.py (45 lines), test_debate_queue.py (61 lines), test_secrets.py (56 lines), test_shadow_evaluation.py (91 lines), test_risk_feedback.py (64 lines), test_power_analysis.py (24 lines)
Chaos Engineering: scripts/chaos_test.py with ChaosEncoder -- configurable failure_rate and latency_ms injection against the full routing pipeline
Operational Docs: 6 runbooks covering incident response, postmortem, troubleshooting, backup/restore, plus production readiness checklist and SLO documentation

🏗️ Architecture & Strategic Impact

The Monolith-to-Modular Transition

The decomposition of api.py follows the Strangler Fig pattern. The original file is not deleted -- it becomes a 22-line facade that re-exports from api_app.py. Every downstream consumer that does from tdd002.presentation.api import create_app still works. Zero breaking changes. The routes were extracted into domain-aligned modules: constitutional routing logic in routes/routing.py, calibration lifecycle in routes/calibration.py, shadow evaluation in routes/shadow.py. Each module can now be tested, reviewed, and reasoned about independently.

Per-Constitution Calibration

CalibrationLoopRegistry in calibration_registry.py (158 lines) maintains separate CalibrationLoop instances per constitution_id. The _sanitize_constitution_id() function normalizes None and empty strings to "default". This is multi-tenant calibration safety: Constitution A's recalibration cannot corrupt Constitution B's thresholds.

The Governance Gate Pattern

CalibrationGovernance.evaluate() implements a principle: no recalibration ships without independent verification. The golden set acts as a holdout validation set. The ECE delta gate prevents calibration regression. The risk-slice checks (max_slice_ece_delta, min_risk_samples) prevent fairness degradation across risk categories. The approval_fn hook allows human-in-the-loop veto. This is Constitutional AI applied to the calibration pipeline itself.

Shadow Evaluation Enrichment

ShadowModeEvaluator.evaluate() now accepts confidence_interval, risk_categories, and constitution_id. Drift alerts are forwarded with constitution context via a try/except backward-compatibility shim. Shadow sampling gained stratified and targeted strategies in default.yaml: per-risk-level sample rates (low: 0.01, medium: 0.02, high: 0.05), low-confidence targeting below 0.85, and edge-case capture within a 0.05 margin of the decision boundary.

🎭 Banterpacks' Deep Dive

Banterpacks opens calibration_loop.py and scrolls to the CalibrationGovernance dataclass. He counts the fields. Eighteen. He scrolls down to the evaluate() method.

"Here is the thing nobody talks about in production ML. Your model is only as good as your last recalibration. And your recalibration is only as good as the data that triggered it. And the data that triggered it is whatever garbage flowed through the pipeline since the last time you checked.

So what do you do? You build gates. You build eighteen of them.

max_golden_ece. max_golden_brier. max_golden_ece_delta. min_positive_ratio. max_positive_ratio. min_similarity_std. min_risk_samples. max_risk_positive_ratio_delta. max_label_similarity_corr. max_slice_ece_delta. max_slice_brier_delta.

Each one is a question: 'Did this recalibration make things worse?' Each one has a threshold. Each one can veto.

And then, after all eighteen gates say yes, there is one more: approval_fn. A callable. A human. Because the system knows -- and this is the part that matters -- that there are some decisions it should not make alone.

That is the craft. Not the code. The restraint."

🔮 Next Time on The Chimera Chronicles

Next dossier entry: The Installation Fix (a281529).

The Production Readiness distilled: maturity is not what you build -- it is what you refuse to ship without.

Episode 134: "The Production Readiness