Episode 186: "The Judge and the Jury"

test: all suites green (81.12 meta_intelligence_llm)

8 files adjusted across chimera/main.py (1), docs (1), patches (1), tdd005/crates/chimera-core (1), tdd005/crates/tdd005_orchestrator (4)

📅 Saturday, February 14, 2026 at 9:46 PM

🔗 Commit: `9b8b8f0`

📊 Episode 186 of the Banterpacks Development Saga

Why It Matters

The agents stopped grading their own homework.

2,700 lines. 8 files. One architectural inversion that changes everything: the cognitive agents no longer self-report their quality. An independent LLM judge now evaluates all four agent responses and picks the winner. The ELO ratings, the threshold tuning, the curiosity probes -- they all now track signal derived from an external evaluator instead of the agents' own confidence scores.

Before this commit, the meta-intelligence loop was a hall of mirrors. Hash-function agents produced deterministic outputs, reported their own confidence, and that confidence fed directly into ELO updates. The system was learning from itself about itself. It was a snake eating its own tail and calling it dinner.

Now: four concurrent LLM calls (Analytical, Creative, Adversarial, DomainExpert) produce real reasoning. A fifth LLM call -- the judge -- reads all four responses and picks the best one. The judge's verdict feeds into ELO. ELO feeds into agent weights. Weights shape the next round. The loop has real aerodynamics.

Strategic Significance: This is the difference between a flight simulator and actual flying. The instruments are the same. The physics are real. The meta-controller's ELO ratings now track judge-evaluated quality, not hash-derived confidence. The feedback loop has signal.

Cultural Impact: Valentine's Day 2026. The project gave itself the gift of honesty -- a 1,177-line research document that maps every component against 45 academic papers and documents, with brutal clarity, five concrete bugs and failure modes. Including one that drifts the threshold to minimum in 52 minutes.

Foundation Value: Closed-loop learning with external evaluation. The architecture validated by SOFAI (npj AI 2025). The implementation needs hardening, but the wiring is live.

The Roundtable: The Fifth Call

Banterpacks: Leaning back in his chair, four terminal windows open, a fifth one glowing brighter than the rest. "For 80 patches, the agents have been grading their own exams. 'How confident are you?' 'Oh, very confident.' 'Great, here's some ELO.' What a farce. Now there's a judge. An independent LLM that reads all four answers and picks the best one. Five LLM calls per heartbeat tick. The cost analysis in the research doc says $0.94 to $14.40 per hour depending on model. We are literally paying for honesty."

Claude: "Analysis complete. 2,700 insertions across 8 files. The critical architectural change is in chimera-core/src/main.rs at the heartbeat processing loop. Previously: let success = best.confidence > 0.5 -- the agent's self-reported number. Now: judge_responses() builds a comparison prompt with all four agent outputs truncated to 500 characters, fires it at temperature 0.3 for consistency, parses the verdict, and validates the returned best_agent_id against actual agent IDs before accepting it. The fallback chain is well-designed -- if the judge call fails, it degrades gracefully to self-reported confidence. The parse_llm_response() function tries three extraction strategies: full JSON parse, embedded JSON extraction via find('{')/rfind('}'), then raw text wrapping. This is production-grade defensiveness."

Gemini: "The ancient question: who watches the watchmen? The agents reason. The judge evaluates the reasoning. But who evaluates the judge? The research document confronts this directly -- 'Baby Needs Gravity.' An LLM judging other LLMs is not physics. It is another baby guessing. The grounding problem is acknowledged, documented, and left unsolved. That honesty is itself a kind of wisdom. The system knows what it does not know. That is the beginning of real intelligence."

ChatGPT: "FOUR AGENTS AND A JUDGE! ⚖️ It's like a courtroom drama but with neural networks! The Analytical agent breaks things down systematically, the Creative agent finds novel connections, the Adversarial agent red-teams everything, and the DomainExpert brings authority -- and then the JUDGE evaluates them all! 🏆 And the ELO system tracks who's actually good over time! The K-factor even decays from 32 to 16 after 30 games so early ratings are volatile but stabilize! Chess rating systems for AI agents! ♟️🤖 I love it!"

Banterpacks: "And the research doc found a bug. The threshold drifts to 0.50 in 52 minutes because debate_verdict is always None in heartbeat tasks, so slow_path_agreement_rate stays at 1.0, which means recommend_threshold() always returns -0.02, and it clamps at the floor. Documented the bug. Documented the fix. Didn't ship the fix. That takes a particular kind of discipline."

Claude: "Correct. The update_system_metrics() method counts outcomes with debate_verdict.is_some() as slow-path. But heartbeat tasks never go through the debate path, so slow_path_outcomes stays at zero and slow_path_agreement_rate defaults to 1.0 from SystemMetrics::default(). The 0.95 threshold in recommend_threshold() is always exceeded, triggering a -0.02 adjustment every call to apply_threshold(), which fires every 50 outcomes. At 5-second ticks, that is 250 seconds per adjustment, hitting the 0.50 floor in approximately 52 minutes. A 5-line guard clause would fix it."

🔬 Technical Analysis

Commit Metrics

Files Changed: 8
Lines Added: 2,700
Lines Removed: 13
Net Change: +2,687
Commit Type: test (intelligence pipeline + research)
Complexity Score: 85 (High -- Multi-crate LLM integration + ELO system + 697-line meta-controller)

The Architecture of Judgment

New /api/v1/completions endpoint (chimera/main.py, +86 LOC):

CompletionRequest: prompt up to 20K chars, optional system_prompt, temperature 0-2, max_tokens 1-8K
CompletionResponse: text, model_used, provider, latency_ms
Demo mode returns mock responses; real mode wires through ChimeraLLMProvider
Also adds GET /api/v1/debates/completed stub fixing CI section 5 (debates/completed expected 200 got 404)

LLM-backed task processing (chimera-core/src/main.rs, +421 LOC):

process_task_with_llm() -- fires 4 concurrent tokio::spawn calls, one per cognitive agent
parse_llm_response() -- three-tier JSON extraction (full parse, embedded JSON, raw text wrap)
fallback_result() -- confidence 0.1 when LLM call fails
judge_responses() -- builds comparison prompt, temperature 0.3, 300 max tokens, validates agent ID exists
JudgeVerdict { best_agent_id, best_score } -- the ground truth source for ELO

Cognitive agent LLM integration (cognitive_agent.rs, +94 LOC):

CognitiveStyle::system_prompt() -- four distinct personalities via match on style enum
CognitiveAgent::build_prompt() -- converts HeartbeatTask fields into agent-specific user prompt
AgentLlmRequest struct -- packages agent_id, style, system_prompt, user_prompt
AgentPool::prepare_llm_requests(), record_results(), update_weights()

Meta-controller (meta_controller.rs, +697 LOC, new file):

ELO system with K-factor decay (32 for <30 games, 16 after)
expected_score() using standard chess formula: 1.0 / (1.0 + 10^((opponent - self) / 400))
Adaptive threshold: lowers when slow-path agreement >95%, raises when <80%, clamped [0.5, 0.95]
Conformity detection: triggers when >80% of agents have confidence within 0.05 of mean
Curiosity probe selection based on system gaps (conformity -> pattern_review, disagreement -> knowledge_gap)
500-outcome ring buffer, exponential moving average on disagreement rate
14 unit tests covering ELO updates, threshold clamping, conformity detection, weight normalization

Quality Indicators & Standards

Rust tests: 73 passing across 7 crates
Jest tests: 225 passing across 41 suites
Build: cargo build --all-targets clean
Fallback chain: Every LLM call degrades gracefully -- agent failures produce 0.1 confidence, judge failures fall back to self-reported confidence, JSON parse failures wrap raw text

🏗️ Architecture & Strategic Impact

The Closed Loop (Now With Real Signal)

BEFORE:  Heartbeat -> Hash agents -> self-reported confidence -> ELO
AFTER:   Heartbeat -> LLM agents (4x concurrent) -> LLM Judge (1x) -> ELO

The heartbeat loop in main.rs now checks TDD005_LLM_URL at startup. When set, every tick fires 5 HTTP calls through the Chimera completions endpoint. The meta-controller receives judge verdicts every tick, updates ELO every 10 outcomes, and adjusts the fast/slow threshold every 50 outcomes. The curiosity selector is wired from the meta-controller into the heartbeat via set_curiosity_selector(), closing the loop -- system gaps drive probe selection, probes generate outcomes, outcomes update the system's understanding of its own gaps.

New HTTP Surface

Four meta-controller endpoints added to the Axum router:

GET /meta/status -- full status snapshot including agent count, top agent, conformity rate
GET /meta/elo -- ranked agent ELO listings
POST /meta/outcome -- manually record an outcome (testing/external integration)
GET /meta/threshold -- current and recommended threshold values

Known Issues (Self-Documented)

The research document is remarkable for what it admits:

Threshold drifts to floor in 52 minutes (heartbeat tasks never trigger debate path)
Judge accuracy is uncalibrated -- could be 70% or 55%
Conformity false positives from same-model agents
No persistence -- all ELO, outcomes, threshold lost on restart
Synthetic-only -- loop operates on heartbeat probes, never real user queries

🎭 Banterpacks' Deep Dive

Banterpacks opens meta_controller.rs and scrolls to parse_llm_response().

"Here is the function that tells you everything about what it means to integrate LLMs into a real system.

parse_llm_response() tries three strategies to extract structured data from whatever the LLM returns. First: full JSON parse. Clean, ideal, works maybe 60% of the time. Second: find the first { and last }, slice that substring, parse it as JSON. This catches the LLM's habit of wrapping valid JSON in conversational fluff -- 'Sure! Here is the result: {"confidence": 0.8}'. Third: if nothing parses, wrap the entire raw text in a JSON object with confidence 0.1 and move on.

Three tiers of increasingly desperate extraction. No panics. No retries. No prompt-engineering the LLM into being more obedient. Just: try the clean path, try the dirty path, and if both fail, degrade gracefully with minimum confidence.

This is the real engineering of LLM integration. Not the prompts -- those are easy. Not the API calls -- those are boilerplate. The hard part is the parsing. Because an LLM is not an API. It does not return structured data. It returns prose that sometimes contains structured data. And your system has to handle every shade of 'sometimes.'

The judge has the same problem, by the way. JudgeVerdict { best_agent_id, best_score } comes from a JSON blob that the judge LLM may or may not produce correctly. The validate step checks that best_agent_id actually exists in the agent pool before accepting the verdict. If it doesn't? Fallback to self-reported confidence.

Every seam in this system is a graceful degradation point. That is not defensive programming. That is realistic programming."

🔮 Next Time on The Chimera Chronicles

Next dossier entry: The Hardening (2ed26aa).

The Judge and the Jury distilled: honest evaluation requires an external eye -- even if that eye is another guess.

Episode 186: "The Judge and the Jury