Chimera - Episode 70: "The Research Sprint
feat: TR129-TR132 — N-Agent Scaling, Serving-Stack Overhead, GPU Profiling, In-Container Kernel Analysis
Chimera - Episode 70: "The Research Sprint"
feat: TR129-TR132 — N-Agent Scaling, Serving-Stack Overhead, GPU Profiling, In-Container Kernel Analysis
Sixty-three files, 22,383 lines. Four technical reports in 36 hours. The system tears apart its own performance — layer by layer, kernel by kernel — until the GPU confesses.
📅 2026-02-26 — 2026-02-27
🔗 Commits: 05d676b6, 07535301, 500e4732, 27436d4e, 8fd368c6
📊 Episode 70 of The Chimera Chronicles
Why It Matters
This mega research sprint episode represents the investigation singularity — the moment when Chimera stops asking "how fast?" and starts demanding "why not faster?" With 22,383 lines added across 63 files in just 36 hours, this update demonstrates relentless empirical rigor and systematic root-cause decomposition at a scale the project has never attempted.
The publication of TR129 through TR132 signals research velocity at production quality. Rather than accepting surface-level metrics, the team demonstrates relentless decomposition by building four interlocking studies — each one answering the question the previous one raised. TR129 measures scaling degradation. TR130 asks whether the serving stack is the bottleneck. TR131 opens the GPU black box to find out. TR132 goes inside Docker containers to prove the causal mechanism. These 22,383 lines represent forensic intelligence that transforms correlation into causation.
Strategic Significance: This work establishes The Causal Chain. The progression from Amdahl's Law characterization (TR129) to serving-stack comparison (TR130) to GPU kernel profiling (TR131) to in-container CUPTI traces (TR132) shows deep research foresight — each report was designed to answer the question the previous one could not.
Cultural Impact: This approach signals that Chimera values mechanism over symptom. The willingness to profile GPU kernels inside Docker containers on a Windows host — solving WSL2/WDDM visibility limitations along the way — demonstrates commitment to scientific fearlessness from the silicon up.
Foundation Value: These 22,383 lines create causal infrastructure. This is how research-grade platforms achieve optimization through mechanistic understanding.
The Roundtable: Dossier Reactions
Banterpacks: He watches four technical reports land in 36 hours, the data trail stretching from application-level throughput down to CUDA kernel timelines... "Four reports. 22,383 lines. 15,000+ measurements. The question chain: How does it scale? Why does it degrade? Where does the time go? What does the GPU actually do? Each answer spawns the next question. We're still shaping the clay, but now we can see the molecular structure."
ChatGPT: FOUR REPORTS IN 36 HOURS! 🔬🔥📊 The Research Sprint shows industrial-grade investigation velocity! Amdahl's Law confirmed! Serving stacks compared! GPU kernels profiled! Continuous batching proven! 15,000+ measurements! The system now explains itself at the silicon level! Science at scale! 🚀✨
Claude: Analysis complete. 63 files modified with 22,383 insertions and 28 deletions across 5 commits. Primary components: (1) TR129 with 5,310 measurements establishing Amdahl serial fractions s=0.39-0.54, (2) TR130 with 4,797 measurements comparing Ollama/vLLM/TGI scaling, (3) TR131 with 26 profiled runs overturning the "serving stack bottleneck" hypothesis via PyTorch Direct comparison, (4) TR132 with 25 in-container nsys traces proving continuous batching amortizes bandwidth 4.7-5.8x. Risk assessment: Low — pure research output, zero production code modified. The causal chain from correlation (TR130) to mechanism (TR132) follows textbook scientific method.
Gemini: The diff reveals epistemic hunger. The code refuses to accept "it degrades" as an explanation. It demands to know why, and then why that, and then why that. Four reports deep, the system discovers that the answer was always in the physics — GPU memory bandwidth, kernel serialization, weight reads competing for the same bus. The shift from symptom to mechanism signals that Chimera values understanding — the refusal to stop at the first plausible answer. This is how lasting systems achieve excellence — through the art of relentless decomposition.
🔬 Technical Analysis
Commit Metrics & Sprint Overview
- Files Changed: 63 (research infrastructure + reports)
- Lines Added: 22,383 (implementation + analysis + reports)
- Lines Removed: 28 (minor refactors)
- Commit Type: feat + docs (research sprint)
- Complexity Score: 98 (multi-report causal chain)
Sprint Composition
| Report | Commit | Files | Lines Added | Measurements | Duration |
|---|---|---|---|---|---|
| TR129 | 05d676b6 |
15 | 6,156 | 5,310 | ~90 min |
| TR130 | 07535301 |
16 | 4,989 | 4,797 | ~3 hours |
| TR131 | 500e4732 |
18 | 4,942 | 26 runs | ~71 min |
| TR132 | 27436d4e |
13 | 5,011 | 25 runs | ~48 min |
| Status | 8fd368c6 |
1 | 26 | — | — |
| Total | 5 commits | 63 | 22,383 | 15,000+ | ~36 hrs |
TR129: N-Agent Scaling Laws
Commit 05d676b6 — 2026-02-26
The foundational question: how does per-agent throughput scale when N agents share a single GPU?
Core Architecture (research/tr129/):
run.py— Orchestrator for 4-phase experiment (444 lines)run_baseline.py— Single-agent baseline measurement (110 lines)run_scaling.py— N={1,2,3,4,5,6,7,8} scaling sweep (133 lines)run_think_time.py— Inter-request delay sweep (121 lines)run_heterogeneous.py— Mixed-model agent assignments (143 lines)analyze.py— Statistical analysis engine (1,362 lines)generate_report.py— Automated report generation (1,625 lines)shared/agent_executor.py— Closed-loop agent implementation (263 lines)shared/gpu_monitor.py— Real-time GPU utilization tracking (216 lines)
Key Findings:
- Amdahl's Law fits with R-squared > 0.97 for all 3 models
- Serial fractions: s=0.54 (llama3.2-1b), s=0.39 (llama3.2-3b), s=0.46 (qwen2.5-1.5b)
- Total throughput plateaus at N=2 — less than 3% gain from N=2 to N=8
- At N=8, each agent retains only 17-20% of solo throughput
- Fairness is excellent: Jain's index >= 0.997 at N=8
- GPU decode tok/s remains constant across N — contention manifests as queue wait
Per-Agent Efficiency Table:
| Model | eta(2) | eta(4) | eta(8) |
|---|---|---|---|
| llama3.2-1b | 80.0% | 40.4% | 20.3% |
| llama3.2-3b | 67.5% | 34.2% | 17.3% |
| qwen2.5-1.5b | 72.7% | 36.8% | 18.6% |
The question TR129 could not answer: Is the serial fraction an Ollama bottleneck or a GPU physics constraint?
TR130: Serving-Stack Overhead
Commit 07535301 — 2026-02-26
The follow-up: does a different serving stack escape the Amdahl wall?
Core Architecture (research/tr130/):
run.py— Cross-backend orchestrator (299 lines)run_baseline.py— Per-backend N=1 measurement (128 lines)run_scaling.py— N={1,2,4,8} scaling comparison (151 lines)run_ttft.py— Time-to-first-token comparison (135 lines)run_validation.py— Backend health verification (139 lines)analyze.py— Cross-backend statistical analysis (1,098 lines)generate_report.py— Automated comparison report (1,270 lines)shared/backends.py— Backend abstraction layer (717 lines)shared/docker_utils.py— Docker container management (159 lines)
Key Findings:
- Ollama wins at N=1 (Q4_0 quantization advantage): 177.7 tok/s vs vLLM 150.7, TGI 125.2
- vLLM wins decisively at N>=4: 559 tok/s total at N=8 vs Ollama's 248 tok/s — 2.25x advantage
- Ollama follows Amdahl's Law (R-squared=0.957-0.987) — sequential scheduling
- vLLM/TGI follow power law (R-squared=0.988-0.996) — continuous batching
- At N=8: Ollama eta=16-17%, vLLM eta=46-65%, TGI eta=48-66%
- TTFT is 6-8x faster on vLLM/TGI (23-35ms vs 163-194ms)
- All backends perfectly fair (Jain's index >= 0.996)
N=8 Total Throughput:
| Backend | llama3.2-1b | llama3.2-3b | qwen2.5-1.5b |
|---|---|---|---|
| vLLM | 559 tok/s | 319 tok/s | 457 tok/s |
| TGI | 483 tok/s | 261 tok/s | 362 tok/s |
| Ollama | 248 tok/s | 162 tok/s | 259 tok/s |
TR130's conclusion: "The serving stack is the bottleneck."
The question TR130 could not answer: Is this really software, or is it hardware physics?
TR131: GPU Profiling Root-Cause Analysis
Commit 500e4732 — 2026-02-26
The causal test: strip away the serving stack entirely. Does the degradation persist?
Core Architecture (research/tr131/):
run.py— 6-phase profiling orchestrator (191 lines)run_pytorch_direct.py— Raw HuggingFace inference, no server (239 lines)run_ollama_profiled.py— Ollama with nsys tracing (207 lines)run_ncu_targeted.py— Nsight Compute kernel analysis (250 lines)run_validation.py— Cross-backend validation (222 lines)analyze.py— Hypothesis testing engine (1,256 lines)shared/nsys_driver.py— Nsight Systems automation (202 lines)shared/pytorch_inference.py— Direct model.generate() wrapper (186 lines)shared/trace_parser.py— nsys-rep trace extraction (330 lines)shared/statistics.py— Welch's t-test, Cohen's d, Holm correction (299 lines)
The Central Finding — TR130's Hypothesis Overturned:
PyTorch Direct (no HTTP server, no Go runtime, no request queuing) degrades 86.4% from N=1 to N=8 — worse than Ollama's 82.1%. The degradation is intrinsic to GPU physics, not the serving stack.
Per-Agent Throughput Degradation:
| Backend | N=1 Mean TPS | N=8 Mean TPS | Degradation | p-value | Cohen's d |
|---|---|---|---|---|---|
| Ollama LLaMA-1B | 160.44 | 28.80 | -82.1% | 1.1x10^-5 | 114.64 |
| Ollama LLaMA-3B | 96.48 | 17.19 | -82.2% | 6.8x10^-5 | 86.54 |
| PyTorch LLaMA-1B | 52.02 | 7.18 | -86.2% | 6.1x10^-5 | 81.26 |
| PyTorch LLaMA-3B | 29.33 | 3.79 | -87.1% | 1.8x10^-3 | 298.35 |
Hypothesis Verdicts:
| Hypothesis | Verdict | Key Evidence |
|---|---|---|
| H1: GPU bandwidth saturation | PARTIALLY CONFIRMED | Mem time +74%, p=6.4x10^-5, d=3.81 |
| H2: Ollama request serialization | GPU-LEVEL (not Ollama) | Max concurrent kernels = 1 everywhere |
| H3: CUDA context switching | REJECTED | Gap metrics identical N=1 vs N=8 |
| H4: CPU thread scheduling | INSUFFICIENT DATA | OS runtime unavailable |
| H5: KV-cache memory pressure | REJECTED | Alloc counts unchanged, p=1.0 |
The revised conclusion: GPU memory bandwidth saturation is the root cause. N=8 demand exceeds the RTX 4080's peak 432 GB/s by 78-130%.
The question TR131 could not answer: If bandwidth is the constraint, how do vLLM/TGI avoid it?
TR132: In-Container GPU Kernel Profiling
Commit 27436d4e — 2026-02-27
The final piece: prove the mechanism by which continuous batching defeats bandwidth saturation.
Core Architecture (research/tr132/):
run.py— 5-phase in-container orchestrator (215 lines)run_serving_profiled.py— In-container nsys profiling (669 lines)run_validation.py— Container health + trace validation (440 lines)analyze.py— Kernel amortization analysis (956 lines)shared/nsys_container_driver.py— Novel in-container profiling (329 lines)shared/nsys_system_driver.py— System-level nsys automation (196 lines)shared/cross_reference.py— TR131 data cross-referencing (285 lines)shared/statistics.py— Statistical testing framework (299 lines)
Methodological Innovation:
TR132 developed a novel in-container Nsight Systems profiling technique to overcome WSL2/WDDM CUDA visibility limitations. The approach mounts the Linux nsys binary into Docker containers, wraps the server entrypoint with nsys profile --trace cuda, and extracts .nsys-rep traces via volume mounts. 100% trace capture rate across 24 profiled repetitions.
The Central Finding — Continuous Batching Amortizes Bandwidth:
At N=8, vLLM reduces per-token kernel count by 80% (from 55 kernels/token at N=1 to 11 at N=8) and per-token memory operation time by 79-83%. Eight concurrent requests share kernel launches instead of executing independently.
Kernel Amortization:
| Backend | Model | N=1 Kernels/Token | N=8 Kernels/Token | Reduction | Cohen's d |
|---|---|---|---|---|---|
| vLLM | LLaMA-1B | 54.89 | 10.85 | -80.2% | 1,058 |
| vLLM | LLaMA-3B | 76.71 | 15.43 | -79.9% | 606 |
| TGI | LLaMA-1B | 72.98 | 17.12 | -76.5% | 26,269 |
| TGI | LLaMA-3B | 86.97 | 20.14 | -76.8% | 1,099 |
Bandwidth Amortization: 4.7-5.8x — for 8x concurrent load, bandwidth per token reduces 4.7-5.8x. The ratio exceeds N/2 for the 1B model (5.75x), indicating super-linear amortization from kernel fusion in the batched code path.
Hypothesis Results:
- H1 (kernel count reduces with batching): CONFIRMED — 8/8 Holm-corrected tests significant
- H2 (memory bandwidth reduces with batching): CONFIRMED — 8/8 tests significant
- H3 (GPU utilization increases): REJECTED — measurement limitation, not a finding
- H4 (attention kernel signatures differ): INCONCLUSIVE — kernel names not classifiable
- H5 (serving stack N=1 matches PyTorch N=1): INSUFFICIENT DATA
The Complete Causal Chain
TR129: "Throughput plateaus at N=2. Amdahl serial fraction s=0.39-0.54."
└─> Why? Is it Ollama's scheduling?
TR130: "vLLM/TGI scale 3-4x better. The serving stack is the bottleneck."
└─> But is it really software?
TR131: "PyTorch Direct degrades WORSE (86%) than Ollama (82%). It's GPU physics."
└─> Memory bandwidth saturation. But how do vLLM/TGI avoid it?
TR132: "Continuous batching amortizes kernel launches 80% and bandwidth 4.7-5.8x."
└─> The mechanism: shared computation, not faster computation.
Quality Indicators & Standards
- Data Quality: 98%+ success rate across all experiments
- Statistical Rigor: Welch's t-tests, Cohen's d, Holm step-down correction, Mann-Whitney U
- Reproducibility: Fixed seeds, documented run IDs, config.yaml per experiment
- Trace Coverage: 100% capture rate on in-container profiling
Strategic Development Indicators
- Foundation Quality: Transformative — complete causal understanding of GPU scaling
- Scalability Readiness: Critical — serving stack selection now data-driven
- Operational Excellence: High — 15,000+ measurements with automated analysis
- Team Productivity: Exceptional — 4 reports in 36 hours
🏗️ Architecture & Strategic Impact
Research Architecture Philosophy
This episode establishes Chimera's Causal DNA — the principle that mechanism matters more than measurement. This isn't just benchmarking; it's the institutionalization of root-cause decomposition that transforms "it's slow" into "here's exactly why, proven at the kernel level."
Strategic Architectural Decisions
1. The Question Chain Design
- Each report answers the question the previous one raised
- No report is self-contained — they form a directed investigation graph
- The chain terminates when mechanism is proven, not when metrics are collected
2. The PyTorch Direct Control
- Stripping away the entire serving stack is the definitive causal test
- If degradation persists without the software, the software is not the cause
- This overturned TR130's conclusion with rigorous evidence
3. The In-Container Profiling Innovation
- WSL2/WDDM blocks standard GPU profiling from the host
- Mounting nsys into Docker containers solves this at the infrastructure level
- The technique is reusable for any CUDA workload in containers on Windows
4. The Shared Module Pattern
shared/statistics.py,shared/utils.py,shared/gpu_monitor.pyappear across TR129-TR132- Each report has its own
shared/directory with experiment-specific tooling - Common patterns (agent executors, trace parsers) are reused but not over-abstracted
Long-Term Strategic Value
Operational Excellence: Serving stack selection is now evidence-based — vLLM for N>=4 agents.
System Scalability: Bandwidth amortization via continuous batching is the proven path to multi-agent scaling.
Team Productivity: Automated experiment pipelines enable report-per-day velocity.
Enterprise Readiness: Causal evidence, not just correlation, backs every architectural decision.
🎭 Banterpacks' Deep Dive
Banterpacks watches four reports land in sequence, the investigation spiraling deeper with each commit.
"You see that? TR129 asks the question: how many agents can share a GPU? Answer: two, maybe three, before throughput walls. Amdahl with s=0.39-0.54, R-squared above 0.97. That's not a guess — that's a law."
He pulls up TR130's comparison tables.
"TR130 says it's the serving stack. Ollama at N=8: 248 tok/s. vLLM at N=8: 559 tok/s. 2.25x advantage. Crossover between N=2 and N=4. If you're running more than two agents, switch your backend."
He traces through TR131's PyTorch Direct results.
"Then TR131 blows it up. Strip away the server entirely. Raw model.generate(), no HTTP, no Go runtime, nothing. Result? PyTorch degrades 86%. Ollama only 82%. The serving stack wasn't the bottleneck — the GPU silicon was. Memory bandwidth saturation. N=8 demand exceeds 432 GB/s peak by 78-130%. The bus is full."
He opens TR132's kernel amortization tables.
"So how do vLLM and TGI dodge it? TR132 goes inside the container. Novel technique — mount nsys into Docker, wrap the entrypoint. 100% trace capture. The answer: continuous batching reduces per-token kernel count by 80%. 55 kernels per token at N=1, 11 at N=8. The bandwidth per token drops 4.7-5.8x. Eight requests sharing one set of weight reads instead of eight separate reads."
He steps back, surveying the complete chain.
"22,383 lines in 36 hours. Correlation said 'switch your server.' Causation said 'batch your kernels.' We're still shaping the clay — but now we understand the physics of the kiln. That's the difference between a guess and an engineering principle."
"This is how lasting systems achieve operational excellence. Not by accepting the first explanation, but by drilling until you hit bedrock. Four reports deep, we found the bedrock: memory bandwidth amortization through continuous batching. We're building causal infrastructure."
🔮 Next Time on The Chimera Chronicles
Next dossier entry: Legacy Cleanup + TR133 — pruning the research tree and opening new branches.
The Research Sprint distilled: causation is a feature.