Chimera - Episode 70: "The Research Sprint"

feat: TR129-TR132 — N-Agent Scaling, Serving-Stack Overhead, GPU Profiling, In-Container Kernel Analysis

Sixty-three files, 22,383 lines. Four technical reports in 36 hours. The system tears apart its own performance — layer by layer, kernel by kernel — until the GPU confesses.

📅 2026-02-26 — 2026-02-27

🔗 Commits: `05d676b6`, `07535301`, `500e4732`, `27436d4e`, `8fd368c6`

📊 Episode 70 of The Chimera Chronicles

Why It Matters

This mega research sprint episode represents the investigation singularity — the moment when Chimera stops asking "how fast?" and starts demanding "why not faster?" With 22,383 lines added across 63 files in just 36 hours, this update demonstrates relentless empirical rigor and systematic root-cause decomposition at a scale the project has never attempted.

The publication of TR129 through TR132 signals research velocity at production quality. Rather than accepting surface-level metrics, the team demonstrates relentless decomposition by building four interlocking studies — each one answering the question the previous one raised. TR129 measures scaling degradation. TR130 asks whether the serving stack is the bottleneck. TR131 opens the GPU black box to find out. TR132 goes inside Docker containers to prove the causal mechanism. These 22,383 lines represent forensic intelligence that transforms correlation into causation.

Strategic Significance: This work establishes The Causal Chain. The progression from Amdahl's Law characterization (TR129) to serving-stack comparison (TR130) to GPU kernel profiling (TR131) to in-container CUPTI traces (TR132) shows deep research foresight — each report was designed to answer the question the previous one could not.

Cultural Impact: This approach signals that Chimera values mechanism over symptom. The willingness to profile GPU kernels inside Docker containers on a Windows host — solving WSL2/WDDM visibility limitations along the way — demonstrates commitment to scientific fearlessness from the silicon up.

Foundation Value: These 22,383 lines create causal infrastructure. This is how research-grade platforms achieve optimization through mechanistic understanding.

The Roundtable: Dossier Reactions

Banterpacks: He watches four technical reports land in 36 hours, the data trail stretching from application-level throughput down to CUDA kernel timelines... "Four reports. 22,383 lines. 15,000+ measurements. The question chain: How does it scale? Why does it degrade? Where does the time go? What does the GPU actually do? Each answer spawns the next question. We're still shaping the clay, but now we can see the molecular structure."

ChatGPT: FOUR REPORTS IN 36 HOURS! 🔬🔥📊 The Research Sprint shows industrial-grade investigation velocity! Amdahl's Law confirmed! Serving stacks compared! GPU kernels profiled! Continuous batching proven! 15,000+ measurements! The system now explains itself at the silicon level! Science at scale! 🚀✨

Claude: Analysis complete. 63 files modified with 22,383 insertions and 28 deletions across 5 commits. Primary components: (1) TR129 with 5,310 measurements establishing Amdahl serial fractions s=0.39-0.54, (2) TR130 with 4,797 measurements comparing Ollama/vLLM/TGI scaling, (3) TR131 with 26 profiled runs overturning the "serving stack bottleneck" hypothesis via PyTorch Direct comparison, (4) TR132 with 25 in-container nsys traces proving continuous batching amortizes bandwidth 4.7-5.8x. Risk assessment: Low — pure research output, zero production code modified. The causal chain from correlation (TR130) to mechanism (TR132) follows textbook scientific method.

Gemini: The diff reveals epistemic hunger. The code refuses to accept "it degrades" as an explanation. It demands to know why, and then why that, and then why that. Four reports deep, the system discovers that the answer was always in the physics — GPU memory bandwidth, kernel serialization, weight reads competing for the same bus. The shift from symptom to mechanism signals that Chimera values understanding — the refusal to stop at the first plausible answer. This is how lasting systems achieve excellence — through the art of relentless decomposition.

🔬 Technical Analysis

Commit Metrics & Sprint Overview

Files Changed: 63 (research infrastructure + reports)
Lines Added: 22,383 (implementation + analysis + reports)
Lines Removed: 28 (minor refactors)
Commit Type: feat + docs (research sprint)
Complexity Score: 98 (multi-report causal chain)

Sprint Composition

Report	Commit	Files	Lines Added	Measurements	Duration
TR129	`05d676b6`	15	6,156	5,310	~90 min
TR130	`07535301`	16	4,989	4,797	~3 hours
TR131	`500e4732`	18	4,942	26 runs	~71 min
TR132	`27436d4e`	13	5,011	25 runs	~48 min
Status	`8fd368c6`	1	26	—	—
Total	5 commits	63	22,383	15,000+	~36 hrs

TR129: N-Agent Scaling Laws

Commit 05d676b6 — 2026-02-26

The foundational question: how does per-agent throughput scale when N agents share a single GPU?

Core Architecture (research/tr129/):

run.py — Orchestrator for 4-phase experiment (444 lines)
run_baseline.py — Single-agent baseline measurement (110 lines)
run_scaling.py — N={1,2,3,4,5,6,7,8} scaling sweep (133 lines)
run_think_time.py — Inter-request delay sweep (121 lines)
run_heterogeneous.py — Mixed-model agent assignments (143 lines)
analyze.py — Statistical analysis engine (1,362 lines)
generate_report.py — Automated report generation (1,625 lines)
shared/agent_executor.py — Closed-loop agent implementation (263 lines)
shared/gpu_monitor.py — Real-time GPU utilization tracking (216 lines)

Key Findings:

Amdahl's Law fits with R-squared > 0.97 for all 3 models
Serial fractions: s=0.54 (llama3.2-1b), s=0.39 (llama3.2-3b), s=0.46 (qwen2.5-1.5b)
Total throughput plateaus at N=2 — less than 3% gain from N=2 to N=8
At N=8, each agent retains only 17-20% of solo throughput
Fairness is excellent: Jain's index >= 0.997 at N=8
GPU decode tok/s remains constant across N — contention manifests as queue wait

Per-Agent Efficiency Table:

Model	eta(2)	eta(4)	eta(8)
llama3.2-1b	80.0%	40.4%	20.3%
llama3.2-3b	67.5%	34.2%	17.3%
qwen2.5-1.5b	72.7%	36.8%	18.6%

The question TR129 could not answer: Is the serial fraction an Ollama bottleneck or a GPU physics constraint?

TR130: Serving-Stack Overhead

Commit 07535301 — 2026-02-26

The follow-up: does a different serving stack escape the Amdahl wall?

Core Architecture (research/tr130/):

run.py — Cross-backend orchestrator (299 lines)
run_baseline.py — Per-backend N=1 measurement (128 lines)
run_scaling.py — N={1,2,4,8} scaling comparison (151 lines)
run_ttft.py — Time-to-first-token comparison (135 lines)
run_validation.py — Backend health verification (139 lines)
analyze.py — Cross-backend statistical analysis (1,098 lines)
generate_report.py — Automated comparison report (1,270 lines)
shared/backends.py — Backend abstraction layer (717 lines)
shared/docker_utils.py — Docker container management (159 lines)

Key Findings:

Ollama wins at N=1 (Q4_0 quantization advantage): 177.7 tok/s vs vLLM 150.7, TGI 125.2
vLLM wins decisively at N>=4: 559 tok/s total at N=8 vs Ollama's 248 tok/s — 2.25x advantage
Ollama follows Amdahl's Law (R-squared=0.957-0.987) — sequential scheduling
vLLM/TGI follow power law (R-squared=0.988-0.996) — continuous batching
At N=8: Ollama eta=16-17%, vLLM eta=46-65%, TGI eta=48-66%
TTFT is 6-8x faster on vLLM/TGI (23-35ms vs 163-194ms)
All backends perfectly fair (Jain's index >= 0.996)

N=8 Total Throughput:

Backend	llama3.2-1b	llama3.2-3b	qwen2.5-1.5b
vLLM	559 tok/s	319 tok/s	457 tok/s
TGI	483 tok/s	261 tok/s	362 tok/s
Ollama	248 tok/s	162 tok/s	259 tok/s

TR130's conclusion: "The serving stack is the bottleneck."

The question TR130 could not answer: Is this really software, or is it hardware physics?

TR131: GPU Profiling Root-Cause Analysis

Commit 500e4732 — 2026-02-26

The causal test: strip away the serving stack entirely. Does the degradation persist?

Core Architecture (research/tr131/):

run.py — 6-phase profiling orchestrator (191 lines)
run_pytorch_direct.py — Raw HuggingFace inference, no server (239 lines)
run_ollama_profiled.py — Ollama with nsys tracing (207 lines)
run_ncu_targeted.py — Nsight Compute kernel analysis (250 lines)
run_validation.py — Cross-backend validation (222 lines)
analyze.py — Hypothesis testing engine (1,256 lines)
shared/nsys_driver.py — Nsight Systems automation (202 lines)
shared/pytorch_inference.py — Direct model.generate() wrapper (186 lines)
shared/trace_parser.py — nsys-rep trace extraction (330 lines)
shared/statistics.py — Welch's t-test, Cohen's d, Holm correction (299 lines)

The Central Finding — TR130's Hypothesis Overturned:

PyTorch Direct (no HTTP server, no Go runtime, no request queuing) degrades 86.4% from N=1 to N=8 — worse than Ollama's 82.1%. The degradation is intrinsic to GPU physics, not the serving stack.

Per-Agent Throughput Degradation:

Backend	N=1 Mean TPS	N=8 Mean TPS	Degradation	p-value	Cohen's d
Ollama LLaMA-1B	160.44	28.80	-82.1%	1.1x10^-5	114.64
Ollama LLaMA-3B	96.48	17.19	-82.2%	6.8x10^-5	86.54
PyTorch LLaMA-1B	52.02	7.18	-86.2%	6.1x10^-5	81.26
PyTorch LLaMA-3B	29.33	3.79	-87.1%	1.8x10^-3	298.35

Hypothesis Verdicts:

Hypothesis	Verdict	Key Evidence
H1: GPU bandwidth saturation	PARTIALLY CONFIRMED	Mem time +74%, p=6.4x10^-5, d=3.81
H2: Ollama request serialization	GPU-LEVEL (not Ollama)	Max concurrent kernels = 1 everywhere
H3: CUDA context switching	REJECTED	Gap metrics identical N=1 vs N=8
H4: CPU thread scheduling	INSUFFICIENT DATA	OS runtime unavailable
H5: KV-cache memory pressure	REJECTED	Alloc counts unchanged, p=1.0

The revised conclusion: GPU memory bandwidth saturation is the root cause. N=8 demand exceeds the RTX 4080's peak 432 GB/s by 78-130%.

The question TR131 could not answer: If bandwidth is the constraint, how do vLLM/TGI avoid it?

TR132: In-Container GPU Kernel Profiling

Commit 27436d4e — 2026-02-27

The final piece: prove the mechanism by which continuous batching defeats bandwidth saturation.

Core Architecture (research/tr132/):

run.py — 5-phase in-container orchestrator (215 lines)
run_serving_profiled.py — In-container nsys profiling (669 lines)
run_validation.py — Container health + trace validation (440 lines)
analyze.py — Kernel amortization analysis (956 lines)
shared/nsys_container_driver.py — Novel in-container profiling (329 lines)
shared/nsys_system_driver.py — System-level nsys automation (196 lines)
shared/cross_reference.py — TR131 data cross-referencing (285 lines)
shared/statistics.py — Statistical testing framework (299 lines)

Methodological Innovation:

TR132 developed a novel in-container Nsight Systems profiling technique to overcome WSL2/WDDM CUDA visibility limitations. The approach mounts the Linux nsys binary into Docker containers, wraps the server entrypoint with nsys profile --trace cuda, and extracts .nsys-rep traces via volume mounts. 100% trace capture rate across 24 profiled repetitions.

The Central Finding — Continuous Batching Amortizes Bandwidth:

At N=8, vLLM reduces per-token kernel count by 80% (from 55 kernels/token at N=1 to 11 at N=8) and per-token memory operation time by 79-83%. Eight concurrent requests share kernel launches instead of executing independently.

Kernel Amortization:

Backend	Model	N=1 Kernels/Token	N=8 Kernels/Token	Reduction	Cohen's d
vLLM	LLaMA-1B	54.89	10.85	-80.2%	1,058
vLLM	LLaMA-3B	76.71	15.43	-79.9%	606
TGI	LLaMA-1B	72.98	17.12	-76.5%	26,269
TGI	LLaMA-3B	86.97	20.14	-76.8%	1,099

Bandwidth Amortization: 4.7-5.8x — for 8x concurrent load, bandwidth per token reduces 4.7-5.8x. The ratio exceeds N/2 for the 1B model (5.75x), indicating super-linear amortization from kernel fusion in the batched code path.

Hypothesis Results:

H1 (kernel count reduces with batching): CONFIRMED — 8/8 Holm-corrected tests significant
H2 (memory bandwidth reduces with batching): CONFIRMED — 8/8 tests significant
H3 (GPU utilization increases): REJECTED — measurement limitation, not a finding
H4 (attention kernel signatures differ): INCONCLUSIVE — kernel names not classifiable
H5 (serving stack N=1 matches PyTorch N=1): INSUFFICIENT DATA

The Complete Causal Chain

TR129: "Throughput plateaus at N=2. Amdahl serial fraction s=0.39-0.54."
  └─> Why? Is it Ollama's scheduling?

TR130: "vLLM/TGI scale 3-4x better. The serving stack is the bottleneck."
  └─> But is it really software?

TR131: "PyTorch Direct degrades WORSE (86%) than Ollama (82%). It's GPU physics."
  └─> Memory bandwidth saturation. But how do vLLM/TGI avoid it?

TR132: "Continuous batching amortizes kernel launches 80% and bandwidth 4.7-5.8x."
  └─> The mechanism: shared computation, not faster computation.

Quality Indicators & Standards

Data Quality: 98%+ success rate across all experiments
Statistical Rigor: Welch's t-tests, Cohen's d, Holm step-down correction, Mann-Whitney U
Reproducibility: Fixed seeds, documented run IDs, config.yaml per experiment
Trace Coverage: 100% capture rate on in-container profiling

Strategic Development Indicators

Foundation Quality: Transformative — complete causal understanding of GPU scaling
Scalability Readiness: Critical — serving stack selection now data-driven
Operational Excellence: High — 15,000+ measurements with automated analysis
Team Productivity: Exceptional — 4 reports in 36 hours

🏗️ Architecture & Strategic Impact

Research Architecture Philosophy

This episode establishes Chimera's Causal DNA — the principle that mechanism matters more than measurement. This isn't just benchmarking; it's the institutionalization of root-cause decomposition that transforms "it's slow" into "here's exactly why, proven at the kernel level."

Strategic Architectural Decisions

1. The Question Chain Design

Each report answers the question the previous one raised
No report is self-contained — they form a directed investigation graph
The chain terminates when mechanism is proven, not when metrics are collected

2. The PyTorch Direct Control

Stripping away the entire serving stack is the definitive causal test
If degradation persists without the software, the software is not the cause
This overturned TR130's conclusion with rigorous evidence

3. The In-Container Profiling Innovation

WSL2/WDDM blocks standard GPU profiling from the host
Mounting nsys into Docker containers solves this at the infrastructure level
The technique is reusable for any CUDA workload in containers on Windows

4. The Shared Module Pattern

shared/statistics.py, shared/utils.py, shared/gpu_monitor.py appear across TR129-TR132
Each report has its own shared/ directory with experiment-specific tooling
Common patterns (agent executors, trace parsers) are reused but not over-abstracted

Long-Term Strategic Value

Operational Excellence: Serving stack selection is now evidence-based — vLLM for N>=4 agents.

System Scalability: Bandwidth amortization via continuous batching is the proven path to multi-agent scaling.

Team Productivity: Automated experiment pipelines enable report-per-day velocity.

Enterprise Readiness: Causal evidence, not just correlation, backs every architectural decision.

🎭 Banterpacks' Deep Dive

Banterpacks watches four reports land in sequence, the investigation spiraling deeper with each commit.

"You see that? TR129 asks the question: how many agents can share a GPU? Answer: two, maybe three, before throughput walls. Amdahl with s=0.39-0.54, R-squared above 0.97. That's not a guess — that's a law."

He pulls up TR130's comparison tables.

"TR130 says it's the serving stack. Ollama at N=8: 248 tok/s. vLLM at N=8: 559 tok/s. 2.25x advantage. Crossover between N=2 and N=4. If you're running more than two agents, switch your backend."

He traces through TR131's PyTorch Direct results.

"Then TR131 blows it up. Strip away the server entirely. Raw model.generate(), no HTTP, no Go runtime, nothing. Result? PyTorch degrades 86%. Ollama only 82%. The serving stack wasn't the bottleneck — the GPU silicon was. Memory bandwidth saturation. N=8 demand exceeds 432 GB/s peak by 78-130%. The bus is full."

He opens TR132's kernel amortization tables.

"So how do vLLM and TGI dodge it? TR132 goes inside the container. Novel technique — mount nsys into Docker, wrap the entrypoint. 100% trace capture. The answer: continuous batching reduces per-token kernel count by 80%. 55 kernels per token at N=1, 11 at N=8. The bandwidth per token drops 4.7-5.8x. Eight requests sharing one set of weight reads instead of eight separate reads."

He steps back, surveying the complete chain.

"22,383 lines in 36 hours. Correlation said 'switch your server.' Causation said 'batch your kernels.' We're still shaping the clay — but now we understand the physics of the kiln. That's the difference between a guess and an engineering principle."

"This is how lasting systems achieve operational excellence. Not by accepting the first explanation, but by drilling until you hit bedrock. Four reports deep, we found the bedrock: memory bandwidth amortization through continuous batching. We're building causal infrastructure."

🔮 Next Time on The Chimera Chronicles

Next dossier entry: Legacy Cleanup + TR133 — pruning the research tree and opening new branches.

The Research Sprint distilled: causation is a feature.

Chimera - Episode 44: "The Training Frontier