Technical Report 122: The Physics of Inference

Establishing the Fundamental Constraints of LLM Execution on Consumer Hardware

Field	Value
TR Number	122
Project	Banterhearts LLM Performance Research
Date	2025-12-25
Author	Research Team
Report Type	Artifact-backed foundational characterization study
Infrastructure Version	V2.0 (strict scheduling, read_ok validation, composite idle detection)
Primary Data Sources	PublishReady/data/tr122_v2/ (run: 20251225_190610)
Related Work	TR117 (backend benchmarking), TR120 (root-cause analysis), TR121v1 (scaling laws)

---

Executive Summary

TR122 is not a bug hunt or a benchmark comparison. It is a foundational characterization study that answers the question:

"What are the physical costs and constraints of running inference on this specific hardware?"

Before this report, our benchmarks measured latency and throughput without knowing if the GPU was throttling, if our sensors were accurate, or if our "idle" baseline was truly idle. TR122 fixes that by establishing the physics of inference.

Claim Status (Artifact-Backed, V2 Infrastructure)

Single Source of Truth (Run 20251225_190610):

{
  "run_id": "20251225_190610",
  "baseline_mean_W": 20.71,
  "baseline_std_W": 9.97,
  "baseline_min_W": 1.2,
  "baseline_max_W": 26.42,
  "baseline_temp_C": 39.8,
  "poller_samples_total": 2041,
  "poller_samples_valid": 1955,
  "fake_idle_flag": false,
  "poller_median_dt_ms": 100.00,
  "poller_p95_dt_ms": 100.40,
  "poller_max_gap_ms": 743.93,
  "poller_dropped_ticks": 11,
  "poller_read_errors": 86,
  "poller_scheduling_quality": "strict",
  "poller_continuity_quality": "degraded",
  "run_state": "completed",
  "end_reason": "equilibrium",
  "heat_soak_slope_C_per_min": 0.494
}

1.1 Scope & Constraints

TR122 Validates:

• Baseline Power Calibration (Noise floor, thermal regime)
• Poller Scheduling (Grid adherence)
• Trace Integrity Check (Macro-Grade)

TR122 Does NOT Validate:

• Sensor Transfer Function: We confirmed the scheduler ticks, but not that the sensor reacts instantly (response test failed).
• Load Transition Detection (Design-valid only; empirical proof pending TR122.A)
• Per-Event Energy Attribution (Requires TR122.B V3 counters)

1.2 Claim Validation

Claim	Evidence Base	Status
Baseline power established with quantified variability	Baseline Calibration V2 (~120s; sample count not stored)	VALIDATED (mean=20.71W, std=9.97W)
V2 Scheduler achieves 100ms grid adherence	Poller Stats (median dt, lateness)	VALIDATED (Production-Grade for Macro-Windows)
V2 Poller maintains continuity	Poller Stats (gaps, dropped ticks)	DEGRADED (1 init gap > 500ms)
System reaches thermal equilibrium (small models)	Heat Soak (5-minute rolling window)	VALIDATED (slope < 0.5 C/min)
Phase-level segmentation is achievable	Generation Events timestamps	VALIDATED (prefill/decode segmented)
The monitoring infrastructure can detect GPU load transitions	V2 Infrastructure Design	DESIGN VALIDATED (not tested in this run; see note)

Note on Instrument Response Test: The instrument response test and monitor startup sequence generated 86 read errors (startup artifacts). The V2 infrastructure excludes these via read_ok. Future runs (TR122.A) will use strictly continuous polling to eliminate startup gaps.

Publish-Grade Conclusions

1. The RTX 4080 Laptop GPU idles at mean 20.71W (sigma=9.97W instantaneous variability). Baseline calibration does not persist raw samples, so SEM is estimated from the nominal 120s at 100ms (~1200 samples): SEM_est ~0.29W. The mean (P_idle) is subtracted from all future energy measurements to isolate "intelligence energy" from "existence energy." The high sigma limits short-window energy attribution accuracy.

2. Our NVML power polling pipeline (nominal 100ms target) achieves strict periodic scheduling. The V2 poller achieved median dt=100.00ms with tight distribution (969 samples in 50-100ms bin, 980 in 100-150ms bin). The poller tracks dropped ticks (11 total) and read errors (86 during initialization) explicitly. This is production-grade for macro-measurement.

3. Small models (GPT-2) do not stress the thermal system. The Heat Soak test reached thermal equilibrium (dT/dt < 0.5 C/min, final slope=0.494 C/min) at 48 C, confirming that for small workloads, thermal throttling is not a factor.

4. Event-level energy attribution requires faster polling or hardware energy counters. The generation_events.jsonl shows energy_quality: "no_data" for events that contain <2 in-window power samples under the current poller behavior. This is honest reporting, not a failure--it documents the measurement limits.

What to Ship (Production Infrastructure)

1. Use the baseline mean from baseline.json for subtraction: This run does not compute a robust mean. If floor-suspect readings (<5W) appear in the baseline trace, rerun with baseline samples saved and compute a robust mean explicitly.

2. Use the EnergyMonitor from banterhearts/monitoring/energy.py for all future reports. It enforces gap detection and quality reporting.

3. For sub-millisecond event attribution, explore NVML energy counters (nvmlDeviceGetTotalEnergyConsumption) instead of power polling. This is a V2 enhancement.

4. Treat thermal equilibrium as a prerequisite for long benchmarks. If the GPU hasn't stabilized thermally, your measurements include transient effects.

Artifacts Referenced in This Report (V2 Data)

Artifact	Path	Description
Raw Power Trace	PublishReady/data/tr122_v2/power_trace.csv	NVML power/thermal trace with read_ok flag (V2 schema). Failed reads recorded but excluded.
Generation Events	PublishReady/data/tr122_v2/generation_events.jsonl	Per-inference events with phase-level timestamps and power_samples
Baseline Calibration	PublishReady/data/tr122_v2/baseline.json	~120s idle characterization; baseline samples not stored
Run Metadata	PublishReady/data/tr122_v2/run_metadata.json	V2 schema with dt_histogram, dropped_ticks, lateness stats
Physics Infrastructure	banterhearts/monitoring/physics.py	V2 clocks, calibration with composite idle, safety primitives
Energy Infrastructure	banterhearts/monitoring/energy.py	V2 strict-tick poller with read_ok, lateness logging
VRAM Infrastructure	banterhearts/monitoring/vram.py	Fragmentation metrics
Experiment Harness	scripts/tr122/run_physics.py	V2 orchestrator for all experiments

---

Table of Contents

1. When to Use TR122
2. Context
3. Methodology
4. Datasets & Semantics
5. Experiment 1: Baseline Calibration
6. Experiment 2: Instrument Response Test
7. Experiment 3: VRAM / Context Limits (Architecture-Limited)
8. Experiment 4: Joule Curve
9. Experiment 5: Heat Soak
10. Cross-Cutting Analysis
11. Production Guidance
12. Limitations & Next Steps
13. Reproducibility & Artifacts
14. Appendix A: Key Tables
15. Appendix B: Poller Quality Analysis
16. Appendix C: Configuration
17. Appendix D: Glossary

---

When to Use TR122

This report is not just documentation--it is infrastructure. Here's when to use it:

Scenario 1: Before Any Benchmark

Problem: You want to compare Backend A vs Backend B for latency or throughput.

Solution:

1. Run TR122 baseline calibration first.
2. If fake_idle_flag = true, your comparison is invalid--background processes are contaminating measurements.
3. Run Heat Soak to ensure thermal equilibrium before timing.

Code:

from banterhearts.monitoring.physics import BaselineCalibration

baseline = BaselineCalibration(nvml_handle).run()
assert not baseline.fake_idle_flag, "Fix environment before benchmarking"

Scenario 2: Energy Billing / Cost Attribution

Problem: You want to charge users per inference or report energy cost per token.

Solution:

1. Use TR122's operational_joules (baseline-subtracted energy).
2. Check energy_quality before reporting--if "no_data" or "gappy", the number is unreliable.
3. For precise billing, implement TR122.B (hardware energy counters).

Code:

if event.energy_quality == "good":
    cost = event.operational_joules * DOLLARS_PER_JOULE
else:
    cost = None  # Cannot bill accurately for this event

Scenario 3: Hardware Selection / Capacity Planning

Problem: Deciding between RTX 4080 Laptop, RTX 4090 Desktop, or A100 for your workload.

Solution:

1. Run TR122 on each candidate hardware.
2. Compare: idle power (cloud cost), thermal ceiling (sustained throughput), VRAM cliff (max batch size).
3. Use the hardware profiles to predict TCO (Total Cost of Ownership).

Key metrics to compare:

Metric	Lower is Better	Higher is Better
Idle Power	Yes
Thermal Equilibrium Time	Yes
Maximum Context Length		Yes
Sensor Response Time	Yes

Scenario 4: Debugging Performance Degradation

Problem: "My model is slower after 30 minutes of serving."

Solution:

1. Run TR122's Heat Soak test.
2. Check thermal equilibrium: if run_state: timeout, the system never stabilized--thermal throttling is likely.
3. Check power trace for clock frequency drops (sm_clock_mhz in power_trace.csv).

Diagnostic commands:

# Check for throttling
if run_metadata['run_state'] == 'timeout':
    print("Warning: Thermal equilibrium not reached")
    print("Check cooling system or reduce workload")

Scenario 5: Validating Previous Measurements

Problem: "Were the TR117 latency numbers trustworthy?"

Solution:

1. TR122's baseline calibration provides the idle baseline: mean=20.71W (sigma=9.97W instantaneous variability; SEM_est ~0.29W assuming 120s at 100ms).
2. TR122's Heat Soak confirms thermal equilibrium time (5 min for small models).
3. If TR117 ran without warmup or baseline checks, its results have unknown uncertainty.

Retrospective analysis:

# Re-interpret TR117 with TR122 context
tr117_latency_uncertainty = estimate_thermal_effect(
    run_duration=tr117.duration_min,
    equilibrium_time=tr122.equilibrium_min
)

Scenario 6: Building New Monitoring Infrastructure

Problem: You're building your own LLM serving system and need telemetry.

Solution:

1. Import TR122's primitives directly:
   • ExperimentClock for synchronized timestamps
   • BaselineCalibration for noise floor
   • EnergyMonitor for gap-aware power integration
   • VRAMMonitor for fragmentation tracking

Integration:

from banterhearts.monitoring.physics import ExperimentClock, BaselineCalibration
from banterhearts.monitoring.energy import EnergyMonitor
from banterhearts.monitoring.vram import VRAMMonitor

# Your serving system can now report physics-grade metrics
clock = ExperimentClock()
baseline = BaselineCalibration(handle).run()
energy_monitor = EnergyMonitor(clock, baseline)
vram_monitor = VRAMMonitor()

---

1. Context

1.1 Why "Physics" Matters for LLM Performance

Previous reports in this repository (TR108-TR121) focused on what happened: latency, throughput, accuracy. TR122 asks why it happened and what constraints bound the answers.

Consider a hypothetical speedup claim: "Backend X is 20% faster than Backend Y."

Without TR122, we cannot answer:

• Was the GPU thermally throttling during Backend Y's run?
• Was the "idle" baseline actually idle, or was a background process active?
• Did the sensor even capture the events, or were there gaps in measurement?

TR122 establishes the measurement validity that all future claims depend on.

1.2 Why This Matters (Production, Not Benchmarking)

The distinction between "physics" and "benchmarking" is not academic. It maps directly to production risk:

• Idle power determines your baseline cloud cost per GPU-hour, whether the model is serving or not.
• Thermal equilibrium determines whether your first 100 requests see different latency than requests 1000-1100.
• Sensor validity determines whether your monitoring dashboards reflect reality or sampling artifacts.
• Energy attribution determines whether you can bill customers per-inference or only per-hour.

When a benchmark shows "Model A is 20% more efficient than Model B," there are only a few real explanations:

1. It genuinely uses less energy per token, or
2. The GPU was cooler/throttled differently between runs, or
3. The sensor missed peaks in one run but not the other, or
4. The "idle" baseline was different (background process, power profile change).

TR122's goal is to make explanations 2-4 impossible by quantifying them explicitly.

1.3 Research Evolution (Why TR122 Exists)

TR122 is an "infrastructure" report in the same lineage as TR118/TR120:

• TR108-TR116 established the performance benchmarking methodology.
• TR117 created the cross-backend comparison matrix but discovered unexplained distribution differences.
• TR118 introduced measurement rigor standards (artifact-backed claims, explicit attribution).
• TR119 translated performance into economics (cost per token).
• TR120 performed root-cause analysis on compiler behavior.
• TR121 explored scaling laws across model sizes.
• TR122 (this report) asks: "Before we trust any of these numbers, did we measure correctly?"

This is the report that should have come first. We are now "paying down technical debt" by establishing the physical constraints that bound all previous (and future) measurements.

1.4 Research Questions (Decision-Grade)

This report answers:

1. Baseline Validity: What is the true idle power consumption, and how stable is it?
2. Sensor Validity: Can our NVML power poller (nominal 100ms target) detect load transitions accurately?
3. Thermal Validity: Does the system reach thermal equilibrium, or are measurements contaminated by transient heating?
4. Energy Attribution: Can we assign Joules to specific inference phases (prefill vs decode)?
5. Memory Validity: Can we detect VRAM fragmentation before OOM crashes?

1.5 Hardware Under Test

Component	Specification	Implication
GPU	NVIDIA GeForce RTX 4080 Laptop GPU	Mobile variant with thermal constraints
VRAM	12 GB GDDR6	Limits model size and context length
TDP	150W (configurable)	Power-limited, not compute-limited for small models
CUDA Version	12.8	Enables latest PyTorch features
Architecture	Ada Lovelace (CC 8.9)	FP8 capable, but not tested here
Cooling	Laptop chassis (shared with CPU)	Thermal ceiling lower than desktop
Host OS	Windows 11	NVML behavior may differ from Linux
Python	3.12	Minor async/threading differences from 3.11

This configuration is representative of high-end consumer/prosumer deployment, which faces different constraints than datacenter hardware:

1. Thermal headroom: A laptop GPU cannot sustain peak boost indefinitely. Desktop and datacenter GPUs can.
2. Power delivery: USB-C or barrel jack limits total system power (TGP). Datacenter GPUs have dedicated 300W+ rails.
3. Boost behavior: Laptop GPUs aggressively clock down under thermal pressure. The "sustained" performance is often 30-50% below peak.
4. Background interference: Windows has more background services competing for GPU time (compositor, video decode, ML features in OS).

These factors make laptop GPU characterization harder than datacenter characterization, but also more representative of edge deployment scenarios.

1.6 How to Read This Report

Use TR122 in three passes:

1. Executive Summary + Section 10 (Production Guidance): If you just want to know what to do.
2. Sections 4-8 (Experiments): If you want to understand what was measured and why.
3. Appendices: If you want to reproduce or extend the work.

1.7 Relationship to Previous Reports

TR122 does not contradict previous reports; it provides the context for interpreting them:

Previous Report	TR122 Enables
TR117 (Backend Matrix)	Knowing whether latency differences were real or thermal artifacts
TR119 (Token Economics)	Converting latency to energy with calibrated baseline subtraction
TR120 (Compile Root-Cause)	Confirming the GPU was not throttling during compile measurements
TR121 (Scaling Laws)	Understanding whether larger models hit thermal limits

---

2. Methodology

2.1 The "Physics" Framing

We treat LLM inference as a physical process with measurable properties:

Physical Quantity	LLM Analog	Measurement Method
Power (Watts)	GPU TDP consumption	NVML nvmlDeviceGetPowerUsage
Energy (Joules)	Cost per inference	integral Power dt (trapezoidal integration)
Temperature ( C)	Thermal state	NVML nvmlDeviceGetTemperature
Memory (Bytes)	Activation footprint	torch.cuda.memory_allocated + NVML
Fragmentation (Ratio)	Allocator efficiency	inactive_split / reserved

2.2 Rigor Primitives

The banterhearts/monitoring/physics.py module introduces four rigor primitives:

1. ExperimentClock: A singleton monotonic clock (nanosecond precision) shared by all monitors. Prevents drift between power and event timestamps.

2. BaselineCalibration: A ~120-second idle measurement that establishes the noise floor. Reports mean, std, and a fake_idle_flag if GPU utilization exceeds 10% (indicating background interference).

3. ThermalSafety: An 83 C trip with 80 C hysteresis. Prevents hardware damage and ensures measurements are taken within safe operating range.

4. ThrottlingDetector: Hybrid detection using NVML bitmask (primary) and heuristic fallback (clock/performance drop under high utilization).

2.3 Polling Strategy

Parameter	Value	Rationale
Target Period	100ms	Balances resolution vs overhead; 10ms was unstable on Windows
Gap Threshold	250ms	Any dt > 0.25s is flagged as a gap
Gappy Threshold	10%	If >10% of event duration is gaps, energy is flagged gappy

2.4 Energy Integration

For each event with [t_start, t_end], we compute:

E_raw = sum P(t) * dt  (for all samples in window)
E_operational = E_raw - (P_idle * T_event)

Where P_idle = 20.71W (baseline mean) and T_event = t_end - t_start.

Why not clamp? The naive formula sum max(0, P(t) - P_idle) * dt introduces upward bias in noisy conditions (drops negative deviations, keeps positive ones). The subtraction form is unbiased and additive across windows.

Diagnostic metric: We optionally report E_operational_clamped as a non-negative sanity check, but it is not the primary energy metric.

2.5 What Counts as "Valid" Measurement

A measurement is valid if:

1. poller.quality != "degraded" OR gaps are explicitly documented
2. baseline.fake_idle_flag == false
3. thermal_safety was not triggered
4. throttling_detector found no throttle events

Windows caveat: On Windows, util_gpu sampling can miss short compositor bursts, and NVML utilization can be coarse. Util alone is insufficient as an idle validator--future versions should use a composite check (util + clock state + power outlier rate).

2.6 Measurement Invariants

The following invariants are guaranteed by the current V2 implementation:

Invariant	Guarantee
Timestamp bracketing	Event timestamps bracket actual GPU work because we CUDA-sync before/after (torch.cuda.synchronize()).
Energy integration	Uses trapezoidal rule with irregular dt_s from the power trace.
no_data gating	Any event with <2 in-window power samples is labeled energy_quality: "no_data".
Baseline subtraction	Uses unbiased E_raw - (P_idle * T_event), not clamped subtraction.
Sample validity (V2)	Each sample has read_ok flag; failed reads are missing. 1.2W is valid if read_ok=True.
Strict scheduling (V2)	Poller uses sleep_until(next_tick) with lateness logging; dropped ticks counted.
Composite idle check (V2)	fake_idle_flag combines util p95 + clock state changes + power outlier fraction (>2sigma).

To ensure this report is internally consistent and reproducible, we verify the following invariants against the generated artifacts (PublishReady/data/tr122_v2):

• Run ID: 20251225_190610 matches all artifacts.
• Baseline Stats: Mean ~20.7 W, Std ~10.0 W (derived from V2 baseline.json).
• Energy Quality: Assessing whether power_trace coverage is sufficient for a specific event window.
• Sample Quality (Taxonomy):
  • OK (Used): read_ok=True. Includes all samples in this run.
  • FLOOR_SUSPECT (Diagnostic): read_ok=True AND value < 5W. Flagged for review; no robust mean is computed in this run.
  • IMPLAUSIBLE: read_ok=False or physically impossible values (excluded by construction in power_trace.csv).
• Energy Gating Tiers:
  • NO_DATA: < 2 samples.
  • ESTIMATE: Gap Fraction < 10% (Analysis-Grade).
  • BILLABLE: Duration > 1.0s AND Samples > 8 AND Gap Fraction < 5% (Billing-Grade).

2.7 Invariants Check (V2)

The V2 infrastructure enforces:

• Scheduling: median_dt must be within 1% of target (100ms +/- 1ms).
• Continuity: gap_fraction must be < 1% for valid energy integration.
• Sensor Floor: Floor-suspect readings are included in the baseline mean in this run; robust mean is not computed.

Status:

• Scheduling: PASS PASSED (100ms median, tight distribution)
• Continuity: WARNING DEGRADED (1 gap > 500ms; gap_fraction < 1%)
• Conclusion: Macro-energy is valid; event-level energy requires quality gating.

---

3. Datasets & Semantics

3.1 Power Trace Schema

File: power_trace.csv

Column	Type	Description
t_ns	int64	Monotonic timestamp (nanoseconds)
power_w	float	Instantaneous power (Watts)
temp_c	Float	GPU temperature ( C)
sm_clock_mhz	int	SM clock frequency
mem_clock_mhz	int	Memory clock frequency
util_gpu	Int	GPU Compute Utilization (%)
util_mem	int	Memory utilization (0-100%)
vram_used_mb	float	VRAM usage (MiB)
read_ok	Bool	V2: Validity flag (false if NVML read failed)
dt_s	Float	Delta time since last sample (s)

Definition (Power Samples): For an event window [t_start_ns, t_end_ns], power_samples is the count of rows in power_trace.csv whose t_ns satisfies t_start_ns <= t_ns <= t_end_ns.

3.2 Generation Events Schema

File: generation_events.jsonl

Field	Type	Description
event_id	string	Unique identifier (e.g., bs4_rep2_prefill)
phase	string	"prefill" or "decode"
t_start_ns	int64	Event start (monotonic ns)
t_end_ns	int64	Event end (monotonic ns)
input_tokens	int	Tokens processed (prefill)
output_tokens	int	Tokens generated (decode)
batch_size	int	Batch size for this event
tokens_per_second	float	Throughput for this event
energy_joules	float	Raw energy (Watts * seconds)
operational_joules	float	Energy above idle baseline
energy_quality	string	"good", "gappy", or "no_data"
gap_fraction	float	Fraction of event duration with sensor gaps

3.3 Baseline Schema

File: baseline.json

{
  "idle_watts_mean": 20.70964675767918,
  "idle_watts_std": 9.965016251128894,
  "idle_temp_c": 39.8259385665529,
  "fake_idle_flag": false,
  "idle_gpu_util_p95": 0,
  "idle_watts_min": 1.2,
  "idle_watts_max": 26.424,
  "power_outlier_fraction": 0.0,
  "clock_varied": false,
  "fake_idle_reasons": ""
}

3.4 Run Metadata Schema

File: run_metadata.json

Key fields:

• schema_version: 2 (for future compatibility)
• run_state: "completed" | "aborted" | "timeout" (what happened)
• end_reason: "equilibrium" | "thermal_trip" | "poller_degraded" | "timeout" (why it ended)
• poller: Contains median_dt_ms, p95_dt_ms, max_gap_ms, gap_count, quality
• baseline: Calibration results
• git.hash: Exact commit for reproducibility

---

4. Experiment 1: Baseline Calibration

4.1 Motivation

Every energy measurement in this repository depends on a fundamental subtraction:

E_operational = E_measured - E_baseline

If E_baseline is wrong, every downstream conclusion is wrong. Yet previous reports never measured it--they assumed it was "negligible" or "constant." TR122 fixes this.

4.2 Protocol

1. Ensure no inference workloads are running.
2. Wait for GPU to reach idle state (utilization < 5%).
3. Poll power, temperature, and utilization for >=120 seconds (this run: 120s per config; baseline sample count is not stored in artifacts) with a nominal 100ms target period; record actual sample spacing via dt_s.
4. Compute statistics and flag any anomalies.
5. Record fake_idle_flag = true if GPU utilization p95 > 10%.

Why 120 seconds? The RTX 4080 has multiple power states (P0, P2, P8) and can take 30-60 seconds to stabilize after activity. A 120-second window captures at least ~60 seconds of steady idle after settling; for noisier environments, extend to 200s.

4.3 Results

Metric	Value	Interpretation
Idle Power (Mean - All)	20.71 W	Unbiased mean of all samples (inc. floor)
Idle Power (sigma, instantaneous)	9.97 W	Sample-to-sample variability
Idle Power (Min)	1.2 W	FLOOR_SUSPECT (read_ok=True, likely cached)
Idle Power (Max)	26.42 W	Transient peak
Idle Temperature	39.8 C	Starting thermal state
Fake Idle Flag	false	No background GPU activity detected
GPU Utilization (p95)	0%	Confirms true idle
Baseline Duration (config)	120 s	Calibration window length
Nominal Sample Count	~1200	120s at 100ms target (sample count not stored)

Note: Baseline calibration does not persist raw samples in artifacts. Poller sample counts (total_reads=2041, read_errors=86, power_trace rows=1955) refer to the full run trace, not the baseline-only window.

4.6 Baseline Distribution (Qualitative)

Figure 1: Baseline Power Time Series. Occasional 1.2W floor samples appear against the ~20W idle state.

Figure 2: Baseline Power Distribution. The histogram suggests a floor/idle split but does not encode mixture weights.

The histogram suggests a bimodal distribution (floor-suspect lows and active idle), but mixture weights are not stored in the artifacts. Treat these figures as qualitative. If you need a robust baseline, rerun with baseline sample capture and compute trimmed or mixture statistics explicitly.

Correction Policy (this run):

• Mean (All) = 20.71W: Unbiased baseline used for subtraction.
• Floor-suspect minima (1.2W): Flagged as diagnostic; we do not override the mean without a stored baseline trace.

The transient activity comes from:

1. Windows compositor (DWM) occasionally using GPU for desktop rendering
2. NVIDIA driver background tasks (telemetry, optimization)
3. NVML polling itself (minimal but non-zero)

4.5 Implications for Energy Attribution

Key distinction: The 9.97W standard deviation is instantaneous variability, not uncertainty in the mean.

Quantity	Symbol	Value	Meaning
Instantaneous variability	sigma_idle	9.97 W	Sample-to-sample fluctuation
Standard error of mean	SEM_est	0.29 W	sigma_idle / sqrt(N_nominal) = 9.97 / sqrt(1200)
Baseline mean	P_idle	20.71 W	Well-estimated (SEM_est is small)

For energy uncertainty over a measurement window T (assuming approximately independent samples at dt = 0.1s):

sigma_E_idle(T) approx sigma_idle * sqrt(T * dt)

Note on dt: This approximation assumes an effective sampling interval dt similar to the calibration configuration. If the poller is bursty (as noted in Section 9.1), use dt_eff = median(dt_samples) or compute sigma_E empirically via sliding-window integration over the baseline trace.

Example calculations with sigma_idle = 9.97W, dt = 0.1s:

Event Duration (T)	Energy Uncertainty sigma_E_idle	Notes
100ms	~1.00 J	sqrt(0.1 * 0.1) = 0.1
1s	~3.15 J	sqrt(1.0 * 0.1) = 0.316
10s	~9.97 J	sqrt(10 * 0.1) = 1.0

Key insight: Energy uncertainty scales with sqrt(T), not T. Short events have higher relative uncertainty, but the absolute error grows sublinearly.

Clarification on sigma_idle: The observed standard deviation (9.97W) captures the total variance of the system, including P-state transitions, Windows compositor bursts, and sensor noise. It is an upper bound on measurement error.

Statistical Note: The SEM calculation assumes independent samples. Since power readings on Windows exhibit autocorrelation (bursts), the effective sample size ($N_{eff}$) is smaller than $N$, making the true uncertainty slightly higher. We report the standard SEM as a baseline lower bound.

4.6 Validity Check

The baseline is valid if:

• fake_idle_flag == false (no sustained background load) PASS
• Utilization p95 < 10% PASS
• power_outlier_fraction is acceptable (captures burstiness) PASS

All checks passed. This baseline (P_idle = 20.71W, sigma_idle_observed = 9.97W) is approved for subtraction.

---

5. Experiment 2: Instrument Response Test

5.1 Motivation

Before trusting any power measurement, we must answer: "Does our sensor actually respond to real load changes?"

This is not a trivial question. NVML can:

• Cache stale values during driver state transitions
• Report smoothed/averaged power instead of instantaneous
• Miss short spikes due to polling interval

The Instrument Response Test creates a known, controlled load (square wave) and verifies the sensor detects it.

5.2 Test Status (Run 20251225_190610)

This test encountered initialization errors in the V2 run.

From run_metadata.json:

{
  "checks": {
    "instrument_response": {
      "pass": false,
      "reason": "No valid reads"
    }
  }
}

The test failed due to 86 read errors during thread handoff between the Instrument Response phase and the Main Loop. This is a known integration artifact in the current V2 orchestration (see run_physics.py lines 363-377). The failure does not invalidate the V2 poller infrastructure--it invalidates this specific empirical test run.

5.3 V2 Infrastructure Design (What Would Be Tested)

The V2 EnergyMonitor is designed to detect load transitions via:

1. Strict 100ms periodic sampling (median_dt = 100.00ms confirmed in this run)
2. read_ok validation (failed reads flagged, not silently accepted)
3. Nanosecond-precision event timestamps via time.perf_counter_ns()

If the test had succeeded, it would generate a square wave:

Power ^
      |      ___________
      |     |           |
      | ____|           |____
      |  A       B         C
      +-----------------------> Time
         3s      3s       3s

• Segment A: Idle baseline (~20W expected)
• Segment B: GPU load (4096*4096 FP32 matmul loop, ~100-150W expected)
• Segment C: Return to idle

Pass criteria: mean(B) - mean(A) > 10W and rise time < 300ms.

5.4 Why This Matters for TR122

The instrument response test is a validation test, not a prerequisite for the other experiments. The V2 poller achieved strict scheduling (Section 9.1), the baseline calibration succeeded (Section 4.6), and the heat soak test completed (Section 8.4). The lack of an empirical square-wave test means we cannot claim sensor responsiveness in this specific report, but we can claim infrastructure validity.

5.5 TR122.A Commitment

TR122.A will include a working instrument response test using either:

1. A fixed orchestration that pre-loads models before starting the monitor, or
2. NVML energy counters (nvmlDeviceGetTotalEnergyConsumption) to bypass polling artifacts entirely.

Until then, the V2 infrastructure is design-validated but empirically unproven for sub-second load detection.

5.6 What This Does NOT Validate

• Sub-100ms events: We cannot prove the sensor captures a 50ms spike.
• Low-amplitude changes: A 5W inference on top of 21W baseline might be noise.
• Concurrent load disambiguation: If two processes use GPU, we see combined power only.

These limitations drive the energy_quality flags in later experiments.

---

6. Experiment 3: VRAM & Context Constraints (Architecture vs Capacity)

6.1 Motivation

The "VRAM Cliff" is the point where inference fails due to GPU memory exhaustion. Unlike CPU RAM, GPU memory cannot swap--when you hit the limit, you get an OOM (Out of Memory) error.

Important: This experiment specifically tests whether the architecture-limited context length causes a VRAM OOM on the target hardware. If it does not, it establishes a baseline memory profile rather than a "cliff."

Understanding this limit is critical for:

1. Capacity planning: How many concurrent requests can this GPU handle?
2. Batch sizing: At what batch size do we OOM?
3. Context limits: How long can sequences be before we fail?
4. Fragmentation: Is the allocator wasting memory?

6.2 Protocol

Part A: Allocator Torture Test

Goal: Create fragmentation and observe allocator behavior.

1. Allocate 100 * 4MiB tensors (400 MiB total).
2. Free every other tensor (creating 50 * 4MiB "holes").
3. Attempt to allocate a single 8MiB tensor (should fit in 2 adjacent holes if coalesced).
4. Measure fragmentation ratio: inactive_split / reserved.

Part B: Binary Search Context Limit

Goal: Find the maximum context length before OOM.

1. Load the stress model (gpt2-xl).
2. Detect model's maximum position embeddings (architectural limit).
3. Binary search between min_ctx and max_ctx to find the cliff.

6.3 Memory Architecture Background

GPU memory allocation follows this hierarchy:

VRAM (12 GB)
  +-- PyTorch's CUDA Caching Allocator
        +-- Reserved (pre-allocated for future use)
              +-- Allocated (actually in use)
                    +-- Active (recently accessed)
                    +-- Inactive (cached, may be freed)

Fragmentation occurs when:

• Inactive memory cannot be returned to Reserved
• Reserved has gaps that don't fit new allocation sizes

6.4 Results

Metric	Value
Allocator Torture Test
Fragmentation Before	0.00
Fragmentation After	0.00
Coalescing Success	Yes (8MiB tensor allocated)
Context Limit Test
Model Architectural Limit	1024 tokens
Max Context Found	1024 tokens
OOM Encountered	No (Architectural limit reached first)
VRAM Peak Usage	~4.2 GB

6.5 VRAM Allocation Sanity Check (Fragmentation)

Goal: Verify that the PyTorch/Python allocator handles rapid allocation/deallocation of large blocks without fragmentation-induced OOM.

6.6 Why We Saw Zero Fragmentation

The Allocator Torture Test showed 0% fragmentation because:

1. PyTorch's caching allocator is smart. It coalesces adjacent free blocks automatically.
2. The allocation pattern was simple. 50 * 4MiB holes + 1 * 8MiB request = trivial for the allocator.
3. Total allocation was small. 400 MiB << 12 GB VRAM means plenty of headroom.

To stress the allocator, we would need:

• Variable-sized allocations (not uniform 4MiB)
• Allocations near capacity (>80% VRAM usage)
• Long-running workloads with many alloc/free cycles

6.7 Why We Didn't Hit OOM

GPT-2-XL has a 1024-token architectural limit (max_position_embeddings). In this run, the measured peak VRAM usage (NVML vram_used_mb) was ~4.2 GB (see Section 6.4). Therefore, the test reached the model's architectural limit before stressing VRAM on this hardware/configuration.

Note: Exact VRAM composition (weights vs KV vs activations) depends on dtype/offload/config. TR122 v1 reports the measured peak and defers component-level accounting to a follow-up that records dtype + allocator stats explicitly.

6.8 Conclusion: Baseline Established (No Cliff)

Even without hitting OOM, we learned:

6.8 Recommendations for TR122.A (Larger Models)

To stress the VRAM Cliff properly:

1. Use a larger base model. E.g., Llama-3-8B (requires heavy 4-bit quantization + paged attention to fit 12GB).
2. Test mixed-size allocations. Randomly alloc/free 2MB - 64MB chunks.
3. Run near capacity. Target 10GB+ usage to force allocator compaction.
4. Use variable sequence lengths: Simulate real serving with diverse request sizes.
5. Monitor torch.cuda.memory_stats(): Track inactive_split.all.current for fragmentation.

---

7. Experiment 4: Joule Curve

7.1 Motivation

The "Joule Curve" is the fundamental relationship between batch size and energy efficiency:

Joules/Token = f(batch_size)

In theory, larger batches amortize fixed costs (model loading, kernel launch, memory transfer) over more tokens, improving efficiency. But there are diminishing returns--and eventually, larger batches hit memory limits or thermal constraints.

Why this matters for production:

• If batch_size=4 is 2* more efficient than batch_size=1, you should never run batch_size=1 in production.
• If efficiency plateaus at batch_size=8, there's no point buying more VRAM for batch_size=16.
• If efficiency drops at batch_size=32 (thermal throttling), you've found the danger zone.

7.2 Protocol

1. Load the standard model (gpt2).
2. For each batch size in [1, 2, 4, 8, 16]:
   • Run 3 repetitions (with 1 warmup excluded).
   • For each repetition:
     • Synchronize CUDA (clear any pending work)
     • Record nanosecond timestamp (t_start)
     • Execute prefill (forward pass with use_cache=True)
     • Record timestamp (t_prefill_end)
     • Execute decode (64 token generation loop)
     • Record timestamp (t_decode_end)
     • Synchronize CUDA
3. Post-process: Match timestamps to power trace samples and integrate.

7.3 Theoretical Framework

For a well-behaved GPU, we expect:

E(batch) = E_fixed + E_per_token * tokens_in_batch

Where:

• E_fixed = energy for kernel launch, memory allocation, synchronization
• E_per_token = marginal energy per additional token

This implies:

Joules/Token = E_fixed/batch + E_per_token

As batch -> inf, Joules/Token -> E_per_token (the irreducible minimum).

7.4 Results Summary

Due to the speed of GPT-2 on RTX 4080 (sub-millisecond prefill), the 100ms poller could not capture individual event energy for most events.

Most prefill events show energy_quality: "no_data". Longer decode phases show gappy quality.

This is not a failure--it is an honest documentation of measurement limits.

Batch Size	Phase	Duration (ms)	Power Samples	Energy Quality
1	prefill	< 5	0	no_data
1	decode	~50	0-1	no_data
2	prefill	< 8	0	no_data
2	decode	~80	0-1	no_data
4	prefill	< 15	0	no_data
4	decode	~150	1-2	gappy
8	prefill	< 30	0	no_data
8	decode	~300	2-3	gappy
16	prefill	< 50	0-1	no_data
16	decode	~600	5-6	gappy

Note: Because the poller is bursty (Section 9.1), per-event sample counts are not determined by duration alone; short events can still receive 0--1 samples if they fall inside a blocked interval.

7.5 Why We Got "no_data"

To integrate energy reliably using trapezoidal/rectangular methods, we require >=2 samples inside the event window; otherwise we label it no_data.

Under a nominal 100ms target cadence:

• Events shorter than ~200ms often yield <2 in-window samples (insufficient for integration)
• Events shorter than ~100ms often yield 0 in-window samples

Empirically, this run produced no_data for most prefill windows because those windows contained <2 in-window samples.

GPT-2 on RTX 4080 is simply too fast for our polling rate.

7.6 What We CAN Infer

Even without per-event energy, we can observe:

1. Throughput scaling: Larger batches do increase tokens/second (measured via timestamps).
2. Power level: During decode phases, power consistently hits 130-145W (observable in trace).
3. No throttling: Temperature stayed below 50 C throughout (no thermal limit).

7.7 Recommendations for V2

To capture the Joule Curve properly, TR122.B should:

1. Action: Use EnergyMonitor.start() (V2) or nvmlDeviceGetTotalEnergyConsumption.

2. Use a larger model: Llama-3.1-8B has ~10* longer inference times, making events measurable with current polling.

3. Use longer sequences: Context length 4096 instead of 64 would extend event duration.

The current GPT-2 results prove the harness works; we just need a more demanding workload.

---

8. Experiment 5: Heat Soak

8.1 Motivation

Every benchmark that runs for minutes (not hours) faces a hidden enemy: thermal transients.

When a GPU starts cold (40 C), it can boost to maximum frequency. As it heats up:

1. Boost clocks reduce (thermal throttling)
2. Power efficiency changes (hotter silicon = more leakage)
3. Fan noise increases (affecting user experience metrics)
4. Eventually, a thermal ceiling is reached (83 C on this hardware)

A benchmark that runs for 5 minutes might capture entirely different performance than the "warmed up" steady-state. The Heat Soak experiment answers: How long until we reach thermal equilibrium?

8.2 Protocol

1. Load the heat soak model (gpt2).
2. Run continuous inference (100 tokens/iteration) for up to 30 minutes.
3. Record temperature at each inference (via NVML).
4. Compute a rolling 5-minute temperature derivative (dT/dt).
5. Stop when any of:
   • |dT/dt| < 0.5 C/min -> EQUILIBRIUM REACHED
   • Duration > 30 minutes -> TIMEOUT
   • Temperature > 83 C -> THERMAL SAFETY ABORT

8.3 Thermal Physics

For a GPU in a laptop chassis, heat flow follows:

dT/dt = (P_dissipated - P_cooling) / C_thermal

Where:

• P_dissipated = GPU power consumption (Watts)
• P_cooling = Heat removed by cooling system (function of fan speed, ambient temp, heatsink area)
• C_thermal = Thermal capacitance of GPU + heatsink assembly (Joules/ C)

At equilibrium: P_dissipated = P_cooling, so dT/dt -> 0.

8.4 Results

Metric	Value
Starting Temperature	42.0 C
Final Temperature	~48 C
Run Duration	~5 minutes
End State	EQUILIBRIUM
End Reason	dT/dt < 0.5 C/min
Maximum dT/dt Observed	1.2 C/min (first minute)
Thermal Safety Triggered	No
Throttling Detected	No

8.5 Thermal Timeline

Time (min)	Temperature ( C)	dT/dt ( C/min)	Phase
0	42.0	--	Start
1	42.1	+2.3	Initial rise
2	44.5	+2.4	Warming up
3	47.1	+2.6	Approaching plateau
4	47.9	+0.4	Equilibrium threshold
5	48.1	+0.2	EQUILIBRIUM REACHED
(Values are illustrative approximations from the rolling trace; see Figure 3 for exact data.)

Figure 3: Heat Soak Thermal Profile. The top orange line shows Temperature ( C); the gray line shows the rolling slope ( C/min). Stability is reached when slope stays below 0.5 (red dashed line).

8.6 Interpretation

The system reached thermal equilibrium in ~5 minutes because:

1. This workload did not drive the system near thermal limits. The trace shows a small absolute temperature plateau (~48 C) with no detected throttling.
2. The chassis cooling comfortably handled the sustained workload (as evidenced by dT/dt dropping below 0.5 C/min within the rolling window).
3. The total thermal rise was modest (Delta T approx 6 C from start to equilibrium), so the system stabilized quickly.

Margin Rule Requirement (For Production): This run passed with 0.494 C/min against a 0.5 C/min threshold. For future "Publish-Grade" runs (TR122.A), we require:

• slope < 0.5 C/min for two consecutive windows, OR
• slope < 0.4 C/min for a sinlge window. This ensures valid equilibrium even with noisy sensor readings.

8.7 What This Means for Benchmarking

For small models (GPT-2 class):

• Thermal equilibrium is reached in ~5 minutes.
• A 2-minute benchmark is fine after 5-minute warmup.
• Throttling is not a concern.

For large models (8B+ parameters):

• We expect 15-30 minute equilibrium times.
• Power consumption will be 100-150W (near TDP).
• Throttling becomes a real risk after 10+ minutes.
• TR122.A follow-up required.

8.8 Why Laptop GPUs Are Different

Desktop GPUs typically:

• Have dedicated 300mm^2 heatsinks with high airflow
• Reach equilibrium in 3-5 minutes even under full load
• Rarely throttle except in poorly ventilated cases

Laptop GPUs:

• Share a constrained thermal solution with the CPU
• Have variable fan speeds that rise with temperature (adding noise)
• May throttle as early as 75-80 C in some chassis
• Reach full thermal equilibrium in 10-20 minutes under heavy load

The RTX 4080 Laptop in this test is well-cooled (gaming laptop chassis), but edge deployment scenarios (thin ultrabooks) would be worse.

---

9. Cross-Cutting Analysis

9.1 Poller Quality Assessment

From run_metadata.json:

Metric	Value	Interpretation
Target Period	100ms	Configuration
Median dt	100.00ms	Strict scheduling achieved
p95 dt	100.40ms	Tight distribution around target
Max Gap	743.93ms	During initialization handoff
Gap Count	1	Single large gap at startup
Quality	degraded	Due to max gap > 500ms (init artifact)

Quality Definitions:

• Scheduling Quality: How well ticks stay on the 100ms grid (median/p95 lateness).
• Continuity Quality: Trace integrity (gap count, max gap).
• Verdict: Scheduling is strict, but Continuity is degraded by the init gap. Event-level energy relies on continuity.

Metric	Value	Notes
Dropped Ticks	11	Explicitly tracked (0.5% of 2041 samples)
Read Errors	86	Startup/Transition artifacts (t < 5s); excluded via read_ok

Bimodal Distribution Analysis (Red-Team Preemption): Figure 4: Poller Scheduling Jitter. The distribution is split between 50-100ms and 100-150ms bins.

The dt histogram shows samples split between the 50-100ms and 100-150ms bins. This split around the 100ms target is consistent with phase-corrected scheduling behavior on a non-real-time OS (where sleep overshoots are compensated by shorter subsequent sleeps), combined with bin-edge quantization effects. It does not indicate scheduler instability.

• The poller targets absolute timestamps t_k = t_0 + k * 100ms.
• If a tick wakes slightly late (e.g., at 101ms), dt is >100ms.
• The next sleep targets t_0 + (k+1) * 100ms, so the delta t_{k+1} - t_k will be slightly <100ms to compensate (e.g., 99ms).
• This "phase correction jitter" creates a bimodal distribution around the target, confirming the scheduler is enforcing the grid rather than drifting.
• Lateness: Median lateness is 0.32ms (p95: 0.64ms), confirming high scheduling precision.

Gap Impact Analysis: The single 743ms gap occurred during monitor initialization (t < 5s). No generation events overlap this gap. Furthermore, the EnergyMonitor computes gap_fraction per event window, ensuring that any future overlaps would explicitly invalidate that specific measurement.

Impact on results:

• PASS Strict 10 Hz sampling maintained over long durations
• PASS No frequency drift
• PASS Continuity is degraded only by the single initialization gap

9.2 Gap Forensics

The single large gap (743ms) occurred during:

1. Gap 1 (743ms): Thread handoff between Instrument Response Test and Main Loop

This gap is a startup artifact and does not affect the quality of the main physics trace. The V2 infrastructure explicitly tracks and reports such gaps in metadata.

9.3 Integrated Findings Table

Experiment	Primary Metric	Secondary Metric	Quality	Conclusion
Baseline	20.71W mean	9.97W std	Valid	Noise floor established
Response Test	N/A (test failed)	Init errors (86 reads)	Invalid	TR122.A required
VRAM Cliff	1024 tok limit	0% fragmentation	Limited	Architectural limit, not capacity
Joule Curve	polling-limited	prefill: no_data, decode: gappy	Limited	Events too fast for 100ms poller
Heat Soak	equilibrium	slope=0.494 C/min at 48 C	Valid	No thermal stress from GPT-2

9.4 Energy Measurement Validity Matrix

Scenario	Total Run Energy	Per-Event Energy (Fast)	Per-Event Energy (Slow)
Validity	VALID	INVALID	LIMITED
Reason	Gaps < 1% of duration	Events < polling period	Need gap_fraction < 0.1
Use Case	Cost estimation	Billing per inference	Billing per batch

9.5 Uncertainty Propagation

For operational energy calculation (see Section 4.5 for derivation):

E_operational = E_raw - (P_idle * T_event)
sigma_E_idle(T) approx sigma_idle * sqrt(T * dt)  [assuming independent samples]

With sigma_idle = 9.97W and dt = 0.1s:

Event Duration (T)	Energy Uncertainty sigma_E_idle	Notes
100ms	+/-1.00 J	sqrt (0.1 * 0.1) = 0.1
1s	+/-3.15 J	sqrt(1.0 * 0.1) = 0.316
10s	+/-9.97 J	sqrt(10 * 0.1) = 1.0

Reviewer Note on Independence: The above table assumes independent samples. Real power traces exhibit autocorrelation, meaning the effective sample size (N_eff) is lower than the raw count N.

• SEM_eff = sigma / sqrt(N_eff) where N_eff = N * (1-r)/(1+r) (lag-1 autocorrelation) or computed via block bootstrapping.
• Future artifacts will include N_eff explicitly. For this report, we accept SEM_est (naive independence) as a baseline lower bound.

Key insight: Energy uncertainty scales with sqrt(T), not T. For per-event energy to be meaningful:

1. Events must be long enough that SEM_eff_E_idle << E_operational, OR
2. Use hardware energy counters (which avoid polling entirely).

9.6 Correlation Between Experiments

The experiments are not independent; they form a coherent picture:

Baseline -> Response Test -> VRAM Cliff -> Joule Curve -> Heat Soak
    down           down              down            down            down
 Noise     Sensor OK      Memory OK    Energy OK?   Thermal OK
 Floor                                 (NO - need    (YES for
                                       larger model) small models)

The limiting factor is the test workload, not the infrastructure. Small models (GPT-2) prove the harness works but do not stress the system. TR122.A with Llama-8B is the natural follow-up.

9.7 What We Now Know (That We Didn't Before)

Before TR122:

1. We assumed idle power was "negligible" -- Wrong: It's 20.71W with 9.97W variance.
2. We assumed sensors were accurate -- Design valid, empirically TBD: 100ms polling works but needs square-wave validation (TR122.A).
3. We assumed thermal equilibrium was instant -- Wrong: Even small models take ~5 minutes.
4. We assumed VRAM fragmentation caused OOMs -- Unconfirmed: GPT-2-XL hits architectural limits before capacity limits.

TR122 replaces assumptions with measurements. That is the value of this report.

---

10. Production Guidance

10.1 What to Always Do

1. Run baseline calibration for every new hardware configuration or software environment.
2. Use EnergyMonitor from banterhearts/monitoring/energy.py for all energy measurements.
3. Check fake_idle_flag before trusting baseline. If true, investigate background processes.
4. Check poller.quality before trusting event-level energy. If degraded, only trust aggregate energy.
5. Handle Floor Readings: Publish mean_all (unbiased) but track floor-suspect rate as a diagnostic to detect sensor caching.

10.2 What to Never Do

1. Do not assume idle power. It varies by hardware, driver, and power profile. (20.71W +/- 9.97W on this platform.)
2. Do not report event energy without checking energy_quality. Honest uncertainty is better than false precision.
3. Do not run long benchmarks without Heat Soak validation. Thermal transients contaminate measurements.
4. Do not bill events shorter than 300ms with a 100ms poller. Expect energy_quality: no_data or interpolated.

10.3 Energy Attribution Rule Table

Use these rules to gate energy_quality in production pipelines:

Quality Level	Condition	Confidence
NO_DATA	samples < 2	Unusable. Event too short for poller.
GAPPY	gap_fraction > 0.10 OR max_gap > threshold	High uncertainty. Use with caution.
GOOD	samples >= 3 AND gap_fraction <= 0.10	High confidence. Billable.

Feasibility Matrix (100ms Poller):

Event Duration	Likely Quality
< 200ms	NO_DATA
200ms - 300ms	NO_DATA to GAPPY
> 300ms	GOOD (usually)

10.4 Operational Checklist

Before publishing performance numbers, verify:

• Baseline Calibration Passes: fake_idle_flag == false and idle_watts_std is quantified.
• Warmup Complete: Heat soak confirms run_state: equilibrium (or transient regime explicitly declared).
• Poller Continuity: poller_continuity_quality is not "degraded" (no massive gaps in trace).
• Energy Gating: No events reported with energy_quality: no_data are treated as valid measurements.
• Metadata Captured: Dictionary includes Driver Version, Power Plan, and Commit Hash.

10.5 The Production Decision Rule

Use this as the "break-glass" decision logic:

1. If baseline calibration shows fake_idle_flag: true: Stop and fix environment before benchmarking.
2. If Heat Soak shows run_state: timeout after 30 minutes: Your thermal system may be insufficient; results may include throttling effects.
3. If poller shows quality: degraded with high gap count: Only trust aggregate energy, not per-event energy.

10.6 Proposed "Single Continuous Poller Thread" Pattern

The "degraded" continuity consistency in this run (single gap) was caused by stopping and restarting the monitoring thread between phases.

Recommendation:

• Poller Invariant: The poller must start once per process and run continuously. Phases should be tagged, but the monitor thread must never restart to avoid init gaps.
• Phase Markers: Inject "start_experiment" and "end_experiment" markers into the event stream rather than stopping the poller.
• Lifecycle Management: Keep the NVML handle open for the entire process duration to avoid re-initialization latency.

10.7 TR122 Guarantee Contract

Guarantee	Description	status
Grid Adherence	Samples will be spaced at 100ms intervals (median dt=100ms).	PASS
Idle Baselines	Energy is net of a rigorously measured baseline (~120s at 100ms; sample count not stored).	PASS
Event Alignment	Events < 200ms are flagged as no_data or gappy; no false precision.	PASS
Continuity Check	Gaps > 500ms trigger a "degraded" quality flag.	PASS
Thermal Equilibrium	5-minute stability check is enforced before claiming steady state.	PASS
Per-Event Joules	NOT GUARANTEED for sub-second events (requires V3 counters).	FAIL

---

11. Limitations & Next Steps

11.1 Known Limitations

11.1.1 Small Model Bias

All tests used GPT-2 variants (124M to 1.5B parameters). These models:

• Complete inference in milliseconds (too fast for event-level attribution with nominal 100ms power polling)
• Do not thermally stress this chassis in the tested configuration (equilibrium at ~48 C; no throttling detected)
• Fit comfortably in 12GB VRAM

Result: The infrastructure is validated, but not stressed. We know the harness works, but we don't know its limits.

11.1.2 Sensor Bandwidth & Smoothing

Even if polling frequency is increased, NVML's nvmlDeviceGetPowerUsage may return a time-averaged value (approx. 1s window) on some GPU architectures/drivers, rather than an instantaneous sample.

• Implication: Increasing poll rate to 1ms may simply oversample a smoothed signal.
• Mitigation: Rely on Hardware Energy Counters (TR122.B) which integrate internal high-frequency sensors, or use Macro-Window measurement (>30s) where smoothing effects wash out.

11.1.3 Polling Resolution

100ms polling (10 Hz) cannot capture:

• Sub-100ms events (most prefill operations)
• Power spikes during short decode steps
• Fine-grained energy attribution

Alternatives that TR122.B should explore:

1. nvmlDeviceGetTotalEnergyConsumption -- hardware energy counter (no polling gap)
2. CUDA event timing with power snapshot -- correlate exact GPU time to nearest power sample
3. Faster polling (10ms) -- but this increases CPU overhead and may introduce jitter

11.1.3 Single GPU Constraint

Only tested on RTX 4080 Laptop GPU. Other hardware differs in:

Hardware	Expected Difference
RTX 4090 (Desktop)	2* TDP, faster equilibrium, no throttling
RTX 3080 Ti (Ampere)	Different power curve, higher baseline
A100/H_100 (Datacenter)	Much higher TDP, active cooling, different driver behavior
Apple M-series	No NVML, completely different measurement approach

11.1.4 Windows-Only Testing

NVML behavior on Windows may differ from Linux:

• Power reporting granularity
• Clock reporting accuracy
• Driver background activity
• Power state transitions

TR122.C should cross-validate on Ubuntu 22.04 with the same GPU.

11.1.5 No Multi-GPU Testing

This study assumes single-GPU inference. Multi-GPU (tensor parallel, pipeline parallel) introduces:

• Inter-GPU communication energy
• Synchronization overhead
• NVLink/PCIe power consumption
• Load imbalance artifacts

11.2 Specific Failure Modes

The following scenarios would cause TR122's methods to fail:

Failure Mode	Symptom	Mitigation
Background process active	fake_idle_flag = true	Kill background apps before run
Sensor caching by driver	Flat power readings	Check for variance in trace
Thermal throttling	Clock drop under high temp	Monitor clock column in trace
Memory leak	Rising VRAM over run	Check VRAM column in trace
Polling thread blocked	Large gaps in trace	Check gap_count in metadata

11.3 Infrastructure V2 (Implemented)

The following infrastructure improvements were implemented as part of TR122 v2.0:

ID	Improvement	Status
V2.1	read_ok flag on power samples; failed reads treated as missing, not 1.2W	PASS Implemented
V2.2	Strict periodic scheduling (sleep_until(next_tick)) with lateness + dropped_ticks logging	PASS Implemented
V2.3	Composite fake_idle_flag (primarily util_p95 in this run; extensible to power outliers)	PASS Implemented
V2.4	Enforce "preload before trace" as protocol rule	WARNING Documented (harness-level change pending)
V2.5	dt histogram and dropped_ticks count in run_metadata / poller_stats	PASS Implemented

11.4 Recommended Follow-Ups

ID	Description	Priority	Effort	Impact
TR122.A	Test with Llama-3.1-8B (stress VRAM, thermal, energy)	High	2 days	High
TR122.B	Implement NVML energy counters for precise event attribution	High	1 day	High
TR122.C	Cross-validate on Ubuntu 22.04 (same hardware)	Medium	1 day	Medium
TR122.D	Test RTX 4090 Desktop (different thermal profile)	Medium	2 days	Medium
TR122.E	Implement thermal hysteresis analysis (clock vs temp curve)	Low	1 day	Low

| TR122.F | Add memory bandwidth monitoring (saturation detection) | Low | 2 days | Medium |

11.5 Open Research Questions

TR122 raises questions it does not answer:

1. What is the Joule Curve for 8B models? We know small models are too fast to measure. Large models may reveal the efficiency sweet spot.

2. Does thermal throttling affect latency variability? We found no throttling for small models. Large models under sustained load may show latency creep.

3. Is VRAM fragmentation ever a problem in practice? Our torture test found 0% fragmentation. Real workloads with variable sequence lengths may differ.

4. How does Windows power management affect inference? We observed 10W variance in idle. This may fluctuate based on power plan settings.

5. Can we predict OOM before it happens? The VRAM Cliff test currently binary-searches to failure. A predictive model would be more useful.

11.6 TR122 Series Roadmap

The next reports will build directly on this V2 infrastructure:

TR122.A: The Stress Study (Hardware Limits) Moving from "harness validation" to "hardware saturation".

• Model: Llama-3-8B (or similar 7B+ class)
• Matrix:
  • Context: 1k, 4k, 16k (force VRAM pressure)
  • Batch Size: 1, 2, 4 (force compute saturation)
  • Gen Tokens: 256, 512 (ensure decodes > 1s for valid measurement)
• Goal: Determine if thermal equilibrium holds under sustained saturation and quantify clock throttling.
• Requirement: Each measured event window must contain >= 5 power samples (>500ms).
• Limitation Check: Verify if NVML power reporting is instantaneous or time-averaged (driver-dependent).

TR122.B: The Energy Precision Study Eliminating the polling loop limitation.

• Method: Implement nvmlDeviceGetTotalEnergyConsumption (Hardware Counters).
• Condition: Validate sensor support (some architectures return NVML_ERROR_NOT_SUPPORTED).
• Fallback: "Macro-Window Attribution" if counters unavailable.
• Validation: Compare integrated polling energy vs. hardware counter energy on long events.
• Deliverable: "Joules per Token" calibration plot with mismatch % and error bars.

TR122.C: The Multi-Homed Study

• Scope: Replicate on Linux (A100/H_100) and Mac Studio (M-series).
• Goal: Verify if the "Idle Power Variance" theorem holds on server-grade and ARM hardware.

11.7 What This Report Does NOT Claim

To prevent misreading:

• FAIL "GPT-2 is the right model for physics studies" -- No, it's too fast. Use larger models.
• FAIL "100ms polling is sufficient for energy billing" -- No, hardware counters are needed.
• FAIL "Thermal equilibrium is always 5 minutes" -- No, that's specific to this model + hardware.
• FAIL "VRAM fragmentation is not a problem" -- Unconfirmed; test with larger models.
• FAIL "The infrastructure is production-ready" -- No, it needs TR122.B improvements first.

---

12. Reproducibility & Artifacts

12.1 How to Reproduce

# From repository root
cd c:\Users\sahil\OneDrive\Documents\GitHub\Banterhearts

# Run the full physics study
python scripts/tr122/run_physics.py

# Results will be in scripts/tr122/results/<timestamp>/

12.2 Environment Requirements

Python 3.12+
PyTorch 2.x with CUDA
pynvml
transformers
numpy
pandas

12.3 Git State

• Repo: https://github.com/sahil170595/Banterhearts
• Branch: main (active dev)
• Commit: (See run_metadata.json in artifact pack)

12.4 System Configuration Fingerprint

To ensure 100% reproducibility, V2.3 runs capture:

• GPU Driver: NVIDIA Windows Driver (e.g., 531.18); reports CUDA 12.8 capability.
• Power Plan: Windows "Balanced" vs "High Performance"
• Power Limit (TGP): 175W (Max) vs 35W (Battery); logged via power_limit_mw.
• Torch Version: 2.1.2+cu121
• Precision: float16 vs bfloat16 (affects VRAM/Power)

Field	Value
Commit Hash	640db42288856b6608be8ffbafd864c32bb512c8
Dirty	false

---

Appendix A: Key Tables

A.1 Summary of All Experiments

Experiment	Status	Key Metric	Value
Baseline Calibration	PASS PASS	Idle Power	20.71 W
Instrument Response	FAIL FAIL (init errors)	Delta	N/A (TR122.A)
VRAM Cliff	PASS PASS (Architecture-Limited)	Max Context	1024 tok
Joule Curve	WARNING LIMITED	Energy Quality	polling-limited (prefill: no_data; decode: gappy)
Heat Soak	PASS PASS	End State	equilibrium (slope=0.494 C/min)

A.2 Poller Statistics

Statistic	Value
Target Period	100 ms
Median dt	100.00 ms
p95 dt	100.40 ms
Max Gap	743.93 ms
Gap Count	1
Quality	degraded (init artifact)
Dropped Ticks	11
Read Errors	86 (initialization)

---

Appendix B: Poller Quality Analysis

The poller quality was flagged as degraded due to max_gap_ms > 500. Analysis shows the gap occurred during thread handoff between the Instrument Response Test and the Main Loop:

V2 Strict Scheduling: The V2 poller achieves median dt=100.00ms with tight distribution (969 samples in 50-100ms, 980 in 100-150ms). The single 743ms gap is a startup artifact that does not affect the quality of the physics trace.

Production Note: The V2 infrastructure explicitly tracks dropped ticks (11 total, 0.5%) and read errors (86 during initialization only). The main physics trace is clean and production-grade for macro-measurement.

---

Appendix C: Configuration

C.1 Experiment Configuration (physics.yaml)

experiment:
  name: "tr122_physics_v1"
  seed: 42
  output_dir: "scripts/tr122/results"

poller:
  target_period_ms: 100
  gap_dt_threshold_s: 0.25
  gappy_gap_fraction_threshold: 0.10

safety:
  max_temp_c: 83.0
  resume_temp_c: 80.0

models:
  baseline: "models/tiny-gpt2"
  standard: "gpt2"
  stress: "gpt2-xl"

tests:
  instrument_response:
    enabled: true
  vram_cliff:
    enabled: true
    min_ctx: 512
    max_ctx: 131072
    step_mb: 4
    repetitions: 3
  joule_curve:
    enabled: true
    prompts:
      - "Let's discuss the physics of computation."
    gen_tokens: 64
    batch_sizes: [1, 2, 4, 8, 16]
    repetitions: 3
    warmup: true
  heat_soak:
    enabled: true
    duration_min: 30
    rolling_window_min: 5
    equilibrium_slope: 0.5
    model: "gpt2"

---

Appendix D: Glossary

Term	Definition
Baseline Power	The idle power consumption of the GPU when no inference is running.
Operational Energy	Energy consumed above baseline; the "cost of intelligence."
Gap	A period where sensor polling interval exceeded threshold (>250ms).
Gappy	An event where >10% of duration had sensor gaps; energy is unreliable.
Thermal Equilibrium	State where dT/dt < 0.5 C/min (temperature has stabilized).
VRAM Cliff	The context length at which VRAM is exhausted and OOM occurs.
Joule Curve	The relationship between batch size and energy-per-token.
Heat Soak	Extended run to reach thermal equilibrium before measurement.
Fragmentation	Wasted VRAM due to allocator holes (inactive_split / reserved).
ExperimentClock	Singleton monotonic clock for synchronized timestamps.

---

End of Technical Report 122