Skip to content
Chimeraforge

Edge LLM Inference Under Real-World Constraints

How fast local inference can get, and how safe it stays at the edge. Independent research by Sahil Kadadekar.

  • 1.46M+ measurements
  • 55 reports
  • 6 whitepapers

Synthesis and technical reports

Chimeraforge Whitepaper: High-Performance LLM Agent Orchestration

Rust vs. Python for production AI orchestration — hybrid architecture and Dual Ollama pattern achieving 58% latency reduction.

Synthesis Compendium

Phase 1 Decision Whitepaper

Foundation synthesis — model loading, ONNX conversion, quantization baselines, and security analysis across 9 technical reports.

Synthesis Phase 1

Phase 2 Decision Whitepaper

Benchmarking synthesis — cross-backend inference parity, TensorRT compilation, and scaling laws across 7 reports.

Synthesis Phase 2

Phase 3 Decision Whitepaper

Optimization synthesis — KV cache tuning, INT8/FP8 quantization, context scaling, and deployment pipeline across 11 reports.

Synthesis Phase 3

Phase 4 Decision Whitepaper

Safety-pivot synthesis — alignment erosion under quantization, concurrency invariance, and backend template divergence across 4 reports.

Synthesis Phase 4

Phase 5 Decision Whitepaper

Attack-surface synthesis — batch perturbation, multi-turn jailbreaks, cross-architecture fragility, and composition effects across 306K+ samples. TR138 Study D batch-invariant-kernel ablation as standalone addendum.

Synthesis Phase 5

Phase 6 Decision Whitepaper

Serving-state safety certification — measurement-validity substrate (judge triangulation, KV-cache safety null, speculative-decoding safety-invariance screen, mechanistic probing, portability validation) + FP8 KV-cache standardized batteries + serving-state factorial.

Synthesis Phase 6

JTPv2 — Predictive Validity of Cheap Pre-Rejudge Signals

Predictive-validity follow-up to the Judge Triangulation Protocol (TR148): can a cheap pre-rejudge signal predict a cell's judge-sensitivity on a leave-one-family-out hold-out before paying for the second-judge pass? On the GGUF-local rlhf-only hold-out the class label saturates (all-positive → AUC undefined) — a structural property of JTP on production-quantization substrate, not a power failure; directional signs and a pool-robustness secondary finding hold.

Latest TR167 · Phase 9

Serving-Stack Physics — GIL-Attributable Concurrency Collapse

346-cell matched study across 3 model tiers × 4 workloads × 5 concurrency levels on a consumer RTX 4080: a uniform parallel-efficiency breakdown at N=2, P95 latency multipliers up to 1446×, and six deterministic N=16 hangs attributed to Python GIL serialization, with kernel-level nsys evidence and cross-backend (vLLM / SGLang / TGI) validation.

TR164 · Phase 8

Cross-Backend Serving-Stack Boundary Matrix (vLLM vs SGLang)

Matched A100 80GB PCIe head-to-head — 5 models × 5 workloads × 6 concurrency levels × 2 phases per backend (1,800 cells / 189,000 rows, ok_rate 1.0). Continuous batching holds 0.47–0.62 parallel efficiency at N=32, an order-of-magnitude shift up from V1 pytorch_direct's 0.056 collapse. vLLM/SGLang deltas are small (1.5–7.6pp) but Holm-significant on 4 of 5 workloads — a workload-conditional routing signal, not an engine ranking.

TR164 V3 · Phase 8

The Amortization-Breakdown Surface — A Predictive Bandwidth Model for Continuous Batching

A 672-cell offline static-batch grid (3 models × A100-80GB + H100 × vLLM 0.10.2, 2,016 timed decode generations, ok_rate 1.0) showing the continuous-batching free lunch is bounded by KV-cache read bandwidth. A one-parameter model η(B) = (1+r)/(1+Br) fits the 96 efficiency curves at median R² 0.93, and its zero-fitted-parameter architectural form predicts the breakdown knee at Spearman ρ = 0.84 — off only by a GPU-specific batch-saturation factor α (0.52 A100, 0.33 H100; a roofline check rules out compute-overlap and VRAM-capacity). Complements V3's closed-loop boundary — grounds it, does not revise it.

TR164 V4 · Phase 8

Evidence Closure for Serving-Stack Physics

Reviewer-facing evidence-closure layer over the V4 amortization model: explicit predictor baselines + failure-mode accounting for the 672-cell static grid, served SGLang validation against the static-knee upper-bound, and true-decode weight-axis matrices on a single H100 (Qwen2.5 / Mistral / Gemma 3). Written from the generated artifacts, not manual transcription.

TR164 V5 · Phase 8

Python GIL Ablation for Direct PyTorch Inference Boundaries

Matched-pair nogil falsification of TR164 V1's GIL-attribution hypothesis — same hardware/models/workloads under a Python 3.14t free-threaded build, one knob flipped. Verdict H2_partial: removing the GIL recovers ~+17.9pp of N=2 parallel efficiency (19/24 combinations) and resolves 2 of 6 deterministic hangs, so the GIL is a mechanism behind the breakdown, not the sole cause.

TR165 · Phase 8

RTSI-Gated Quantization Routing

Offline LOOCV feasibility study over the TR142 RTSI table — routing the highest-instability (model × quantization) configurations to a safe baseline recovers ~76% of the quantization refusal-loss gap at ~20% of configs rerouted (out-of-sample ROC-AUC 0.84). A proof-of-mechanism defense, circularity disclosed; paper-grade expansion specified.

TR163 · Phase 7

Speculative Decoding x Safety

Protocol and publication contract — does speculative decoding change safety outcomes via draft model token shaping?

TR144 · Phase 6

KV-Cache Quantization x Safety

FP8 KV-cache as a silent safety degradation vector — five-phase null result across 24,054 records on 3 models. Workload-specific paired eval recipe.

TR145 · Phase 6

Mechanistic Safety Probing Under Quantization

Why standard mechanistic probes fail to predict quantization-induced safety degradation. 5,100 forward passes across 4 phases.

TR146 · Phase 6

Final Portability Validation for Benchmarking Integrity

Full v1–v4 integration plus external case study using TR126 as reference standard. 52,410 measurements across 4 GPU regimes and 3 Triton minor versions.

TR147 · Phase 6

Multi-Judge Reliability for Refusal-Axis Safety Classification

68,620 judge rows across 5 judges on the TR145 safety subset. κ = 0.6917 (triangulate verdict) — single-judge labels insufficient. Plus a dual-axis methodology finding: safety-specialist judges measure a different axis than general LLM judges.

TR148 · Phase 6

Standardized Safety Battery — FP16 vs FP8 KV-Cache

7,578 records across 3 models × 4 standardized batteries (HarmBench, JailbreakBench, StrongREJECT, XSTest) × 2 KV-cache dtypes. Replicates TR145’s FP8 null on literature-comparable corpora; corrected paired-odds-ratio estimator. Local-only judging, $0 external API.

TR149 · Phase 6

The Serving-State Safety Factorial (v2 Local Expansion)

FP8 KV-cache folded across batch size, prefix-caching, and temperature — 45,000 responses, 20,754 matched FP16-vs-FP8 pairs across 5 models × 4 batteries × 6 serving-state contexts (XSTest uncapped to 450/cell). Harmful-prompt refusal perfectly invariant under every serving state (0/8,976 discordant pairs); only footprint is a sub-percentage-point over-refusal lean on the Qwen family on XSTest (Mantel–Haenszel pooled OR 1.88 [1.32, 2.69]). 117/120 cells TOST-equivalent at ±3pp.

TR152 · Phase 6

Batch Inference Safety Under Non-Determinism

Audit-layer flip adjudication + 7,257-sample reduced replication on enriched 187-prompt subset, with corrected refusal detector (v2.2). Study D batch-invariant-kernel ablation lives as a standalone addendum.

TR138 · Phase 5

Batch-Invariant Kernel Ablation

Full-depth mechanism report for batch-conditioned refusal robustness. 110-record H100/vLLM ablation across 55 candidate score-flip pairs: standard vLLM reproduces 22/55 label flips and 25/55 text changes; the VLLM_BATCH_INVARIANT=1 environment flag reduces both to 0/55. Kernel-path mechanism evidence outside the original 306,996-sample synthesis.

TR138 Study D Addendum · Phase 5

Multi-Turn Jailbreak Susceptibility Under Quantization

Conversational attack sweep — 10,600 conversations across 4 models, 6 quant levels, 8 attack strategies.

TR139 · Phase 5

Many-Shot & Long-Context Jailbreak Under Quantization

15,000 scored samples across 4 models, 6 quant levels, 5 shot counts, and 3 context-length profiles.

TR140 · Phase 5

Cross-Architecture Refusal Fragility Under Batch Perturbation

127,224 records across 18 models, 10+ families, 4 alignment types — batch-induced safety flip asymmetry on Blackwell GPU.

TR141 · Phase 5

Quality-Safety Correlation Under Quantization

Multi-format synthesis across GGUF, AWQ, and GPTQ — 6 models, 51 model-quant cells, quality-safety divergence analysis.

TR142 · Phase 5

Cross-Request Safety Leakage Under Continuous Batching

14,250 records — batch composition effects on safety in multi-tenant vLLM inference.

TR143 · Phase 5

Alignment Robustness Under Quantization

GGUF + AWQ + GPTQ safety evaluation across 6 models (1.2B–7.6B) — refusal template destabilization and deployment taxonomy.

TR134 · Phase 4

Safety Under Multi-Agent Concurrency

Does running N concurrent agents on a shared backend degrade model safety?

TR135 · Phase 4

Cross-Backend Safety Consistency

Ollama vs vLLM vs TGI safety comparison across 3 models, 4 backends, and 6 benchmarks.

TR136 · Phase 4

The Safety Tax of Inference Optimization

Unified synthesis of quantization, concurrency, and backend effects on LLM safety — 74,254 samples.

TR137 · Phase 4

KV-Cache Production Economics

Phase-split $/token with cached decode across MHA and GQA architectures.

TR123 · Phase 3

Quality & Accuracy Baseline

Backend equivalence, quantization impact, and sampling variance across 5 models.

TR124 · Phase 3

Quantization Decision Matrix

Production quant selection across 6 models, 4 families, 9 formats (GGUF + AWQ + GPTQ) with quality and safety evaluation.

TR125 · Phase 3

Linux/Triton Compile Validation

Cross-platform A/B confirming torch.compile benefits under real Inductor+Triton on consumer GPU.

TR126 · Phase 3

Long-Context Performance

Consumer GPU context scaling from 512 to 32K tokens with two-regime VRAM analysis.

TR127 · Phase 3

Production Workload Characterization

Concurrency, saturation, streaming, and multi-turn performance of Ollama on consumer GPU.

TR128 · Phase 3

N-Agent Scaling Laws

Closed-loop multi-agent throughput scaling on consumer GPU with Ollama.

TR129 · Phase 3

Serving Stack Benchmarking

Ollama vs vLLM vs TGI — multi-agent throughput scaling comparison.

TR130 · Phase 3

GPU Kernel Profiling

Root-cause analysis of multi-agent throughput degradation via NVIDIA Nsight Systems.

TR131 · Phase 3

Serving-Stack GPU Kernel Profiling

In-container Nsight Systems analysis of vLLM and TGI under multi-agent load.

TR132 · Phase 3

Predictive Capacity Planner

Operationalising 70,000+ measurements into a decision tool for deployment planning.

TR133 · Phase 3

Cross-Backend Inference Benchmark

Frontier benchmark comparing eager, JIT, torch.compile, ONNX, and TensorRT across architectures.

TR117 · Phase 2

Efficiency Anomalies

Root cause analysis of the Python ceiling, Qwen mystery, and ranking flip in multi-agent inference.

TR117 Multi-Agent · Phase 2

Model Scale Comparative Analysis

ONNX Runtime + TensorRT performance across 1,210x parameter scaling.

TR118 · Phase 2

Cost & Energy Analysis

Local-first inference TCO with telemetry — prefill and generate cost breakdowns.

TR119 · Phase 2

The Compile Paradox

Why a compile backend can win the mean while losing the median — root-cause audit.

TR120 · Phase 2

Scaling Laws & Capacity Planning

How latency, throughput, and cold-start risk change from ~0.1M to ~20.9B parameters.

TR121 · Phase 2

The Physics of Inference

Establishing the fundamental constraints of LLM execution on consumer hardware.

TR122 · Phase 2

LLM Performance Analysis

Single-agent benchmarking across 158 runs and 6 models — optimal configs for real-time workloads.

TR108 · Phase 1

Agent Workflow Optimization

Process-isolated agent workflow benchmarking; GPU=60, CTX=512, TEMP=0.8 for stable output quality.

TR109 · Phase 1

Concurrent Multi-Agent Performance

Parallel agent execution study (150 runs) establishing Python multi-agent efficiency baselines.

TR110 · Phase 1

Rust Agent Workflow Performance

Comprehensive parameter optimization and Python performance parity validation for Rust agents.

TR111 · Phase 1

Rust vs Python Agent Comparison

Cross-language analysis for production LLM deployments — latency, throughput, and resource efficiency.

TR112 · Phase 1

Rust Concurrent Multi-Agent Analysis

Rust concurrent multi-agent performance profiling and contention analysis.

TR113 · Phase 1

Dual Ollama Architecture

Rust concurrent multi-agent performance with dual Ollama instances — near-zero contention.

TR114 · Phase 1

Async Runtime Deep Dive

Rust async runtime performance analysis for multi-agent LLM workloads.

TR115 · Phase 1

Cross-Model Benchmarks

Qwen 2.5 vs Gemma 3 vs Llama 3.1 8B — comprehensive multi-agent performance study.

TR116 · Phase 1

Gemma 3 Benchmark Report

Performance benchmarks for Google Gemma 3 models — baseline measurements, parameter tuning, and quantization.

Baseline · Phase 0

Ollama LLM Benchmark: Quantization & Runtime Analysis

Llama 3.1 8B across q4_0 / q5_K_M / q8_0 quantizations — runtime characterization on consumer GPU that motivated the TR108+ Rust-vs-Python program.

Baseline · Phase 0

Performance Deep Dive: Quantization & Kernel Optimization

CUDA kernel deep-dive across quantization schemes on an RTX 4080 — the baseline characterization that anchored the TR123-133 optimization phase.

Baseline · Phase 0

Key findings

Concrete results pulled from the published reports. Numbers, not narrative.

100% ASR

Q2_K collapses refusals on the worst-affected model — 100% attack success on qwen2.5-1.5b. Not uniform: effects vary by model.

TR139
p = 0.942

Alignment type does not predict batch-induced safety fragility (RLHF, SFT, DPO, distilled — none differ).

TR141
25pp

Backend migration moved safety 7–25pp, peaking at 23–25pp on Llama 3.2 1B. Chat template divergence, not the framework.

TR136
13.9×

Quality metrics are not safety proxies. Safety degraded 13.9× faster than quality on llama3.2-1b at Q3_K_S.

TR142
99.4%

Dual Ollama reached 99.4% coordination efficiency on the best config, and cut contention to near zero. Architectural fix, not code fix.

TR114
+74%

GPU memory bandwidth is the multi-agent bottleneck — not the serving stack. Overturned the TR130 conclusion.

TR131
2.25×

Continuous batching delivers 2.25× throughput at N=8 via 77-80% kernel reduction.

TR132
Q4_K_M

The safe GGUF default — established across 5 models, extended to 7 in v2. 30-67% cost savings.

TR125
NULL

FP8 KV-cache produces no Holm-significant safety effect across 24K paired records on 3 models. Not pre-approved, not pre-banned — workload-specific paired eval required.

TR145
κ = 0.69

Cross-LLM judge agreement is "triangulate" — single-judge labels are insufficient for safety classification. 68K judge rows over the TR145 safety subset. Plus: safety-specialist judges measure a different axis than general LLMs.

TR148

Conclusive reports and appendices

Dissertation-style synthesis documents consolidating findings across multiple technical reports.