quantfit
Quantize an LLM — and check it still refuses what it should. Quantization makes a model cheaper to serve and can quietly strip safety behavior; a 4-bit model that answers what the full-precision model refused is a regression no perplexity number will show you.
quantfit verify-safety --demoProbe set
40 curated prompts: some to refuse, some to answer
Generate
the unquantized baseline and the quantized model
Judge
refusal or compliance, by a local classifier
Two axes
refusal-robustness drift and over-refusal drift
Verdict
bounded by a Wilson 95% CI, never absolute
Safety drift is a vector, not a number
verify-safety generates from both the unquantized baseline and the quantized model over a curated probe set, judges each response with a local classifier, and reports two axes: refusal-robustness drift (did the quant start complying with what should be refused — the dangerous direction) and over-refusal drift (did it start refusing what should be answered — the usability direction). A scalar refusal-delta can read zero while both axes move in opposite directions. Verdicts are bounded, never absolute: a no-detection result bounds the drift, it does not certify safety.
Features
- 01
Method × scheme matrix
AWQ, GPTQ, SmoothQuant, FP8, and RTN through one llm-compressor backend (vLLM-loadable), plus GGUF via llama.cpp. Schemes span W4A16 through NVFP4 and MXFP4.
- 02
Honest capacity, up front
check reads HuggingFace metadata without downloading and sorts the job into fits-VRAM, fits-RAM-via-sequential-onloading, or refuse — naming the real limit. No OOM twenty minutes into a run.
- 03
Built to be checked
Reports follow a written spec (QSR), reproduce decides whether one run reproduces another, and audit holds the documentation to the code — exit code 3 when they drift apart.
Commands
12 commands. Full flags and output samples live in the README.
verify-safetydid quantization break refusals? two-axis drift
checkwill it fit? read from HF metadata, no download
planwhat config a heuristic would pick, and why
quantizerun the quantization across the method matrix
probeper-bit-width quantization sensitivity
gatepre-release CI check that refuses to pass on thin evidence
screenrun a target set to QSR spec
reproducedecide whether one report reproduces another
audithold the docs to the code — exit 3 on drift
emitrender any drift report as a model card
listthe supported method × scheme matrix
calibratebuild and ingest calibration sheets
Evidence
Each capability traces to the measurements behind it. These are the published reports, not a summary of them.
| Capability | What it rests on | Reports |
|---|---|---|
| Quantization erodes alignment | The finding the tool exists to operationalize: alignment behavior degrades under quantization in ways quality metrics do not surface. | TR134 |
| Quality is not a safety proxy | The quality-safety divergence work — and the refusal-threat index behind the judging methodology — established that a good perplexity number tells you nothing about preserved refusals. | TR142 |
What it does not do
The limits the tool states about itself.
- It ships transparent config help, not auto-quantization: you pass --method. Learned routing exists as published research but is explicitly out of scope.
- At over-VRAM sizes use gptq — AWQ’s grid search is transfer-bound under sequential onloading.
- FP4 schemes need Blackwell to serve, though quantfit can produce them anywhere.
- The probe set is 40 curated prompts; a no-detection result bounds drift rather than certifying safety.