LLM Evaluation Harness

Hi, I’m Roger. Here are some experiments I’ve run on LLMs running on my own machine, for fun.

Experiments

Exp 001: Quantization Cliff

• Qwen2.5-7B-Instruct across four llama.cpp k-quants (Q3_K_M → Q8_0) on math, factual QA, and structured extraction (n=600 per cell).
• “Q4 is fine” mostly holds. Only significant cliff: TriviaQA at Q3_K_M (−3.2pp vs Q8_0, p=0.011).

Exp 002: LoRA vs API

• LoRA-tuned Qwen2.5-1.5B beats gpt-5.4-mini zero-shot by +0.215 F1 (p=0.0001) on a 6-field email extraction task.
• Cost story is the inverse of the usual pitch: break-even ~19M inferences at 2026 pricing.

Exp 003: Layer Probes

• Linear probes on every layer of Qwen2.5-1.5B-Instruct, predicting whether the model will refuse a prompt.
• Token embeddings alone (layer 0) capture 95% of the best probe accuracy (69.8% vs peak 73.1%). Refusal looks mostly lexical.