LLM Distillation 2026: Shrink Qwen3 with Axolotl & DPO

You’ve watched the same bill three quarters running: a frontier API endpoint doing work a 4B model could handle, at 40x the cost and 900ms of p95 latency you can’t engineer away because the round trip is somebody else’s data center. Meanwhile Qwen3 and DeepSeek-V4 checkpoints sit on Hugging Face for free, and every “just distill it” thread on X skips the part where your student model scores 82% of teacher quality on your eval set but collapses on the three edge cases your product actually depends on. You’ve got a rented 8xH200 node approved for two weeks, a teacher model you can query, and no reliable answer to whether logit-level, sequence-level, or on-policy distillation is the one that gets you shipped.

This is for developers who already fine-tune — you’ve run a LoRA job, you know what a tokenizer mismatch looks like, you can read a loss curve without panicking. You should be comfortable with PyTorch, CUDA memory errors, and renting GPUs by the hour. It is not a course on transformer internals, not an intro to Python, and not a pretraining guide. We stay on Axolotl, TRL, and Unsloth as the working toolchain; if your stack is JAX or a bespoke trainer, the concepts port but the commands don’t.

Honest framing: AI is genuinely good at the mechanical middle of this — generating teacher traces at volume, sweeping hyperparameters, writing eval harness boilerplate, catching a malformed dataset schema before you burn a node-hour. It is bad at telling you whether your student actually got worse. Benchmark deltas hide catastrophic forgetting; a model that gains two points on MMLU while quietly losing the ability to follow your system prompt will pass every automated gate you write. Human review is non-negotiable on three things: the eval set itself (only you know what failure costs you), the regression check before you promote a checkpoint, and any preference data you use for alignment. Automate the runs, not the judgment.

What This Guide Covers

• The 2026 cost math that decides whether distillation beats just paying for the big model — with the break-even points spelled out
• How to pick a teacher/student pair that won’t fight you, across the Qwen3, DeepSeek-V4, and Llama 4 Maverick families
• Building a distillation dataset that holds up: trace collection, top-K logit capture, and the deduplication step most people skip
• A clear-eyed comparison of logit-level, sequence-level, and on-policy distillation — what each one buys you and what it costs
• A complete first Axolotl run you can follow end to end, including the config decisions that quietly determine your outcome
• On-policy distillation with TRL’s GKDTrainer and Unsloth, and when the added complexity earns its keep
• Hyperparameter guidance for temperature, alpha, sequence packing, and LR schedules — ranges that work, not vibes
• LoRA versus full fine-tune decided on real numbers: hours, dollars, and quality delta on a rented 8xH200 node
• Layering DPO and KTO onto a distilled student without undoing the distillation you just paid for
• Catastrophic forgetting: how to detect it early, how replay buffers help, and the regression guards that stop a bad checkpoint from shipping
• Getting the student into production — GGUF, AWQ, vLLM, and llama.cpp tradeoffs for your serving profile
• A benchmark harness that measures the three things that matter together: quality, latency, and cost per million tokens
• Case studies of distillation runs that worked and runs that didn’t, with the diagnosis for each
• A decision framework for the question underneath all of it — whether this project should be distilled at all

Instant online access the moment checkout completes. One purchase, the complete guide, no upsell and nothing held back for a “pro” tier.