Nvidia Nemotron 3.5 Lightning NVFP4 2026: QAD Tested

Nvidia has promised since Blackwell’s announcement that FP4 is the future of inference, and for two years the honest answer was “yes, once someone ships a real model in it.” That week arrived: Nemotron 3.5 Lightning NVFP4 is a production checkpoint trained with quantization-aware distillation (QAD), not a post-hoc squeeze of an existing BF16 model. It landed on Amazon SageMaker JumpStart essentially the same week, turning a Blackwell-native 4-bit format into a one-click deploy instead of a paper with a GitHub repo attached. The benchmark chart isn’t the interesting part. NVFP4 roughly halves weight memory and roughly doubles throughput versus FP8 on GB200/B200 hardware, which resets the GPU math for anyone writing a 2026 inference budget. If you sized capacity around FP8 tokens-per-dollar, that plan is now stale.

What’s actually new about Nemotron 3.5 Lightning NVFP4

Three things shipped together, and the combination matters more than any one piece. First, the model: Nemotron 3.5 Lightning is Nvidia’s latency-optimized member of the Nemotron 3.5 family — a hybrid Mamba-Transformer design where most layers are linear-time state-space blocks and only a minority are full attention. That architecture already cuts KV cache pressure hard at long context.

Second, the format: NVFP4 is Nvidia’s 4-bit floating point encoding (E2M1 elements) with a two-level scaling scheme — a small FP8 (E4M3) scale per 16-element micro-block, plus an FP32 per-tensor scale. Compared to the older MXFP4 approach with power-of-two block scales across 32 elements, the finer block and the higher-fidelity scale factor recover most of the accuracy that naive 4-bit throws away.

Third, and this is the part that changes the argument: the checkpoint was produced with quantization-aware distillation, not just post-training quantization. In PTQ you calibrate on a few hundred samples and hope the error stays small. In QAD, the quantized student trains against the full-precision teacher’s output distribution with the fake-quant operators inline, so the weights actively learn to land where the 4-bit grid can represent them. Nvidia reports near-parity with the BF16 teacher on reasoning and math evals — a claim that was never credible with plain PTQ at 4 bits, and the reason “4-bit is for hobbyists running on a 3090” is now an outdated take.

The SageMaker JumpStart listing is the distribution story. You no longer need a Blackwell dev box, a TensorRT-LLM build, and a weekend to evaluate this. You pick a ml.p6e-gb200-class or B200-backed instance, deploy from the JumpStart catalog, and get an OpenAI-compatible endpoint. That collapses the evaluation cost from “engineering project” to “an afternoon and a few hundred dollars of instance time,” which is exactly how a format goes from interesting to default.

Why it matters

• Memory per parameter drops to ~0.5 bytes. Weights that needed roughly 1 byte each in FP8 now need half that plus scale overhead. A model that barely fit one GPU in FP8 fits comfortably with room for a much larger KV cache — and KV cache headroom determines your concurrent request ceiling.
• Throughput roughly doubles on Blackwell. The GB200/B200 tensor cores execute FP4 natively at approximately 2× the FP8 rate. This is not a software trick you can port to Hopper; H100 has no FP4 datapath, so NVFP4 weights there get upconverted and you lose the win. The FP8 vs NVFP4 throughput gap is a hardware-generation argument, not a kernel-tuning one.
• Fewer GPUs per replica means less tensor parallelism. Dropping from TP4 to TP2, or TP2 to TP1, removes all-reduce traffic from the critical path of every token. The end-to-end latency improvement often beats what the raw FLOP ratio suggests, particularly at small batch sizes where communication dominates.
• QAD makes 4-bit defensible for agentic workloads. Agent loops amplify small quality regressions — one bad tool-call argument in step three poisons the whole trajectory. PTQ 4-bit models were disqualified here. A quantization-aware distilled checkpoint with near-teacher eval parity changes the risk calculus enough to warrant an A/B.
• Your 2026 capex model needs rebuilding. If you sized a Blackwell purchase or a reserved-instance commitment on FP8 tokens/sec, you may be over-provisioned. Conversely, if you concluded Blackwell wasn’t worth the premium over Hopper, NVFP4 is the variable that flips that spreadsheet.
• The tooling is open even though the hardware is not. Nvidia Model Optimizer (modelopt) ships the NVFP4 quantization and QAT/QAD recipes under an open license, so you can apply the same treatment to your own fine-tunes rather than waiting for someone to publish a quantized checkpoint.

How to use Nemotron 3.5 Lightning NVFP4 today

1. Check that you actually have FP4 silicon. Compute capability 10.0 (Blackwell datacenter) or higher is the gate. On anything older this exercise is academic.

   nvidia-smi --query-gpu=name,compute_cap,memory.total --format=csv
   
   py -c "import torch; print(torch.cuda.get_device_capability())"
   # (10, 0) or higher -> native NVFP4. (9, 0) is Hopper: FP8 only.

2. Deploy from SageMaker JumpStart if you want the fastest path to a working endpoint. The Nemotron SageMaker JumpStart listing wraps the NIM container, so you get an OpenAI-compatible route without touching TensorRT-LLM.

   from sagemaker.jumpstart.model import JumpStartModel
   
   model = JumpStartModel(
       model_id="nvidia-nemotron-3-5-lightning-nvfp4",
       instance_type="ml.p6e-gb200.36xlarge",
   )
   predictor = model.deploy(accept_eula=True)
   print(predictor.endpoint_name)

3. Or run it locally with vLLM if you own the hardware. vLLM reads the NVFP4 scale metadata straight out of the checkpoint config — you do not pass a quantization flag.

   pip install -U vllm
   
   vllm serve nvidia/Nemotron-3.5-Lightning-NVFP4 \
     --tensor-parallel-size 1 \
     --max-model-len 131072 \
     --gpu-memory-utilization 0.90 \
     --port 8000

4. Hit the endpoint and sanity-check output quality before you trust the benchmark numbers. Run your own hardest prompts, not MMLU.

   curl http://localhost:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "nvidia/Nemotron-3.5-Lightning-NVFP4",
       "messages": [
         {"role": "system", "content": "You are a precise reasoning assistant. Show your work."},
         {"role": "user", "content": "A batch job costs $0.42/GPU-hour on FP8 and finishes in 9.5 hours across 4 GPUs. NVFP4 doubles throughput and lets us drop to 2 GPUs. What is the new total cost, and what is the percentage saving?"}
       ],
       "temperature": 0.2,
       "max_tokens": 800
     }'

5. Measure throughput honestly. Compare against the FP8 checkpoint on the same hardware and the same request mix. Vendor numbers assume favorable batch sizes; yours probably differ.

   vllm bench serve \
     --model nvidia/Nemotron-3.5-Lightning-NVFP4 \
     --dataset-name sharegpt \
     --num-prompts 1000 \
     --request-rate 20 \
     --metric-percentiles 50,95,99

6. Quantize your own fine-tune with NVIDIA Model Optimizer when the stock checkpoint isn’t domain-tuned enough. Start with PTQ to get a baseline, then escalate to QAT/QAD only if the accuracy delta is unacceptable.

   pip install nvidia-modelopt[all]

   import modelopt.torch.quantization as mtq
   from transformers import AutoModelForCausalLM, AutoTokenizer
   
   model = AutoModelForCausalLM.from_pretrained("./my-finetune", torch_dtype="bfloat16")
   tok = AutoTokenizer.from_pretrained("./my-finetune")
   
   def forward_loop(m):
       for text in calibration_texts:   # 256-512 in-domain samples is plenty
           m(**tok(text, return_tensors="pt").to(m.device))
   
   model = mtq.quantize(model, mtq.NVFP4_DEFAULT_CFG, forward_loop)
   mtq.print_quant_summary(model)

   If PTQ loses more than a point or two on your evals, wrap that quantized model in a distillation loop against the BF16 original. That is the QAD recipe in miniature, and modelopt ships reference scripts for it.

7. Pin your serving config. Record the exact vLLM/TensorRT-LLM version alongside the checkpoint revision. NVFP4 kernel implementations are moving fast, and a silent kernel change between minor versions can move your latency numbers more than the model does.

How it compares

Format	Bits/weight	Scaling scheme	Native hardware	Typical accuracy retention	Relative throughput
BF16	16	None	Ampere and later	Baseline	0.25×
FP8 (E4M3)	8	Per-tensor or per-channel	Hopper, Blackwell	Near-lossless	1.0× (reference)
NVFP4 + PTQ	4	FP8 scale per 16-element block + FP32 per-tensor	Blackwell only	Small but measurable loss	~2×
NVFP4 + QAD	4	Same, trained with fake-quant inline	Blackwell only	Near-parity with BF16 teacher	~2×
MXFP4	4	Power-of-two scale per 32-element block	Blackwell, some AMD	Coarser; larger loss	~2×
INT4 (AWQ/GPTQ)	4	Integer group scales	Ampere and later	Varies widely by model	Memory-bound gains only

The INT4 row deserves the most attention. AWQ and GPTQ have served 4-bit weights for years, but they win on memory bandwidth while still computing in higher precision — the matmul itself doesn’t get faster. NVFP4 on Blackwell differs in kind: the tensor cores multiply in 4-bit, so you get the compute win as well as the memory win. That’s why Blackwell FP4 inference is a hardware story and INT4 was a software one.

What’s next

The obvious next domino is FP4 training, not just inference. Nvidia has published recipes for FP4 pretraining with selective higher-precision layers, and the moment large-scale runs land in production, the economics of the whole stack shift again. QAD is the bridge technology: it proves the gradient signal survives the 4-bit grid well enough to teach a model where to sit, which is the same core question training has to answer.

Watch the ecosystem catch-up too. TensorRT-LLM, vLLM, and SGLang all have NVFP4 paths now, but kernel maturity varies a lot by shape and batch size, and the MoE path in particular is still improving quickly. Expect meaningful throughput gains from software alone over the next couple of quarters without touching the checkpoint. Watch also whether NVFP4 stays proprietary or converges with the OCP microscaling standard that MXFP4 belongs to. If the industry consolidates on one 4-bit float, portability improves and Nvidia’s format advantage narrows; if it doesn’t, NVFP4 becomes another sticky reason to stay on CUDA.

Finally, watch the cloud catalogs. SageMaker JumpStart carrying a Blackwell-native FP4 checkpoint on day one signals that the hyperscalers now consider FP4 serving mainstream rather than experimental. When Azure and GCP list equivalent one-click NVFP4 deployments — and when the per-token pricing on those endpoints undercuts FP8 endpoints by a visible margin — FP4 stops being the interesting option and becomes the default one. Budget accordingly.

Frequently Asked Questions

Does NVFP4 work on H100 or older GPUs?

Not natively. FP4 tensor core support arrives with Blackwell (compute capability 10.0+). You can load an NVFP4 checkpoint on Hopper in frameworks that support dequantization, but the weights get upconverted before the matmul, so you keep some memory savings and lose the throughput win entirely. If your fleet is H100, FP8 remains the right target.

What is the difference between quantization-aware distillation and quantization-aware training?

QAT inserts fake-quantization operators during fine-tuning so the model adapts to the quantization grid using the standard task loss. QAD adds a full-precision teacher and trains the quantized student to match the teacher’s output distribution as well. The teacher signal is richer than hard labels, so QAD typically recovers more accuracy at the same training budget — which makes it the technique of choice at 4 bits.

How much accuracy does NVFP4 actually cost versus FP8?

With plain post-training quantization, expect a small but real drop that shows up first on multi-step reasoning, math, and long-context tasks. With the QAD-trained Nemotron 3.5 Lightning checkpoint, Nvidia reports near-parity with the BF16 teacher. Treat both as claims to verify: run your own domain evals, and weight agentic tool-calling accuracy heavily if that’s your workload.

Can I quantize my own model to NVFP4?

Yes. NVIDIA Model Optimizer (nvidia-modelopt) exposes NVFP4_DEFAULT_CFG for post-training quantization with a short calibration loop, plus QAT and distillation recipes when PTQ isn’t good enough. Start with PTQ — it takes minutes — and only escalate to QAD if your evals demand it, since distillation needs real training compute.

Why does the Mamba-Transformer hybrid architecture matter here?

Because quantization gains compound with architectural ones. NVFP4 shrinks the weights; the hybrid design shrinks the KV cache by replacing most attention layers with linear-time state-space blocks. Together they free far more memory for concurrency than either does alone, which is why Lightning hits low latency at high batch sizes rather than only in single-stream benchmarks.

Is SageMaker JumpStart the cheapest way to run this?

No — it’s the fastest. JumpStart on GB200-class instances carries a managed-service premium over raw EC2 or a self-hosted cluster. Use it to validate quality and measure real throughput on your traffic in an afternoon, then decide whether to move to self-managed vLLM or TensorRT-LLM once you know the workload is worth committing capacity to.