Async LLM Batch Jobs 2026: Modal & Ray Data Hands-On

You’ve got 5 million rows sitting in a Parquet file and a synchronous inference script that would take nine days to chew through them. So you rewrite it with asyncio, crank the concurrency, and hit 429s at row 40,000. You switch to a rented H100, watch nvidia-smi report 30% utilization, and realize you’re paying $3/hour for a GPU that’s mostly waiting on Python. Meanwhile the job dies at hour six from a spot preemption with no checkpoint, your provider’s batch endpoint returned a JSONL file where 3% of the lines are malformed and you can’t tell which requests they map to, and finance is asking why the “cheap” offline pipeline cost more than the real-time API it was supposed to replace.

This is for developers who already ship production Python and have called an LLM API — you know what a token is, you’ve used a dataframe, you can read a stack trace without flinching. You should be comfortable with the shell, environment variables, and reading library docs. This is not an ML theory guide: no training, no fine-tuning, no model architecture. It’s not a prompt engineering book either. And it assumes you want throughput on jobs measured in millions of rows, not a faster chatbot.

Honest framing: LLMs are genuinely excellent at the bulk classification, extraction, summarization, and enrichment work that batch pipelines exist to do — at volume, at a cost per row that makes previously impossible projects viable. They are bad at telling you when they’ve quietly degraded. A batch job that returns 5 million well-formed, confidently wrong outputs looks identical to a successful one on your dashboard. Schema validation catches malformed JSON; it does not catch semantic drift, silent truncation on long inputs, or a system prompt change that shifted your label distribution. Human review of a stratified sample is non-negotiable before any batch output touches a downstream system, and cost estimates should always be validated against a real pilot run rather than a spreadsheet.

What This Guide Covers

• How to decide, with a clear framework, whether a workload belongs on a batch API, rented GPUs, or serverless — before you commit engineering time to the wrong one
• The throughput-versus-latency economics of large token volumes, so you can estimate a job’s real cost per million tokens instead of guessing
• Working through both major provider batch APIs end to end, including the operational gotchas that only surface at scale
• Job design patterns that make a multi-hour run survivable: sharding strategies, checkpointing, and idempotency so a restart doesn’t re-bill you for finished work
• Running burst inference on serverless GPUs — spinning up capacity for a job and paying nothing when it’s idle
• Streaming execution over datasets far larger than available RAM, so dataset size stops being the constraint on what you can process
• The specific throughput knobs that separate a 30%-utilized GPU from a saturated one, and how to reason about which to turn first
• Two techniques that reclaim large speedups for free when your requests share a common system prompt — widely available, widely unused
• Keeping millions of outputs reliably parseable, and what to do about the fraction that still won’t validate
• Failure engineering as a first-class concern: tiered retries, partial-batch error handling, and surviving spot preemption without losing hours of work
• Building the embedding half of the pipeline, which is usually where the row counts and the surprises are biggest
• A cost worksheet that compares batch-API discounts against rented H100 and L40S hours on the same workload, so the tradeoff becomes arithmetic rather than opinion
• Observability that actually tells you something: throughput dashboards, cost-per-row tracking, and forensics for diagnosing a job after it’s already failed
• Where offline inference tooling is heading, and how to make architectural choices now that won’t need unwinding in a year

Instant online access the moment checkout completes — the full guide, available immediately, no waiting on an email. One purchase, no upsells, no subscription, no add-on tiers.