Async LLM Batch Jobs 2026: Modal & Ray Data Hands-On
Learn how to run LLM batch inference at scale in 2026 with Modal and Ray Data — async offline jobs that cut token costs and beat real-time APIs.
Chapter 1: The Offline Inference Shift: Why Batch Beats Real-Time in 2026 For most of the LLM era, "calling a model" meant one thing: an HTTPS request, a token stream, a spinner in a UI. That pattern was never wrong, but it quietly became the default for workloads it was never designed to serve. Somewhere around 2024, teams started noticing that the majority of their token spend had nothing to do with a user waiting on a response. It was enrichment jobs, document classification, synthetic data generation, evaluation harnesses, embedding refreshes, and content pipelines — work that runs on a...