Third Moment Research 2026: Running 10,000 AI Agents for 36 Hours

Most agent demos last ninety seconds and run one process. Third Moment Research has published something much closer to production: 10,000 AI agents running in parallel for more than 36 hours straight. The hard problems in running thousands of AI agents in parallel are no longer about prompts. They are about orchestration, state, cost and recovery. If your team plans to move long-running AI agents from a notebook to a cluster in 2026, this is one of the first public case studies at this scale worth learning from.

What’s actually new: running thousands of AI agents in parallel, for days

Third Moment Research has shared how it keeps a fleet of 10,000 agents working at the same time over a run of more than 36 hours. Plenty of teams have run big batches of short LLM calls. Plenty have shown a single agent working on its own for hours. Few have done both at once, with a huge number of agents over a long horizon, and real-world write-ups at this scale are rare.

The useful part is the operations, not the headline number. At this scale, failure is normal. Across 10,000 agents and 36 hours, you will hit rate limits, provider timeouts, malformed tool outputs, out-of-memory containers and agents that loop on the same step. Third Moment’s account centers on four disciplines: orchestration (deciding what runs, where and when), checkpointing (saving agent state so work survives a crash), cost control (stopping runaway spend before it compounds) and failure recovery (resuming or retrying without redoing finished work).

One caveat: Third Moment has not open-sourced a turnkey framework, and the public details stay at the architecture level. The playbook below is our own. It follows the same principles, uses tooling you can deploy today, and does not claim to copy Third Moment’s internal stack.

Why it matters

• Long-horizon agents are now an infrastructure problem. Once a run outlasts a deploy cycle, a token refresh or a provider outage, your durability layer matters more than your agent framework.
• Checkpointing is required. If an agent crashes at hour 30 with no saved state, you lose 30 hours of compute and tokens. Agent checkpointing and recovery is what makes long runs economically possible.
• Cost scales faster than you think. Ten thousand agents each burning a modest number of tokens per minute add up to a very large bill over 36 hours. Scaling autonomous agents without per-agent budgets means scaling your losses.
• Rate limits become the real scheduler. At this concurrency, provider quotas decide your throughput, not your CPU count. Multi-agent orchestration has to be quota-aware.
• Observability has to be aggregate. You cannot read 10,000 transcripts. You need fleet-level metrics such as step rate, error class distribution, cost per completed task and stuck-agent counts.
• It sets a bar for AI agent infrastructure in 2026. Buyers and investors will start asking agent vendors the same questions: what happens at hour 20, and what does a failure cost?

How to use it today: a playbook for long-running AI agents at scale

You don’t need 10,000 agents to benefit. The same patterns hold at 50 or 500. Here is a practical build order.

1. Treat every agent step as a durable, idempotent unit of work. Each step (one LLM call plus its tool calls) reads state, does work, then writes state atomically. A replayed step should produce the same result or detect that it already ran. Here is a minimal Python pattern using SQLite for local testing. Swap in Postgres for production:

   import json, sqlite3, hashlib
   
   db = sqlite3.connect("agents.db", isolation_level=None)
   db.execute("""CREATE TABLE IF NOT EXISTS checkpoints (
     agent_id TEXT, step INTEGER, state TEXT, step_hash TEXT,
     PRIMARY KEY (agent_id, step))""")
   
   def load_latest(agent_id):
       row = db.execute(
           "SELECT step, state FROM checkpoints WHERE agent_id=? ORDER BY step DESC LIMIT 1",
           (agent_id,)).fetchone()
       return (row[0], json.loads(row[1])) if row else (0, {"history": []})
   
   def save(agent_id, step, state):
       blob = json.dumps(state, sort_keys=True)
       h = hashlib.sha256(blob.encode()).hexdigest()
       db.execute("INSERT OR IGNORE INTO checkpoints VALUES (?,?,?,?)",
                  (agent_id, step, blob, h))
   
   def run_agent(agent_id, max_steps=500):
       step, state = load_latest(agent_id)
       while step < max_steps and not state.get("done"):
           state = agent_step(state)   # your LLM + tool logic
           step += 1
           save(agent_id, step, state)

2. Use a durable workflow engine for orchestration. Don’t hand-roll retry loops across thousands of processes. A workflow engine such as Temporal persists workflow history, so a crashed worker resumes from the last completed activity. Wrap each LLM call as an activity with explicit timeouts and retry policies:

   from datetime import timedelta
   from temporalio import workflow
   from temporalio.common import RetryPolicy
   
   @workflow.defn
   class AgentWorkflow:
       @workflow.run
       async def run(self, task: dict) -> dict:
           state = {"task": task, "history": [], "done": False}
           for _ in range(500):
               state = await workflow.execute_activity(
                   "agent_step", state,
                   start_to_close_timeout=timedelta(minutes=5),
                   retry_policy=RetryPolicy(
                       initial_interval=timedelta(seconds=2),
                       backoff_coefficient=2.0,
                       maximum_attempts=6,
                       non_retryable_error_types=["BudgetExceeded"]),
               )
               if state["done"]:
                   break
           return state

   Very long runs need continue_as_new every few hundred steps to keep workflow history bounded.

3. Put a global, quota-aware rate limiter in front of every model call. Ten thousand agents hitting a provider at once will trigger cascading 429 errors. Use a shared token bucket (Redis is fine) sized to your real quota, and add jitter to retries:

   -- Redis Lua token bucket: KEYS[1]=bucket, ARGV: capacity, refill_per_sec, now_ms, cost
   local b = redis.call("HMGET", KEYS[1], "tokens", "ts")
   local cap, rate, now, cost = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3]), tonumber(ARGV[4])
   local tokens = tonumber(b[1]) or cap
   local ts = tonumber(b[2]) or now
   tokens = math.min(cap, tokens + (now - ts) / 1000 * rate)
   if tokens < cost then
     redis.call("HMSET", KEYS[1], "tokens", tokens, "ts", now)
     return 0
   end
   redis.call("HMSET", KEYS[1], "tokens", tokens - cost, "ts", now)
   return 1

4. Enforce per-agent and fleet-wide budgets. Every agent carries a token and dollar budget in its state. When the budget runs out, the agent stops cleanly and saves its partial results. A fleet-level circuit breaker halts new work if spend per completed task drifts above a threshold:

   class BudgetExceeded(Exception): pass
   
   def charge(state, usage, price_in=0.0, price_out=0.0):
       cost = usage["input_tokens"] * price_in + usage["output_tokens"] * price_out
       state["spent"] = state.get("spent", 0.0) + cost
       if state["spent"] > state["budget"]:
           state["done"] = True
           state["stop_reason"] = "budget"
           raise BudgetExceeded(state["agent_id"])
       return state

5. Detect stuck and looping agents. Long-running agents often fail silently: they repeat the same tool call or make no new progress. Hash the last N actions and flag an agent if the hash repeats, or if a progress metric hasn’t moved in K steps. Then restart it from an earlier checkpoint with a perturbed prompt, or retire it.

   def is_looping(history, window=6):
       recent = [json.dumps(h["action"], sort_keys=True) for h in history[-window:]]
       return len(recent) == window and len(set(recent)) <= 2

6. Run workers as horizontally scaled, preemptible jobs. State lives in the checkpoint store, so workers are disposable. That lets you use spot or preemptible capacity safely. A Kubernetes Job example:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: agent-workers
   spec:
     parallelism: 200
     completions: 200
     backoffLimit: 1000
     template:
       spec:
         restartPolicy: OnFailure
         terminationGracePeriodSeconds: 60
         containers:
         - name: worker
           image: your-registry/agent-worker:latest
           env:
           - name: AGENTS_PER_WORKER
             value: "50"
           resources:
             requests: { cpu: "500m", memory: "1Gi" }
             limits:   { memory: "2Gi" }

   Each worker runs many agents with async I/O. Agents mostly wait on network calls, so one process can multiplex dozens of them.

7. Instrument at the fleet level. Emit a few metrics per step: agent_id, step, latency, tokens, error class and cost. Build four dashboards: active agents, steps per minute, error rate by class, and cost per completed task. Alert on trends, not individual failures.

How it compares

Third Moment’s disclosure is a case study, not a product. The useful comparison is between the orchestration approaches you could use to reproduce this kind of run:

Approach	Durability / checkpointing	Scale to thousands of agents	Best for	Main drawback
Durable workflow engine (e.g., Temporal)	Built in; replays from event history	Strong, with history size management	Long-horizon agents that must survive crashes	Learning curve; workflows must be deterministic
Distributed compute (e.g., Ray)	Manual; you persist state yourself	Strong for actor-based parallelism	Compute-heavy agents, local model inference	Recovery semantics are your problem
Queue + workers (Redis/SQS + containers)	Manual; checkpoint store plus idempotent steps	Strong and simple	Teams that want minimal moving parts	You build retries, timeouts and visibility
Agent frameworks with persistence (e.g., LangGraph checkpointers)	Built-in per-graph checkpoints	Moderate; needs an external scheduler	Complex agent logic with branching	Not a fleet scheduler on its own
Managed batch APIs from model providers	Provider-managed per request	Very high throughput, high latency	Single-shot tasks, not multi-step agents	No interactive tool loops

Our take: for most teams, the winning combination is an agent framework for the logic inside each agent, plus a durable workflow engine or a disciplined queue for the fleet. Ask one layer to do both jobs and it will fail somewhere around hour six.

What’s next

Expect more public numbers. Once one team publishes a 10,000-agent, 36-hour run, others will publish bigger ones, and the metrics will sharpen. Watch for cost per completed task, the share of agents that needed recovery, and how much work failures destroyed. Those numbers say more than raw agent counts and will become the real benchmarks for AI agent infrastructure in 2026.

Tooling should catch up too. Today, most teams bolt durability onto agent frameworks after the fact. We expect frameworks to ship first-class integrations with durable execution engines, standard checkpoint formats and fleet-level budget controls. Model providers face pressure as well: per-organization rate limits sized for chat traffic will look outdated once customers routinely run thousands of concurrent long-horizon sessions.

The open question is evaluation. Running 10,000 agents for 36 hours is impressive engineering. Proving the output was worth the compute is harder. The next useful disclosures, from Third Moment Research or anyone else, will tie fleet operations to output quality: how many agent-hours a useful result took, and how much of that was waste.

Frequently Asked Questions

What did Third Moment Research actually announce?

Third Moment Research described how it runs 10,000 AI agents in parallel for more than 36 hours continuously. It is one of the first public, real-world case studies of long-horizon, massively parallel agent work. The focus is orchestration, checkpointing, cost control and failure recovery, not a new model.

What is the biggest challenge in running thousands of AI agents in parallel?

Durability. At this scale, failures happen constantly: rate limits, timeouts, crashes and looping agents. Without agent checkpointing and recovery, every failure throws away completed work. Rate limits are a close second, because provider quotas, not hardware, usually cap your throughput.

How do you control the cost of long-running AI agents?

Give every agent a hard token or dollar budget stored in its state. Add a fleet-wide circuit breaker based on cost per completed task. Kill looping or stuck agents, and cache repeated context where your provider supports it. Track spend per outcome, not just total spend.

Do I need Kubernetes or Temporal to do this?

No. A queue, a pool of workers and a Postgres checkpoint table get you surprisingly far. Durable workflow engines and container orchestration pay off once runs span many hours, involve hundreds of agents or more, or must survive deploys and preemptions without manual cleanup.

How many agents can a single worker process handle?

Agents spend most of their time waiting on model and tool calls, so one async worker can often handle dozens at once. The real limits are memory for agent context, CPU-heavy tool execution and your shared rate limit. Benchmark your own workload before you size the fleet.

Is multi-agent orchestration at this scale only for research labs?

No. The patterns apply whenever a workload splits into many independent, multi-step tasks: large-scale data extraction, code migration across many repositories, test generation, market research or document review. The scale changes; the discipline stays the same.