Contract Testing LLM Tool Calls 2026: Pydantic AI & Braintrust

Your agent calls get_refund(order_id, amount_cents) a thousand times a day and the eval dashboard says 98% tool-call success. Then finance flags a batch of refunds issued in dollars instead of cents, because a model version bump quietly started coercing a float where your schema expected an int — and your test suite only ever asserted that the right function name got picked. That’s the 2026 gap: teams measure whether the model called the tool, not whether the arguments it passed were correct, complete, and in-contract. Add parallel calls, nested tool chains, and providers shipping silent point-release changes to their function-calling behavior, and you get production incidents that no unit test, no golden transcript, and no vibes-based eval ever catches.

Written for backend and ML engineers shipping LLM agents that touch real systems — payments, tickets, databases, internal APIs. Assumes you’re fluent in Python, comfortable with type hints and pytest, and have already wired up function calling with at least one provider. It assumes familiarity with CI basics (GitHub Actions or equivalent) and JSON Schema. Out of scope: prompt engineering for better tool selection, RAG retrieval quality, fine-tuning, agent framework selection wars, and end-to-end UX evaluation. This is narrowly about testing the contract between your model and your functions.

Honest read: models are genuinely good at picking the right tool and filling obvious required fields, and they’re excellent at generating the adversarial fixture cases you’d never think to write by hand. They are unreliable at units, enum boundaries, optional-versus-null semantics, ID formats, and anything requiring restraint — like declining to call a tool at all. LLM-as-judge scoring of argument correctness drifts and should never gate a destructive-action test on its own. Human review is non-negotiable on three things: the schema definitions themselves (a wrong contract tested perfectly is still wrong), the pass/fail thresholds you wire into CI gates, and any test asserting behavior around irreversible operations. Automate the loop; own the contract.

What This Guide Covers

• A working mental model for why tool calls fail in production even when call-rate metrics look healthy — and which failure classes actually cost money
• How to adapt contract testing from microservices to LLM function calling, so provider changes surface in CI instead of in your incident channel
• Designing Pydantic AI argument schemas that constrain aggressively — units, formats, enums, and cross-field rules the model can’t shrug off
• Structuring validation-driven repair loops with Instructor and retries so recoverable mistakes self-heal without masking real regressions
• Generating adversarial argument fixtures at scale: near-miss values, boundary conditions, and the ambiguous prompts that expose brittle contracts
• Building deterministic replay suites in Braintrust so the same inputs produce comparable results across runs, branches, and model versions
• Using Promptfoo for fast local contract checks that run in seconds, before you burn CI minutes or provider credits
• Metrics that reflect reality — argument-level accuracy, field-wise precision, and null-handling correctness instead of a single call-rate number
• Detecting silent schema drift when a provider ships a point release, including what to snapshot and how to diff behavior version over version
• Testing parallel, nested, and multi-turn invocations where ordering, state, and partial failure make naive assertions useless
• Keeping suites fast and cheap: caching, sampling strategy, tiered test depth, and where cheaper models are safe substitutes
• Wiring the whole thing into CI with hard cost caps and regression gates that block a merge without becoming flaky-test theater
• The specific pitfalls that burn teams — over-permissive schemas, judge-only assertions, fixtures that leak into training-adjacent workflows, and thresholds nobody revisits
• Case studies from production agent deployments, plus where tool-call contract standards are heading and what to design for now

Instant online access the moment checkout completes — read it immediately, no waiting on a download email. One purchase, complete guide, no upsell and no tiered “pro” version held back.