Contract Testing LLM Tool Calls 2026: Pydantic AI & Braintrust
LLM tool call testing catches what eval dashboards miss: wrong arguments, bad timezones, malformed filters. Contract-test your agents with Pydantic AI and...
Chapter 1: Why Tool Calls Break in Production: The 2026 Reliability Gap Your agent works. You demoed it, the tool fired, the customer smiled, and you shipped. Three weeks later support tickets are stacking up: refunds issued for the wrong order, calendar invites landing in the wrong timezone, a search endpoint hammered with malformed filter strings that return zero results and burn tokens on retries. You check your eval dashboard. Tool selection accuracy: 97%. Everything is green. This is the 2026 reliability gap, and it is the single most expensive blind spot in production LLM engineering. The model picked the...