Your fallback chain works in staging and then a Friday-night provider incident hits production: the primary 429s, your gateway retries into the same rate limit, streaming responses arrive with mangled tool-call deltas, and your prompt cache silently stops hitting because the shim stripped cache_control. Your bill for the week is 40% above forecast and you cannot attribute a dollar of it to a team, a route, or a retry storm. Meanwhile the decision you actually need to make — OpenRouter vs LiteLLM vs Portkey — is buried under vendor benchmarks that never publish their methodology, marketing pages that all claim “sub-millisecond overhead,” and Reddit threads from 2024 describing products that have since been rewritten twice.
This is written for backend and platform engineers who already ship LLM features to real users and are choosing or replacing a gateway layer. You should be comfortable reading HTTP traces, reasoning about p99 versus p50, and running a self-hosted service behind your own load balancer. It assumes working knowledge of at least one provider SDK and basic observability practice. Out of scope: prompt engineering, model quality evaluation, fine-tuning, agent frameworks, and vector database selection — this guide is about the plumbing between your application and the model, not the model itself.
An honest boundary: AI is excellent at generating the boilerplate config, translating a routing policy from one gateway’s schema to another’s, and drafting migration adapters. It is unreliable at anything requiring current, verified fact — pricing tables shift monthly, feature parity claims go stale, and models will confidently invent config keys that were deprecated or never existed. Every benchmark number here came from instrumented runs, not model output. Human review is non-negotiable on three things: spend caps and virtual key scoping before they touch production, data residency and compliance claims before you sign anything, and the tool-call and thinking-block fidelity of any OpenAI-compatible shim you adopt — that one you must verify against your own payloads, because a passing smoke test proves nothing about a nested tool schema.
What This Guide Covers
- A clear-eyed decision framework for choosing between six gateways — including when the honest answer is “no gateway yet”
- Side-by-side comparison of OpenRouter, LiteLLM, Portkey, Cloudflare AI Gateway, Helicone, and Kong across the dimensions that actually differ
- Working fallback chains built in each major gateway, so you can feel the ergonomics before you commit a quarter to one
- Retry and failover semantics under load — where naive retry configs amplify an incident instead of absorbing it
- Weighted, latency-based, and conditional routing strategies, with the failure mode each one introduces
- Cache economics done properly: exact-match versus semantic, and the hit-rate math that tells you whether caching pays at your volume
- Prompt-cache pass-through testing — how to detect a gateway that quietly breaks provider-side caching and what it costs you
- A reproducible benchmark methodology plus measured results for streaming overhead and p50/p99 latency across all six
- BYOK versus marked-up credit pricing modeled at realistic volumes, with the break-even points made explicit
- Cost attribution patterns that let you report margin per customer, per feature, and per team instead of one opaque invoice
- Governance mechanics: virtual keys, spend caps, and per-team rate limits that hold up when a team ships a runaway loop
- Self-host versus SaaS weighed against data residency, SOC 2 posture, and how deep the observability actually goes
- The compatibility-shim pitfalls that corrupt tool calls and thinking blocks, and the test payloads that expose them early
- Case studies, a weighted scoring rubric you can adapt to your constraints, and migration playbooks for moving between gateways without a rewrite
Delivered as instant online access the moment checkout completes — no waiting, no shipping, no upsell sequence, no follow-on course. You buy the guide, you get the guide.
Reviews
There are no reviews yet.