For most of the last decade, an API gateway was a pretty simple idea: take an HTTP request, authenticate it, maybe rate limit it, send it to the right backend. Boring, important, mostly invisible. Apigee was good at this. Then generative AI happened, and suddenly the API gateway has a much more interesting job.
Google Cloud has been quietly turning Apigee into something more like a control plane for AI, and the scope of what it now handles is worth understanding.
The Problem with Calling LLMs Like They Are Regular APIs
A large language model (LLM) is not like a database query. A database query costs roughly the same every time. An LLM call can cost a penny or it can cost a dollar, depending entirely on how many tokens are in the prompt and the response. Tokens are the unit of work for LLMs: roughly 4 characters each, and you pay for every one, input and output combined.
This creates a governance problem that standard API management tools were not built for. Limiting requests per minute does nothing to stop a single verbose user from sending a 10,000-token prompt every minute. Standard auth controls unauthorized access, but not a prompt injection attack where a user tricks the model into ignoring your instructions. And logging the request tells you an API call happened, not what was actually sent or returned.
Apigee now handles all of this natively, in the proxy layer, before the request ever reaches the model.
What It Actually Does Now
The centerpiece of the AI management story is the LLM gateway capability. Apigee sits in front of any model endpoint, including Vertex AI, Gemini, OpenAI, Anthropic, and self-hosted models, and applies policies to every call. Token quotas enforce per-user or per-tenant spend limits on both input and output tokens. Semantic caching uses Vertex AI embeddings to detect when two different prompts are asking essentially the same question, and returns the cached answer instead of making a redundant model call. Model Armor, which runs natively in the proxy layer, validates prompts and filters outputs for prompt injection, jailbreak attempts, and sensitive data exposure.
The semantic caching piece is worth dwelling on for a second. Traditional caching is exact-match: the same string returns the cached result. Semantic caching is different. If one user asks “what is the refund policy” and another asks “how do I get my money back,” those are different strings but the same question. Semantic caching catches that, returns the same answer, and saves the model call. For products with high query overlap, this can cut LLM costs significantly without touching a line of application code.
The Agentic API Problem
The more interesting frontier is what happens when AI agents start calling your APIs. Agents are not passive. They reason, plan, and take actions: calling external systems, retrieving data, triggering workflows. The Model Context Protocol (MCP) is the emerging standard for how agents describe and call external tools, and Apigee now manages MCP servers natively.
What this means in practice: an ISV with an existing REST API catalog can expose those APIs as MCP tools through Apigee, without rewriting anything. The agent ecosystem can discover and call them. Apigee handles authentication, rate limiting, and observability on every tool call, the same way it handles every other API call. The ISV’s existing API surface becomes an agentic integration layer by default.
This matters because agent-driven traffic behaves differently from human-driven traffic. An agent making autonomous decisions can generate bursts of rapid, sequential API calls that look nothing like a human user session. Without policy enforcement at the infrastructure layer, a misbehaving agent can hammer a backend system in ways that are hard to detect and expensive to recover from.
Why This Shows Up in Enterprise Deals
For software vendors selling into regulated industries, the compliance question around AI features is real. When a healthcare company or a bank asks “how do you ensure sensitive data does not get sent to the model,” the answer “we train our developers to be careful” does not close the deal. The answer “we enforce prompt sanitization and output filtering at the infrastructure layer, with audit logs on every call” is a different conversation.
Apigee makes that second answer possible without requiring the application team to build it themselves. The governance layer is in the proxy. The audit trail is in Cloud Logging. The token budget enforcement is in the policy. None of it requires application code changes.
A few things worth thinking about: If your AI features scale to 10x current usage tomorrow, do you have visibility into which customers are consuming what? If an agent in your product starts looping and making thousands of API calls, what stops it? And when your next enterprise prospect asks how you govern your AI infrastructure, what is your answer?
Want to go deeper? Here are a few links worth your time
- Apigee AI Gateway overview, The full capability set: token quotas, semantic caching, Model Armor, multi-model routing, and observability.
- MCP support for Apigee, How Apigee manages remote MCP servers and surfaces existing APIs as agent tools.
- Using Apigee for AI, Engineering deep dive on multi-model routing, RAG integration, and agentic API patterns.