Your AI API keys are leaking money, and your reverse proxy doesn’t care. BFE (Beyond Front End), the CNCF sandbox project born at Baidu to handle traffic that would melt most load balancers, just dropped v1.8.3 — and it’s not another round of config tweaks. This release turns BFE into something it was never marketed as: an AI gateway.
Think of BFE as a traffic cop standing between your users and your backend services. It inspects, routes, and shapes every request that passes through. Originally built to serve billions of requests per day across Baidu’s empire, it handles layer-7 load balancing with a plugin architecture that makes it stupidly extensible. Platform engineers use it when they outgrow NGINX or Envoy in raw throughput scenarios, or when they need fine-grained request routing that doesn’t require a PhD in proxy configuration.
So why should you care about v1.8.3? Because while everyone else is scrambling to build AI API gateways from scratch, BFE just quietly shipped one — with token-aware rate limiting, session affinity for AI workloads, and protobuf access logging — all wired up and ready to go.
The Headliners: What’s New in BFE v1.8.3
Token-Aware AI Rate Limiting
This is the big one. BFE now ships mod_ai_rate_limit, a brand-new module that doesn’t count requests like every other rate limiter on the planet — it counts tokens. If you’ve ever burned through an OpenAI budget because a misbehaving script fired 500 requests in a minute (each processing 4K tokens), you know why this matters.
The module tracks tokens per minute (TPM), requests per minute (RPM), and max concurrency per API key, with all state backed by Redis for distributed coordination across multiple BFE instances. You can even set different limits per model — so your team’s GPT-4 access can be tighter than their GPT-3.5 playground.
{
"product": "my-ai-gateway",
"AIApps": [{
"Name": "openai-proxy",
"RateLimit": {
"Default": {
"TPM": 100000,
"RPM": 60,
"MaxConcurrency": 10
},
"Models": {
"gpt-4": {
"TPM": 40000,
"RPM": 20,
"MaxConcurrency": 5
}
}
}
}]
}The module is toggled by the new EnableAiGateway flag, which acts as a master switch for all AI gateway capabilities. Flip it on, and BFE transforms from a generic reverse proxy into something that understands LLM traffic patterns natively.
Implementation: PR #1255
Session Sticky Routing for AI Workloads
AI inference is stateful, even when the APIs pretend it isn’t. If you’re running a multi-model setup where requests need to hit the same backend instance for caching, streaming consistency, or warm-model affinity, mod_session_sticky is your new best friend.
This module introduces session affinity with Redis-backed caching, so sticky routing survives across multiple BFE instances. The sticky identifier can be extracted from a JSON body field — not just headers or cookies — which is exactly what AI workloads need since most LLM APIs pass their context in the request body.
{
"SessionSticky": {
"StickyIDSource": {
"Type": "json_body",
"Path": "conversation_id"
},
"Redis": {
"Addr": "redis://127.0.0.1:6379",
"DBNum": 0,
"MaxAge": 3600,
"RenewWindow": 600
}
}
}The MaxAge setting controls how long a sticky mapping lives, while RenewWindow refreshes it near expiry — so long-lived conversations stay pinned without manual intervention. For teams running inference clusters behind a proxy, this removes a whole class of routing headaches.
Implementation: PR #1262 (core module) and PR #1263 (Redis cache and JSON body sticky IDs)
Enhanced AI Token Authentication
The existing mod_ai_token_auth module got a significant upgrade in this release. It now supports SetAiAuthInfo for dynamic authentication context injection, meaning AI-specific auth metadata can be set and propagated through the request pipeline alongside the standard token validation flow.
This matters because AI APIs often have auth patterns that don’t map cleanly to traditional bearer tokens — some use API keys in headers, others in query parameters, and a few need per-request signed payloads. The updated module handles this complexity without requiring you to duct-tape together multiple auth plugins.
Implementation: PR #1256 (token auth update) and PR #1265 (SetAiAuthInfo)
Protobuf Access Logging
Rounding out the release is mod_access_pb, a protobuf-based access logging module. If you’re pushing BFE into high-throughput environments (and let’s be honest, that’s the whole point), standard text-based access logs become a bottleneck. Protobuf encoding cuts log size dramatically and makes downstream parsing in your observability stack significantly faster.
Implementation: PR #1257
Bug Fixes and Improvements
The release also includes several targeted fixes that shore up the new AI gateway modules:
- AiError integration — rate limit violations now return proper AI-specific error responses instead of generic proxy errors (PR #1259)
- Redis error handling — new
is_redis_errordetection ensures Redis connectivity issues don’t silently break rate limiting (PR #1264) - MaxConcurrency validation — fixed a bug where missing MaxConcurrency config would cause a panic at runtime (PR #1267)
- SessionStickySecure test fix — corrected test coverage for secure session sticky behavior (PR #1266)
- String-only tag values — enforced that tag values in AI basic info are always strings, preventing type confusion (PR #1260)
BFE v1.8.3 just made a strong case that your next AI gateway doesn’t need to be a startup — it can be the load balancer you already know.
