The Load Balancer That Learned to Speak AI
BFE · release
BFE, the open-source load balancer born at Baidu (the name is short for Baidu Front End), just shipped v1.8.7 — the latest in a streak of releases aimed squarely at AI traffic. If your mental model of BFE is “yet another reverse proxy,” this release wants a word. It now speaks the Gemini protocol, meters video generation and image tokens, and routes inference traffic (requests asking an AI model to generate an answer) with circuit breakers and health-checked failover.
What is BFE? In plain English: the calm, extremely fast customs officer that stands in front of your servers. It inspects every incoming request and decides which backend gets to handle it — routing, rewriting, load-balancing and logging at layer 7, the layer that understands actual HTTP conversations instead of just shoveling bytes. It was born inside Baidu, where it has balanced production load for years, and it lives as a CNCF sandbox project: early-stage, publicly governed, free to adopt.
Why care about v1.8.7 specifically? Because of what lands in it: Gemini protocol support with a three-stage fallback chain for usage extraction, billing for video generation and image input tokens (tokens being the chunks of text and pixels AI providers bill on), length-tier pricing that mirrors how frontier model vendors actually charge, and a production-hardened EPP integration (Endpoint Picker — explained below) with health checks, failover hysteresis and a circuit breaker, for inference-aware load balancing. If you route LLM (large language model) traffic, this release hands you new metering and failover tools you can put to work immediately.
Think of BFE as the customs officer for your API traffic: the one who inspects every shipment, checks papers, and waves it to the right gate. v1.8.7 teaches that officer to speak Gemini, appraise video, and never route to a dead inference pool.
What’s New in v1.8.7
Gemini Protocol Support
BFE now speaks the Gemini protocol — Google’s dialect for its Gemini model APIs — natively. Every AI provider wraps usage reporting differently, so BFE extracts usage through a dedicated adapter per protocol; when the first adapter comes up empty, a three-stage cross-protocol usage fallback chain re-reads the response in a different dialect before concluding there is no usage data. The practical result: metering keeps working even when the response does not look like what the route expected — a very common situation when one OpenAI-style client gets answered by an Anthropic-style backend. Merged in PR #1367.
Video, Images, and Length-Tier Pricing
The billing engine grew up. v1.8.7 adds support for OpenAI’s responses and video_generation request modes, bills image input tokens, and prices generated video with output_cost_per_video. It also adds length-tier pricing: frontier vendors charge different rates above certain input-token thresholds, and BFE can now mirror those tiers exactly instead of smearing them into one average price. The supported pricing keys, expanded from the release shorthand (input/output_ prefixes cover both directions):
input_cost_per_token_above_200k_tokens
output_cost_per_token_above_200k_tokens
input_cost_per_token_above_256k_tokens
output_cost_per_token_above_256k_tokens
input_cost_per_token_above_272k_tokens
output_cost_per_token_above_272k_tokens
input_cost_per_token_above_512k_tokens
output_cost_per_token_above_512k_tokens
cache_creation_input_token_cost_1hThat last key bills 1h-TTL cache writes — providers cache repeated prompts for up to an hour (TTL = time-to-live) and bill that write at a different rate from the standard short-lived cache. Three new access-log fields — ai_cache_write_1h_tokens, ai_image_input_tokens and ai_video_count — make all of this visible per request. And because a single token can cost a millionth of a cent, model prices are now kept as float64 and converted per billing item, so tiny per-token rates survive the arithmetic. Merged in PR #1366 and PR #1350, with the pricing-precision change in PR #1362.
Production-Grade EPP Integration
The biggest operational upgrade in this release. BFE’s ext-proc integration — external processing, the pattern where a proxy outsources routing decisions to a helper service over gRPC — with an EPP (Endpoint Picker, from the Kubernetes Gateway API inference extension: the component that decides which inference pod serves a given request) is now production-hardened. That means inference-pool metadata, primary/backup EPP addresses with gRPC health checks, failover hysteresis so a flapping endpoint does not cause traffic storms, and a circuit breaker that stops routing to an endpoint that keeps failing until it recovers.
There is also TLS and timeout configuration for the EPP connection, plus a new /monitor/epp_metrics endpoint that exposes EPP counters in Prometheus format. If your EPP dies at 3am, BFE now notices and acts instead of blindly forwarding into the void. Merged in PR #1360, with a companion fix in PR #1365 that now sends the request’s path and method (the :path/:method pseudo-headers) so the EPP can correctly parse chat-style requests.
Bug Fixes and Security-Adjacent Improvements
API keys stop leaking into access logs. A unified credential masking gate recursively scans the entire request log via protobuf reflection: authenticated requests get the raw key replaced with the internal key_id, brute-forged keys get their fields emptied, and the raw Authorization header is no longer logged at all. Operators take note: anything downstream that parses raw keys out of access logs will need to switch to key_id — that breakage is the point. Merged in PR #1358.
Billing gets accurate. A duplicate-deduction guard, a fix for truncated cache-read token counts (tokens served from a provider’s prompt cache), and a skip for the /count_tokens endpoint landed in PR #1346. Client-aborted requests no longer get billed for a full-request estimate (PR #1353), and cross-protocol non-streaming responses now bill their complete usage (PR #1365).
Fail-closed quota handling. A missing Redis balance is now treated as exhausted instead of returning a 500 — quota enforcement fails in the safe direction. Zero-total-token quota plans also load correctly. See PR #1354.
And the rest, AI side:
- fresh input token billing for Anthropic (maker of Claude) high cache hits (PR #1350)
- AI protocol handling converged into the bfe_model_protocol adapter layer (PR #1351)
- recognition of Anthropic message_delta as final usage in the OpenAI adapter (PR #1372)
And the rest, plain-ops side:
- CA/CRL (certificate authority / certificate revocation list) base directories now relocate correctly on TLS config reload (PR #1356)
- a mod_header panic on internally-constructed responses is fixed (PR #1360)
- path joining is Windows-safe (PR #1360)
- timezone data is baked into the official Docker image (PR #1347)
Closing
BFE v1.8.7 is what happens when a battle-tested edge proxy takes AI traffic seriously — the customs officer just learned to appraise art. The release notes list no breaking changes, so for most deployments this is a routine upgrade: pull the new image or binary from the release page and reload — just plan around the access-log key masking above if anything downstream parses raw keys. If you meter mixed Gemini, Claude, or OpenAI traffic at the edge, this is the release worth upgrading for.