5,000-RPM Vision-LLM Serving

Highlights

1. Dual-backend routing with automatic failover: Gemini ⇄ AWS Bedrock Qwen3-VL, threshold-based traffic splitting on a 10-second window.
2. Quota-aware circuit breaker: on Gemini 429s, retry on Bedrock; after N quota errors disable Gemini until the next UTC day.
3. Redis-backed distributed inflight queues and token-bucket rate limiting, with graceful in-memory fallback if Redis dies.
4. Per-call token and cost metering accumulated in Redis; a custom async load tester that empirically found the real limits.