Respan released an open-source AI gateway built in Rust on Tuesday, replacing a system that had grown to more than 100,000 lines of wrapper code around LiteLLM after two years of development. The company spent those years building atop LiteLLM before deciding to scrap the entire architecture in favor of a compact core with plugins that manage model providers, routing, MCP tools, realtime voice capabilities, and security features. The shift came after maintenance problems made upgrading the underlying framework too risky to continue.
The tipping point arrived in January 2026, when a routine LiteLLM upgrade triggered sporadic connection failures. Respan reversed the update but couldn't identify the root cause—"We still haven't figured out the cause," the company wrote in its Hacker News launch post. Since that incident, its engineers have abandoned direct LiteLLM upgrades entirely, instead cherry-picking upstream fixes individually. To sidestep another maintenance crisis, the company took a page from Nginx's playbook, shrinking the gateway's core and relocating features like routing, model providers, and security into plugins. Some provider integrations turned out surprisingly compact—Respan's Mistral provider requires only 22 lines of code, while quirks such as Groq reporting token usage in a non-standard field can be isolated and handled independently.
When an HTTP or WebSocket request arrives, the gateway passes it through plugins sequentially. Each plugin can select which model processes the request, modify the response, initiate extra calls, or halt the request before it ever reaches a provider. If both the client and provider speak the same format, the gateway preserves the original request and its provider-specific fields without alteration. The gateway can also bridge protocols—an application designed for OpenAI's Realtime API can connect to Gemini Live without client-side changes, according to Respan, though some functionality may not transfer. Custom plugins get compiled directly into the gateway, meaning developers who add one must maintain the code and rebuild the binary each time. Because plugins operate inside the gateway process, they have access to model requests and other sensitive information. Rust prevents memory-safety errors, but it can't guard against malicious dependencies or flawed application logic—a concern that surfaced during the Hacker News discussion following the launch.
Respan isn't the only company convinced that Rust belongs at the heart of an AI gateway. LiteLLM is porting its own gateway to Rust, though where Respan rebuilt from scratch, LiteLLM is migrating one route at a time with the Rust core disabled by default and any unsupported request falling back to the existing Python path. The company outlined the plan in June and aims for a pure Rust server handling every hot path, authentication, and rate limiting—a target the current beta hasn't yet reached. LiteLLM has also published benchmarks claiming its Rust gateway carries roughly seven times less overhead and consumes nine times less memory than Bifrost, though those figures come from the company's own tests using a benchmark harness it developed. Respan's gateway remains pre-alpha, but developers can download a prebuilt binary from GitHub and run it independently of the company's paid platform—custom plugins still require compiling a separate build. For teams that have hit the limits of wrapper code and incremental patches, a small core with inspectable plugins offers a way to regain control without betting the whole stack on someone else's upgrade cycle.