Google has open sourced a new tool that eliminates Python from the machine learning serving pipeline, delivering two to five times the throughput of its best optimized Python server while using just a quarter of the memory, according to a report published Tuesday in The Register. The tool, called XSE (XLA Serving Engine), already processes more than six million queries every second across Google services including YouTube Shorts, Google Ads, and Search. The release marks the latest step in Google's strategy of training models in Python but deploying them in C++ to avoid interpreter overhead in production environments.
XSE targets a different use case than the large language model inference systems that dominate current attention. The engine is designed for smaller, task-focused models that handle ranking, recommendations, embedding, and vision tasks—workloads characterized by brief requests, extremely high query volumes, and strict latency requirements. These models don't engage in extended conversations; instead, they rank search results, suggest videos, and execute other split-second decisions that users rarely notice. In this workflow, data scientists build their model code using Python and Google's JAX framework, then an export utility packages the model's calculations in StableHLO—a portable intermediate format—alongside trained weights in what Google calls an "XLA SavedModel." The C++ serving runtime then uses XLA to compile those calculations into native code for the target processor, with no Python interpreter involved.
Nathan Beach, Google's director of product management, wrote in a LinkedIn post Tuesday that "LLM serving gets most of the attention right now, but a significant amount of production inference is still ranking, recommendation, embedding, and vision models." These models, he noted, "tend to be short requests with very high [queries-per-second] and tight latency budgets." The report also states that Google plans integration with TorchTPU, allowing PyTorch users to export models for C++ serving on TPU accelerators.
The approach reflects a broader industry pattern in which Python's sluggishness has become a persistent problem for machine learning deployment. While XLA SavedModels can run on GPUs and CPUs—with Google partnering with both Arm and Intel to develop the XLA ecosystem—the company clearly anticipates adoption on its proprietary TPU accelerators. Other companies have pursued similar strategies: PyTorch offers LibTorch, a native C++ runtime, while Microsoft and collaborators provide the ONNX Runtime with support for an open graph format. Even Python's own reference implementation leans heavily on C, with Rust now gaining ground.
The performance gap between interpreted Python and compiled native code becomes critical at Google's scale, where a Python interpreter evaluating each request would create crippling bottlenecks for services handling millions of queries per second. Google worked with component partners including Arm and Intel to build out the XLA ecosystem, making the technology portable across different processor architectures while steering users toward its own TPU hardware. The industry-backed OpenXLA Project, which provides tools for compiling machine learning computations into native code for various processors, forms the foundation of this setup. For organizations running high-volume inference workloads on smaller models, the choice between developer convenience and production efficiency may no longer require compromise. The tooling ecosystem now supports writing in familiar Python frameworks while deploying with the speed and resource efficiency that only compiled code can deliver.