Small language models are entering mainstream corporate use, but executives running them in production describe expenses and complexity that don't surface until after deployment, according to an InformationWeek report published this week. While the models deliver efficiency gains, the report finds that organizations face a more complicated reality than vendors suggest, with costs showing up in data stewardship, monitoring, and troubleshooting across multiple specialized systems.
Gartner projected in July that spending on domain-specific and specialized AI models will jump 210% in 2026 alone, outpacing foundation model growth by more than double. Last year, the research firm predicted companies will operate specialized, task-focused models at roughly three times the volume of general-purpose large language models by 2027. Enterprise adoption has shifted from pilot programs toward mainstream architecture, with growth picking up speed, according to technology leaders interviewed for the report.
Ameet Ubhayaker, director of enterprise data analytics and AI at Protiviti, told the publication he isn't observing wholesale replacement of large models, but rather an emerging tiered stack: small models handle routing, classification, and structured output, while frontier models are reserved for ambiguous, multistep work. Jason James, previously CIO at Aptos, said his team ran three or four specialized models as prototypes on local hardware rather than in the cloud to measure whether on-premises costs and effort beat what they pay frontier models in token costs. "It is really a math exercise," James said. "What is the cost to implement? What is the cost to run? And what is the volume of use? If these are all significantly less to run locally, only then should it be considered."
The report highlights trouble spots that surface after go-live. Eric Pace, who recently left an enterprise IT leadership role and now advises companies on AI deployment, described a small, fine-tuned Llama model his team deployed to curate marketing messaging that was trained on historical data labeled good or bad, with testing that looked strong and a fast, inexpensive path to production. Then the data that appeared reliable in testing didn't hold up at production scale, and the team eventually pulled the model out. Organizations should optimize for cost per successful task, not simply cost per token, Ubhayaker said, because a smaller model may appear cheaper, but retries, human review, and escalation to larger models can quickly erase those savings. Running several specialized models introduces troubleshooting complexity, because when a single workflow passes through more than one, tracing a failure means working through multiple model owners to find where the problem originated. Every specialized model needs its own evaluation and monitoring to catch behavioral drift, the report notes, and a fleet of them adds complexity around routing, upgrades, and lineage, with base model updates and downstream fine-tuning requiring retraining and re-benchmarking.
The value of small language models is conditional, based on the task, the data, and who owns what happens to the model after it ships, according to the report. Narrow, repeatable, verifiable tasks such as classification, routing, or structured extraction, in environments with real data-boundary or compliance requirements, are where small models shine, but this doesn't mean they're replacing frontier models—the destination is a layered architecture, not wholesale migration, Ubhayaker said. Whether a task belongs on a small or large language model should be a per-use case call, with deciding factors including whether the task is bounded and verifiable, whether the organization has the engineering capacity to maintain the model, and whether a smaller model can actually achieve the required accuracy, not just which architecture is cheaper on paper. Model cost isn't total cost, and it's essential to budget for governance, management overhead, and training before tallying up any savings, James said. For enterprises weighing the shift, the lesson from production deployments is clear: small language models demand capabilities in data stewardship, monitoring infrastructure, and cross-team coordination that may dwarf the token savings they promise. The business case hinges less on choosing small versus large than on matching model architecture to organizational readiness and task boundaries—a discipline many companies are still building as they scale beyond pilots.