Response and resolution speed can make the difference between a smooth customer experience and a frustrating one.
With AI customer service agents customers no longer have to wait for a human, but there's still a wide variance of response times and other delays and impediments that can slow down responses and frustrate customers.
Where chatbot delays come from
We split this into two big categories: speediness of responses and UX issues resulting in a poor interface.
Some delays occur before the AI ever receives a message. These are largely controlled by the design and implementation of the chat interface. Some of the major mitigations that can prevent a poor experience are:
- Remove unnecessary pre-chat forms. Do you actually need the customer's name and email before showing the chat interface? Information can often be collected during the conversation, only when it's needed to resolve the request.
- Make the chat interface immediately usable. Customers should not click the launcher only to encounter a blank loading panel.
What about the actual chat response times
Once the customer has gotten into your chat interface and submitted a question, the system has an opportunity to demonstrate that automation can provide a faster and more effective experience, not merely act as a deflection layer.
For a straightforward conversational response, a major component is model inference latency: the time required for the model and its hosting provider to begin and complete the response.
LLM completion times can vary based on a number of factors: the size of the model being used, the underlying compute, the size of the input context being passed in, current load that the provider is experiencing, and more. It's also worth noting that often, frontier models (like those provided by OpenAI and Anthropic) don't tend to be competitive from a speed perspective. Open source models on third party inference platforms typically win here.
In many cases open source models are just as if not more effective in terms of performance and the ability to resolve a customer's inquiry, but they can be considerably faster. Choosing the right model for the task is important, and if speed is a big consideration then an open source model powered by a wafer-scale compute provider is often the answer.
Comparing open source provider performance
Once you've decided to use an open-source model, you're confronted with tens or even hundreds of different provider options. You can't necessarily go by their claims because each website will tell you they offer the fastest inference, so you'll have to test and benchmark them over time. We've benchmarked some of the top providers using the same model and workload to demonstrate which is most performant.
It turns out, the variance is quite large. The response time difference between the fastest and slowest provider, for the same model, can be over 10x. That's a difference of waiting half a second for a response to your question, or waiting over five seconds.
In our testing one provider, Cerebras, consistently beat out others on generation speed and reliability.
We benchmarked some of the leading inference providers (Baseten, Cerebras, Cloudflare, Fireworks AI, and Together AI) using the same input and model (gpt-oss-120b) over a period of 24 hours, triggering a number of requests per provider and averaging the complete response time. Some providers are missing data for periods where their API did not return any successful responses.
What about reliability?
While these providers are fast, in our testing we observed considerable variation in reliability. All of the providers we tested returned occasional unexpected errors or out of capacity errors. Without appropriate handling, these failures leave the customer without a response.
Handling these errors is a matter of having the right fallback mechanisms in place: knowing when to retry a request on the same provider or switch to a different model or inference provider.
Other important technical considerations
While inference is likely going to make the biggest difference in terms of response speed, internal architecture is just as important.
- How many database lookups are happening each time a user sends a chat?
- If external services are involved, is caching used effectively to reduce delays?
- How effectively is context managed?
Putting it all together
A fast AI support experience requires more than selecting a fast model. It depends on the entire system: the chat interface, prompt and context design, retrieval, tool orchestration, inference infrastructure, retries, and fallbacks.
Valiopt optimizes these components as a unified system, helping ecommerce brands deploy AI support agents that respond quickly and resolve customer requests end to end. We do the benchmarking, model selection, and continuous improvement for you while you focus on running your business. Get in touch to see how we can help.