Walk The Talk: Measuring Chatbot Success

Chatbots have played the long-game in the AI renaissance. These little guys never got the glamorous press launch of ChatGPT, but have been babies in the corners of websites for decades. During their pop-up career, they’ve learned deeply and have grown to be greeted with relief rather than resignation by customers. And sweet mother of Norman, they’ve managed it. According to a study by Drift, these days 90% of chatbot experiences are billed as either neutral or positive by customers. By 2024, the global chatbot market is expected to surpass $995 million, which is a compound annual growth rate of 27%.

But yeah yeah with the big impressive numbers. These sweeping stats still beg the question: how do we ensure that our own chatbots are measurably successful? In this blog, we’re whittling down a few key performance indicators (KPIs) that will help us know exactly that.

1) Compare with other channels

Sorry Roosevelt, but comparison is not the thief of joy when it comes to chatbots. We absolutely want to know whether our chatbot is resolving customer queries more quickly than our phone calls, emails or other channels might. A big green flag for a successful chatbot is the savings in our agents’ time, which equally speeds things up for our customers.

For instance, a garden centre’s bot tracking the delivery of tropical plants would free up agents from constantly locating palm trees on the M6. Let’s say that about 150 people per day call to ask where their plants are, and these inquiries take an agent about 3 minutes to iron out, a bot capable of conveying this information offers a potential saving of 3×150 minutes per day (aka 7.5 hours!).

But it’s not all about speed of interaction: if we discover that Tilly has spent 8 minutes asking our bot when her palm tree will arrive, before being told how much a cheese plant is, there are other metrics than speed to consider.

2) Identify the metric, but resist using just one

Here at Adexchange we can help identify which metrics to size in on, without getting target fixation on one (like time savings). Although we might want to nail first-contact resolution rates, we should always try to balance this with aspects like customer satisfaction. Why? Because this ensures that customer abandonment is positive and that we’re curating a more holistic bot experience for them, which avoids skewed outcomes.

Other rates to focus on might include the number of cases a chatbot can handle, how long customers spend with the bots, their re-use rates or their escalation rates. We rate all these rates, but we’ll also deal with them so that you don’t have to.

walk the talk adexchange blog 2

3) Mystery shop

It’s handy to refer to NPS and CSAT scores to get an overview for how our chatbots are doing, but a mystery shopper approach can offer more of a deep dive into their success. Getting an independent party like us to run through a myriad of questions and scenarios with your chatbot is a great way to identify its overall fitness.

A bit like sadist CrossFit coaches, we love a spicy bit of bot bleep testing. And just like you might do an arms or legs day in the gym, we can design mystery shops which isolate a bot’s weaker areas as well as assess its overall health. Let us loose on chatbot skill sets like navigating reservation booking systems, comprehending customer misspellings, or charismatically handling complaints.

4) Agent workload

Chatbots thrive best when they free up our agents from monotonous tasks releasing them to handle higher value and more fulfilling work. This ties in happily with the fact that 62% of customers, according to Tidio, are happy to use a bot rather than speak to an agent. But the value of time savings doesn’t just boil down to one number. A bot’s success can also be measured by the qualitative value of the tasks the agents are now free to manage. We appreciate this can feel a little grey and nuanced; but don’t worry it’s the kind of data we love to dig out and talk to you about.

5) Re-engagement

People tend not to pursue confusing situationships when it comes to chatbots (unless much bigger issues are at play or we’re watching the film Her). So, if our customers are repeatedly re-engaging with our bots, this is indicative of a healthy and fulfilling customer-bot relationship. By which token, measuring total interactions against new interactions with chatbots is a handy little statistic to size up their overall success.

walk the talk adexchange blog 3

6) Customer intent

To know how successful our chatbots are, we need to know why our customers are using them. Say we’re a beauty company. Do we know whether our regular Ruhama is using the chatbot to rebook her eyebrow threading appointment, or to find out about new treatments? Because if she just wants those brows on fleek, but our chatbot is relentlessly upselling her the new Indian head massage, our girl is left screaming into a void. And that void will quickly become an agent. And then everyone will need an Indian head massage which isn’t financially viable.

7) When’s it popping

Life is all about timing, and chatbots are no exception. Generally, none of us LOVE it when a chatbot starts popping off at us as soon as we arrive on a webpage. We like a moment to take our metaphorical coats off and settle in. Equally, we want the chatbot to be ready to fire when our customers do have questions. The art of chatbot positioning is a biggie.

8) What’s the actual problem?

Once we have identified our chatbot’s wobbly spots, it’s not about writing them off to a terminal diagnosis resembling “this is a bit rubbish”. Instead, it’s about understanding why a point is pinching so hard. If our customers are constantly escalating to agents, is it because our chatbot isn’t understanding them? Or is it because it’s giving them poorly worded answers? Or is it that the solution to the customer’s question is technical enough to require an agent anyway? It’s this kind of investigative thinking that we can get our teeth into, before offering you a road map out of the rut.

The clearest marker of chatbot success is that there isn’t one SINGLE one. Which is why it’s worth taking a more panoramic look at our chatbots. Currently, according to Drift, the number one reason people lean towards chatbots is because of their 24/7 availability. But with a bit of TLC, our chatbots could surpass that baseline of convenience into marked and measurable success when it comes to customer service. With well-oiled chatbots, our businesses can save time and money whilst enabling high-value human workforces. And that’s when statistics like a chatbot’s potential to save businesses 30% in costs (IBM) can really start to sing.

Now is a great time to do the chatbot work and reap the rewards. Or, if you’re into the time saving metric, let us do the work for you. It’s our field of expertise – so we’ll be efficient.

Picture of Julia Davies

Julia Davies

Account Manager
With years of Account Management experience behind her, Julia works closely with our clients to understand their problems and offer real solutions, improving customer journeys across every channel and touchpoint.
Picture of Julia Davies

Julia Davies

Account Manager
With years of Account Management experience behind her, Julia works closely with our clients to understand their problems and offer real solutions, improving customer journeys across every channel and touchpoint.

FREE RESOURCE DOWNLOAD

Our 'Power of Video' Top 10 Takeaways

We're committed to your privacy. Adexchange uses the information you provide to us to contact you about our relevant content and services.You may unsubscribe from these communications at any time. For more information, take a look at our Privacy Policy.