Chatbots - Measuring for Success - Insight into working life at Stream
Stream’s unique software solutions, combined with a clear understanding of the significance of data, enables us to work with clients to help build brand loyalty, increase sales, drive customer satisfaction and drive employee engagement. Stream helps build trust that delivers brand loyalty and commercial rewards and, through this, can prove that the cost of inertia far outweighs the price of change.
Stream have just won a contract to develop an online coaching and professional development tool for a major healthcare company. Watch this space for more information.

We have recently completed 5 incentives for Anglian Home Improvements taking guests to London, Loutraki, Dubrovnik, Dubai and Rio de Janeiro.
Chatbots - Measuring for Success
19 September 2017

Chatbots - Measuring for Success

So you’ve decided to build a chatbot, now how are you going to measure its success? Chatbots are unique and require their own defined KPI’s to measure whether and how they have improved the customer experience. Think of them as one of your team and measure how effectively they have engaged the customer to create lasting relationships.

In a previous blog, we talked about how to design and build a chatbot. We suggested that you start simply, by designing the bot to focus on just one use case - for example, helping customers to book appointments. Before you start working out the metrics, you first need to decide what your goals are. The simpler the chatbot, the easier the goals are going to be. These will be unique to your business, so today, I just want to provide you with some ideas on what you can measure and how useful those measurements are.

With chatbots, it is best to focus on a few measurements to get an holistic picture of the success of your chatbot. Just as you would with one of your team, you need to look at the whole customer experience your bot is providing.

Rate of activation or number of users

Whilst this is still a metric widely used for companies to measure the number of customers they have using their website, app or bot, it isn’t really useful unless it is measured against how engaged those customer are.

Chatbots are still new to a lot of industries, so you are likely to see a high level of activation, but also an equally high drop off rate.

Number of volunteers

This metric looks at the number of times a user voluntarily activates your chatbot without being reminded. A useful statistic to understanding how open your customers are to interacting with messaging services. You can also track where and when your customers activate the chatbot to see whether there can be improvements made to the information available to them at that point in their journey.

Number of interactions/user

Again, this is the type of stat we all love for our websites and apps, as it generally means a user has interacted with more of our brand. Whilst the same can sometimes be said with a chatbot, it is unlikely that the customer wants to spend too long interacting with a robot (however engaging you might have made yours). In the same way as with websites, although the high number of page views your website is getting might sound great, if no one is ultimately buying, then those page views count for nothing.

The quality of each interaction is the most important metric to study. With a chatbot, it is easy to measure the number of conversation steps a customer takes. It will depend on the service your chatbot is providing as to what you set as the average number of steps a customer should take to achieve the desired goal. Each interaction is counted as a step; for example, if the customer says ‘Hi’ and then the chatbot says ‘Hi’ that would count as two steps. Before developing your chatbot, have fun with role-playing conversations that you would like the chatbot to handle. This will give you a good feel for how many steps a customer should take. You can then analyse the interactions that fall below or above the average number of steps as these will be an indicator of the customer’s experience.

Rate of retention

The rate of retention will vary depending on the service you want your chatbot to perform. If the chatbot is expected to help customers with local news, events then you would expect to be working on a one day retention rate, whereas in a B2B environment or large retail purchase, customers may not need to interact more than once a month. If you liken the rate of retention to how often you would expect your customers to be actively interacting with your brand, you can then get a good idea of what your average metric should be.

This is a fairly critical metric, so you should make sure you are recording this weekly, monthly and quarterly, to see how the chatbot is developing over time.

Retreat rate

Retreat rate is one of the key metrics to focus on with your chatbot, as this can tell you how often the customer was provided with a poor experience. This is a wide and complex subject of its own that requires in-depth analysis to understand how it can be improved.

There are two ways of measuring the retreat rate, the first being how often your customer had to be transferred to a live support agent when the chat was deemed to be reducing in quality.

The second is disorientation. We have all been there when you are using a chatbot and suddenly you must have asked something completely out of its scope of understanding, because it either starts asking you the same question again or provides you with information you didn’t ask for. The best way to measure this is to divide the number of times your bot had to retreat by the number of messages sent.

However well you develop your chatbot, there will inevitably be times when it has to respond with “I don’t understand” or similar messages. Being able to track this and analyse what the customer needed/asked that led to the confusion will enable you to better train the chatbot in the future.

Making sure this doesn’t end up with the customer having a poor experience can be helped by good chatbot design. Be honest with your customer and either ask them to rephrase the question, provide them with helpful buttons they can use, or let them know you are going to pass their chat through to a support agent to help them.

Chatbots are a relatively new tool in the marketer’s toolkit. We at Stream love to kick ideas around on this and other new marketing opportunities and we’d love to talk it through with you over a coffee.