Welcome to our conversational guide to “conversation metrics”. You're about to learn all the essential aspects of measuring conversations between customers and your business. We'll start by defining what makes a good metric, go through some basic definitions and move towards more advanced concepts. Feel free to read the contents of this guide in order or jump straight to the section that sparks your interest. Here's a list of topics covered in this guide:
- Characteristics of Good Metrics
- The Definition of Conversation Metrics
- Ways to Collect Conversation Metrics
- Important Conversation Metrics
- Conclusion
What Are the Characteristics of Good Metrics?
Creating effective metrics is not an easy task and requires a sound understanding of the processes you want to measure and improve. Here’s a list of ten characteristics a good metric should have:
Why Do Metrics Have to Be Strategic?
To create a set of effective metrics for your business, you need to begin with the end in mind. To do that, you have to use business goals and objectives as starting points and then work back to find ways to measure the outcomes. Doing this will help you monitor if you’re on track to achieving your plans.
Why Do Metrics Have to Be Specific?
If you want to come to valuable conclusions, you need to clearly define the rules for measuring the outcomes of your actions. You have to know what needs to be measured and how it relates to the process you want to monitor and improve over time. Doing this will ensure that you have a reliable source of information to validate your ideas and evidence of how significant your impact is.
Why Do Metrics Have to Be Simple?
A famous Leonardo Da Vinci quote says that “simplicity is the ultimate form of sophistication”. It’s not only a catchy phrase. The elegance of simple metrics comes from the instant ability for anyone to grasp the concepts behind them and, more importantly, take actions in the desired direction.
Why Do Metrics Have to Be Owned?
Every metric needs to have an owner who is accountable for the results. Depending on the size of your business, it can be just one person or a group of people, but without anyone being responsible, a metric lacks meaning. Working as a collective isn’t a must, but it can improve the outcomes through collaboration.
Why Do Metrics Have to Be Actionable?
When a metric changes its value beyond an acceptable threshold, it should be clear for everyone involved what actions to take to improve it. Measuring conversations with customers without understanding how to improve the outcomes doesn’t bring any benefits and leads to frustration.
Why Do Metrics Have to Be Timely?
While comparing current data with the past might be helpful to understand how your performance evolves, it’s the timely changes that bring the most significant opportunities for improvement. Business reps who understand how the measured process works should easily access timely data to connect the results with the recent actions or inactions.
Why Do Metrics Have to Be Accurate?
It’s hard to create metrics that reliably measure business performance because accuracy depends on the raw measurements and the formula used to calculate derived values. Therefore, collecting precise data is only half of the equation, and the other half is correctly using them to measure intended objectives accurately. It's impossible to draw meaningful conclusions without ensuring that the underlying data and calculations are correct.
Why Do Metrics Have to Be Correlated?
While direct measuring complex behaviours or the desired outcomes might not be possible, metrics should answer how well the process meets customer expectations. Therefore metrics shouldn’t just show some numbers, but those numbers have to provide a clear picture that reflects the reality as closely as possible and help drive the anticipated results.
Why Do Metrics Have to Be Aligned?
It’s vital to understand that all the metrics have to amplify the desirable outcomes. It’s easier to look at each one of them in isolation. But, to ensure that some metric driving anticipated effects in one area doesn’t undermine the results in others, you need to look at them from a connected system perspective where each one affects all the other metrics and make sure that this influence is positive.
Why Do Metrics Have to Be Standardised?
You need to agree on the definitions of the metrics you use to assess business performance to ensure the same meaning across the company. Without shared understanding, you risk a lot of confusion, misunderstandings, difficulty in comparing the results and sharing insights.
What are Conversation Metrics?
Conversation metrics are quantitative measures of various aspects of how a company exchanges messages with its consumers. They are an invaluable source of quantitative information about what consumers are trying to do, where they come from, and when they perform these activities while letting you also monitor and improve customer service experiences to suit their needs and expectations better.
How to Collect Conversation Metrics?
In many situations, even if it’s not scalable, manually collecting information is possible. However, this isn’t the case for capturing conversation metrics. The amount of data is just too vast for any human to collect it reliably. Leveraging a fully automated solution is a must to collect data about conversations effectively. The process for collecting conversation metrics works like this:
- A chatbot captures the raw data.
- Anynymises that information.
- Stores it in a database for future retrieval, processing, and analysis.
Why is it Important to Collect Conversation Metrics?
Conversation metrics enable observing customers' behaviour from a birds-eye perspective during their everyday interactions with your business. Collecting them helps answer the following questions:
- How many customers engage in conversations with your business?
- When and where do these conversations take place?
- In how many online chats do customer service agents actively participate?
- What’s the percentage of conversations handled entirely by a chatbot?
- How often do support agents work in tandem with chatbots?
What is the Risk of not Collecting Conversation Metrics?
Companies that don’t collect information about their daily interactions with customers don’t have any visibility into the authentic voice of their customers. By not collecting conversation metrics, you can only have a vague idea of how good or bad service your business provides, which limits your understanding and can effectively block any improvement.
What are the Most Important Conversation Metrics?
When you try to understand how you can do a better job via real-time conversations, it makes sense to learn everything you can about the conversations you're having. Here’s a list of key conversation metrics worth tracking:
- Volume
- Time of the Day
- Communication Channel
- Modality
- First Reply Time
- Average Response Time
- Number of Responses
- Task Completion Rate
- Dropout Rate
- Number of Errors
- Conversation Distribution
What’s the Volume of Conversations Metric?
The volume of conversations is a metric that continuously tracks the number of conversations over digital channels between your business and its customers.
How to Analyse Changes in the Volume of Conversations?
When analysing the differences in the volume of conversations, it’s important not to jump to conclusions too quickly because the change itself is just a signal for further investigation.
Is it Good or Bad when the Volume of Conversations Increases?
An increase in incoming conversations is neither good nor bad but signals a need for further investigation to understand the reasons behind the upsurge. Here’s a list of several potential causes:
- Your demand generation strategies work and bring more visitors.
- You added a new product or service to your offerings, and it’s gaining traction in the market.
- Your product or service quality has decreased, and customers either need help to consume the value or complain more often.
How to Increase the Volume of Desirable Conversations?
To increase the volume of desirable conversations, you can do several things:
- Focus on demand generation strategies that increase the flow of visitors interested n your offerings.
- Improve website targeting and welcome messages to align them better with customer intent.
- Introduce a new product or service that’s more effective than the industry standard.
Is it Good or Bad when the Volume of Conversations Drops?
A considerable drop in incoming conversations is neither good nor bad but signals a need for further investigation to understand the reasons behind the decreased numbers. Here’s a list of several potential reasons behind this type of change:
- The flow of incoming visitors has decreased.
- Your product or service quality has improved, which caused customers to need help from service agents less often.
- People are less interested in the products or services you offer because another, newer, or more effective option became available on the market.
How to Decrease the Volume of Undesirable Conversations?
You can reduce the number of undesirable conversations by improving the quality of your offerings and by letting your chatbot do the initial talking and filter incoming messages based on their relevance to your business. Here’s a list of several ideas on how to do that:
- Build and continuously improve your knowledge base to anticipate frequent customer questions and answer them instantly without the need for repeated customer service rep attention.
- Include a sales qualification chatbot in your conversation flow to reduce the number of conversations with people who are not the right fit for your business.
- Improve the quality of products and services in response to customer needs to reduce the number of customer complaints.
Why is Volume of Conversations an Important Metric?
By comparing how many conversations your customers start day-to-day, week-to-week, month-to-month, you'll be able to measure the pulse of your omnichannel conversation strategy and make adjustments accordingly.
What’s the Time of the Day Metric?
Looking at the volume of conversations gives you an overview of how many conversations you have, but measuring the time of these interactions gives you additional insight into when these conversations happen during the day.
How to Visualise Conversation Demand Throughout the Day?
One of the best ways to visualise customer demand for conversations during the day is in a heatmap, which uses more saturated colours to indicate that more conversations are happening in a given timeframe, and less saturated ones to show smaller demand.
Why is the Time of the Day an Important Conversation Metric?
Monitoring this information will provide you with information about peak times that can help to understand and adapt to customer needs by intelligently scheduling messaging shifts.
What’s the Communication Channel Metric?
Knowing how many conversations you have and when they happen both provide you with a wealth of knowledge, but it’s also essential to know where these conversations occur. Communication channel metric tracks which communication channels are used for conversations between customers and your brand.
Why is Communication Channel an Important Conversation Metric?
Communication channels vary slightly in terms of functionality and customer expectations. Tracking them makes it easier to make conclusions about what works and doesn’t and let you make selective improvements. For example, websites enable tracking user behaviour before engaging in conversation, encouraging website-specific improvements.
What Additional Information to Track During a Conversation on a Website?
A website provides unique opportunities to track additional details about the visitor’s steps before engaging in a dialogue. It often includes the page that triggered the conversation, providing further contextual cues that help derive the customer's intent and help to validate and improve conversational marketing and sales efforts on those pages.
Why is it Important to Track Conversations on Every Page of a Website?
It’s tempting to focus only on top-performing pages, but tracking conversations on every page of your website allows you to analyse conversations on a page-by-page basis. You can drive more customer engagement by enhancing targeting and welcome messages on low-performing pages on your website.
What’s the Modality Metric?
The modality metric measures methods of interaction clients use to communicate with your business. Customers can ask questions or request actions by either typing or using their voice.
How do Customers Choose Modality?
Preferred modality depends on personal preferences, customer location, and time of the day. For example, when a customer drives a car, they will choose voice over typing, and it might be exactly the opposite in a cinema or a library.
Why is Modality an Important Conversation Metric?
Knowing the most common modality used by your consumers may help you plan and prepare better responses to their questions. For example, when there's a point in the conversation flow where customers usually speak, you might want to remove visual cues or use them only as a fallback mechanism when the speech recognition fails.
What’s the First Reply Time Metric?
The first reply time measures time it takes customers to receive an initial response when they engage with your business. In other words, it measures how long your clients have to wait before being served.
What’s a Good First Reply Time?
You should invest energy to reduce the first reply time to a bare minimum. Generally speaking, if it’s greater than 2-3 minutes, you risk losing the client's attention forever.
How to Improve First Reply Time?
Enabling chatbot-powered automation helps offload your customer service representatives and reduce the time clients wait for the initial response. Chatbots can help to improve first reply time in the following ways:
- Engage customers in real-time with personalised welcome messages
- Provide guidance and answer simple questions
- Perform simple tasks like appointment booking, rescheduling and cancellations
- Route conversations to the most appropriate human agent in situations that require higher cognitive abilities or complex judgement
Why is First Reply Time an Important Conversation Metric?
Reply time can make or break your relationship with a customer in the first few moments of interaction. Speedy replies make customers feel listened to, valued, and appreciated, making the time to deliver the initial response one of the most important metrics to track and improve.
What’s the Average Response Time Metric?
Average response time is a metric that enables you to understand how long, on average, it takes to respond to a customer query or perform the desired action.
What’s Good Average Response Time?
Good average response time will vary across niches and situations. In general, it’s vital to decrease the average time it takes to respond to a customer, but it’s also worth noting that some queries are more complex than others and require more time to respond. No matter the complexity of the customer query, good response time should at least keep them engaged, which translates to answers in under a minute.
Is it Good or Bad when the Average Response Time Drops?
At first glance, it seems reasonable to decrease the average response time. While it’s true in many cases, it’s vital to combine this information with customer feedback to paint a fuller picture of the overall customer satisfaction to ensure that it’s not a result of cutting corners.
How to Decrease the Average Response Time?
There are a few ways that help to decrease the average response time:
- Delegate an increasing number of questions and supporting activities to a chatbot
- Collect more relevant information before transferring the conversation to a human
Is it Good or Bad when the Average Response Time Increases?
An increase in average response time doesn’t necessarily have to be a bad thing since it can result from many external factors, including increased query complexity. It’s advisable not to consider this metric in isolation and combine it with customer satisfaction metrics to determine if the increase in response time correlates with customer delight.
Why is Average Response Time an Important Conversation Metric?
It’s essential to remember that this metric, while important, provides only a high-level view of the actual performance since it combines all of the times it took to respond to queries that vary in complexity into one number.
What’s the Number of Responses per Conversation Metric?
This metric helps to understand the average number of responses in dialogue. In other words, how many questions have been answered and actions performed during a conversation.
What’s a Good Number of Responses per Conversation?
It’s challenging to define a specific number that will apply to various situations. A better way to establish a desirable number of responses per conversation is by analysing actual interactions with customers and asking the following questions:
- Was it possible to satisfy customer needs in fewer responses?
- Would fewer responses improve customer experience?
If you answered yes to both of these questions, a shorter conversation would better suit customer needs.
Is it Good or Bad when the Number of Responses per Conversation Rises?
In many situations, an increase in the number of responses per conversation signals inefficient conversation flow, but it doesn’t always have to be the case. A sudden rise in the number of replies per conversation doesn’t give a definitive explanation but signalises a need for further investigation. Some customers have special emotional needs or ask complex questions that require additional clarification steps before coming up with a helpful response.
Is it Good or Bad when the Number of Responses per Conversation Goes Down?
It’s usually a good sign when the number of responses per conversation goes down, but it’s vital to ensure that customer needs are fully satisfied during a short conversation.
What’s a Good Way to Increase the Number of Responses per Conversation?
In some niches, it’s vital to establish an emotional rapport with customers, and techniques used to achieve this goal are a productive step towards reaching it. Another good way of increasing the number of responses per conversation is to provide additional information to customers with an investigative mindset to satisfy their thirst for knowledge.
What’s a Bad Way to Increase the Number of Responses per Conversation?
Increasing the number of responses by including replies that are not relevant to the conversation is a sure way to customer dissatisfaction.
What’s a Good Way to Decrease the Number of Responses per Conversation?
The best way to decrease the number of responses per conversation is by providing precise answers to customer queries.
What’s a Bad Way to Decrease the Number of Responses per Conversation?
Ignoring some customer requests or cutting corners to end the conversation quickly are examples where a decrease in the number of responses has detrimental effects.
Why is the Number of Responses per Conversation an Important Metric?
The number of responses per conversation helps you measure the pulse of conversations you’re having with your customers. Many situations are similar, so they should fall within the average, but it’s also important to remember the outliers that might require additional clarification or emotional support.
What’s the Task Completion Rate Metric?
Task completion rate is a measurement of success or failure in completing the desired action or a sequence of steps. A task completion rate measures how many started tasks have been completed.
What are the Examples of Tasks that Can be Completed During a Conversation?
Here’s a list of examples of successful tasks:
- Elicited Engagement
- Question Answered
- Prospect Qualified
- Appointment Booked
- Customer Survey Completed
- Appointment Rescheduled
- Appointment Cancelled
- Reminder Sent
- Customer Feedback Collected
- Referral Received
What’s a Good Task Completion Rate?
Although it might not be possible to complete a task in rare situations because of unpredictable external factors, a good completion rate should be as close to 100% as possible.
How to Improve Task Completion Rate?
Each task should be predefined to be measured. If it’s complex, it’s vital to break it down into several smaller tasks that can be completed independently to capture completion rates at a more granular level. Doing this will help you find problems sooner and simplify making necessary improvements.
Why is Task Completion Rate an Important Conversation Metric?
High completion rates ensure that your customers are getting what they want from a conversation with your company, which impacts their perception of the quality of services your business provides and improves their level of satisfaction and loyalty to your brand.
What’s the Dropout Rate Metric?
In addition to measuring the task completion rates, it’s also essential to know where users drop out. The dropout rate is a supplementary metric that will help you pinpoint the exact moments in your conversation flows that need further improvements.
How to Use Dropout Rate to Find Problems in the Conversation Flow?
Usually, most dropouts happen at a particular step of the conversation that causes problems to your customers. Dropout rate helps to find these moments quickly and easily, letting you channel your energy precisely towards fixing them.
What Are the Reasons for High Dropout Rate?
The following list presents several reasons why customers might decide to drop off a conversation:
- Clients have to wait too long during peak times or off-hours to complete a task.
- Customers complete successfully a lower-level task that’s not tracked and decide to stop there.
- The prompt is confusing or seemingly irrelevant.
- Customers say something unexpected. For example, clients answer a question with “yes” or “no” without listening to it completely, and you haven’t anticipated this type of answer, e.g. “Would you like to reschedule or cancel your appointment?”
- Customers don’t feel that they’re on the right track to completing the task.
How to Decrease the Dropout Rate?
Depending on the reasons for the high dropout rate, several common remedies can help:
- Include a chatbot into your conversation flow that enables customers to complete everyday tasks quickly and easily.
- Break complex tasks into a sequence of lower-level steps that can be tracked and completed independently.
- Examine and Improve the phrasing of the prompts leading to the problematic state. Avoid prompts that customers might consider ambiguous even without listening to the question entirely.
- Check the way customers phrase their answers and include them in the list of expected responses if relevant.
- Stay focused on the task, avoid any digressions, and aim to complete the task in the fewest number of steps.
Why is Dropout Rate an Important Conversation Metric?
High dropout rates are an alarming signal that customers can’t complete the tasks they’re after and require immediate attention because it can cause frustration and overall dissatisfaction among your prospects and customers, which can have disastrous effects on your bottom line. Many reasons cause high dropout rates, and it’s vital not only to track this metric but also to make improvements quickly.
What’s the Number of Errors Metric?
The number of errors shows moments in conversation flow where customers encountered problems. There are several common reasons:
What are Typos and Speech Recognition Errors?
These errors occur when a customer makes a mistake during typing, autosuggestion rewrites the words it considers wrong, or when the speech recognition engine wrongly recognises spoken words.
How to Reduce Typos and Speech Recognition Errors?
The first step to mitigating this problem is usually asking the customer to rephrase their answer, but if it happens repeatedly, it becomes frustrating, so another technique comes into play. It’s about showing a list of best matches for what was understood and letting them to select the most appropriate option.
What’s a No Speech Detected Error?
No speech detected happen when the customer didn’t respond to a query within a defined timeframe and can indicate that:
- The prompt was confusing
- The customer doesn’t have the necessary information at hand
How to Reduce the Number of No Speech Detected Errors?
Depending on the reason that’s causing the no speech detected error, it might be a good idea to:
- Rephrase the prompt
- Enable asking for help where to find the requested information
What’s the No Matches Error?
No matches error indicates that client words have been recognised, but the chatbot didn’t know what to do with this information. There are two possible scenarios:
What’s a Correct Reject in a No Matches Error?
Correct reject means that the customer query was correctly not matched because it’s not a valid response in that state of a conversation. For example, when the prompt asks: “Which location would you like to choose?” and the customer responds with “I like apples”, the chatbot won’t be able to continue, nor should it.
How to Reduce the Number of Correct Rejects?
Since the customer doesn’t want to cooperate and tries to stray the conversation in an irrelevant direction, there’s no need to reduce the number of correct rejects.
What’s a False Reject in a No Matches Error?
“False rejects” happen when the customer’s response is valid, but the chatbot can’t match it with any of the expected responses. This error usually occurs in the early stages of development, when a chatbot cannot correctly understand all ways in which customers phrase their answers.
How to Reduce the Number of False Rejects?
It’s vital to monitor the no matches error metric, and when the customer phrased their question or answer in a way that should have been understood, add it to the list so that next time it’s going to be correctly matched.
Why is the Number of Errors an Important Conversation Metric?
The number of errors metrics helps identify points in which an input from a user didn’t result in any response or action. Resolving these errors is crucial to have effective conversations with customers, and they will quickly lose trust if you don’t.
What’s the Conversation Distribution Metric?
Conversation distribution provides an overview of the split between customer service agents and chatbots responding to customer queries.
What’s a Good Conversation Distribution?
A good conversation split between chatbots and customer service agents varies slightly between industries, but a good rule of thumb is 50/40/10, where:
- 50% of conversations are fully automated
- Chatbots assist 40% of conversations
- Customer service reps fully manage 10% of conversations
Why is Conversation Distribution an Important Conversation Metric?
Analysing conversation split can give a wealth of information about how conversations are distributed in your company and help make informed decisions about automating the most prominent candidates.
Conclusion
It’s vital for the long term success of your business to test, learn, and improve your products, services and supporting activities regularly. Conversation metrics can be an invaluable source of quantitative information about your customer needs. To properly use conversation metrics, you need to make hypotheses for improvement, put them into action, and then let the metrics validate your ideas to set a direction for the future.