Ellen Williams — August 9, 2018 — August 9, 2018
MaritzCX clients often ask us “why are our scores on the same question different between these two studies?” We commonly find that mode effects explain the differences, however there are other aspects of research that may be at play, two of which we discuss below:
Relationship vs. Transactional
Relationship studies are conducted periodically to explore the holistic customer relationship, and are designed to guide strategic decisions. Competitive or benchmarking studies also fall into this category. Questions on a relationship study ask the respondent to consider all of their experiences with your company. The customer’s ratings may be influenced by recent experiences, something that happened long ago, or on a particularly memorable experience. It may be based on interactions across various points of contact. Regardless, the responses reflect the customer’s collective experiences and measures the overall relationship with your company.
On the other hand, transactional studies are ongoing, focus on customer experience with a specific interaction, and are intended to facilitate process improvement and coaching. A metric on a transactional study asks the respondent for a rating based solely on that particular experience, one point in time, perhaps with a single channel, not considering other interactions with your company. This data is collected very close to the actual interaction. A company may have many different scores from transactional surveys across multiple touchpoints.
If a respondent had an excellent experience in one instance and was surveyed about that interaction they may give a very high rating on that transactional survey. However, if over time a respondent’s experiences have varied they may give a lower rating on a relationship survey. The opposite can also happen, such as when a customer is surveyed about an interaction when they happened to experience your website being down, which likely causes them to give a very low rating to that transactional survey. But if the same customer doesn’t regularly experience website outages their cumulative rating on a relationship study may be much higher. Variance in scores can also be driven by sample sizes. Relationship studies generally have a robust sample of all customers, whereas a transactional study may have very few respondents depending on how many customers had an interaction in the time frame you are observing and how narrow that window is. Given these differences, we do not recommend the scores from relationship studies and transactional studies be directly compared.
Blind vs. Branded or Sponsored
A blind customer experience study is one that does not show any brand names, logos, or any other identifying criteria to the respondent. Branded studies are just the opposite where the invitation or survey itself may have brand name, logo or imagery of the company sponsoring the study.
Customer studies are usually branded in order to help boost the response rates (particularly when looking for hard to find respondents), and in some cases, give credibility to the study or mitigate privacy concerns. Particularly with online surveys, branding a study may make respondents more likely to respond, feeling that their answers are going to a company they know and trust.
With a branded study, respondents are likely to have positive or negative perceptions, feelings and loyalty towards the brand sponsoring the study which may bias their responses. If the respondent is surveyed about a brand they have an affinity for, they are likely to give more positive answers regardless of the questions.
Conversely, blind studies give the customer no context or preconceived notion in which to base their rating, therefore they may tend to be more honest, without bias. Competitive studies are blinded so when respondents rate their satisfaction with multiple different companies the results will be less biased if they do not know which one of them is the sponsor. Competitive studies may also ask respondents to rate brands of which they are customers and non-customers, which could also impact each company’s scores. Typically, a customer’s score would be higher than a non-customer, but not always.
Conclusion
Rather than trying to compare scores from two studies that may have differing designs, researchers should focus on the patterns. For a competitive or relationship study this would mean looking at the scores from various questions across the competitive landscape. If the study has been conducted previously, focus on whether there have been changes over time. Customer or transactional studies are more often conducted on a regular basis and allow you to view longitudinal trends within and across measures. The focus should center on why there might be differences between your company and competitors or why your score changed from time period to time period and what actions can be taken to make improvements for the future.
Business & Finance Articles on Business 2 Community
(36)
