Sampling in CX Research
The phrase “statistics don’t lie” carries an important truth — yet in research analytics, it's surprisingly easy to draw incorrect conclusions even from accurate data. The ways in which data is collected, interpreted, and compared all come with a risk of error. For a CX analyst, the safeguard lies in a solid understanding of statistics and a disciplined application of rules that help distinguish real trends from mere artifacts.
Let’s begin with a fundamental principle: a statistical study does not need to cover the entire population (i.e., the full set of elements we want to draw conclusions about) in order for the results to be valid. In CX research, this means you don’t have to survey all customers who purchased a product or used a service. In statistics, when information is gathered from only a subset of the population, we call this a sample survey. The group selected for study is referred to as a sample, and if that sample is representative, it allows us to generalize the findings to the broader population.
There are two main methods for selecting a sample. The first is purposeful sampling, where the researcher selects specific individuals to form the sample. For the results to be statistically valid, the sample must reflect the true structure of the entire population.
For example, when drawing conclusions about the population of Poland, a sample consisting only of urban residents with higher education would not be representative. To accurately reflect Polish society, the sample must mirror key characteristics: nearly 40% of the population lives in rural areas, and no more than 21% hold a university degree. Knowing such demographic distributions allows us to recreate them in the sample.
This approach requires careful attention to who is invited to participate. Even with meticulous planning, there is still a significant risk of bias, and that risk is difficult to quantify.
In Customer Experience research, random sampling — where each individual has an equal chance of being selected — is often the best approach to ensure representativeness. However, the required sample size depends on several key factors:
Acceptable margin of error (smaller error = larger sample size),
Variance of the measured characteristic in the population (higher variability = larger sample),
Confidence interval (narrower interval = larger sample),
Population size (the larger the population, the smaller the relative sample percentage needed).
When conducting CX research in digital channels — such as e-commerce websites, corporate sites, or mobile apps — it's important to use analytics data (e.g. traffic volumes, unique visits, and session counts) to define the population. To draw conclusions representative of the customer base, it’s usually necessary to collect a large number of responses from users interacting with the website or app.
A common pitfall among online marketing specialists is underestimating sample size requirements. Many in-house surveys aim to optimize site conversion using A/B testing, but incorrectly assume that comparing two groups of 100 users each is statistically sufficient. In reality, statistical significance typically requires hundreds or thousands of responses per variant.
Avoid drawing conclusions from dozens of surveys collected over just a few days. Low satisfaction scores from such small samples can easily be a result of random fluctuation, and acting on them risks misinterpretation and biased decisions.
There’s also a widespread belief in the CRO (conversion rate optimization) community that frequent, small tweaks are always worthwhile. Consider a scenario where a website change is made, and 5,000 responses are collected in the following month, with slightly improved ratings. Can we declare the experiment a success? Not necessarily.
Another analytical trap is relying solely on sample size, while ignoring whether the sample is properly selected. After implementing changes intended to improve the site experience, one cannot simply compare satisfaction ratings month-over-month. Users only perceive a change when they can compare the “before” and “after”. This is where web analytics and cohort analysis become critical.
New users who never saw the earlier version of the site may rate it highly — not because of improvement, but because they never experienced the prior issues. With cohort analysis in YourCX, you can track the same returning users over time. This is the only reliable way to evaluate whether a change made at a specific point in the customer journey truly impacted the experience, especially for users who previously reported a problem.
Since it can take weeks or even months to gather a representative cohort of return users, it is best to conduct ongoing research and analyze trends over longer time periods.
What About Net Promoter Score (NPS) Analysis?
Just like a company’s valuation, any other metric can be measured in different ways. That’s why comparing two identical indicators — measured using different methods — may be completely misleading. In CX research, the Net Promoter Score (NPS) methodology is widely used, with industry benchmarks often published publicly. This frequently triggers a desire to compete for higher scores, but monitoring competitors without the proper context can be deceptive.
While the NPS methodology aims to objectively reflect the strength of customer advocacy based on overall experience, comparing results without understanding the underlying data collection methods is problematic. More important than the score itself is the context behind customer feedback at different touchpoints. For example, an NPS measured immediately after purchase and one collected after product delivery may reflect entirely different experiences, despite having the same score.
Just like on the stock market, in CX research it rarely makes sense to compare results day by day — especially when small sample sizes increase the risk of statistical noise. Every investor knows that trends matter more than snapshots, and CX analytics is no different. Even if your NPS remains unchanged from the previous year, it may still represent progress — particularly if industry benchmarks are trending downward, as often happens in the software sector due to rising customer expectations.
Measuring Specific Customer Touchpoints
YourCX enables you to create an unlimited number of context-specific surveys, tailored to different stages of the customer journey. These can be combined with real-time behavioral data to gain meaningful insights. This means you can design focused micro-surveys asking only about a specific touchpoint, such as a return or refund page.
Additionally, you can gain a better understanding of negative feedback by linking it to real user behavior on your e-commerce platform. For example, feedback collected on return policy pages can be analyzed together with user history — enabling you to identify not only the touchpoint itself, but also whether the respondent is a returning customer or someone who made a purchase and then returned to the site.
Suppose you hypothesize that negative feedback is due to an ineffective return policy. Even if you receive 10,000 total CX responses in a month, that might not be enough to evaluate this particular issue. What matters isn’t how much you ask — it’s what customers are actually rating. In this case, proper sampling becomes critical.
Why? Imagine that 30% of respondents gave negative feedback, but only 20% of those actually made a purchase, and just 5% returned an item. That results in a sample size of just 30 people for return-related feedback. This is why more time is needed to collect a statistically meaningful sample.
Validating Improvements
Let’s say you’ve identified the return policy as a pain point and implemented improvements — like a simplified return process and a dedicated phone line to avoid frustrating customers with a generic call center. In your next CX survey, the NPS score increases. But how can you be sure it’s a result of that specific change?
You can’t simply compare monthly NPS scores, as some feedback may still be coming from customers affected by the old return process. Instead, apply cohort analysis: isolate new and returning users, and analyze their experiences over time. A large enough sample allows you to filter results in YourCX — for example, excluding customers who made a return under the old process.
The most insightful cohort consists of customers who experienced the old return policy, and are now reviewing the new experience. Once you collect at least 500 responses from this group, you can calculate statistical confidence levels. If you reach a 95% confidence interval, you can confidently report the improvement as significant.
In this way, NPS is not just a number — it becomes a tool for actual experience improvement.
Want to learn more about statistical sampling in CX? Visit our dedicated guide on sample selection for further details.
