User Research and Statistical Confidence Intervals
In the realm of User Experience (UX) research, confidence levels play a crucial role in guiding decision-making processes. These levels, calculated using the science of statistics, help researchers gauge the reliability of their findings.
A confidence level of 51% or higher is considered more reliable than making decisions based on a coin toss, especially for decisions that impact a small percentage of users and carry minimal risk. For instance, if 42 out of 50 people in a survey do something the same way, the confidence interval suggests that between 71% and 91% of people will approach the task that way with a 95% confidence level.
However, in some industries such as pharmacy and autopilot manufacturers for planes, a higher degree of certainty is required, often approaching 100%. Industries like semiconductor manufacturing, medical research (e.g., clinical cancer research and biomarker studies), construction engineering (e.g., textile membrane modeling), and automotive (e.g., real-time behavior analysis) also require high accuracy in their research results. They should adopt conventional safety levels that approach near 100% certainty by using robust, validated, and precise measurement and simulation systems, often involving advanced materials, rigorous testing, and validated AI/machine learning models.
Achieving a high degree of certainty (95% or greater) is often costly and time-consuming, making it difficult for UX researchers to attain except in specific circumstances. For smaller UX teams or solo researchers, a confidence level of 80% or greater is often acceptable before pursuing research or development. In corporate UX research, a confidence level of 90% or greater is considered sufficient to make major user-facing decisions.
Reaching a 90% confidence level can be challenging and requires careful attention to research design. A confidence level of 80%, on the other hand, offers a 4 in 5 chance of being correct and is easier and less costly to achieve compared to a 90% confidence level. This level is useful for rapid iteration and ideation without descending into a guessing game about user needs.
The purpose of a confidence interval is to help manage risk, not eliminate it entirely. A rational approach to decision-making is enabled by reporting research results with confidence values clearly stated, reducing career risk for researchers. Journals and political polling organizations tend to require a high degree of certainty (95% or greater) in the accuracy of the data for publication.
In conclusion, understanding and applying confidence levels in UX research can lead to more informed and reliable decisions, ultimately improving the user experience. By being aware of the trade-offs between cost, time, and certainty, researchers can make more effective use of their resources and contribute to more successful projects.
