‍

I. Introduction

In today’s hyper-competitive market, businesses are constantly seeking deeper, more nuanced insights into their customers’ minds. Traditional market research methods, often reliant on static surveys and limited data points, struggle to keep pace. At Glaut, we believe that AI holds the key to unlocking a new era of qualitative data collection at scale – one that’s faster, more efficient, and yields richer, more actionable insights. But can AI-moderated research truly outperform traditional surveys when it comes to gathering those deep, qualitative insights that drive real business decisions? To answer that question, we conducted a rigorous head-to-head research study, pitting our AI-powered voice interviews platform against a leading traditional survey platform.

‍

II. AI vs. Traditional Surveys: A Head-to-Head Performance Comparison

To put AI-moderated interviews (aka, “AIMIs”) to the test, we designed a research study directly comparing Glaut's capabilities with those of a leading provider of traditional surveys. We recruited two groups of 100 participants each, ensuring balanced representation across demographics. One group engaged with Glaut's AIMIs, while the other tackled the same questions in a traditional survey format. 

Ensuring a Fair Comparison: To isolate the impact of AI moderation, we meticulously replicated the research questions between both research methods. We focused on questions related to trust and loyalty towards brands—a topic known to elicit rich, qualitative data. All potential follow-up questions that a researcher might typically include in a traditional survey were already embedded within the survey builder platform. This approach ensured that any observed differences in response depth and richness could be confidently attributed to Glaut’s AI agent and its real-time, context-sensitive interactions and follow-ups.

‍

To measure the effectiveness of each approach, we focused on five key performance indicators:

• Completion Rate: How engaging and user-friendly was the experience? A higher completion rate indicates a more positive user experience.
• Words Count: Did the surveys elicit detailed and expressive responses? More words suggest richer linguistic data.
• Number of Themes Extracted: How much depth and breadth of insights did we gather? We performed the same thematic analysis on the responses from both approaches, interview transcripts from Glaut and open-ended verbatims from the traditional surveys. More codes point to a more comprehensive understanding of the topic.
• User Experience Rating: How satisfied were participants with the overall survey experience?
• Transcript Quality: Which platform generated transcripts deemed higher in quality by an independent AI-powered evaluator?‍
• Transcript classification as gibberish or not: Which mode generated the highest number of not-gibberish transcripts according to a large language model (“LLM”) judge?

By meticulously analyzing these metrics, we aimed to determine if AI could truly elevate the quality, depth, and richness of qualitative data collected in market research.

Leveraging LLMs for Consistent Analysis: To maintain rigor and consistency in our analysis, we turned to the power of LLMs. Recent studies have shown that LLMs are capable of performing thematic analysis, text quality evaluation, and text classification with a level of sophistication comparable to human researchers (Dai, Shih-Chieh et al., 2023; Paoli, Stefano De., 2023; Chiang, C., & Lee, H., 2023). By employing an LLM, we ensured a standardized, unbiased assessment of both the AI-generated and traditional survey transcripts.
‍

III. The Results Are In: AIMI Takes the Lead

The results of our study were conclusive: AIMIs delivered on their promise of richer, more insightful qualitative data. Here are some of the key findings:

• AIMIs sparked deeper conversations: Participants using Glaut's AI-moderated interviewsprovided significantly more detailed responses, with an average word count 129% higher than those using traditional surveys.
• AIMIs unlocked hidden layers of insight: Our analysis, conducted by an AI-powered evaluator, revealed that AI-moderated interviewsgenerated a significantly higher number of unique themes, indicating a deeper and broader understanding of customer opinions and motivations.
• AIMIs delivered exceptional quality: A staggering 66% of the time, the AI-moderated interviews generated transcripts that were rated as higher quality than those from traditional surveys.‍
• AIMIs encouraged more thoughtful responses: The AI-moderated interviews delivered 74 not-gibberish transcripts out of 100, far surpassing the traditional method’s 44%.

‍

‍

And the best part? This significant leap in data quality didn’t come at the expense of user experience. Despite interacting with novel technology, participants rated their satisfaction with the AI-moderated interviews very highly, averaging an impressive 8.48 out of 10. Moreover, the higher completion rate, adjusted for meaningful interactions (61% for AI vs. 39% for traditional surveys), demonstrates that AI can drive engagement without sacrificing user-friendliness.

But were these improvements truly due to the unique capabilities of AIMIs? To be certain these performances were not due to random chance, we conducted rigorous statistical testing. The results showed a clear and significant correlation: the AI-powered approach — with the use of voice interaction and contextual follow-ups — was correlated for the increased word counts, greater number of codes, and higher-quality transcripts in a statistically significant way.
‍

Figure 1: Distribution of Word Counts per Respondent, Grouped by Completion Mode (Glaut vs Traditional Survey)

‍

‍

Figure 2: Distribution of Themes Counts per Respondent, Grouped by Completion Mode (Glaut vs Traditional Survey)

‍

Figure 3: Distribution of User Experience Ratings, Grouped Completion Mode (Glaut vs Traditional Survey)

‍

‍

Figure 4: Better Transcripts Counts, Grouped by Completion Mode (Glaut vs Traditional Survey)

‍

‍

Figure 5: Transcripts Classification Results, Grouped by Completion Mode (Glaut vs Traditional Survey)

‍

IV. The Power of Voice and Follow-Up Questions

So, what exactly makes AI-moderated interviews so effective at unlocking deeper insights? The answer lies in two key innovations: voice interaction and dynamic, personalized follow-up questions.

• The Touch of Voice: Unlike traditional text-based surveys, Glaut's AI-moderated interviews leverage the natural flow of voice, humans’ most natural medium of communication. Participants respond to questions using their voice, creating a more engaging and personalized experience. This conversational approach, similar to how we naturally interact in everyday life, encourages participants to open up, express themselves more fully, and provide richer, more nuanced answers.
• Follow-Ups that Guide the Conversation: Glaut AI-moderated interviews go beyond simply asking predetermined questions. It actively listens to and understands participant responses in real-time. If an answer is incomplete, unclear, or requires further exploration, AIMIs seamlessly pose dynamic follow-up questions, just as a skilled interviewer would. This ability to dynamically adapt the conversation ensures that no valuable insight is left unearthed. This is radically different from a pre-determined question, which by design does not change or request further probing based on the respondent’s answer.

By combining the engaging power of voice with the intelligence of dynamic follow-up questions, Glaut's AI-moderated interviews platform empowers researchers to gather deeper, more meaningful data, ultimately leading to a richer and more complete understanding of their target audience.

‍

V. A Closer Look at the Research

For those seeking a deeper understanding of our study, we believe in transparency. Here’s a brief overview of our methodology:

• Participants: Our study involved 200 participants, carefully divided into two groups of 100, ensuring balanced representation across key demographics such as age, gender, location, and occupation.
• Market Research Design: Both the AI-moderated interviews and traditional surveys were meticulously designed to cover the same thematic areas and elicit comparable responses, ensuring a fair and accurate comparison.‍
• Data Analysis: We employed a robust analytical approach, leveraging both statistical testing and the advanced capabilities of large language models to evaluate survey performance and ensure the statistical significance of our findings.

We encourage you to delve deeper into the research! You can download the complete research paper, authored by our generative AI researcher, just clicking here.

‍

VI. Don't Just Survey, Glaut it!

Ready to revolutionize your market research with the power of AIMIs? Visit our website to learn more about Glaut's cutting-edge AI-powered research platform. Explore the features of AIMIs, discover real-world case studies, or request a personalized demo to experience the future of market research firsthand. 

Don't just collect data - with Glaut, you will be able to understand people beyond the numbers.

‍

VII. References

1. Villalba, A.C., Brown, E.M., Scurrell, J.V., Entenmann, J., & Daepp, M.I. (2023). Automated Interviewer or Augmented Survey? Collecting Social Data with Large Language Models. ArXiv, abs/2309.10187.
2. Chiang, C., & Lee, H. (2023). Can Large Language Models Be an Alternative to Human Evaluations? Annual Meeting of the Association for Computational Linguistics.
3. Paoli, S.D. (2023). Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the model. ArXiv, abs/2305.13014.
4. Dai, S., Xiong, A., & Ku, L. (2023). LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis. Conference on Empirical Methods in Natural Language Processing.
5. Kheiri, K., & Karimi, H. (2023). SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learning. ArXiv, abs/2307.10234.