This blog guides you through what concept testing survey questions are, which questions to ask in your concept testing survey, how concept testing is beneficial, and how quantilope’s automated concept testing approach can help.
Table of Contents:
- What are concept testing questions?
- Considerations when drafting concept testing questions
- Formats for concept testing questions
- How to conduct automated concept testing with quantilope
What are concept testing questions?
Concept testing questions are types of market research questions used to gather current or potential customers’ views about a proposed product, service, prototype, or any other type of collateral that could eventually go to market.
Testing new concepts before they go to market is a crucial part of the product/service development process. Without knowing if your idea has validity you’d be leaving its potential success entirely to chance by trusting only your own judgment; this can create a biased concept, especially if you’re not the prime target audience for the final product, service, etc.
The most important type of information while generating a concept idea is whether your concept is likely to be a hit with consumers who are truly in the market for it. To determine that, brands use concept testing surveys to ensure that information is reliable, useful, and actionable. The type of concept testing questions you ask, and how you ask them, need to be carefully thought through before your concept testing survey is launched.
In the next section, we’ll explore things to keep in mind when drafting concept testing questions.
Back to table of contents
Considerations when drafting concept testing questions
The exact concept testing questions you choose to include in your survey will depend on your business objective and the concept you’re testing, but there are some general factors to bear in mind when deciding what to ask.
Study goals
First, decide what the goal(s) of your study are. What kind of information about your concept do you want to gather from your respondents? You might want to know how unique the concept feels, which features of the concept respondents like the most (and why), which features they would remove (and why), how impactful certain design elements are, or how likely your target market would be to buy the product or service. Once you’ve established your concept testing study goals, these should serve as a guide for which metrics to measure and which are the right questions to ask.
Target market
Second, think about who your target market is. The type of person you want to attract with your final concept will help inform your questioning; think about what they’re like as people, what their lifestyle is like, and what they might need from the product or service. Figuring out who your target audience is might even be a goal of your concept testing study, but anything you can gather in advance to steer your questioning will help make sure your concept will appeal to the right audience.
Concept testing methods
Finally, the content of your study questions will be partially determined by the concept testing method you use. There are three main methodologies to consider for concept testing:
A/B Monadic Testing
A/B Monadic Testing involves asking respondents to evaluate one concept in isolation. With monadic testing, equally structured respondent groups are each shown just one concept to assess, with the advantage that brands can directly compare concept feedback to choose one winner, based on respondents’ in-depth reactions to a single concept (rather than reviewing multiple at a time).
As an example of monadic A/B Testing, say respondents are evaluating a new concept for a carbonated beverage. The beverage brand would show equally-structured groups of respondents a different version of the concept, and ask follow-up questions about what they’ve seen (do they remember certain elements of the drink, would they likely buy it from the shelf, what did they dislike about the concept, and so on.)
Conjoint analysis
Conjoint Analysis is another type of advanced method commonly used for concept testing. With Conjoint Analysis, respondents are presented with a number of different concepts at once - each with a different variety of features. With this method, brands can design their ideal products and see which portion of the audience each concept captures.
Respondents will choose which concepts they prefer, which (via automated analysis) reveals which individual product features they place the most importance on for a certain type of concept. The cool part about Conjoint Analysis is that though respondents are choosing an overall concept as their most preferred, brands can still deduce which aspects of the concept led them to that decision.
Using sneakers as an example, there might be three concepts shown: Concept A is a white sneaker with black shoelaces, a silver logo, and regular insoles; Concept B is a black sneaker, with white shoelaces, a purple logo, and cushioned insoles; Concept C is a white sneaker, with white shoelaces, a purple logo, and cushioned insoles. There may be more concepts than just these three, but the analysis would reveal which of the attributes (color of the shoe, shoelaces, logo, and sole type) most strongly influence the final purchase decision. With this advanced method, respondents are asked which ‘concept’ they would choose, and their individual preferences would then be inferred in the automated analysis.
Qualitative research
In contrast to quantitative research methods like A/B Testing and Conjoint Analysis, qualitative research allows for a more spontaneous, in-depth approach to gathering feedback - especially useful early on in the product or service development process.
Questions are open-ended, and follow the lines of: ‘What do you make of this concept idea?’, ‘Which specific elements of this concept do you like?’, or ‘Which parts of it don’t you like?’.
Consumer feedback gathered from qualitative research can reveal aspects of the concept’s appeal that a design team might not have thought of, which can be really useful for feeding into further quantitative rounds of research.
Back to table of contents
Types of concept testing survey questions
After determining your study goals, target audience, and types of advanced research methods you might want to leverage in your concept test, you’ll start to draft your actual concept testing survey questions that respondents will see.
Screening questions
Screening questions ensure that the respondents who complete your survey actually fit your target market credentials.
If you want to know whether a new sports sneaker concept will be successful in the market, there’s no point in asking consumers who don’t have any interest in buying or wearing sports sneakers. That being said, you might want your screener to allow for consumers who don’t wear sneakers themselves but do buy them for others. After all, you’re looking to get opinions from the person who will be making the purchase, whether they personally use your product/service or not. In this case, you’d ask a screening question like ‘Do you personally buy/wear sports sneakers?’ If yes, that respondent will continue on in the survey. If they answer no, you’d ask a follow-up screening question such as ‘Have you purchased (or plan to purchase) a pair of sports sneakers for yourself or someone else?’. If they still answer no to this question, this respondent is likely not a good fit to provide feedback on your concept and will be ‘screened out’ of taking the survey.
You’ll also want to screen respondents by demographics so that you’re selecting people who fall into your target audience age range or so that you have a representative sample group. You can set quotas on these demographical questions so that a certain percentage come from each age group, gender, region, etc. You can personally decide on those individual percentages based on your unique audience demographics, or by following a representative sample based on something like the US census (if fielding in the US).
Concept validation questions
Concept validation questions are the questions that gather feedback on your concept - both the good and the bad. They will be guided by the metrics you want to measure (e.g. relevance, usability, appeal, uniqueness...) and will give you a good idea of which parts of your concept are liked most by your target audience, which ones need improvement, and which should be dropped altogether. This feedback will often feed into further iterations of your concept, which can be put into subsequent rounds of quantitative testing until you narrow down your best-performing features.
The kinds of questions you might ask for concept validation are:
-
Which of the following best describes how you feel about this idea? (Love it, like it, neutral, dislike it, hate it)
-
How important are the following features? (Very important, somewhat important, neutral, not that important, not at all important)
-
How relevant is this concept to your lifestyle? (Very relevant, somewhat relevant, neutral, not that relevant, not at all relevant)
For a deeper understanding of consumer responses, it’s useful to ask ‘why?’ as much as possible within your concept testing survey. This might mean including some open-ended answer boxes as follow-ups to rating scales, using logic around their responses (i.e. if a respondent answers 'not that relevant' or 'not at all relevant' - ask why it’s not relevant). It’s useful to know the reasons behind people’s preferences as this guides future concept ideation/creation.
Purchase intent questions
What you really want to know when you’re testing product ideas is whether your target market is likely to buy the finished product. This is particularly important at later stages of concept testing when you have a polished concept with all its features that you want to take to market. Like the examples we saw above for concept validation questions, a purchase intent question is often structured according to the 5-point Likert scale (also see ‘Formats for concept testing questions’ below):
-
How likely would you be to purchase this product?
-
Would definitely buy
-
Would probably buy
-
Might or might not buy
-
Probably won’t buy
-
Definitely won’t buy
-
General market research questions
Getting a broader sense of the market into which you intend to launch your product will set the scene for how it will fit into a category. For example, is your assumed target market the right one? Might your product attract consumers you hadn’t initially anticipated? How will your product be used - alongside others or as a replacement?
You can ask consumers about their shopping habits and typical spending - anything that you want to know to give yourself a clearer idea of how your new product will fare in the market.
For example:
-
Which are your favorite brands [in X category]?
-
How much do you usually spend on this type of product?
-
When/how would you use this product?
Formats for concept testing questions
There are a number of ways in which concept testing questions can be formatted.
5-point Likert scale
The 5-point Likert scale is frequently used in market research as it gives a good overall idea of where respondents stand on an idea - as you’ll see in some of the example questions above.
This format consists of 5 possible answers, often structured as follows:
-
How much do you agree or disagree with [X]?
-
Strongly agree
-
Agree
-
Neither agree nor disagree (sometimes displayed as ‘neutral’)
-
Disagree
-
Strongly disagree
-
One advantage of the Likert scale is that it shows how many people are neutral about an idea, which in itself is revealing about how strongly consumers feel about your product or service.
This format can also be applied to other metrics, including likelihood to buy (very likely, somewhat likely, neutral, somewhat unlikely, very unlikely); frequency (every time, almost every time, sometimes, almost never, never); and satisfaction (very satisfied, satisfied, neutral, dissatisfied, very dissatisfied). These are just a few examples of how the answers can be written; you can include more or fewer levels of agreement/disagreement as you prefer, and tweak the language as it fits best to your question.
Ranking questions
Ranking questions ask respondents to rank attributes or other items in order of preference. For example, you might ask them to rank different features of a sneaker from most to least important - like color, style, price, etc.
You can decide how many items you want respondents to rank, and how many they have to rank (i.e. respondent must select their top 3 or respondent can select up to 3).
Rating questions
Rating questions are another kind of descriptive metric question that asks respondents to score a concept or its attributes in terms of appeal, relevance, importance, and so on. This can be a numerical score (1-5), a star rating system (i.e. 1-3 starts), or any other form of rating.
To see what ranking and rating metrics look like in a real study, check out quantilope’s grocery loyalty program study dashboard which compares these descriptive metrics to an advanced method output: MaxDiff.
NPS questions
NPS (Net Promoter Score) questions ask how likely people would be to recommend a product (in this case, based on a concept) to other people. NPS is a great way to gauge the value consumers place on an idea, in terms of their confidence in telling others to buy it.
To learn more about NPS, check out this page on the method.
Open-ended questions
As mentioned above, qualitative research is all about getting unbiased responses to open-ended questions, which is important for an authentic picture of appeal and purchase intent. However, you don’t always have to conduct a qualitative survey to ask these questions - you can include this style of questioning in a quantitative project too.
Instead of providing a list of possible responses to a question (like the Likert scale, rating scale, and ranking scale questions), you include an open text field and ask respondents to type in their answers.
It’s worth noting that open-ended questions are harder to analyze than pre-coded questions, so keep in mind the number of these questions that you include, especially if you have a really large sample.
Some examples of open-ended questions you could ask are:
-
What do you like about this idea?
-
What do you dislike about this idea?
-
How would you improve this idea?
How to conduct automated concept testing with quantilope
quantilope offers an automated approach to reliable product concept testing. Through its online Consumer Intelligence Platform, quantilope users can choose to start their concept testing survey based on a survey template or build their questionnaire from scratch with drag & drop modules. With a library of pre-programmed screeners, descriptive questions, and advanced research methods, users have high-quality concept testing right at their fingertips. Even survey templates are fully customizable to ensure your concept testing is uniquely tailored to your business.
Along with quantitative survey templates and a question library, quantilope’s qualitative video research solution - inColor, offers brands the opportunity to gather concept testing feedback in consumers’ own voices. inColor works by prompting respondents with video questions and receiving video answers back. The automated technology on the backend then analyzes each video for things like keywords, facial expressions, and sentiments.
As concept testing is often an iterative process, with successive versions of an idea being tested and built upon as consumer views come in, quantilope’s real-time platform is the ideal way to manage the concept testing process. Monitor consumer feedback as it comes in, share the study with colleagues or other product managers with a single link, and generate actionable dashboards for stakeholders all in one end-to-end platform.
quantilope offers three levels of research support based on researchers’ needs - from a DIY approach, to Do-It-Together with quantilope’s research team, to a fully managed study, you pick what works best for your needs.
To learn more about how quantilope’s concept testing tools can lead to sound concepts and successful product launches, get in touch below!
