Concept Testing and Validation: Methods, Survey Design, and Examples

In this post we explore concept testing and concept validation, both highly valuable research approaches to screen ideas before they go live.

Table of Contents: 

• What is concept validation?
• What is concept testing?
• Benefits of concept testing
• Concept testing methods
• Concept testing use cases
• How to validate new ideas using concept testing
• Concept testing examples
• Concept testing best practices
• Concept testing mistakes you should avoid

• Get started on your concept testing with quantilope
  
  

What is concept validation? 

Concept validation is the initial stage of development where you gauge the overall potential of an idea - rather than gather feedback on specific features and prototypes (which is reserved for actual concept testing). Concept validation might involve brainstorming sessions, internal discussions, qualitative research like focus groups, preliminary quantitative online surveys, or a mix of each approach to understand if the concept idea addresses a true consumer need and aligns with business goals. In many cases, concept validation helps decide if the idea is worth pursuing further through more rigorous concept testing.

Concept validation can be used across a range of concept types, such as website layouts, ad campaign ideas, logo drafts, or marketing messages.
Back to Table of Contents

What is concept testing? 

Concept testing is a research approach focused on obtaining feedback for products, services, or even marketing campaign pitches. This helps businesses avoid investing heavily in offerings that won't actually resonate with their target market once they go live. Starting with a well-defined concept, brands will present their offer to a panel of research participants using descriptions, visuals, or prototypes. They'll collect consumer feedback through methods like surveys, focus groups, or interviews, assessing the concept's overall appeal, value proposition, features, and pricing. Many times, brands can leverage an advanced method like an A/B test in their concept testing study to compare versions of ideas to see which has the most potential and what learnings can be gained from each individually.

From the study, insights teams can determine whether to continue with their new development process, iterate and re-test, or pivot their approach entirely.
Back to Table of Contents

Benefits of concept validation and concept testing 

Brands use concept testing because it effectively allocates budget to the right concepts, at the right time, for the right group of consumers and their needs. If you have a number of potential product routes in mind and don’t know which one to invest in, concept validation is the best way to identify the strongest ideas and prioritize those concepts to work on further. You might also find that some elements of concept A work well, but concepts B and C contain other appealing elements that could be combined to create a whole new concept for further testing - showing the benefit of developing multiple concepts for testing.  

As well as testing the inherent ideas and elements within a concept, concept validation also helps pinpoint motivating language to be used throughout later concept descriptions. For example, if you’re thinking about testing a new body spray, respondents might be noticeably drawn to simpler words like ‘fresh’ or ‘clean,’ but less attracted to words like ‘reviving’ or ‘invigorating’; this helps guide product decisions around packaging and communications both for the current concept test and potentially for future ones as well.

After all, it’s said that over 30,000 new products are launched each year, and 95% of them fail. Concept validation (and later concept testing) will help put you in the best position to land in that successful 5%.

Concept testing specifically examines elements like whether an idea solves a genuine problem for the target market, how it stands out from competitors, which features are most appealing, and if the price aligns with perceived value. It reduces business risk by highlighting potential issues before major investments and allows businesses to focus resources on the most promising ideas. Concept testing can also be used to identify improvements for existing products - when product concepts have been in the market for a while, it can be easy to lose sight of what the consumer really wants. 

Concept testing ultimately builds a stronger customer connection by demonstrating a commitment to fulfilling unmet needs. This is especially important in today’s crowded marketplace, where it's becoming harder for products and services to stand out with consumers having access to so many choices. 

Below are just a few examples of how concept validation and testing can benefit your business: 

Improve products and services 

Successful service and product launches come from thorough research to make them the best they can be. There are several areas of the concept that a business might want to research, including:

• Features: what the concept is offering that is useful, original, or otherwise interesting to the consumer
• Brand fit: whether the idea feels like it belongs to the brand, and whether it fits with the brand‘s image
• Usage occasions: when consumers can see themselves using the product/service, and whether these are numerous and varied enough for the product to achieve high enough sales
• Fit in competitive set: how the concept differs from other products or services in the category, positively or negatively
• Price: what consumers would expect it to cost, and why - as well as reactions to any price points included in the concept

Gain actionable insights

Insights from concept testing help guide researchers from an early-stage product idea to a proposition that's market-ready. For example, a product might have an initial positive response, but consumers may suggest improving the colors and font, making it even more preferable. The more positive a response the concept can generate, the better it will theoretically perform in market. These kinds of tangible, actionable insights feed into the development and refinement of concepts - guiding more effective marketing strategies and future projects.

Be cost-effective

Get buy-in from stakeholders

It’s not unusual for team members to differ in their opinions on which concepts will be most successful with consumers. The only people who can really tell you whether they’d buy something are consumers themselves. Even if they don’t like it personally, once stakeholders have data-backed evidence that a concept is appealing to consumers, they are much more likely to support it.
Back to Table of Contents

Concept testing methods

When it comes to concept testing, there are many methods one can leverage to capture actionable feedback from respondents. Below are a few examples: 

Monadic (A/B test)

With a Monadic (A/B) test, respondents are shown a single concept and typically asked to rate it in a number of different areas (purchase intent, brand fit, uniqueness, etc.). The advantage here is that respondents devote time to giving thorough feedback on the concept they see, with no bias affecting their responses as they don’t have other concepts with which to compare it. The response rate tends to be good, as evaluating just one concept means a shorter questionnaire, allowing them to stay fully engaged for the entire experience. On the downside, this concept testing method means a greater number of respondents must be recruited to get enough feedback for all of the concepts. Costs are therefore slightly higher and fieldwork could take slightly longer than other methodologies.

Sequential monadic

With this method, respondents see a single concept at a time and answer questions about it before moving on to the next one. This still allows them to consider each concept separately and in detail (like a Monadic A/B test), but also introduces the element of inherent comparison - which some brands may be aiming for. Because of the introduction of multiple concepts, some sequential monadic tests can become lengthy which has the potential to affect completion rates. Other elements to consider with sequential monadic tests are the potentials for first-position-bias and respondent fatigue. 

Comparative

This method involves showing several concepts to each respondent before asking them to answer questions. This approach asks respondents to compare different concepts, considering the advantages and disadvantages of each. It can be very effective in understanding key successes and failures in concepts and provides fast fieldwork with high response rates. However, with comparative testing, it's less feasible to gather the finer details of each concept.

Proto-monadic

This approach begins with a sequential monadic method - several concepts are seen in turn and each one is evaluated before moving on to the next - followed by an overall evaluation of which concept is the best one. While it has similar disadvantages associated with sequential monadic testing, it does lead to an understanding of preferred concepts.

Qualitative

In contrast to the first four quantitative methods described above (which capture numerical responses from respondents), qualitative research (open-forum feedback) gives greater leeway for spontaneous reactions to a concept. You can ask open questions such as ‘What do you think about this concept?’, ‘What is your favorite idea within this concept?’ or ‘Tell me about when you might use this product.’ Qualitative research is often more effective in the earlier phases of the product development cycle to gather initial feedback and help craft concept details. Qualitative research also tends to be done amongst a smaller sample size than is used for quantitative online surveys, yet it provides rich information that might not be captured using a questionnaire with pre-defined question parameters.
Back to Table of Contents

Concept testing use cases

Product development

Designing new products or further improving existing ones can be a challenge. Which features should it include? Which characteristics are most important to consumers? Does the color matter? Which price point will make the product most competitive? Putting product concepts in front of respondents and gaining their reactions will quickly tell you which areas you need to focus on to build a successful offer.

Imagine you have an idea for a new smoothie. There are endless flavor combinations, ingredients, and benefits that you could offer. To narrow down the possibilities, you could start with concept validation by conducting focus groups or in-depth interviews among those who buy smoothies. You would explore how satisfied they are with existing smoothie products in the category and what they typically look for in a smoothie (taste? a certain consistency? health benefits?). You could ask participants to create their own ideal smoothie concept and use these insights in a more built out concept test later on.

Logo design

A logo is such a crucial piece of branding to get right. It should be recognizable, instantly associated with your brand, and say something about your brand values. Logo testing with your target audience helps determine which designs stand out and communicate the right message about your brand.

There are millions of logos out there, each evoking a different emotion, response, or recognition among consumers. Logo concept testing will help you decide on a logo that stands out in your category, communicates what you want to say about your product (e.g. the finest ingredients, freshness, great taste), and is recognizable in stores and on marketing communications.

Website design

Brands can screen their concepts for website design to ensure the layout, navigation, content sections, and color themes are a good fit for your brand and for the purpose of your site. Concepts can be limited to screenshots or brands can opt to trial more sophisticated digital mock-ups.

Package testing

Different options for packaging - whether they are images or mock-ups - can be shown to respondents for their opinions on standout, brand fit, and general appeal. Testing mock-ups has the added advantage of exploring potential usage, material preference, and size perceptions.

Continuing with the above example of smoothies - say you’re now at a stage where you’re starting to think about smoothie packaging. Should that smoothie be sold in a carton, a pouch, a plastic bottle, glass, or something else? What should the opening mechanism be - twist off, screw cap, straw? How do environmental concerns, ease of use, and pricing expectations feed into people’s preferences?

Concept testing for packaging will tell you which packaging design fits best with the brand, what different packaging options communicate about your product, and how each packaging concept is viewed among consumers.

Name testing

What’s in a name? A lot, if you’re a company or product that wants to communicate your strengths and make sure people remember you. To be the next Adidas, Hershey’s, or Twitter, it’s imperative that you get feedback from customers to make sure their associations with your name options are positive and what you intended.

How do you make your smoothie brand and/or product stand out amongst the array of competitors on the market? Which product name will resonate most with consumers, fit with what your brand is all about, and be memorable enough for your consumers to search for it on the store shelf? Names are an important part of the marketing mix, and you need to make sure you choose a strong one that is relevant. You might be surprised at the associations people have with different names, making them more or less appealing.

Name concept testing will tell you what each idea communicates, the associations they each conjure up, and which is perceived as the most original and attractive within the category.

Marketing messages

You might have your smoothie recipe and packaging solidified, but how are you going to tell your target audience about it? Which language will you use to make sure perceptions of your brand are aligned with brand values? Which imagery will work best to make consumers crave your smoothies? Which messages will strike a chord with consumers, which words will capture their interest, and which will tell them that this product will fulfill their needs gaps? These are all elements to include in a marketing concept testing study - perhaps leveraging a Conjoint, A/B test, or some other advanced methodology.

Pricing structures

Getting the price point right for your product is crucial. You might have a great product with eye-catching packaging and a strong marketing campaign, but if the price is too high it will lower purchase intention, and if it’s too low it will affect quality perceptions.

Pricing concepts can be tested through a variety of methodologies - like PSM/Van Westendorp or Conjoint, each eliciting assumptions that consumers make about the quality and type of product you are offering.
Back to Table of Contents

How to validate new ideas using concept testing

Knowing the benefits of running a concept test and when you might do so, below are the steps brands will typically take when starting their study: 

Start with your concepts 

Remember, the quality of your research will depend on the quality of the concepts you put into it. While you don’t need to have a fully- developed prototype to start your concept test, make sure you have a way of clearly communicating your idea to survey takers (e.g. a detailed description, mock-up illustration, etc.). 

Make sure you have considered the following:

• If the concepts are written, are they written clearly, with no ambiguity? Written concepts should present the overall idea of the product/service, its benefits (in relation to the consumer need), and the unique selling point (USP). 
  

• Are the concepts concise? (i.e. don‘t pack too many ideas into each concept or make each concept difficult to fully grasp)

• Are the concepts consistent in what they communicate to respondents? (Make sure the information and benefits of each concept are comparable - so if you’re testing washing machines, make sure the number of wash cycles, type of cycles, size, color etc., are included in each concept)

• Are all concept visuals high quality? A poor quality stimulus can impact perceptions and take away from the actual concept being tested.  

Establish your hypotheses and desired metrics

With each concept you’re testing (if testing more than one), consider the desired metrics you want to come out of your research. Maybe you want to find out your concept’s appeal (is there a perceived needs gap it will fill?), its viability (will it sell enough units to make it an attractive business proposition?), or its uniqueness (does it solve a problem for customers that competitive concepts can’t?). The objectives will obviously differ depending on what you’re researching. For example, logo research will ask about the appeal of the overall design and the values it communicates, while website research might ask about usability and clarity. These consumer-based metrics will help guide internal decisions later on, like manufacturing feasibility and pricing - given that the perceived success of a product will dictate the resources you can put toward creating it.

Use your desired metrics to formulate a hypothesis (or, hypotheses) that will guide the rest of your concept testing project.

Decide on your research approach

Concepts can be tested using quantitative or qualitative research - and many times, brands opt for both!

Your choice of research approach depends on the stage of your concepts for testing. If you have ideas that are more early-stage/abstract, you might benefit from starting with qualitative concept validation research to gather feedback and opinions that will help shape up your final offer for further testing. Or, say you have a number of defined concepts that you want to narrow down; using qualitative research to discuss the pros/cons of each can help curate a solid list of final concepts to test in a quantitative study. Qualitative studies might involve focus groups (to generate discussion and debate) or in-depth interviews with individuals to delve more deeply into the view of the target customer. Qual often works really well to help determine if there truly is a need for the product or if a product is truly solving a problem. 

If your concepts have already been through qualitative testing or have been honed enough for them to be fully understood by respondents, then you might be ready for a quantitative study. Quantitative research today is mostly in the form of an online survey. The advantage of quantitative research is that you have access to a much larger number of respondents (compared to qual) with statistically reliable analysis. The other benefit of using a quantitative study is the opportunity to leverage advanced research methods like TURF, MaxDiff, Conjoint, and A/B Testing to validate your concept ideas.

Ideally, two or more ideas should be tested per concept study for better chances of optimization; the appealing elements from each idea will provide guidance on what a final winning concept would include. For a greater level of detail on the various merits of each concept - providing there isn't a long list of concepts to test - a sequential monadic approach is a good approach so respondents can focus on giving authentic, genuine responses for each concept before moving on to another. 

Design your survey questions

For qualitative research, this will be a discussion guide: a list of topic areas that you want to cover during focus groups or in-depth interviews. Topic areas will include respondents’ existing beliefs and attitudes, relevant product usage, and any needs gaps they might have. Questions will take an open-ended form - for example ‘tell me about looking after your teeth’ (if your concepts are dental care related) or ‘tell me what you think of this concept overall.’

While there is room for open-ended questions in a quantitative questionnaire, survey results are easier to analyze if the majority of questioning uses scales or ratings-based formulations. This might include likert-scales (those with a pattern such as ‘strongly agree,’ ‘somewhat agree,’ ‘neither agree nor disagree,’ ‘somewhat disagree,’ ‘strongly disagree’) or other scoring systems. The use of single/multi-select, scales, and ratings questions, allow respondents to score/rank ideas or to choose from a list of pre-set answers. Answers to questions like these are easier to quantify than qualitative surveys, and statistical comparisons can be made across a large group of people.

Some ideas to include in your questionnaire might be:

• Overall appeal of the concept

• Likes/dislikes

• Potential usage occasions

• Relevance to consumers' needs (does it fill a need gap?)

• Price and value perceptions

• Intention to purchase/use

• Alignment and/or impact on brand imagery or brand consideration

• Agreement with key messages
  
  

To continue with our dental example above, some quantitative questions around a dental care product may be:

Which flavors do you prefer in a dental care product? [Select all that apply]

• Peppermint

• Spearmint

• Berry

• Citrus

• None of the above 
  
  

How appealing is the following dental care product packaging? [Likert scale]

• Very appealing

• Somewhat appealing

• Neither appealing nor unappealing

• Somewhat unappealing

• Very unappealing

Rank the following colors you’d prefer in a dental care product [ranking question]

• White

• Pink

• Blue

• Striped

• Green

Within your survey, you will also need to collect information on your respondents’ demographics, attitudes, and category purchase behavior. This will help you understand which consumers are most likely to respond positively to your idea once it has been launched, as well as to target communication effectively by consumer type.

One large benefit of concept testing is that questionnaires can (and often are) templated, making efforts much more efficient each time you are looking to run a similar type of testing.

Choose your sample

Once you decide on a methodology and build your research project around your research hypotheses and metrics, it’s time to set up your sample (aka, the people who will actually participate in your research).

The people who review your concepts need to be ones that represent the consumers who are most likely to buy the finished product or service. This means you should send your concept testing survey to people that represent the concept’s desired demographics (age, gender, income, geographical location) and behavioral profile (those who buy existing products from your brand/competitors, or those who undertake activities with which your concept/product is associated).

You can choose to recruit your respondent sample through a panel provider or send your survey to your own business-supplied-list of participants (though remember this might mean missing out on competitors' customers or potential customers not currently buying into your category).

Launch your survey and analyze responses

Once survey results are in, study the commonalities as well as the contrasts in responses.

For quantitative research, this is where you can measure scores and other feedback for each of your desired metrics to learn where each concept’s strengths and weaknesses lie. Statistical testing comes in handy here to point out major differences between concepts as it relates to demographics, features, etc.

For qualitative research, listen back to interviews and review your notes for common highlights or overall themes.
Back to Table of Contents

Concept testing examples

Companies who launch successful products, services, logos, and websites, know how important effective concept testing is. Often they have to work through numerous unsuccessful concepts before they narrow down ideas to the one(s) that will perform well. Magazine publishers test their logos and page designs because they know these are instrumental to the magazine’s personality; haircare brands test concepts for new products to make sure they address the needs of their consumers; and so on. 

Burger King's Satisfries

Sometimes, companies act off their gut instinct but later wish that they‘d done research first. An example of this kind of outcome is Burger King’s Satisfries, which appeared on their menu in 2013.

Positioned as a lower calorie alternative to regular fries, Burger King hadn’t foreseen that lower calorie wasn’t necessarily perceived as more healthy by health-conscious customers. These customers wanted to forgo their diets on the rare occasions that they visited Burger King for a treat, and the higher price point compared to regular fries didn’t help either. For many consumers, they didn't fully understand why they were getting fewer, worse tasting fries, for a higher price point - due to a lack of and/or clarity of Burger King's marketing efforts.

In the end, consumer reception of the product was too low and a year after launch, Burger King had to pull Satisfries from their menu. Had they run a concept test, Burger King would have saved a huge investment. 

Lay's "Do Us a Flavor"

On the other hand, Lay's (Frito-Lay) successfully leveraged concept testing with their "Do Us a Flavor" campaign. Recognizing a consumer desire for new experiences and brand interaction, they launched a contest inviting customers to submit their own flavor ideas for new potato chips. Prior to the public contest, Lay's likely conducted internal concept testing to select a promising shortlist of flavors. Then, the consumer voting and social media buzz surrounding the campaign acted as a large-scale concept test. This allowed Lay's to track the most popular flavors and gather feedback, resulting in millions of submissions and votes.

The campaign was a hit due to its focus on customer involvement, the novelty factor, and a built-in system for gathering direct market insights about flavor concept preferences before moving on to actual product development.
Back to Table of Contents

Concept testing best practices

When conducting your concept testing study, below are a few best practices to be mindful of.

Make your concept testing iterative

Start testing ideas as soon as you have them. The earlier you do so, the more quickly you will catch onto ideas that work and identify those that don’t. Then, as your ideas develop, keep testing and building upon each iteration of your concept.

Have a plan before you start

Get the most out of your concept test by thinking carefully about your target audience, the most appropriate research approach/methodology to use, and which metrics you want to capture. Having the right sample, approach, and metrics will make your research reliable and actionable - rather than jumping right in without any thought.

Benchmark the findings

Decide which thresholds you want to use in determining whether a concept has potential or not. If you have five concepts and want to know which one is the most original, the winner might still only receive a score of 6/10 for originality. Use this as a benchmark for future iterations of testing, to see if any new ideas beat the original winner.
Back to Table of Contents

Concept testing mistakes you should avoid 

Now knowing best practices to put forth in your concept validation research, there are also a few things to be mindful of not including:

Don’t test too many concepts

While it’s great to test several concepts against one another to get an idea of what works and what doesn’t, be careful not to test too many at once among the same group of people. It’s generally best to stick to 6 concepts at most in one test - any more than that and respondents may start to become fatigued. Choose concepts that represent a good mix of elements. You can always run subsequent iterations of concepts to narrow in on final idea.

Don’t ask the wrong people

You might get great feedback on your concepts, but if the people you ask don’t represent the ones who will make the actual purchase decision or aren’t in the market for your idea, it’s time and money wasted. Make sure your sample has sufficient interest in your category and sufficient motivation to pay for the final product you’re testing.

Avoid leading or biased questions

Avoid using questions that suggest a desired answer or frame a certain concept in an overly positive or negative light. You can do this by keeping questions neutral and/or open-ended, and ensuring multiple team members review the study to cross-check any potential biases. 

Don't wait too late in development

Waiting until the product or idea is almost fully developed to conduct concept testing leaves little room for major changes. To avoid this, test your concepts early and often. Test rough sketches, initial prototypes, and core concepts before major investments are made.
Back to Table of Contents

Get started on your concept test with quantilope

Concept testing is an essential component in developing successful brands, products, and communications. It offers valuable insights into consumer perceptions and how their perceptions will affect actual performance in the market.

quantilope’s Consumer Intelligence Platform automates the concept testing process, providing advanced, real-time insights around your concept ideas. Start by using quantilope’s pre-built (yet fully- customizable) template or drag and drop question modules to start your survey from scratch. Additionally, quantilope's inColor solution, a video-based qualitative tool, allows respondents to see, feel, and potentially even use concepts before giving their video feedback.

Connect to any panel of your choice within quantilope’s fielding tab and begin analyzing results as soon as fieldwork begins, using AI-driven analysis for chart titles and descriptions. Save your charts in the reporting tab to include in a final, shareable dashboard with an AI-generated insights summary (thanks to quantilope's AI co-pilot, quinn!).

To learn more about concept testing with quantilope, get in touch below!