The Benefit Real Life Testing Outweighs the Payoff of Looking Good

Having grown up in the more traditional pen and paper world of marketing, I have to constantly remind myself that it’s OK to make a mistake.   I have to keep reminding myself that not only are most marketing tests or published pieces not carved in stone — they aren’t even printed on PAPER!  So what am I freaking out about?

Testing by Launching

I’m not advocating rushing out into the world with just anything, but when you’re at the 80% mark and you need to do some testing — don’t be afraid of testing live or pre-launching your product or service.

That’s what Survey Analytics did with their new SurveySwipe product.  They had developed a mobile market research platform but needed some real life input to tweak the bugs.

They weren’t shy about it either – they decided to pick the State of the Union address as their test.  I think that was BOLD and very visible.  The risk to most of us traditional marketing types might be “BUT WHAT IF WE FAIL”  what if people can’t use the tool?  What if they think badly of us?  What if… What if…

Well – what if we TELL them it’s a test?  And what if we expect that there will be issues and what if we approach the whole thing as a game and as an exciting learning opportunity where we’re are all learning something together?  That would be a whole different kind of event.

The result might even be a more energized and excited consumer base — one that accepts that this is a test and can’t wait to see the bugs worked out.  WHAT A DIFFERENCE!

What Survey Analytics Learned

  • Mobile Platforms are Definitely Exciting. Based on the overall engagement — this is HOT HOT HOT!
  • Email or push – not both. This is a valuable lesson for all of us – how does your market want to be notified?  Be sure to ask what they want and then do it exactly that way.
  • Don’t be afraid to wing it. This is my favorite lesson.  The Survey Analytics team KNEW that one of the features was getting real time data.  Instead of playing it safe – they started PLAYING and polling in real time with their beta testers — asking questions on the fly!

You can read all the details of what they did well and what the Survey Analytics team learned from their SurveySwipe beta test.  But the real lesson here wasn’t in the technology, but in the mindset.

Today’s technology makes it virtually impossible to really test anything to perfection.   In fact, the standard practice for web content is to actually create several versions of the content and split test using tools like Google Web Site Optimizer.

Wondering what to charge for a product or service?  Chances are you can post a description and prototype on ebay at a variety of prices and see which sells best.

It’s clear that the old days of testing something to death have been overtaken by actually putting it out there and working out the kinks in real time.

How do you feel about that?  What are your tips, Do’s and Don’ts about when it’s ok to test a beta in real time and when more traditional pre-launch analysis is required?

Text Analytics – State of the Union – DiscoverText

DiscoverText has released the Raw-Data sets for the State of the Union Text text analytics portion. Dr Shulman is still working on the visualization of such a large data-set and we’ll keep you posted as things get updated. Meanwhile, If you;d like to play around with the data yourself please click on the link below:


Webinar Slides & Recording Available: DIY Market Research Trends to Watch & Take Advantage of in 2011

Thanks to everyone who registered and attended our webinar:

“DIY Market Research Trends to Watch & Take Advantage of in 2011”

We had a wonderful turn-out and discussed many trends from the death-watch of paper surveys, text analytics technologies, crowdsourcing research opportunities,, to the rise of online and offline mobile research platforms. Included in this post are the slides and recording to review and share.


Let us know what you think about these new trends coming down the pipeline. Do you think they will change how you currently conduct research?

Mobile Research – State of the Union – The DataSet

The mobile research initiative using SurveySwipe and partners like, Personal Democracy Forum, Netroots Nation etc. was conducted yesterday.

The methodology/process

We invited (though our partners) users to register and download a mobile app – SurveySwipe. The intent was to try out 3 ideas:

  1. Mobile Polling/Data Collection in general (iPhone, Andriod and Windows Phone 7)
  2. Pulse – On-the-Spot polling using Push Notification (iPhone Only)
  3. Dial-Testing on the mobile phone (iPhone Only)

Here are the results:

We did 5 surveys in all – A PRE, 3 Pulse surveys and a POST survey. I’ve already documented some of the lessons learned with this exercise. This post is about the data we collected.

The SOTU Panel Profile:

The Dial Test

We are still compiling the individual dial-test data – but this is the profile of the users who completed the dial test.

The PRE Survey

Pulse 1 – Seating Arrangement / Employment

Pulse 2 – Wealth/Afghanistan/Humor

Pulse 3 – Broadband / PUblic Transportation

Final – Post Survey

Dial Testing – Raw Data

We are still compiling the results of the dial test. We had ~300 users take the dial-test in real-time. We’ll be posting the raw data of the dial test also in the next day or two.

DiscoverText – Text Analytics

We are still waiting on Dr. Shulman to provide us with the data-set for the DiscoverText data for text mining. We’ll post it on the blog as and when it gets available.

Thank you.

How to Determine if Your Study Should be Conducted Online

The target audience for a survey is an important consideration in deciding whether it should be conducted online.  The target audience must have access and preferably use the Internet on a regular basis.  Until recently, senior citizens were generally not good candidates for online surveys because they were not as familiar with the Internet as younger groups.  Young children that are not yet able to read and type so they would not be good candidates for online surveys because they would need to be coached through the questionnaire.  When conducting business-to-business research, executive-level respondents may be difficult to recruit via email.  Therefore a phone survey may be recommended for certain audiences.

Online surveys were initially conducted among technology workers because they were the first to have a large portion of their population with access to the Internet.  Many companies were hestitant to use online research until recently because it was believed that not a large enough proportion of populations under investigation would be connected to the Internet.  As long as the target population is connected, it is safe to say that an online survey can be conducted and in most cases, is the most cost effective, time efficient way to go.

In research, answers are rarely black and white.  Because research is not a perfect science, answers are typically found through evaluting the different requirements, the pros and cons of an approach, objectively thinking about the impact on the business and research outcomes.  In most cases, it is quite simple to determine if a study should be conducted online, but with certain research areas, it may be more difficult to determine the proper method.

Business needs are the most important consideration when undertaking market research.  The business need drives the request for proposals and defines what type of research is to be conducted, whether it is exploratory, qualitative or quantitative in nature.  Once the business information needs have been defined, then study design can be established.

It is important to always adhere to strict ethical research principles when implementing a study.  This is one reason it is generally recommended to use a third party to conduct a market research survey.  The business needs can sometimes cloud the objectives at hand and a survey can be designed that leads respondents to specific answers.  A professional market researcher should always have the ability to push back if the business need impedes on ethical research design.  The Council of American Survey Research Organization (CASRO) rules outline standards of ethical conduct for the industry.  This information can be found at

It is important to understand the research implications of the survey design regardless of whether the survey is for an Internet or a telephone audience.  A sound methodology ensures that research principles are adhered to in order to report objectives and correct data.  The length of a survey and the impact on customer survey fatigue is one example of what must be taken into account.  The researcher wants to always ensure that the questions can be displayed in a manner that can easily be understood by the respondent.  Evaluating the potential affects alongside technology, administration and business needs will help determine whether a phone or Internet survey should be conducted and will help guide how certain questions are asked.

Technology has improved the quality, speed and efficiency of Internet surveys.  It is important that the programming rules for a study not be overly complex.  Single response, multiple response, scale, open-ended response or even grid questions are the most common types of questions seen in a survey.  Branching rules to skip questions based on respondent qualifications, substituting values into specific questions or answer, or auto-populating variables should all be standard features in a research tool.

Online research studies are almost always inherently cheaper than mail or telephone studies.  It is still important to factor in the cost of the research tool, the time saved doing the study in-house versus outsourcing, any extraneous costs such as language translations, or sample acquisition costs.  One important cost element is that some Internet survey tools have built-in processes that are not always the easiest to administer.  Outsourcing in any type of market research can not only give you piece of mind that your project is being handled by professionals, but that you will also get the best representative sample available.

About the Author: Michael Holmes is the President of EMI Online Research Solutions, an online research panel company with over 80 panels from 80 different countries.

Webinar 1/26/11 9AM PST: DIY Market Research Trends to Watch & Take Advantage of in 2011

Tuesday 1/26/11

9:00AM PST

Sign up here:

As January is speeding past us it’s not too late to learn about new DIY market research trends to look out for in 2011. Interactivity, collaboration and open source concepts are changing how we collect feedback. Marketing research is evolving. It’s more than surveys and focus groups. It’s more than those thousand dollar reports produced by Freedonia or The Conference Board. Today, finding out what your customer is thinking is a hands-on process.

SurveyAnalytics is proud to be a part of this free live webinar featuring panelists: Ivana Taylor, Marketing Strategist and Editor of; and Esther Rmah LaVielle, Chief Education Director of SurveyAnalytics.

In this fast paced session, we’ll be exploring:

– 2011’s Top research trends that have influenced how we collect feedback

– Resources where you can see examples of these trends in action.

– Tools that you can use today to get started

Register today to discover how your peers are leveraging insight into defined opportunities to innovate via today’s technologies for tomorrow’s research!

Sign up here today!

About the Presenters:

Ivana Taylor is the President of Third Force, a strategic marketing firm that helps small to medium sized companies get and keep loyal, profitable customers regardless of price.  She is the publisher of an online publication that provides small business CEOs with low cost marketing strategies, ideas and resources.

Esther LaVielle is the Chief Education Director at QuestionPro and SurveyAnalytics, which was started in 2002 in Seattle and is now one of the fastest growing private companies in the US. Prior to her adventure at SurveyAnalytics she spent 3 years as a market research project manager

Mobile Research – State of the Union Pulse – The Good/Bad and Ugly

So, we just finished up with our first mobile pulse polling project for the State of the Union. This was our first app based real-time research project that we did. I’ll document some of the issues bugs and ideas that got generated as part of this.

Emails + Push Notification

We decided to send email as well as push notification (for iPhone users) — turns out this pisses people (and confuses people) – we should _either_ send push notifications or send emails. Sending both does not make sense (of course in hindsight.)

Opt-Out link on Email:

The Opt-Out link was not getting appended to the emails. Again a rookie mistake. We were so busy focusing on the iPhone/Andriod user experience, we missed out the opt-out piece on the emails. We also need an opt-out on the app itself. Deleting the app (from a technical standpoint) does not help because, we are not guaranteed notification of the deletion of the app. That is a simple technical issue — will be fixed next time around.

Multiple Emails:

OK – some reason some folks were getting multiple emails for each survey. The way we set this up was to send out a “pulse” when we did a survey. I think when we do these live polling models, we should NOT be sending emails – just rely on push notification. A single email with a launch would probably have sufficed. We ended up sending about 5 emails (we did a PRE, 3 pulses and a POST survey) — ok – probably too much – but we had to keep all our partners also on board. Next time, we’d probably limit the PRE event survey to about 5 questions — pulses are not emailed and then a post-event survey also via push notification.

The data-set:

This is where things became interesting. Each of the pulses were filled out almost instantly. We got 85% of the data within the 5 minutes of sending the pulse – . We’ll be sharing the data-set after we scrub the data for privacy artifacts like IP addresses etc.

The other thing we did was to create the pulse questions on the fly. All the partners had a group chat session and we decided on the pulse survey. This is probably where we got carried away and we sent about 3 pulses. For example, after seeing the seating arrangement, we pulsed the question around the seating arrangement. This is where real-time polling gets interesting — where the questions can be defined and executed in real-time.

Top Line data:

We did 5 Surveys all up: PRE, 3 Pulses and a POST. IN addition we also had a dial test running in parallel. (Yes a lot of stuff to happen – but this was a first pilot and we wanted to push the limits to see where the limits are!) If we did not push the  limits, we would never know where things break. We had told everyone that this is a beta test and the first time someone has even attempted (at this scale) a mobile research project of this scale. We had over 20 partners and collectively deciding the questions on the fly on a real-time chat system and pushing the surveys to the users.

Here are some top line reports:



I’ll be posting more data as we clean things up.

Finally, The mobile software SurveySwipe is still in beta; in many ways this project tonight was a full scale field test. With that in mind, I am not surprised that there may be a few bugs yet; I’ve yet to see any of the mobile survey apps that don’t have them. I suspect that the opt-out challenge is just part of that beta process.

SurveySwipe Lets You Tell Obama What You Think of His Speech – IN REAL TIME!

There is a really exciting project going on this evening during the President’s State of the Union Address — YOU will have the opportunity to literally DIAL your response to the speech from your smartphone!

If you’ve got an iPhone, Android, Blackberry or Windows device, you can DOWNLOAD THE MOBILE APP HERE.

Then watch the speech and literally swipe your reaction as the president is speaking :

That is ALL THERE IS to it!

Ahead of the address, people who have downloaded the app will receive a push notification asking if they would like to participate in the State of the Union pulse poll. SurveySwipe will then launch, and the user can agree or disagree with what is being said by during the State of the Union by moving a slider up and down a scale on the mobile screen.

The idea of pulse polling is similar to “dial testing”  to find and study the phrases, the moments, and the inflections in a speech.

With SurveySwipe, “we want to make the technology cheap and simple for everybody to use,” says SurveyAnalytics’ Rob Hoehn. Outside of State of the Union, for example, a small nonprofit could do live polling at their event, so long as the attendees or participants have smartphones and have downloaded the SurveySwipe app.

This is a new and exciting customer survey technology that takes advantage of the power of mobile devices and the wisdom of the crowds.

You Can See How YOUR Community Will React to the Message!

Here’s the really exciting (or scary) part- depending on your perspective.  The person who owns the project (i.e. the state of the union in this case) will have access to how the different communities actually react to a given event.

This project is a joint effort between SurveyAnalytics, SurveySwipe, Cooper Strategies, and UMass Amherst. The SurveySwipe solution is compatible with iPhone, iPod touch, and iPad devices as well as most other major smartphones including WP7, BlackBerry, and Android. The SurveySwipe technology was combined with UMass Political Science professor Stu Shulman’s research tool – DiscoverText™ to analyze the data from social networks in addition to data provided by participants over their devices in real-time.

All data collected in response to the State of the Union address will be “open-sourced.” It will be licensed using the Creative Commons Attribution License and will be available to the public.

“It’s very important to us that we make these innovations both valuable and available,” said Vivek Bhaskaran, President and CEO of SurveyAnalytics, “Everyone that creates a community for the State of the Union will have access to their own data, as well as an aggregate view of data across all communities. And instead of the results being generated by a small sample of a few hundred people, the results will reflect a sample of thousands more.”

“The explosive growth of smartphones makes these devices the perfect platform to collect real-time, participatory feedback during live events,” says Hoehn, “It’s also become more and more important to understand how people feel, think, and respond as events occur and it’s changing the way that brands, companies, and blogs engage with people.”

How to Participate:

  2. Then watch the state of the union
  3. As the president speaks literally swipe your finger over the scale — keep it to the left (low – Nah, you are full of it or I don’t like what you’re saying or how you’re saying it), move it to the middle (ok, or neutral) or move it to the right (I agree completely)

Then stay tuned for the results- we’ll have them published for you right here!

Hot Trends in Business to Business Marketing

The DIY Marketing Trends is a hot hot hot topic these days — in fact, if you want to make sure that you’re on top of what’s going on in the world of DIY Marketing, it would make sense to take a look at what trends are driving business-to-business marketing overall.

In full self-disclosure, this list comes from a series of conversations that I’m following over at a wonderful community of small business experts who ask and answer small business questions.

The first question I’m going to summarize is about what topics will be hot on the marketing blogosphere.  When I looked at the answers, I realized that these topics were really broader trends that will affect the world of Market Research.  Also notice how these question and answer formats are basically research in and of themselves that can be analyzed as “social media chatter.” If you want to learn more — you’d better register for Wednesday’s Webinar NOW.

Trend: In Bound Marketing: If you’re not familiar with the term, it means using marketing strategies and methods that attract your ideal customer to your web site and get them to register or purchase what you are offering.    Pushing your marketing message with an obvious in-your-face-advertising campaign isn’t very popular these days.  Consumers are savvy searchers and only want to receive information from companies and products that they are interested in.

Research Tip: Get to Know Your Ideal Customer Put your research focus on getting to know your respondents and providing them with the information that will help them get educated on how to choose .  Make sure that the “Why they should choose YOU” is highly linked to what your ideal customer values.

Trend: Right Place Right Time.  As if profiling wasn’t enough – you’ve got to hit the right person at exactly the time when they are most likely to be open to what you’re offering.  This is the equivalent of becoming a genie in a bottle hanging off your customers’ key chain.  The company, product, person or offer that is THERE in front of them at the exact time that they are ready to purchase will get the sale.

Research Tip: Find out what triggers your customer’s need for your product. A really great way to do this is to interview your ideal customers and get to the root of what events got them to start searching for your product, how they got choose you and why they choose you.  Once you have that profile set – you can easily put together a survey instrument that takes a snapshot of your target market and identifies those people who are experiencing the trigger event that signals the need for your product.

Trend: Word of Mouth Marketing Evolves. Word of mouth marketing is how busy buyers weed through the myriad options that they are bombarded with daily.  It takes too long to evaluate hundreds of products via the web.  It might just be easier to ask your LinkedIn group or expert communities what they would recommend.

Market Research Tip: Provide Data. There is an opportunity for savvy market researchers to use their skills and resources to profile and research the communities that they know best.  Maybe you can do research on specific market segments and create a public data set (register for the  DIYMarketing Research Trends webinar to learn about this one)

These are just a few trends that smart people are chatting about.  I hope that you’ll join us tomorrow at 9am Pacific and 12pm Eastern to learn about more DIY Marketing Research Trends