So! Continuing with the #stuffiwonder theme…

The telephone vs online survey debate.

The one that goes;

“Really, given that everyone’s moving from landline to mobile/cell these days, telephone survey sample representivity is seriously compromised”.

More often than not (and, of course, depending on who’s doing the debating), it ends with a nod to online panel surveys. In this context, “…they’re probably just as good as – if not better – than telephone surveys”.

Right?

Well, I don’t know.

Panels are opt in. And yes, the same can (and should) be argued about telephone interviews. You most definitely need research participants to opt in beyond a “Bugger off, I’m eating my dinner” response.

But what differences might we see, in terms of motivation and the research output, between a sample comprising individuals who;

• Have been approached randomly (and I get that it’s not really random; the population will be limited to those with landlines), vs
• Sign up to be part of a/several market research panel/s and/to get paid for their opinions?

How do market research online community providers populate their clients’ communities?

(When I say “market research” communities, that’s exactly what I mean; a community used as a market research tool. I’m not talking about online communities that are used in a marketing/customer relations exercise.

I’m not quite sure that the difference is apparent to all, but they’re not the same; not by a long shot.

In one, you’re giving the community members love because you want to make them happy. In the other, the relationship is somewhat more pragmatic; you want to learn from them. Notably, if you’re giving them love to make them happy, you’re not necessarily going to learn much, because they’ll be all nice and lovely back).

Anyway, focusing specifically on market research communities; what checks are in place to ensure that the people who end up in the community represent the people the client actually wants to hear from (ie the population of interest)?

To borrow from the delightful John Lacey, I’m filing this one under #stuffiwonder.

Maybe I should stick to knitting, but despite my obvious leaning towards qualitative (vs quantitative) research, I feel compelled to write about a quantitative research issue that isn’t quite getting the consideration it should; sample representativeness. This, I might add, seems to be a problem particularly – although by no means exclusively – for quantitative research conducted in the online environment.

Sample representativeness

When we (when I say ‘we’, I don’t mean you or me of course, I mean them, but let’s go with ‘we’) undertake a quantitative market research study, it’s rare that we have either the time, or a budget, that would allow us talk to each and every customer/potential customer of interest.

Instead, we choose a selection of those customers/potential customers to represent the greater population of interest to us. In research-speak, this selection is called the sample.

In a quantitative research context, the way you choose your sample, and the structure of that sample, is everything. These two factors will pretty much define the extent to which you can extrapolate your research findings to the population of interest. In non-research speak, that means the extent to which you can have any confidence in the research results.

Making sure that the sample you want to use to generalise to the greater population of interest is representative of that greater population is;

1. Of vital importance

2. Not always easy

3. Of vital importance

Number 3 isn’t a typo. Issues arising from number 2 often mean that the sample may be seriously compromised. I put number 3 there as a reminder.

So if you’re reading this, I’m assuming you didn’t go for the Sony Bravia.

Following on from the last post, good, useful qualitative research output is sample dependent.

If you aren’t talking to the right people, then even the cleverest, most innovative techniques in the world won’t help.

So how do you make sure your sample is a good one? There are 3 things to keep in mind:

- Sample definition

- Recruitment

- The research dynamic

Sample definition is a job for brand and product managers/marketers. Why? They typically know their business best, and it’s absolutely paramount that the sample is aligned with their business and marketing objectives. Anything short of a sample defined according to these objectives will be sub-optimal.

Recruitment is another important factor in getting the right sample. In this case, it’s down to a sharp screener and a fine recruiter (albeit, to get the well-screened and well-recruited participants to actually show up, you really just need luck).

Finally, the research dynamic. The aim here is to manage the research dynamic so that you get the best out of your sample. For example, deciding whether or not it’s appropriate to mix men and women in any particular group, or the best way to split ages across the groups, etc. These considerations play an important role in determining the difference between useful and useless research.

The zebra bite? A well-defined, well-screened sample, set to ‘work’ in an appropriate forum is the starting point for good, useful qualitative research.

Next time – fieldwork!