Coincidentally, after my last post, this came up in my Google Reader today;

More on the Problems with Opt-in Internet Surveys

Here’s the first article;

Study Finds Trouble for Opt-in Internet Surveys

I had the great privilege of attending an AMSRS Professional Development session earlier this year to hear Jon Krosnick speak. He was brilliant.

And so it’s with great interest that I follow this very timely and fascinating debate (hosted on Gary Langer’s excellent blog).

Make sure you follow the links to get the full story/debate. It’s an important one.

P.S And here’s a link to the study itself.

So! Continuing with the #stuffiwonder theme…

The telephone vs online survey debate.

The one that goes;

“Really, given that everyone’s moving from landline to mobile/cell these days, telephone survey sample representivity is seriously compromised”.

More often than not (and, of course, depending on who’s doing the debating), it ends with a nod to online panel surveys. In this context, “…they’re probably just as good as – if not better – than telephone surveys”.

Right?

Well, I don’t know.

Panels are opt in. And yes, the same can (and should) be argued about telephone interviews. You most definitely need research participants to opt in beyond a “Bugger off, I’m eating my dinner” response.

But what differences might we see, in terms of motivation and the research output, between a sample comprising individuals who;

• Have been approached randomly (and I get that it’s not really random; the population will be limited to those with landlines), vs
• Sign up to be part of a/several market research panel/s and/to get paid for their opinions?

Are you measuring what you think you’re measuring?

Raise your hand if you can think of some brands that need to work on customer engagement.

And/or raise your hand if you can think of some brands that should be (more) transparent.

And/or lastly, raise your hand if you can think of some brands who should join the ‘conversation’.

If you’ve got your hand up, maybe you can answer this;

What, exactly, do you mean?

Go on. Define ‘customer engagement’, ‘transparency’ and the ‘conversation’.

No wait. I mean in a useful way; a way that can be operationalised and measured.

The words we use, the way we put them together, and even the words we don’t use play a key role in shaping how we think and what we do. Not just ‘we’ as in you and me, I’m talking bigger picture. My case in point, for which I have a fascinating example, is public heath.

The very clever Dr Julie Smith, from ANU, has (amongst other things), conducted a study to look at the way peer reviewed articles report the health risks of women not breastfeeding.

What she found;

“Just as Voldemort in the popular Harry Potter story was hardly ever named because of fear of attracting attention, researchers fear naming infant formula as a cause of ill health”.

Her analysis found that;

“… even where research shows artificial feeding increases health risks, the researchers systematically do not mention the association of artificial feeding with increased illness and morbidity”

And that;

“…the research mislead readers by associating breastfeeding with the illnesses. Formula is very rarely named as a risk factor for various illnesses.”

The reason I’m blogging about it is that not only is it fascinating research, I also think it’s brilliant research. A well considered, well informed framework for inquiry (Julie knows her stuff). Sharp questions and, in effect, actionable results.

P.S On topic, also interesting, is this piece by Diane Wiessinger. A great example of the incredibly powerful effect language can have on our attitudes and behaviours.

So, what should we be measuring here? What will Web 2.0 research gems look like?

On the surface, it’s an easy enough question to answer. Once the marketing objectives have been defined, then we’ll just go measure how well the brand tracks on those measures. Of course. You nut.

But wait a minute. That assumes you know what you’re trying to achieve in this space. More fundamentally, it assumes you know what brand equity in the Web 2.0 environment even looks like.

And this is the crux of the matter.

With Web 2.0, brand equity is, for the most part, defined by the community. Heavens – it’s even generated by the community.

That’s quite a shift for the way we think about marketing.

Crikey. Just when we researchers thought we had a pretty good grip on this whole measurement thing, along comes Web 2.0 to blog us on the head and throw us sideways.

Budgets are (or will be) moving into this space faster than you can tweet about it.

But what on earth should we be measuring here? What should we track and how will it help us?

What do we need to consider, from a research point of view, when we’re looking to understand this (relatively) shiny new world of Web 2.0?

I love this book.

I think of it as a dear old friend: gentle, kind hearted and dependable.

For those of us who aren’t statistically inclined, it provides easy access to the world of distributions, significance, correlation co-efficients, and the rest.

Just the ticket for a qualitative researcher! Porcupine worthy.

Title: Statistics without tears: a primer for non-mathematicians

Author: Derek Rowntree

ISBN: 0024040908

We’re still on Terrible Questionnaires…

Tip # 3: Pilot the thing! 

I thought everyone knew about this one too. But they can’t be piloting the questionnaires in my Terrible Questionnaire folder. They really, really can’t.

Fill it in yourself. Does it flow? Does it make sense? Is it easy to answer? Are you bored with it?

Then, and this is really important, get other people to fill it in (before you send it out). Which bits are boring? Which bits are difficult to understand? Listen to their feedback. Then fix the questionnaire and pilot it again (and again) until it’s user friendly.

The more user friendly it is, the better the quality of the data you’ll collect.

Anyway, that’s the last of my ramblings on Terrible Questionnaires (for now). I’m moving on in my life. 

See you next time for some thoughts on insights.

The topic today is grids. Specifically, the grids you get in some online questionnaires, with the brands listed along the top of the page, and a string of brand attributes down the side.

I’ve seen grids with 70 odd cells (or more!) to complete. You even have to scroll and scroll down the page to see the whole grid. And then there’s another 70+ cell grid on the next page.

Do you really think respondents will go through each brand and each attribute and rate them thoughtfully?

Chances are you’ll get random answers so they can just get to the next page. And then it’ll be a race to get past all the other grids just to get the incentive. 

Reliable, quality data? No!

If you haven’t been following this thread, it’s all about Terrible Questionnaires. The focus is on online surveys and I’m making suggestions on how to avoid writing or commissioning them.

Here’s the first tip: keep them short.

I thought everyone knew about this. But I keep getting 20 minute long (sometimes longer!) questionnaires. Even if I’m interested in the topic, 20 minutes is too long. I get tired and bored.

“But I can’t make it shorter. I need to know about X, Y and Z!” you say emphatically. Well, you also need to know that if your questionnaire is too long, the quality and usefulness of the data you collect will be compromised.

5 minutes. In most cases, I reckon that’s about how long you’ve got before respondents tune out. Flick the “nice to know” and focus on the “need to know” information.

If it must be longer, schedule several sessions (and incentives) over time. Believe me, if the questionnaire is too long, and your respondents get bored and tired, you’ll get rubbish.

There are two ways a questionnaire might be Terrible:

•    It’s uninteresting and difficult to answer, which is Terrible from a respondent’s point of view

•    It fails to collect useful data, which is Terrible from a marketer’s point of view

I’m going to focus on the uninteresting and difficult to answer problem. Because notwithstanding poorly defined research objectives, I can’t think of any other variable that could have as much impact on the usefulness of the data as an uninteresting and difficult to answer questionnaire.

Most of the Terrible Questionnaires in my folder are similarly Terrible: 

•    They’re too long,

•    They have impossible grids to cope with (sometimes several pages of them) and, 

•    Most of them, quite patently, haven’t been test driven.

Over the next few blogs, I’m going to work on some guidelines to avoid the pitfalls of Terrible Questionnaires.

Stay tuned…

I have a folder on my computer desktop called Terrible Questionnaires. 

It houses a vast collection of questionnaires I’ve been sent that I’ve either got stuck on, got tired of, or thought I could have written better.  

Every time I look at the Terrible Questionnaire folder, I get annoyed.

So I’m writing in the hope that those writing, or commissioning, the Terrible Questionnaires will stop doing it, and start writing interesting and easy to complete questionnaires. I’m hoping to do away with the Terrible Questionnaire folder altogether.

Next time, I’ll explain why I think they’re Terrible (capital T). Stay tuned…