What has Botticelli’s The Birth of Venus got to do with market research? Well, note the crop. But as nice as it is, it’s only the bigger picture that tells us the actual story.

This is at the heart of the next issue for discussion; the ‘new’ (?!) practice of listening online.

Of course ‘listening’ itself, as a method of research inquiry, is pretty obvious and hardly new. What is new* for market research however, is;

• the online location per se,
• some of the online listening technologies, and
• in some cases, the actual content generated online.

But new or not, you still need to know exactly who it is that you’re listening to. And you also need to think about the context.

For example…

If you’re undertaking a market research listening exercise for a client, let’s assume, quite reasonably, that your focus will be on listening to their customers or potential customers.

Is the Internet a good place to listen?

Well, of course it is! But there are some very important questions to ask before you begin;

Of all your client’s customers/potential customers, how many have access to the Internet?

And of all your client’s customers/potential customers who have access to the Internet, how many are confident enough in a) their opinions and b) their writing (or video editing) skills, to express those opinions publicly online?

And of all your client’s customers/potential customers who have access to the Internet, and who are confident enough in a) their opinions and b) their writing (or video editing) skills to express their opinions publicly online, how many bother?

And of all your client’s customers/potential customers who have access to the Internet, who are confident enough about a) their opinions and b) their writing (or video editing) skills to express their opinions publicly online, and who bother, how many express those opinions in an articulate way (ie in a way that a marketer or market researcher might find of use)?

The skews are breaking my brain.

Defining your sample in terms of exactly who it is that you want to get feedback from is absolutely key in terms of determining where and how you should collect your data.

Notably, if your client’s target market comprises a wider group of people than the customers or potential customers who fit the very narrow profile described above, then – and critically – collecting useful data to generate useful output will mean going well beyond the insights you glean from your ‘new’ online listening endeavours.

*relatively speaking and/or as the hype would suggest

There’s been a lot of talk about engaging research participants in this ‘new’ research paradigm.

I’m focusing on qualitative market research here because firstly, that’s my thing…

: )

…and secondly, because I hear that ‘engagement’, within the context of market research online communities, is community-nirvana. The ‘best’ communities are engaged communities.

This strikes me as, paradoxically, both obvious and alarming.

There’s a very fine line between engaging research participants enough to… well…participate in our market research, and over engaging them.

Without due diligence, research effects (pick one of many) are likely to confound the research output in unintended, unexpected and underestimated ways.

I don’t, for one second, suggest that other research methodologies are free of research effects – they aren’t! – but surely this doesn’t automatically generate a license for ‘new’ market research to ignore them.

And while I don’t think you can necessarily control for engagement, some questions;

• To what degree should you try to ‘create’ it?
• How much is too much? When does it start to mess with what you’re looking for from your research?
• How do you disentangle it from your analysis?

I’ve just come back from a fantastic trip to Chicago where I attended the ESOMAR Online Research 2009 conference. You can find Jeffrey Henning’s brilliant recap here (he pretty much live-blogged it – very impressive!).

Anyway, not surprisingly, there was lots of talk of ‘new’.

And it would have been easy to come away with the message that the market research industry really needs to get with the ‘new’ programme or, quite simply, it will wither away and die.

Because the new world of research is here! New methodologies. New technologies. New ways of engaging with respondents (Gasp! Did I say that? I meant ‘participants’, ‘co-creators’ or ‘collaborators’).

All well and good, but what does this actually mean?

New kinds of output?

Let’s all take a moment to think. What, as market researchers, is our purpose?

Market research is about understanding the market. At a very basic level, the end goal is to deliver information that will help our clients make relatively informed/better decisions about how to sell their products or services.

And here’s my point; the output (ie what our clients are paying for) is only ever going to be as ‘new’ as the questions they/we ask*.

Over the next few blog posts I’m going to take a look at some of the elements of ‘new’ I outlined above. Examine them closely. Explore what they mean for researchers at a practical level. Separate, if you like, the hype and theory from the actual task of delivering useful output.

Should be interesting…

: P

*If you’re in the ‘listening’ camp, ie “Oh no, no! We don’t ask questions, we just listen to the conversation!”, I’ll argue that you’re still (implicitly) asking questions when you choose to/not to include any particular content in your analysis.

The term ‘traditional research’ is often used to describe offline methodologies; mostly, I’ve observed, in the context of selling what I assume must be ‘non-traditional’ (??) online methodologies.

In this context, the word ‘traditional’ conjures imagery of antiquated, moth-eaten and fusty research practices.

And clearly, many offline methodologies are anything but. Not to mention that many research solutions require a hybrid of both online and offline approaches.

But if we have to throw labels around, it’d probably be more accurate to substitute the word ‘traditional’ with ‘established’ (as Paul Vittles from TNS usefully suggested during question time at an AMSRS breakfast seminar I attended last week).

So does this make ‘non-traditional’ research ‘unestablished’ research?

: P

Wow. Doesn’t that put a whole new spin on it!?

Great post over at Ray Poynter’s (always) inspiring blog.

The way Ray described a short-term community, I think, delineates a clever, and potentially efficient, research methodology.

But I’m yet to be convinced of the long-term research community concept (although I’ve no doubt some readers are probably pulling their hair out over my apparent inability/reluctance to see the light/‘get it’).

: P

I’m assuming that in a ‘community for market research’ (vs a ‘community for customer relations/brand building’) context, a client will actually have some questions they want to ask and/or issues they’d like to explore.

If they don’t – if it’s a case of just wanting to pick up on issues entirely generated by the research community itself – then I’m guessing that they’d probably get better value by undertaking some basic social media monitoring … (god forbid).

Anyway, assuming we have a community that’s being used for market research, and there are some questions to throw into that research (in whatever shape or form), here are my questions;

1) At what point in time, along the short to long-term spectrum, does the nature of the output you get from a research community change? And probably more importantly, in what way does it change?

(Or is it different from day one because of how the participants are briefed about purpose of the community and their role in it?)

2) Would one analyse the data coming out of long-term vs short-term research communities differently? This, I suppose, brings us to the issue of the research community objectives. From the sales pitch, I sometimes get the impression that the sole objective of the exercise is to get the research community members to bond; never mind the insight, they’re bonding!

: P

Kidding.

But really, what are the issues, the benefits and drawbacks of community members ‘developing a sense of community’ (and from that, I would assume we mean ‘belonging’), particularly over a longer time frame?

Does it make participants more honest, or more willing to share? Maybe. But (it could easily be argued), maybe not. And what impact does, for example, group/clique think etc have on the output? How would you identify/measure the impact in such a wide (uncontrolled? quasi-controlled?) landscape?

(And to take it to the extreme, if one’s aiming for ‘uncontrolled’, then back to the point above; isn’t the ‘community’ just a very limited method for undertaking social media monitoring?)

3) If developing a sense of community/belonging is one of the key operating principles for long term research communities, then how do new recruits (or exiting members for that matter) impact the existing community dynamic and thus affect the output? How about changes in community moderators/managers?

4) Are there any studies looking at the differences in terms of valuable/usable output between short-term vs long-term research communities?

Are there any actual or even theoretical definitions of the ROI (tangible or even intangible) that one might expect from a short-term compared to a long-term research community available?

(I’ll probably come back to this with more questions when I’ve had time to think more).

Coincidentally, after my last post, this came up in my Google Reader today;

More on the Problems with Opt-in Internet Surveys

Here’s the first article;

Study Finds Trouble for Opt-in Internet Surveys

I had the great privilege of attending an AMSRS Professional Development session earlier this year to hear Jon Krosnick speak. He was brilliant.

And so it’s with great interest that I follow this very timely and fascinating debate (hosted on Gary Langer’s excellent blog).

Make sure you follow the links to get the full story/debate. It’s an important one.

P.S And here’s a link to the study itself.

How do market research online community providers populate their clients’ communities?

(When I say “market research” communities, that’s exactly what I mean; a community used as a market research tool. I’m not talking about online communities that are used in a marketing/customer relations exercise.

I’m not quite sure that the difference is apparent to all, but they’re not the same; not by a long shot.

In one, you’re giving the community members love because you want to make them happy. In the other, the relationship is somewhat more pragmatic; you want to learn from them. Notably, if you’re giving them love to make them happy, you’re not necessarily going to learn much, because they’ll be all nice and lovely back).

Anyway, focusing specifically on market research communities; what checks are in place to ensure that the people who end up in the community represent the people the client actually wants to hear from (ie the population of interest)?

To borrow from the delightful John Lacey, I’m filing this one under #stuffiwonder.

My qualitative toolbox has grown with current online technologies. And the possibilities promised by evolving technologies are endless.

It’s intriguing to contemplate how these changes influence not only the way I might do research, but no doubt, the way I think about and frame research issues.

Never a dull moment, that’s for sure.

But exciting as it is, it’s always a good idea to stop and think. And here’s something to think about…

Lunch

You catch up with a friend (in real life) for lunch. You talk about this, that, and the other.

While between catch ups you talk on the phone, email each other often, are Facebooked, and have been members of the same online community for over 5 years, sitting down, face to face, puts a whole different angle on your understanding of their life. It provides a completely different context.

Watching their face light up when you ask after their kids, seeing the micro-frown when you discuss topic X, and observing their extended search through their handbag to avoid discussing topic Y; these are all things you could never pick up online.

The conversation takes paths that your online conversation couldn’t have travelled (and this works the other way too, but stay with me here…).

Take this into a research context, and you realise that while you can get some (often surprisingly) deep and passionate reads on emotion through written words, images etc via online methods, there are times when you’re just going to need more.

Springboards

The nuances of body language provide the researcher with cues and real life stimulus. A pause, a frown, half a giggle; all invaluable springboards to discussion.

And this dynamic is something that, quite patently, offline research can provide over and above an online dialogue.

I stress, this is only important if it’s important; it depends solely on the task at hand.

But for the most part, I like a side of offline context to go with my analysis.

And because of the above, I don’t think qualitative market research methods will change quite as fast as one might be led to desire or believe.

At least not the ones that provide the relatively comprehensive insights I need in order to do my job well.

No doubt market research is evolving. As it does. And as it should.

And a natural and obvious part of that evolution seems to be the move to online qualitative research methodologies; eg qualitative content analysis, bulletin board focus groups, online communities etc.

But is online research necessarily the best or only way forward? Is traditional (offline) research on its way to the research graveyard?

I don’t think so.

For the record, I like online qualitative research (speaking for the most part, with bulletin board focus groups in mind here).

It’s fun to do (albeit time and labour intensive). Most research respondents who participate seem to enjoy it. And importantly, the output, for the task at hand, has been pointed and relevant.

In some cases an online qualitative approach is the optimal methodology. When you need to reach otherwise impossible to reach individuals, generate interaction between typically un-collaborative individuals etc, it’s worth its weight in gold.

But let’s not put the cart before the horse. Let’s take a reality check.

Online for everyone?

Not all people are like us.

We (you and I and other readers of this blog) are not particularly representative of any given market. We’re a highly skewed group in terms of our attitudes, our communication skills and our love (?!) of things collaborative and co-creative.

Believe me. Not all people are like us.

I’m reminded of this every time I do (offline) group discussions or depth interviews.

I’m reminded of this every time I talk to my relatively less-online-focussed clients or friends.

I’m reminded of this every time I walk down the street and see people talking, thinking or engaging in any one of the many offline activities that make up the bulk of their lives.

Neilsen (I’m guessing inadvertently) help me to make my point. According to a recent study, Australians* are spending a whopping 16.1 hours a week online. Up from 13.7 hours in 2007.

But one week = 168 hours.

That means they spend 151.9 hours offline. *And that figure was based on a sample of internet users.

Which suggests that we’re not quite at the point where online conversations are a part of everyone’s everyday life. Not by a long shot.

And that brings me back to the important issue of sample. Are we willing to accept the (well documented) skews that come part and parcel with online samples? While in specific cases an online skew won’t be an issue, more often than not, it will.

So, as much as we might like to, we simply can’t take all our research online (yet). The vast majority of the people we want to understand just don’t hang out there.

(Part two coming soon).

I’ve been quiet. I’ve been thinking.

A click around the online market research community (oh the imminent irony!) tells me that, apparently, market research is – or should be – changing. And I mean really changing; as in beyond all recognition.

In a nutshell, and broadly generalising, here’s the gist;

1. Traditional (offline) research is becoming irrelevant

2. We need to find new ways to engage with  consumers, respondents, research participants

3. In the age of collaboration and co-creation, market research online communities are the way forward

Hmmm…

I started thinking about this here and here.

Over the next few posts, I’m going to look at points 1-3 in a bit more depth. Stay tuned for the possibly uncomfortable ride…

There’s lots of talk about the future of the market research industry and its move (was slow, now fast) to online. And exciting as it all is, something about it bothers me.

I can see two key factors that would logically spur the growth of online qualitative* market research;

1. The perception that online qualitative research can deliver better insights than traditional offline methods (for the purists)

2. The perception that online qualitative research is more efficient than offline methods. Specifically, the perception that you can do more, faster, and for less (for the realists)

And, for each project we undertake, my three key questions;

1. Can it?

2. Is it?

3. Are we thinking about questions 1 and 2 hard enough?

Broadly speaking, and within the context of gathering qualitative consumer insights, online communities fall into two categories of interest;

1. Existing communities: virtual spaces where people with shared interests have gathered

2. Manufactured communities: virtual spaces where people who have been paid, or given some other incentive to offer their opinion come together

Are online communities, as described above, the goldmines for insight they’re being sold as?

Here are my thoughts;

Existing communities

These are communities that have naturally evolved to address member based interests and needs. Sport, parenting, cooking, fashion, music, social media etc – all have online communities that congregate in various virtual spaces across social media land (eg MSN or Yahoo groups, Facebook, blogs etc).

Theoretically at least, existing communities are a great resource for qualitative researchers. ‘Real’ conversations that often happen in relatively real time, minus any research effect. Insight heaven!

But what about in practice? Three key issues come to mind;

1. Sample

My old favourite, sample. Who are the community members? Do they represent the target market? There’s no way of telling.

2. Access

Many existing communities, and possibly the most interesting ones, are private; unsearchable for a start, but also, locked behind membership and passwords. If you can’t see them or hear them, it’s going to be difficult to glean any insights!

3. Professional ethics

Without the context of the market research ‘deal’, where critically, research participants are aware of, understand, and agree to participate in the research process, how should one proceed?

Should the researcher disclose their market research agenda? Is it unethical not to do so?

And if they do disclose their role as research participant/observer, how will the community respond? What effect will the researcher’s presence have on the community’s ecosystem and/or shape of the discussion? These unknowns make analysis a risky business.

Manufactured communities

Many researchers call them communities. They sell them as communities. Quite frankly, I’m not sure that manufactured communities (manufactured for the purposes of market research) should actually be called communities at all. Rather liberal use of the word ‘community’ in my opinion. A more accurate description would be a ‘purpose built environment’.

But notwithstanding the misnomer, from where I sit, manufactured communities are simply an inefficient, high cost, low return version of an online bulletin board focus group.

Here are my questions about manufactured communities;

What’s the cost of manufacturing, hosting, nurturing, monitoring and maintaining them to gather qualitative insights?

Where does analysis begin? Where does it end? What kind of questions are you trying to answer anyway?

You want consumer insights?

I can give you a thousand insights.

Here’s a conversation we once had with a potential client.

Potential Client: “We need insights!”

Us: “Tell us more” (spot the qualitative researchers).

Potential Client, in a louder voice: “We need insights!”

Us: “We can give you a thousand insights”

Potential Client, now smiling broadly and dancing with delight: “Wonderful!”

Us: “What are you going to do with them?”

Potential Client, somewhat soberly: “Oh.”

There’s a happy ending: the Potential Client became an Actual Client and over time, we gleaned the specific kind of insights they needed in a practical sense. Insights with focus and bite.

I tell this story because it highlights one of the issues I have with online communities for harnessing qualitative insights (they’re all the rage, you know) (both communities and insights that is).

My issue is around efficiency. Are online communities the most efficient way to get insights with focus and bite?

Of course, the answer is;

It depends! 

But for the most part, probably not.

Hang on to that cliff; more on this shortly.

Sentiment is the very essence of what we’re trying to understand through market research. And this is something that social media monitors (SMMs) don’t gauge very well.

Although automated sentiment analysis is often sold with the SMM package, there are two things about it that trouble me;

1.    Accuracy

2.    Specificity

Accuracy

There seems to be considerable scope for error in the labelling.

For example, how would automated sentiment analysis label a statement such as “F&*#ing brilliant!”?

Depending on the context, this statement could be;

1.    Dripping with irony

2.    An exclamation of genuine excitement and joy

3.    A description of a high wattage light bulb

So, would it be labelled as negative, positive or neutral? 

Notably, some SMMs claim to be contextually savvy, and that they can identify positive, negative or neutral sentiment with 90% accuracy (is 90% good enough?).

BUT…

Specificity

BUT (note caps), even with 90% accuracy, these labels are still seriously wanting. They don’t provide me with information that’s of much use – if any – because they’re too vague.

It’s the finer points of “sentiment”; the despair, frustration, excitement, boredom, curiosity etc underlying the positive or negative sentiment labels that I’m interested in. This is the level of sentiment I need if I’m to understand what’s going on with any effect. And to get to this level of sentiment, I really need to dig a bit deeper.

Where’s the gold?

I need to dig deeper, but where do I begin? Back to my earlier points about the issues with SMM sample definition and skews; I don’t know where the real gold (vs fool’s gold) lies.

Without spending the time and effort to sort through each and every buzzversation (possibly reaching the millions?), I can’t distinguish between content of import and that of little consequence. I just don’t know where to drill deeper in a meaningful, robust kind of way.

So it’s virtually back to square one.

Itching

SMMs are an exciting idea and I’m itching to find a way to use them.

From a PR or customer service point of view, I imagine they’re worth their weight in (real) gold.

But in my qualitative market research business, I’m not sure of how to use them with either confidence or pragmatic effect.

Sentiment aside, the sample scope/limitations and the unknown skews preclude the output from forming anything approximating a solid foundation for analysis. Bit of an issue for me.

Social media monitors (SMMs) trawl the web to find mentions of your brand or what ever it is that you’re interested in monitoring.

There are many SMM products and services available: some free, some you pay for.

Here’s a very basic example;
http://www.whostalkin.com/

If you type in the name of a brand or topic of interest, you’ll get an idea of the kind of information SMMs return.

Depending on the level of sophistication built into the SMM you use, you can refine your search with key words, run analytics, see where the buzz is happening etc.

There’s a lot of hype around SMMs. Not surprising really. The idea – getting feedback on the cyber-buzz around your brand, product or service – is timely and sounds quite marvelous!

Kind of. Until you think about it a bit more. Which I have. And wearing my qualitative researcher’s hat, SMMs actually fail in two important ways;

1. Sample definition
2. Sentiment

Sample definition

What constitutes a SMM sample? In a nutshell, a SMM sample comprises the searchable/findable content sourced from various online channels. That’s as precise as you can get really. The truth is, you just can’t know who’s represented (or not) within that content.

For example, SMMs can’t identify and screen out marketing blogs, websites or chatter. This means that SMMs don’t distinguish between content generated by marketing folk and content generated by non-marketing folk.

And let’s face it, quite a lot (most?) of the brand chatter out there is actually generated, nurtured and sent bouncing around the interwebs by marketing folk. People like us. The kind of people we try very hard to screen out of market research samples.

Also worth noting is that SMMs can’t distinguish between content generated by core customers, infrequent customers or non-customers. This means that all customer/brand relationship variations are automatically given the same share of voice and weight in the analysis.

Another factor to consider is that the sample will be skewed. And while a sample skew, in itself, is not necessarily a problem, it’s certainly a problem when you don’t know how it’s skewed. Which is the case here.

Without being able to define the sample, and without knowing how the sample is skewed, there’s no foundation or context for meaningful content analysis.

Next time, I’ll take a look at sentiment…