Does Research Work?
Oct 10th, 2007 by Simon
This is really interesting. And, I read it here at The Toad Stool first.
Arnold tested the ad “1984″ - that epic Apple ad that still gets mentioned in research groups and is one of the most famous ads of all time globally.
Except, they didn’t test it as an ad - they tested a re-created ‘animatic’ or ‘board-o-matic’ version.
Here’s the result, including the full animatic version (note how long they run - about 3 mins 30 secs to describe a 60″ commercial) AND the excerpts from the focus groups. Now, although the results are seriously shocking, bear in mind that the animatic they used was really good quality - they’re rarely that good. Also, it takes every bit of a finished film - which usually is better than the original script. So, to all intents and purposes, this is better than most stimulus produced for testing.
So, what does it really tell us?
That 1984, the commercial idea, is no longer relevant to consumers today?
That testing ads doesn’t work?
(both of which probably have a good degree of truth to them)
Or that we’re asking the wrong questions? Because, that for me is the right answer.
Ask people what they understand - dig for the message that is told. Then dig for the values - the softer associations and what they mean for this particular company. These are all questions that people can reasonably respond to. Without speculating or guessing at the answer.
But, DON’T ask people what they THINK of an ad. NOR what they would do to improve it. Don’t get them to list their likes and dislikes. And don’t ask them whether they felt it was clear who it was for. These are all questions that people can only guess at. Or that they’re not really qualified to answer.
We wouldn’t listen to our parents tell us how to edit/adapt/improve/change an ad developed by your best people, unless we heard the comment and said “Damnit, they’re right!” So why do we listen to consumers doing the same thing when we’ve asked them questions that are just as speculative?
Consumers can’t tell us the answers, but they can react to what they see and give us some indication of where we’re doing well and where we need to do some more thinking. Let’s not damn research for being useless if we don’t use it right - but let’s not blindly follow 20 people from the suburbs just because they’re “real people” either!
And, in case you forget the original ad, here it is…
but the excerpts are ‘reconstructed’ at best and fake at worse. look at the clothes on the respondents and the technology in the room. And do we really blieve that the qual debrief recommended chimps? there are many things wrong with research but this is simply a good scam to make us laugh and land a serious point.
In many ways it IS a good scam. And there’s so much to take exception to in the approach. But, it does raise the age-old discussion of whether testing ads can work.
It was bizarre - I found myself becoming increasingly agitated by the whole thing. Logical arguments like: The context has changed - 1984 meant something in 1984 came to mind… (I asked my design research methods class if they had heard of Orwell’s 1984? - not a one!).
But the clip is funny because the dénouement releases the tension perfectly. Funny and frightening at the same time.
The guy with the chimp fetish might be more common in research groups than you think…there could be something in the air tonight.