We’ve been working in the US a lot recently. Whilst looking for a particular type of local partner we came across the website of a ‘branding business’ and their associate research agency. We read their websites with mild interest initially, but with increasing incredulity subsequently.
Extracts from their websites include:
We use CATI (Computer Assisted Telephone Interviewing) systems, computers to automate the activities of the interviewing facility. The system controls the questionnaire, ensures random calling, rotates the questions to account for fatigue, and collects the data……
…..Many research companies make a litany of mistakes that only ensures the data will not be projectable or valuable in developing a brand. Many avoid random sampling, choosing online, email, mail or expert panel surveys which are self-selecting and, therefore, not projectable to the larger target audience. Online surveys, for example, are self-selecting because you only get results from those who would be most likely to fill out a questionnaire from an email request and go online to fill it out……
…..[The agency] also does not recommend focus groups, which can be easily manipulated by both participants and moderators. It is not a scientific approach but rather an anecdotal one, and used by most companies to simply re-affirm what they already want to believe. In the end, focus groups mean nothing. Your brand is far too important for that. It must be supported by objective, projectable and scientific research.
We believe we are a good natured bunch, who largely take a live and let live attitude towards others. But in our opinion the extracts above are quite extraordinary.
A few observations follow:
In the first instance, any form of research is self-selecting unless, of course, you don’t tell people that you’re conducting research. Which we think breaks the market research code of conduct in most markets we know. Isn’t it delusional to think that someone who chooses not to slam down the phone to a random call from a research agency asking for a few minutes of their time is any less self-selecting than someone who chooses to respond to an online email?
In the second instance, any form of research is open to manipulation by both its originators and its users should they be so inclined. Wasn’t it out of the common practice of the manipulation of statistics that the phrase ‘lies, damned lies, and statistics’ was born? It would be naive to suggest that research is not sometimes subject to misuse and abuse, although to suggest that most companies use focus groups to “simply re-affirm what they want to believe” seems to be doing a vast number of intelligent clients a huge disservice.
Surely, the point is that when choosing a research agency and methodology, whether qualitative or quantitative, clients should select those whose people are:
a) experienced enough to be able to interpret the research fairly, and
b) strong minded and authoritative enough to represent those findings and have them accepted, irrespective of external pressures or any desire for convenient answers.
If not, then why waste the money?
Finally, to say that the focus group is “not a scientific approach but rather an anecdotal one” seems to be stating the obvious. But that does not mean to say focus groups “mean nothing.”
We conducted a research project for a service business out of which one person’s anecdote in a focus group provided an insight which led to a complete revision of their business strategy and resulting NPD programme. It did so because to all involved the anecdote made intuitive sense and was subsequently validated by quantitative research.
No form of research, whether qualitative or quantitative, can provide an undisputed whole picture and no form of research should purport to. As is said time and time again, research should only be used as an aid to judgement and as a tool for decision making, alongside many other tools from statistically robust data to, dare we say it, the instinct of those with flair and vision.
There’s a place for both qualitative and quantitative research out there but there are examples of good and bad in each, the lesson being to tread carefully, choose wisely and use any findings conscious of both their value and their limitations.
It is somewhat ironic that we find ourselves writing this as we often challenge the default use of focus groups, but certainly not for the reasons argued in the extracts above.
The positioning of the author of the extracts seems somewhat adversarial. Maybe that’s intentional, or maybe it’s not. If it is, maybe our response is the response they are looking for? But it’s not a response that will see them stealing share from our US partners.