I've just finished a multichoice survey full of 'have you stopped beating your wife' type questions. It really annoyed me!
While I can appreciate multi-choice makes analysis simpler and makes it easy to do the stats, it doesn't make the results any use if the wrong questions were asked!
There really is NO excuse for a badly designed multi-choice:
Strongly agree, slightly agree, slightly disagree, strongly disagree, couldn't give a rat's arse!
I've seen some amazing examples. One online poll stated:
"Would you buy a $100 laptop or is that too cheap?"
Answer options were yes, no, don't know.
What REALLY amazed me was that according to the results, 80% answered 'yes'. Still trying to figure out what they were answering yes to.
I have often written on such surveys why the answers offered do not accurately reflect my answer. Unfortunately this is less easy on online surveys where there may be no chance to do this. In one case I replied that I refused to answer the survey because it was so obviously designed to elicit responses favourable to the sponsoring company.
There are serious cases of this. For example, when Councils were trying to offload Council Housing, surveys asking residents for their opinions were often seriously skewed. I also saw this firts hand when a major Housing Association in South London was doing so called customer satisfaction surveys which were then used to convince the Housing Corporation and Politicians that all was well.
The combination of this type of survey cheating and the selective presentation of the results is one way that so called evidence based policy making is got around. "We carried out extensive consultation." Yep. But it was all bleedin manipulated!!
What can they tell from these questions though?
Strongly agree, slightly agree, slightly disagree, strongly disagree, couldn't give a rat's arse!
They seem a bit non commital, and I could'nt determine how the person thinks about the subject from those questions. How do they collate the real answers? Anyone understand why these particular questions seem so popular?
I avoid these surveys whenever possible. I had a phone call recently about my car service, and I forgot to tell them I hadn't got the time. After the first half dozen irrelevant questions that couldn't be answered sensibly, I said something like "Look, this is pointless. The people weren't polite or helpful because I never met any of them. My car was collected in the morning, and it was delivered back in the afternoon serviced, washed and with two new tyres. I paid for it. End of story."
Starting point here:
http://en.wikipedia.org/wiki/Likert_scale
Whenever I see one, the red mist descends...
I've completed a fair few on-line surveys.
While some are well designed, some are indeed terrible. I completed one where they assume you use a certain product and didn't have a 'not applicable' or 'don't use' option. Really I should have abandoned the survey, but I thought they deserved my 'very dissatisfied' answers for designing such a bad survey.
Which leads to another problem - although I do try to be honest, does everyone? Better designed surveys seem to have repetitive questions that may catch some dishonestly but can't exclude it. I do surveys because, quite sadly, I find some of them fun. I like seeing adverts before they are on TV because, basically, I pretty much hate most adverts. I do like to give my honest opinion on them. The 'reward' is secondary. But for some people, building up reward points is the primary objective. Do they get through the surveys as quickly, rather than as honestly, as possible?
I like to think my opinion matters- I said the Knorr Stock Pot with Marco Pierre White looked cheap and old fashioned when it was a cartoonised version - low and behold - a 'real life' version is used on TV.
Have to say the worst are usually under-grads asking people on forums to complete surveys for their courses - usually about supermarkets for some reason.
However, there is usually an opportunity to give feed back about the survey, which I invariable avail myself of.
I tend to avoid the on-line versions wherever I can as, like DD stated, you can't add your own notes and comments - you know, like "what pillock wrote this survey?"
I remember many moons ago a young lady caught me in a more generous mood than usual in Bury St Edmunds and I agreed to take part in her survey. After a few minutes and several questions she asked me if I drank a certain brand of brandy. Erm, no, I don't drink brandy, I says. Oh, that negates everything I've just asked you, she says in some confusion. I'd go back to the twunt who wrote this and beat him about the head with it, I advised her.
Once, while out shopping, I was asked by a scantily clad young woman if I wanted to pop into a pub and have a free pint of lager followed by a short survey. I was hot, thirsty and knackered so I said yes. After swiftly downing the lager the first question was 'How many pints of lager do you drink a week?' There was no 'none' option. The next question was a 'which of these phrases best describes the product' one. There was no 'stagnant piss' option. There was a question about favourite brands of lager, etc, etc. The woman got quite stroppy with me and demanded to know why I'd accepted the free pint if I don't normally drink lager. But I'm sure many other people would have accepted as I did, it wasn't my fault the questions were so presumptive. Her problem and several of the others described above would have been helped by simply asking one pertinent question right at the start of the process.
I wonder if she normally needs to bribe people to spend time with her?Originally Posted by Croydon Bob
As researchers we have to have some means of assessing opinion and the Likert scale is as good as has been found. However, this doesn't excuse poor question structures or the increasingly common practice of forcing opinion. Moreover, researchers should rely on the old reliability rule of thumb:
Most reliable - I have evidence that you did something (e.g. bought a pint of lager)
Next most reliable - you say you did something in the past (this declines in reliability with time - opinion polling in 1994/5 consistently showed results that indicated a Labour landslide victiry in 1992)
Getting a bit unreliable - you say you will do something in the future (there's quite a deal of solid research evidence indicating the unreliability of future intention answers - this is why opinion pollsters ask how you would vote today rather than at some unspecified future date)
Flaking at the edges reliability-wise - you say you agree or disagree with something happening now or give an opinion on a current activity (e.g. drinking a pint of lager)
Barely reliable at all - you give an opinion about something that might happen in the future
Rubbish - you give an opinion about some ungrounded general, all-encompassing statement
The problem is that loads of survey creation software is out there (Survey Monkey etc.) and people think they know how to construct a market research survey (which they usually don't). They sent us on long courses about behavioural psychology, statistical analysis tools and research methods - little bits of it stuck with me. Certainly enough to get the screaming abdabs when faced with the typical on-line survey. That and most of it's just sugging anyway.
And don't get me started on self-selected samples, non-adjustment for samling error or a host of other failings in much so-called market research.
Most of the online surveys for customer satisfaction I've done seem pretty badly constructed, since they typically include all kinds of questions which only apply to a fraction of people, but where there isn't a 'Not Applicable' box.
Personally, I think if I was running a business online, and actually looking for information rather than mere validation, I'd probably be most interested to know what customers thought about me compared to my competition - if they thought there were other companies better than me, how did they think I could improve, and if there were companies I had an edge on, what was it that they liked about me.
I suspect the trouble is that the survey is being made up by a survey company, not the company requiring the information. This survey company probably assumes - very likely correctly - that they are more likely to get further business if they slant the survey to make the customer look like they are doing well, rather than giving them an honest assessment of what is wrong. It may be bad for the customer's business, but it is probably good for the job security of the manager who is responsible for commissioning the survey.
I suppose that, rather than companies actually wanting to find out where they can improve, most surveys are done because people being trained to be managers are told they should do surveys, and just like employee assessments, it's the collecting that counts, and the results are generally likely to to be ignored and archived, unless they can easily be used as support for some pet project of a manager.
No, market research is a huge and important part of any large brand. Internal surveys, yes, I suspect much of the data is ignored, but on-the-street research data is used and is very valuable.
Research for PR purposes often has biased questions precisely because the outcome is already decided.
Posting Permissions
Reply With Quote
Bookmarks