Chris McGinnis tweeted yesterday about an executive at Accor Hotels who admitted to publishing over 100 fake reviews about his company’s hotels on hotel search and review site TripAdvisor. And this raises an interesting question about how much you should trust online hotel reviews.
Let’s get the executive’s conduct out of the way first. He used an alias, which implied he knew what he was doing was wrong. I don’t really buy his argument that it had no relation to his official business responsibilities and thus didn’t need to be declared. I declare my business relationships here even if they don’t influence my content because the possibility exists and that’s what matters. You can decide for yourself if such bias actually makes an appearance.
What’s more interesting to me — and what inspired this post — is that the types of comments he left were relatively tolerable despite the lack of disclosure. From the article:
A listing for the Sofitel Phnom Penh Phokeethra, for example (noting this is his 137th review on the site), includes his comments, “As a first time visitor to Phnom Penh I didn’t know much about the hotel scene so booked a brand I knew well. It turned out to be a good choice.”
Well you know what? He probably didn’t know anything about Phnom Penh. But he does know that particular brand very well. And he was happy with his stay. All very matter of fact. If that’s the most damning thing they have on him, it’s not enough to make me stop staying at Accor’s properties.
At the same time, he review wasn’t all that informative either. You guys know me. I like to prattle on and probably provide too much detail. I like information, facts, stuff I can work with. But his review was the equivalent of a star rating and nothing more. If he was going to write anything at all, it should at least have something more I can benefit from than “it turned out to be a good choice.”
TripAdvisor remains the only place I still read hotel reviews. I don’t read reviews on chains’ own sites because I don’t trust them. I don’t read reviews on most online travel agencies because I don’t use them. That leaves TripAdvisor, which will link to other OTAs but doesn’t sell any rooms themselves. I realize there are a lot of reasons not to trust what you read there, just like there are a lot of reasons not to trust what you read on Yelp (its food counterpart) and Amazon (its online shopping counterpart).
So how do I filter the reviews to come up with useful information?
I’m going to put on my scientist cap again (actually, PhDs get a hood…) and talk about signal detection theory. This concept has been used extensively in the electronics industry but also has a purpose in psychology — How do you distinguish a valid signal (good or bad) from a bunch of useless noise?
We would expect real reviews to be shaped like a bell curve with a peak somewhere near the “true” quality of the hotel. And then there’s a lot of noise, or reviews that we can’t trust. Some of these might be on the positive side of the curve, written by duplicitous managers and employees. As a result, many people shift their threshold for judging a hotel a little higher. In other words, if we think artificially high reviews are raising the average score by 1 star, we might raise our threshold from 3 stars to 4.
I do this a lot in Europe. I don’t know if it’s because false reviews are more common or because standards are lower, but I never book a 3-star hotel in Europe. Or even most places abroad. Of course, on international trips I also tend to benefit from good exchange rates that let me book luxury for less (Bangkok) or I use lots of points (Paris).
This is a good first-level approach, but it is not comprehensive and you could overlook some good options — what scientists call a “miss.” This is because there is also a threat of artificially low reviews. Competitors, or maybe just some angry and unreasonable guests, could write some very negative reviews in an effort to drag down a hotel’s rating more than it deserves. Surely you’ve seen this before. A hotel has lots of 4-stars, maybe some 5- and 3-star reviews, and very few 2-star reviews. Then there’s a spike in 1-star reviews.
The answer here is to just cut them out. If 1-star reviews don’t follow a trend (in other words, if there is a decline and then a sudden spike) I just don’t trust them at all. I do the same with good reviews. If there are mostly negative reviews, and a few positive reviews, but a lot of very positive reviews, those positive reviews can be equally suspect. These are outliers and I don’t want them contaminating my information.
Combining this approach with the first one now gives me a slightly better method. I completely cut the most negative reviews, because anyone who ranks a hotel as 1-star clearly has an unusual problem if there are mostly positive reviews. More than a few gets the hotel cut altogether. I also raise my threshold to mitigate the risk of fluff on the high end. Finally, I actually read the reviews. I do this last because I don’t want to look at every last hotel. By this point I’ve usually found at least half aren’t worth looking at in depth.
I look at the content of the review to see how much quality is there. Do I learn anything about the hotel itself, or is it merely “good”? Vague descriptions are the hallmark of someone who doesn’t care, views writing the review as a chore, and is otherwise not enthusiastic. Overly enthusiastic descriptions indicate someone who has a very specific purpose without necessarily being honest.
Reliable reviews tend to be those that are mostly positive or negative but manage to have both pros and cons while including evidence of specific incidents that led to those opinions. The maid barged in while the “Do Not Disturb” sign was up? That’s better than saying the staff were rude. The view was impressive? Tell me what you could see! I weed out more than half of reviews by sticking to those that actually help me decide. At this point, I’m no longer caring about the average star rating. I only read the reviews of hotels that are good enough. Now I want to know why it’s good enough.
This is like taking our bell curve of reviews and thinning it out. It’s one reason why Amazon does so well with its rating system, and why Yelp and TripAdvisor are okay but still growing. When you have 100 or 1,000 reviews, you can cut half and still get a good idea of how most people (the trustworthy ones) actually feel. When you have 20 reviews and you cut half, there’s not likely to be any consensus left or what consensus there is can be easily swayed. (Maybe there were just 10 really good fake reviews.) I am usually skeptical of hotels with a small number of reviews no matter how good they are.
So there you have it. I don’t know how helpful I was, but it’s why I still use TripAdvisor even though I know many people who wouldn’t trust it with a 10-foot pole. I don’t trust it either. It’s only useful if you’re willing to put in some work to separate the wheat from the chaff, and even then you can be duped. But reviews like the Accor executive’s aren’t really the sort I use to make a final decision, so they don’t bother me at all.
If you want to read more about signal detection theory, Prof. David Heeger at New York University has a decent online summary of the topic. To follow along with the description in my post, you should modify his illustrations to put noise on both sides of the signal and not just one. It makes the math more tricky, and I thought about working through that, but I don’t really use any math when picking a hotel except to calculate a currency exchange ratio.
And because these things amuse me almost as much, Bernie Garrett at the University of British Columbia comments on his blog about the differences between scientific, quasi-scientific, and pseudoscientific.
Too long? Now you know why I don’t normally drink triple-shot lattes!