How to spot fake reviews

Review websites are like a wife to me: I always check with them first before making a major decision.

I say this jokingly, but it is pretty much true. There is Amazon for consumer products, Yelp for restaurants and site-seeing, TripAdvisor for hotels, and Angie’s List for repair services. Even at the end of the day when I want a beer, there’s BeerAdvocate.

Review websites are a tremendous service and they have helped me make a lot more informed decisions.

Fake reviews spoil the game

The downside is review websites are victims of their own success. There is a huge market for fake positive reviews, ranging from owners that self-promote their business, patrons that are bribed with discounts for good reviews, and professional liars that sell positive reviews for $1 a piece.

Fake reviewers make me extremely mad, and they deserve punishment for the tremendous injustice they are doing to society. But there’s no point in getting angry as retribution is an unrealistic goal.

The best thing to do is to ignore these fake reviews. If fake reviews can be identified, then there will be less of a market for them.

Today I came across a useful and free tool in the battle. This is a computer algorithm that can spot spammers much more accurately than we can. In the study, the algorithm was nearly 90 percent accurate versus humans just about 60 percent. More about this after the jump.

Also a small administrative note: this website is undergoing account maintenance and may be offline from October 29th, from 12:00am to 4:00am EDT.

H/T: New York Times via The Responsibility Project

The Review Sniper

The algorithm is called the Review Sniper and there is a demo version available for free at the website ReviewSniper.org

The way it works is simple: you paste in a review, and click “submit.” The Review Sniper then tells you whether the review falls into the 4 categories of “truthful, mostly truthful, somewhat deceptive, or deceptive.”

The directions are all self-explanatory at the website:

Currently the tool is completely free and it works best for hotel reviews because those are the reviews it was tested with.

The theory of fake reviews

The Review Sniper is the result of work from researchers at Cornell. In a recent paper, the researchers worked out an algorithm that could identify deceptive opinion spam, which are “fictitious opinions that have been deliberately written to sound authentic, in order to deceive the reader.”

The algorithm relies on a set of linguistic clues and word analyses to parse out genuine reviews from deceptive ones. The analysis is way over my head, so I will not conjecture any more. But you can read about the details in the following paper.

To test the accuracy of the algorithm, the researchers compiled a set of reviews. They employed freelance writers to produce 400 deceptive reviews about Chicago hotels. These got combined with 400 reviews mined from TripAdvisor that seemed to be genuine (they were from repeat users, not overly positive, and long).

Three volunteers were asked to judge the reviews as fake or genuine, and the results were compared side by side with the algorithm. On the whole, the volunteers were only about 60 percent accurate in their conclusions. By contract, the algorithm was nearly 90 percent accurate in its identification.

That people are credulous is no surprise, and that is it helps to have emotionless algorithms to parse out fake reviews.

A closing test

The research so far seems limited to hotel reviews, but one hopes further research will make deceptive reviews a thing of the past.

If you want to test your own skill, see if you can identify whether the following reviews are truthful or deceptive.

Here are the reviews, used as examples in the paper.

The first review:

I have stayed at many hotels traveling for both business and pleasure and I can honestly stay that The James is tops. The service at the hotel is first class. The rooms are modern and very comfortable. The location is perfect within walking distance to all of the great sights and restaurants. Highly recommend to both business travellers and couples.

And the second review:

My husband and I stayed at the James Chicago Hotel for our anniversary. This place is fantastic! We knew as soon as we arrived we made the right choice! The rooms are BEAUTIFUL and the staff very attentive and wonderful!! The area of the hotel is great, since I love to shop I couldn’t ask for more!! We will definatly be back to Chicago and we will for sure be back to the James Chicago.

Do you have your answers? I can report that my guesses were right.

The answer is “Gur svefg bar vf zbfgyl gehgushy naq gur frpbaq vf qrprcgvir” (paste into rot13 to decode)

So the next time you come across a product with a suspicious review, you might want to check it out with the current demo version of Review Sniper.



Share this post:

| More

Previous post:

Next post:



  • Chris

    Here is a game theory reason why one might choose to post a deceptive review:
    http://xkcd.com/958/

    If he enjoys the hotel, he posts a negative review to reduce demand, which lowers the price and ensures vacancies next time he visits.

  • Scott

    @Chris, you bastard, I was going to link to that!!

    :)

  • Chris

    :)
    I was going to submit it earlier to Presh, so he could write an article about its logic.

  • Michael

    My reactions before reading which review is which:

    First review:
    Sounds mechanical, does not have much in the way of specifics; my immediate thought is that this review is fake.

    Second review:
    Lots of details, and sounds like a real review at first, but several red flags are raised. First, the reviewer is too specific in mentioning names. How many people refer to a hotel by its full name and repeatedly? (“the James Chicago Hotel” and “the James Chicago” versus the first review which just says “The James”.) Second, “The area of the hotel is great”, what is that referring to? There is a mispelling, “definatly”. Overall, review sounds overly gushy and not genuine.

    Initial impression: first review fake, second genuine
    Final conclusion: both reviews are fake

  • Michael

    It’s interesting to try variations and see what happens. For instance, I took the second review apart and red it to the review sniper one sentence at a time:

    “My husband and I stayed at the James for our anniversary.”
    – slightly deceptive

    “This place is fantastic!”
    – mostly truthful

    “We knew as soon as we arrived we made the right choice!”
    – slightly deceptive

    “The rooms are BEAUTIFUL and the staff very attentive and wonderful!!”
    – mostly truthful

    “The area of the hotel is great, since I love to shop I couldn’t ask for more!!”
    – slightly deceptive

    “We will definatly be back to Chicago and we will for sure be back to the James Chicago.”
    – slightly deceptive

    I also made up my own fake review:
    “The hotel is the best. From the time you arrive at the door until the moment you leave, you will be treated by royalty and will be pampered like a baby. You will also discover that you paid less for this five star hotel than the bum down the street paid to sleep in the alley behind the hiline motel. If you stay at this hotel and don’t agree with me that you have experienced heaven on earth, the manager and i will both give you a million bucks.”
    –mostly truthful

  • http://www.mindyourdecisions.com/blog/ Presh Talwalkar

    Thanks for that xkcd, that is hilarious though as pointed out similarly destructive!

    Michael, nice one about parsing the false review. I see there is a long way to go before algorithms are fool-proof.

Previous post:

Next post:

This site is for recreational and educational purposes only (privacy policy)

Powered by WordPress