The Underground Economy of Company Reviews
Online company reviews are high stakes.
Top reviews on sites like Glassdoor and Google can get thousands of impressions each month and are major drivers of brand perception.
Employers know this. And when I come across multiple 5 star reviews left with no cons, or a Pulitzer worthy essay from a former intern, I become suspicious.
These reviews start to resemble 30 under 30 lists: so artificially constructed that you begin to question their credibility in the first place.
The scrutiny around company reviews is well documented; some companies file lawsuits worth over a million dollars to reveal anonymous reviewers that complain about their jobs.
Whilst it's the flashy lawsuits that make the headlines, there also exists an underground economy of company reviews operating quietly every single day.
In this underground economy, some companies pay over $150 to freelancers to try and get a negative review removed. If they want “better” results, they go to the plethora of Online Reputation Management services (ORMs) in the United States that can charge retainers worth thousands of dollars.
The supply of positive reviews exists too. My research led me to find companies, including a prominent Y-Combinator backed startup, that solicit fake positive reviews from online freelancers to improve their rating.
Many of these mercenary fake reviewers, often based in South East Asia, make a full time living doing this, netting over $2,000 per month.
Some of these run such sophisticated operations that they’ve even created their own pricing tiers (e.g $35 per original review, $20 to post an already created review from an email address), a la SaaS offering.
Others operate on a contingency fee agreement model, where they only get paid if they’re able to take a negative review down.
The underground economy of company reviews is well and truly alive. And today we’re going to find out how it operates.
The barriers to entry for adding fake reviews are much lower than for getting reviews removed, so that’s where we’ll start.
To write an employer review, all you really need is the ability to create an email address. For most sites, you don’t need any proof of employment (say like a company specific email address).
I went on a gig marketplace site and posted a pretty vague post related to wanting to find out more on how to improve a company’s online presence.
Within minutes of posting a gig, my inbox was flooded with proposals:
After a bit of chatting, I narrowed the scope of their services and summarized their rates into the table below:
Adding Fake ReviewsChannelCostTimelineModelFreelancer #1$10 per reviewMonthlyUnlimitedFreelancer #2$35 per original review, $20 per already created reviewMonthlyUnlimitedFreelancer #3$25 per reviewMonthlyUnlimitedFreelancer #4$25 per reviewMonthly10 ReviewsFreelancer #5$20 per reviewMonthlyUnlimitedOnline Reputation Management Agency$300 SubscriptionMonthly8 Reviews
Let’s dive a bit deeper into the services that Freelancer #5 offered.
Freelancer #5 explained to me he had been writing reviews for one particular company for the past 4 months now. Each month he wrote them 10 reviews.
In another message, he tells me he’s offering the same services to 5 other companies. Doing some quick math:
5 companies x 10 reviews per company x $25 per review = $1,250 per month
Considering the average person in Pakistan earns $150 per month, that’s not bad change at all.
One of the companies that he’s offering his services to includes a Y-Combinator backed startup. I won’t name the company, but here’s what its average Glassdoor review rating distribution looks like:
5 star reviews account for over 77% of the company’s total reviews. Obviously, no one is buying fake reviews that make them look bad.
But here’s the thing: freelancers are getting quite smart when it comes to writing reviews that don’t look too fishy. They tend to do this by spacing the reviews out (so that they don’t come in “spikes” – more on this later) and they also make sure that they’re not always leaving the “cons” section blank.
Don’t get me wrong, if you come across this company’s reviews, it’d be pretty easy to tell they’re quite strange. In fact, I can’t even post some screenshots here because it’d give the company away immediately.
But it would be challenging to conclude that the above company is buying reviews just by analyzing review volume and distribution without actually reading some of the reviews.
The same company is also buying reviews on Google Reviews.
Sidenote: I got curious about how he’s been writing 50 reviews from 50 different emails per month. Would he actually create 50 different email addresses? And what about the IP address – doesn’t Glassdoor flag multiple reviews from the same IP?
One of the freelancers answered my question:
Moving on – another company that seems to buy fake reviews seems to be having some more trouble. Approximately a month after a freelancer linked me to fake reviews he had written for this company, all five reviews that he had linked me to had been removed:
Based on this Glassdoor webinar from 2018, “if it is found that a user has created multiple email accounts to submit reviews, then ALL submissions from that user are deleted” – so likely Glassdoor’s content moderation team flagged one of the initial reviews and the same freelancer who was writing reviews for that company had all the fake reviews deleted.
So far, it looks like the key to an effective fake review creation strategy lies in:
Spacing the fake reviews out
Writing each review from a different IP address (i.e benefit of being part of a team)
Using language that isn’t an obvious giveaway
On that third point: the reality is that many of these freelancers’ first language is not English.
As an experiment, I turned to everybody’s favorite new toy, ChatGPT, and asked it to write me a positive Glassdoor review:
And I’d say that the above answer was better than 95% of the fake reviews I came across.
The process for removing an employer review usually works like this:
You identify one or multiple reviews that you want removed
You verify whether the review violates the site's Guidelines, or whether there’s something else about the review(s) that could get it removed.
You file an appeal to get it removed.
As an example, Glassdoor’s Review guidelines can be found here. Mainly, they forbid mentioning anyone by name who’s not an executive and revealing proprietary or confidential information, amongst a host of other things.
Sounds simple enough right? Well, according to one of the freelancers I messaged:
After some research, I summarized the different vendors and prices in the table below:
Removing ReviewsChannelCostTimelineModelSelf-Reported Success RateFreelancer #1$100 per review3 DaysContingency Agreement Model100%Freelancer #2$30 per review7 DaysContingency Agreement Model100%Reputation Management Service #2$450 per review21 Business DaysContingency Agreement ModelUnknownReputation Management Service #3$1000 per reviewUndefinedContingency Agreement Model100%Reputation Management Service #4 Plan 1$550 per review5-6 WeeksContingency Agreement Model50-75%Reputation Management Service #4 Plan 2$300 Subscription + $100 per each review removedMonthly ServiceSubscription Plan50-75%Freelancer #3$20UndefinedPay regardlessUndefinedFreelancer #4$500UndefinedContingency Agreement ModelUndefined
As you can see, unlike the fake review generation market, the prices vary quite a bit for getting reviews removed.
At one end, you have freelancers on gig marketplaces that will attempt to remove a review for less than $100. And then on the other end, you have ORMs (Online Reputation Management Agencies) that have multiple employees and more comprehensive packages in place. The one constant seems to be that most companies operate on a contingency agreement model (i.e pay only if review gets removed).
I didn’t really understand the variance in prices so much, so I decided to ask.
Nader Nadernejad is 25 years old and the founder of Nadernejad Media, one such online reputation management agency.
Nader, based in Halifax, Canada, started his company almost 7 years ago and works with 3-5 larger corporations each year to help them with their online reputation.
He tells me that about half of those corporations also ask him to help them with their Glassdoor Review presence. The person doing the reach out is often the Chief Employee Experience Officer (CXO) or the Head of Employer Branding.
I hope you're enjoying the article. In my next article, I investigate the Strange World of Referral Bonuses - enter your email and I'll deliver it straight to your inbox when it's out in 3 weeks.
One of the most common requests he gets is when:
“Companies come to us and say they just got a random influx of negative Glassdoor reviews all of a sudden. For example, they’ll get a bad review one day from a disgruntled employee, and then shortly after 7-8 negative reviews come in a day after.
In this case, we’ll contact Glassdoor, tell them that this person was upset, that they went in and made a bunch of accounts. And usually what you’ll notice is that those types of reviews will disappear.”
The problem, Nader tells me, happens when a specific person has a vendetta against the company and knows what they’re doing:
“So in this case, they are able to change their IP address, put a lot of effort into destroying a company – and in that case, our client will go: “What do we do now?” We’ve even seen instances where they’ve suspected it’s a competitor company leaving these reviews.
In those cases, when we can’t get Glassdoor to get the review(s) taken down, we recommend getting reviews in the most organic, authentically way possible. And to create a repeatable process where you can garner those reviews.”
The above is what really points to the difference in price points. Nader’s firm consults more strategically on a company’s review landscape, often offering ways to improve employee morale and to find ways to effectively collect reviews. The freelancers, on the other hand, are just focused on getting a specific review removed.
I decide to ask Nader about what happens when he sees a review that he doesn’t think is actually based on fiction or from one employee who is trying to sabotage a company. Essentially, what happens when a company asks you to take down a (presumably) legitimate review? Does his firm still try to suppress the review?
Nader is emphatic:
“There’s no way for us to remove the review. There’s nothing I can do to get it away. And if we were to flag a review that is legitimate, Glassdoor would just say “nah, this is a real review”.
And there’s easy ways to see that. The IP Address that it comes from, the language that was used, is it similar to previous comments? Is it a duplicate? Is it the same sentiment?
And so our goal becomes to accentuate the positive experiences within the company. Whilst it’s very easy for one employee to have a toxic experience, that won’t necessarily mean the whole company is toxic.“
Glassdoor itself is adamant about the fact that a company can’t just remove reviews because they don’t like them. For a review to be removed, it has to actively break their Community Guidelines.
To test this out, I decided to send one of the larger ORMs in the industry an email about taking down a Glassdoor review for a completely random company.
I don’t have any plans to actually go through with removing this random review, but I’m more curious about whether an ORM would turn me down or not. Would they still try to tell me it can be removed?
The review I pick appears to be perfectly legitimate. It’s balanced, doesn’t seem to break any of Glassdoor’s review guidelines (like naming someone or threatening anyone), and is quite clear to read.
A couple of hours later, I hear back from the ORM:
The ORM also explains to me that this will not be an “easy” removal. Their primary strategy for getting the review removed will be a Letter of Demand, which according to the ORM is one of a review management site's tools to establish communication with the public when something is wrong:
“We will use it to claim that the review contains false information, not necessarily negative, but false, then we'll create a case where they will analyze and we'll negotiate to take it down.”
I also sent the review to a freelancer, just to see what the differences in services offered looked like.
Comparing them:
ORM vs FreelancerVariablesOnline Reputation Management AgencyFreelancerCost$550$150Stated chance of getting review removed50%100%Expected timeline5-6 Weeks3 WeeksPayment modelContingency Agreement (pay only if review gets removed) OR Subscription Plan ($300 per month + $100 per each review that gets removed) Pay now, regardless of whether review gets removed or notContract needed?YesNoPrimary Strategy of RemovalLetter of DemandBad review removal request
Obviously, I have no idea if this review would actually get removed. What is true, though, is that there are a bunch of different services offered to try and get legitimate reviews like the above taken down.
When I ask the ORM on why I should trust them, they get back to me with:
“We have been working with Glassdoor reviews for approximately 8 years, during this time we have helped several companies with their negativity. Each case is different, there are companies that want us to manage all their negative reviews and some others that just want to focus on a few ones. Our success rate is approximately 75%. “
ReviewMeta is a site that analyzes Amazon reviews and tells you how many are legitimate. The creator of the site, Tommy Noonan, mentions in an interview with NPR that the main giveaway that a product is soliciting fake reviews is:
- A large, suspicious flood of positive reviews at the exact same time. For example, a 3 day stretch of time constituting 30% of total reviews.
- Phrases and words that are constantly repeated, especially in the section with no cons
- Brand monogamists (only review products from one company)
Whilst the last two bullets are hard to track, the first can be used to analyze different companies’ reviews and to check if there might be some funky business going on.
After a couple of days, I have the ability to track review volume and review ratings over time for any company that I specify:
Let the games begin.
Voluntary Response Bias
One of the biggest challenges that review platforms face is the Voluntary Response bias.
Research shows many of today’s most popular online review platforms (e.g Amazon) have a distribution of opinion that is highly polarized, with many extreme positive and/or negative reviews, and few moderate opinions.
Think about it: have you ever felt moderately satisfied at your job and thought to yourself, now would be a great time to leave a Glassdoor review? Probably not.
On the other hand, if you’ve had a terrible experience or even just had one thing really flip you off, you might be quite likely to leave an angry review.
Consider when a company goes through layoffs. You’re going to have a flood of angry reviews coming your way and are likely going to experience a “spike” in reviews.
Note: Just like the Wall Street Journal’s methodology described here, I considered there to be a spike if the total number of reviews in the month was greater than three standard deviations above the mean of the surrounding months.
Let’s take the company below. Here’s a graph of of their review volume since Jan 2020, including when they announced one of their first round of layoffs in June 2022:
In June 2022, approximately 19% of this company's 52 reviews were 1 star reviews (compared to an overall average of around 10%). This is what we could call a statistically significant spike in reviews.
It also illustrates how the employees most likely to leave reviews are the ones that obviously had a bad experience (i.e getting laid off).
Here’s another company that had a similar spike in negative reviews due to layoffs in November 2022:
This company had an approximate 20% 1 star review rate (compared to an overall average of 12%) in November 2022, as well as an Avg Rating of 2.96 that month (compared to an overall average rating of 3.73).
Unless HR is proactive, their reviews page risks succumbing to an echochamber of negative reviews that can really tilt one way.
Note: Glassdoor does state (based on this video from 2017) that about 75% of the reviews on their platform are neutral. Their “give to get policy” has helped in keeping the platform from becoming too polarized.
I can understand why HR teams, like the ones that Nader talked to me about earlier, take a proactive stance towards managing their reviews. If they don’t try to control their reputation themselves, then their reputation risks getting controlled by the employees that had the worst possible experience.
Goodhart’s Law
Goodhart’s law states the following:
"When a measure becomes a target, it ceases to be a good measure"
Every October, Glassdoor publishes their Best Places To Work ranking.
In a report that the WSJ did a couple of years ago, they found large spikes in the number of reviews that some companies (e.g SpaceX, Bain & Co, etc) got in September. The logic here is that some companies try to artificially inflate their Glassdoor reviews right before the October deadline.
I decided to revisit some of this analysis with Glassdoor’s 2023 Best Places To Work Ranking.
One of the companies I examined is rated as one of the best places to work in 2023. Let’s refer to this company as FunPlaceToWork.
Here is how their review volume looks like for all of 2022:
FunPlaceToWork got around 50 reviews in September 2022. Of those 50 reviews, 96% were 5 star reviews.
FunPlaceToWork averaged 12 reviews per month up till then in 2022. Also, in the prior six months, the average percent of 5 star reviews received every month was ~75%.
Both the spike in volume of reviews and the spike in percentage of five star reviews are statistically significant.
I find it strange that Glassdoor’s proprietary algorithm and/or Human Content Moderation team did not find a spike of this nature unusual. If we look at Glassdoor’s eligibility criteria for the award, it’s as follows:
The goal, according to Glassdoor, is to collect “authentic and unbiased reviews”.
Whilst there’s nothing against the rules for asking your employees to leave you reviews, I find the statistically significant spike of reviews at odds with the goal of collecting "unbiased and authentic" reviews (which Glassdoor states is the purpose of the awards).
Glassdoor states that an employer is allowed to ask its employees to leave reviews, but that they are not allowed to “coerce” them. Examples of what you can’t do:
Offer incentives like Gift Cards in exchange for positive reviews.
Withholding their reference letter unless they leave you a positive review.
Anything that leads you to require proof for the employee to show you that they wrote a review.
It is possible to play by the rules (i.e not break any of the above rules) and to still in my opinion not collect authentic and unbiased reviews.
They say that you shouldn’t hate the player but the game – I think FunPlaceToWork played by the rules, won fair and square, and that this is simply a perfect example of Goodhart’s Law.
I reached out to Glassdoor ([email protected]) about the above and this is the reply I got:
When I was 22, on an F1 visa with 3 months to find work, I didn’t give a damn about bad reviews. I needed a job and I’d sign any piece of paper you put in front of me.
Compare that to someone at the peak of their career, someone with optionality and a multitude of job offers; an “A-Player”, as the experts call it, would absolutely have the luxury of choice and discard a job offer based on bad company reviews.
For most people, the impact of online company reviews lies somewhere in the middle. In marketing, there’s a concept of a “marketing touchpoint” - an interaction with the brand over the course of the whole buying journey.
Company reviews are one of the many touchpoints a job seeker experiences over their interview process. And with the technology industry booming the past couple of years, companies couldn’t afford to slack on any touchpoints, including this one.
After all, when others start to game the system, you’re at a disadvantage if you don’t. The rewards can be quite high. Certainly higher than just trying to be as transparent as possible.
HR leaders are often more incentivized to inflate their metrics than to get honest feedback. Fake review writers have bills to pay. ORMs know that companies are desperate. And the platforms, well, aren’t always paying attention.
The result is a potluck of interests that leads to an underground economy.
One that ends up hurting the job seeker.