This was a really interesting post. I've long wondered about the utility of Steam reviews, especially given how many people seem to be willing to use the review as either a pat on the back or a...
Exemplary
This was a really interesting post. I've long wondered about the utility of Steam reviews, especially given how many people seem to be willing to use the review as either a pat on the back or a slap in the face, rather than an actual rundown of the videogame. As this article points out, unfortunately the slap in the face is significantly more likely.
This post makes me feel bad for not reviewing the games that I play and also like a bit of a hypocrite for so readily consulting reviews without contributing any of them myself. Nevertheless, I seem to have this really big hangup about writing reviews on any platform. On one hand, I definitely value the presence of reviews and check them regularly for nearly anything I buy. I also greatly appreciate the people who give their time and effort to provide substantive and valuable feedback.
On the other hand, reviewing things myself feels... off? I can't really adequately describe it.
Part of it feels like I'm doing the platform's work for them. Why should I tell you how this thing you're selling is?
Part of it feels like I'm trusting the lion not to eat me when I put my head in its mouth. Why would I trust that you, as a store, will use my or any other reviews truthfully when you have a vested financial interest in selling these items to people?
Part of it feels like feeding even more valuable personal data into some very data-hungry companies. You already track my every interaction with your product, why should I also share how I felt while using it?
Part of it feels like I'm simply spitting into the wind of a hugely broken system. What value does my one little review have against the hundreds of others featuring unfair spiteful takedowns, useless irreverent memes, or fraudulent purchased praise?
I feel like what I want is for a review site to be separate from any storefront, public yet private, with high standards and a large userbase. This is but a pipe dream and would likely never get off the ground, but it's the only way I can see us getting any sort of aggregate review system that comes close to working.
I also would love to see it offer not only aggregate scores but also scores based only on similar profiles or other sectioned breakdowns. I don't care necessarily care how all 20,000 people felt about a certain game if I can instead find out how the 100 whose media diet most matches my own did. There's the now widespread belief that any piece of media has only one score of quality, arrived at by averaging the opinions of all of its critics, because more data allegedly leads to a more precise truth. The reality is that this is a horribly one-dimensional way of looking at complex, multi-dimensional responses, but we do it because it's convenient, easy to understand, and feels fair. At the very least we should be able to tweak the inputs to make that one-dimensional breakdown the very best it can be for us.
In the absence of a site that allows for more measured and thoughtful critical aggregation, however, I'm at the point where I simply don't know if there's value to crowdsourced data of this kind without some significant precautions and balances built into the mix.
I used to use Fakespot before online purchases to guard against products whose scores had been inflated by fraudulent reviews, but then I read an article about how companies would order fake reviews for competitors' items in order to try to sow distrust in their competitor's products and potentially get the competitor flagged for fraud. Life, famously, finds a way, and so, seemingly, does misinformation.
I don't trust the platforms with a vested financial interest in selling me stuff to be completely honest about the stuff they want me to buy, I don't have trust in companies to not try to fudge their products' scores or those of their competitors to their advantage, and I don't have trust that the average reviewers are even doing so correctly -- much less thoughtfully.
Yet, even still, I check reviews. For nearly everything. Why? Because even if they're fake or skewed or ouright inane, they still reduce the possibility space of uncertainty surrounding any product. In the absence of reviews my expectations are anchored to nothing. With reviews, they at least tether to something, awful as that anchor point may be.
I think this phenomenon, more than anything, is probably what the author and other game devs like him are up against. Even if only 30% of the recent reviews for a game are negative, and all of the negative ones are clearly about grinding axes rather than the game itself, a viewer still has their impression anchored to a pessimistic perspective of the game, however undeserved. It's the idea that I can stare at shit and know it's shit but still feel like there's some truth to the shit simply because it's there. After all, it's more real than the absence of shit, which could mean absolutely anything at all, which isn't helpful in the slightest.
I initially saw this on /r/gamedev here and planned to post it then, but got a sidetracked, so this is slightly old. Even still, this is something I'd not thought about at all and it was really...
I initially saw this on /r/gamedevhere and planned to post it then, but got a sidetracked, so this is slightly old. Even still, this is something I'd not thought about at all and it was really interesting, so I figure it's worth posting regardless. Of note, I chose to use the title of the reddit post over the article title itself since I feel the former better represents the article content.
TL;DR from the author via the reddit thread linked above:
For recent reviews, the average is about 10~15 per month (the lower-extreme is from my own experience). Since each review holds 7~10% "weight", it would only take as few as 5~7 negative reviews to drop you from 100% to 50% which is a quality control pool so low that it does not represent any form of accuracy, assuming that 10~15 players is significantly lower than your average MAU.
While most offline games don't experience "burned out" or "revenge reviews", online games suffer hard and every month.
"Burned Out" reviews are 200+, 500+, and often even 2000+ hour reviews that are "negative" due to enjoying the game too much and getting burned out, where it was enjoyable for the first 1999 hours but not the 2000th due to, usually, an obscure reason similar to when you're looking for an excuse to break up with your gf ;D
^ The auto-response to this is "What if they suddenly started being shady, +lootboxes, etc" -- I know. However, when does this actually happen? Everyone knows in 2019 this is indie dev suicide. That's like if 2 people steal a yogurt from your office break room per year, the company would just remove the entire fridge based on that. I get why this is said, and those that do it need to be called out, but what about the 99.99%+ majority that don't? If we gathered a % of all the games that did this on Steam, would it be less than 0.0001%? I'm willing to bet it would be an even smaller # than that.
"Revenge" reviews occur in retort to a moderation action: As small as a warning (even meta; eg, Discord). Even as small as an unlogged "warning for a warning" (we call an "FYI"). These forms of reviews generally appear within 24 hours of a disciplinary action and has the same # of hours as "burned out reviews" and will attack the dev on a personal (RL) level instead of actually reviewing the game, or masking the real reason for the review.
The average revenge reviewer will continue playing after their moderation action is over for up months/years to-come. However, the review will always remain negative.
Example dump of recent high # playtime reviews (ordered by playtime - and only a small sample pool of many more): https://i.imgur.com/XyqUzDl.png
Moderation "reminds" players to revenge review. Online games are social: Expect many revenge reviews to be accompanied by bountiful amounts of comments / other reviews from the entire group that this user players with (including bulk marking the review as "helpful" within a small period of time).
Before our moderation efficiency patch, we held 93% average in both overall/recent reviews. Ever since then, our average "recent" score averages between 30 to 60% due to these two forms of reviews. The only reason our overall is still 84% (still a big drop from 93%) is because we have already listened to the dominant "real" negative reviews.
Here's the gross part: If I had no empathy and ditched moderation practices altogether (we won't), our reviews would be significantly better. Even at the cost of population dropping from toxicity, higher % reviews brings about higher population flows of new players. The fact is, while moderation >actively triggers revenge reviews, toxicity passively hits players. This means if 7% of those that receive disciplinary action revenge review, only about 1% are likely to review for toxicity. This means that the current review system [indirectly] rewards devs that do not moderate their games and take care of their community members.
What's my point? Awareness, curiosity and perspective - consider it a blog of observations.
Maybe a less relevant point but I'll admit that there's been a lot of MMOs on Steam that I've taken an interest to, but refused to pick up and play due to negative reviews complaining about the...
Maybe a less relevant point but I'll admit that there's been a lot of MMOs on Steam that I've taken an interest to, but refused to pick up and play due to negative reviews complaining about the endgame or about the cash shop. They all match this description of having huge amounts of hours played. Two recent examples I can think of are Elsword and Soulworker.
This was a really interesting post. I've long wondered about the utility of Steam reviews, especially given how many people seem to be willing to use the review as either a pat on the back or a slap in the face, rather than an actual rundown of the videogame. As this article points out, unfortunately the slap in the face is significantly more likely.
This post makes me feel bad for not reviewing the games that I play and also like a bit of a hypocrite for so readily consulting reviews without contributing any of them myself. Nevertheless, I seem to have this really big hangup about writing reviews on any platform. On one hand, I definitely value the presence of reviews and check them regularly for nearly anything I buy. I also greatly appreciate the people who give their time and effort to provide substantive and valuable feedback.
On the other hand, reviewing things myself feels... off? I can't really adequately describe it.
I feel like what I want is for a review site to be separate from any storefront, public yet private, with high standards and a large userbase. This is but a pipe dream and would likely never get off the ground, but it's the only way I can see us getting any sort of aggregate review system that comes close to working.
I also would love to see it offer not only aggregate scores but also scores based only on similar profiles or other sectioned breakdowns. I don't care necessarily care how all 20,000 people felt about a certain game if I can instead find out how the 100 whose media diet most matches my own did. There's the now widespread belief that any piece of media has only one score of quality, arrived at by averaging the opinions of all of its critics, because more data allegedly leads to a more precise truth. The reality is that this is a horribly one-dimensional way of looking at complex, multi-dimensional responses, but we do it because it's convenient, easy to understand, and feels fair. At the very least we should be able to tweak the inputs to make that one-dimensional breakdown the very best it can be for us.
In the absence of a site that allows for more measured and thoughtful critical aggregation, however, I'm at the point where I simply don't know if there's value to crowdsourced data of this kind without some significant precautions and balances built into the mix.
I used to use Fakespot before online purchases to guard against products whose scores had been inflated by fraudulent reviews, but then I read an article about how companies would order fake reviews for competitors' items in order to try to sow distrust in their competitor's products and potentially get the competitor flagged for fraud. Life, famously, finds a way, and so, seemingly, does misinformation.
I don't trust the platforms with a vested financial interest in selling me stuff to be completely honest about the stuff they want me to buy, I don't have trust in companies to not try to fudge their products' scores or those of their competitors to their advantage, and I don't have trust that the average reviewers are even doing so correctly -- much less thoughtfully.
Yet, even still, I check reviews. For nearly everything. Why? Because even if they're fake or skewed or ouright inane, they still reduce the possibility space of uncertainty surrounding any product. In the absence of reviews my expectations are anchored to nothing. With reviews, they at least tether to something, awful as that anchor point may be.
I think this phenomenon, more than anything, is probably what the author and other game devs like him are up against. Even if only 30% of the recent reviews for a game are negative, and all of the negative ones are clearly about grinding axes rather than the game itself, a viewer still has their impression anchored to a pessimistic perspective of the game, however undeserved. It's the idea that I can stare at shit and know it's shit but still feel like there's some truth to the shit simply because it's there. After all, it's more real than the absence of shit, which could mean absolutely anything at all, which isn't helpful in the slightest.
I initially saw this on /r/gamedev here and planned to post it then, but got a sidetracked, so this is slightly old. Even still, this is something I'd not thought about at all and it was really interesting, so I figure it's worth posting regardless. Of note, I chose to use the title of the reddit post over the article title itself since I feel the former better represents the article content.
TL;DR from the author via the reddit thread linked above:
Maybe a less relevant point but I'll admit that there's been a lot of MMOs on Steam that I've taken an interest to, but refused to pick up and play due to negative reviews complaining about the endgame or about the cash shop. They all match this description of having huge amounts of hours played. Two recent examples I can think of are Elsword and Soulworker.