Tackling the Comment Voting Problem
I took a break from Tildes for a week and came back to look at things again with a fresh perspective.
One of the things I immediately noticed was how the earliest comments are the ones that get the upvotes to the top of the comment list, and tend to stay there, even when better comments and chains flow below.
I started thinking about why this is so pervasive. Not just on tildes, but everywhere. Reddit and tumblr both suffer this issue to a degree. At the end of the day, going through any comments requires a certain amount of time, and a certain approach to the existing library of commentary. If we lock in the amount of time an average person will examine comments (which...is not much), we’re left with the only thing to address: the approach to going through the existing library.
Plenty of proposals (mostly already done) come to mind. Perhaps you go by most active or most recent comments. Controversial perhaps, or sorting by newest, rather than most popular. Maybe some secret mix of it all (the reddit “hot” formula). What about complete and utter randomness? ...yeah remember that Certain Amount of Time we discussed earlier? It’ll only be a couple posts before the user will switch back to another sort method.
So what should we try? What HASN’T been tried?
What about multiple panes? User-selectable, arrangable, 1-4. Vertical columns of different views, updated dynamically synchronously or asynchronously for the most controversial, new, and active. You could see all the views at once, side by side, so that your time switching between views and waiting for page loads evaporates and 100% of that limited attention span is spent on the comments in each of the sorts.
Having the more rapidly-changing columns (newest, active) update synchronously (every # seconds, configurable) would allow a user to engage those comments in time for the next refresh. The less-rapidly changing columns could be set to be asynchronous- updating as the orders change (top, controversial). This can also be tweaked as the site gets either more or less active as a whole. So what might need to be asynchronous now while things are quiet, can be made synchronous later.
Again, all of this is just a possibility, or perhaps starting point for a way to address the overall issue of the first comments being the most voted on.
I think we can certainly try to find ways to do better, but overall, this is a problem that's extremely difficult to solve. The core problem is that there's an inherent feedback loop—a comment getting votes means that it gets more visibility (if that isn't true, the voting system doesn't have much of a point). More visibility means that more people read the comment, which means that generally more people will vote on it. Votes lead to more votes.
It's a conflict between having a voting system help make it easier to figure out which comments are the most worth reading, and making it "fair" to all comments. It's just not really possible to have it both ways though, leaning more toward one side will hurt the other.
That's probably a bit like how reddit's "best" sorting algorithm for comments works (the default one): https://redditblog.com/2009/10/15/reddits-new-comment-sorting-system/
IIRC correctly, HackerNews uses some form of this, though they are fairly tight-lipped about the exact implementation.
I can think of two things that might help with this problem. Might.
We definitely need to move beyond the basic thread view with some sort of organization. I think the tags can be used to group similar comment chains by topic (such as humor, offtopic, etc) which would then let users set their own preferences for which group they'd like to see first. If groups can add their own tags and sort comments based on those tags we've given people enough to experiment with that they may tease out better ideas for sorting over time.
The other option is to move beyond the standard ranking model in some way, ignoring the votes. Split threads into 'old' and 'recent' views based on time. Mix up the order of the comments so that new ones get some brief time at the top. Editor-level tools such as special tags could be used to alter the comment rankings. Longer comment chains could gain or lose visibility depending on the quality of the discussion and number of replies. Different thread management modes could be active based on the amount of activity in any given thread relative to the mean/median for that group, or based on hard thresholds, because a front-page style discussion with 10k comments is going to need a different mechanism than a smaller thread of 100 comments to reach its full potential.
There's only so much we can do with the 'thread view' model, though. Moving beyond that might be the real win, if we can come up with something better to replace it.
Yes, but there is a temporal inequity involved here, as well: a comment posted earlier has more time to get votes than an equivalent comment posted later. The first person to comment on a thread will always have an advantage over later commenters. That's not to say that a later comment can't make it to the top of a thread (one of my highest-voted comments ever on Reddit was posted in an already-full thread, about 8 hours after the post was made, and still made it to the top, and something similar happened to me here on Tildes a few days ago). However, commenting first or early will always provide some inherent advantage over later comments.
I don't pretend to have a solution, but I think it helps to define the problem.
Votes could be made completely invisible to everyone and only used to sort. The top ten top-level comments would be randomized, or more as a thread grows. I love this place so far, but I already see the votes thing reminding me a bit of (one of the many reasons) why I don't use reddit much anymore: we instinctively keep track of what comments "do well" (whatever the metric; upvotes, votes, position on page). Reddit makes it implicit with karma. Tildes doesn't, but we realize that certain comments "did well" all the same, because we can see the vote count. If we couldn't, we could say "it was in the top-20," but all that meant was that it was at least 20th most-voted. It could have been first; we'd have no idea.
Scrapers or whatever (I'm not a web guy) could probably break that, though. Look at all the sites that track reddit votes and do undeletes of the data.
Edited for clarity
That wouldn't change anything. The comment with the most votes will still be at the top of the page, and will therefore have an advantage in gaining more votes.
I did consider a random sorting, but voting on a comment has to provide some sort of feedback. If the comment with the highest number of votes is buried near the bottom of the page, what's the point of voting on it?
Although... there would still be the aspect of trust and reputation. All votes would contribute in some way to a tilder's trust/reputation, even if the vote counts aren't visible in the thread.
Not if only the "top" is randomized, the top being dynamic by number of top-level comments. Top-10, top-20, etc.
Those probably deserve more votes, although earliest comments still have a large advantage.
It wouldn't be; it would be in the dynamic top-x, and therefore visible to all but the most casual reader of the comments. This probably wouldn't work well with a very large number of comments, because when there's 300 children to every comment, most readers aren't getting to the 20th top-level comment, but at that point, it would be the top-5 top-level comments.
Exactly.
What if the top x comments can't be voted on at all? Let's say the vote button is removed on the top 5 comments. These can't be voted on at all, until a lower-ranked comments gains enough votes to replace one of them. Also there needs to be at least 6 comments in a thread before any of them can be voted on.
Same for child comments, the top 5 child comments of the same comment can't be voted on.
(It would probably be better to calculate the number based on the total number of comments instead of using a fixed number. Might also have to be different numbers for top-level and child comments.)
This would ensure that the top x comments would get replaced continuously.
Not purely for the sake of replacing comment, but for the sake of giving better comments posted later a chance to become the top comment.
In that case it will stay in the top x, being immediately voted back up as soon as it gets replaced.
Because that generally doesn't work well. Look at the top comments on Reddit, YouTube, Imgur, or your favourite online newspaper. Memes, pop-culture references, conspiracy theories, harassment. Usually you have to scroll some way down to find anything of real value. That's the whole point of this thread, we don't want Tildes' comment sections to become like these places.
We've already established that this can and does happen. I think you're overestimating the amount of times that this does not happen and is therefore a problem.
All of these environments have a distinct "society" that exists on them. The way their users are encouraged to interact is not the same as they are being encouraged on Tildes.
There might be a bit of the tragedy of the commons as this site expands, as people who want deep discussion are likely less numerous than those who enjoy memes, pop-culture references, etc. but if we can create a environment where most people don't upvote these posts, report these posts, or use other tools to curtail their pervasiveness I don't see any issue with the current voting system. It allows me to see what people have the most agreement with, or the most detailed comments without having to read through a bunch of trash.
Fuzzing.
Little-seen comments should be considered poorly-classified, and have a large ranking-error value. Try slotting them into different positions on the list.
Limited-exposure placement might also work -- show otherwise unranked comments (or posts) to a subset of users. Increase exposure as votes increase.
Have you considered completely replacing the overall "vote" mechanic with tagging? I know that you intend to implement comment tagging in the future, but from what I've read, your plans are to use it only for negative tags. Both positive and negative tags seem useful and like an overall improvement. Comments could be "Well-sourced" or "Unsupported"; "Verified" or "Misleading"; etc. Then, to help sort the whole thread, users could vote on which tags are more important for the particular topic. In a casual thread, "Humorous" comments might be voted to the top, whereas in a news thread, "Well-sourced" and "Verified" would be the most important, with the number of votes on each comment tag being used to sort it further.
There would be no "Agree" or "Disagree" comment tags, which is often how up and downvoting is used. I like that Tildes has no "Disagree" button, but I'm sure that the general "Vote" button is often used to mean, "Agree."
I think about it quite a bit. My main concern is that I don't want to add too much friction to the voting process. It's good for voting on comments to be a common action that people can do quite freely, and if you have to justify every single vote by selecting a reason it starts to feel like a lot of trouble. Then you end up with the voting system losing efficacy because people don't want to bother using it.
I also think (as I've said before) that the utility of the positive tags is somewhat lower. When I go into a thread, I'd usually never think things like "I only want to read the 'insightful' comments in this thread", but it's very common to think things like "I don't want to read the 'off-topic' comments or the 'meta' ones".
I do think there's value in some positive tags, but I don't think they can replace voting entirely. I think voting is useful for other purposes more similar to "thanks" and "acknowledged" a lot of the time as well - it's fairly common for me to just vote on someone's reply to me as a way of implying it's been seen, instead of feeling like I need to reply to everything.
Exactly what I was thinking. Hence why I started to think...why not both?
Maybe Hacker News' approach of reserving the top comment spot for a random, or more recent, comment. It's been proposed before here on Tildes. I like its simplicity.
The column thing is an interesting idea, but I mainly use this site on mobile and I feel like it'd be horrible on a small screen, so it'd have to be desktop only I think.
This happens on StackOverflow as well, the earliest answers, no matter how outdated they are, somehow still remain on top. There's no good function for keeping the good content up top, while the outdated or less valuable stuff floats to the bottom.
You're on to something here. If one view with one function determining which comments fall and rise isn't the solution, perhaps two or three or some other mix of views and functions are a potential solution.
Heck, why not a TL;DR of 3 comments or sampling of comments? It's started to be done for app store reviews "X out of Y reviews say the graphics of this game are good" or something like that. That could also be a way to aggregate and surface particular comments to support the aggregate view.
I thought Reddit used the 'Wilson Score Interval' for it's Hot posts. It makes a predication of the trajectory of a post and then brings it higher than the first comments. I don't know how effective it is though. This post on Medium describes it in some detail.
I thought Tildes was heading towards giving individuals some kind of weighted score based on quality of previous posts. Has this been implemented yet?
Weighted anything is not yet implemented.
I personally would like no votes for anything, what's the point?
Sorting for one, to keep subjectively "better" content more visible.
Also, @Kat makes a good argument
I think random would be a interesting thing to try because it would create more 1 on 1 communication instead of just getting lost in the crowd. I think that's what Tildes is all about and if we keep bringing in users of the current quality that we have then there won't be a quality of comment issue. However that would be the question, what is the point of having voting on comments in the first place?
My point is if you implemented the random comment sort then the point of voting kinda goes away
In some sense it would be fun to have a toggle where if it's on, a comments would load not based on the total vote counts for a thread, but for the number of votes you have given a user. This would give everyone a different list of the comments and their chains and would encourage users to try and find other users who post high quality content so that their feed is front loaded with high quality content. Meausres would have to be put in place though to prevent a user from being presented as the first comment of every article (maybe just shuffling after categorizing engagement).
Of course, if you use the vote button as an 'I Agree' button, you will effectively create an echo chamber. But if you use the site as an engagement tool, it would really be a decent way to self-filter out content you feel is continually from 'low-effort' users.
That's an interesting idea. It almost reminds me of how, on Reddit using RES, you might tag users who post stuff you like.
This model would allow you to continue to see your own personal "best" content, without your opinions on what's "best" affecting the whole userbase. Or maybe that's only the case in "personal sort" mode and in "popular sort" mode, you'd see an aggregate of eveyone's voting opinions.
There might be a problem in the long run, though, with people creating their own personal echo-chambers and with new users not getting a good experience unless they actually engage with ~ for an extended period.
It would definitely be interesting to see how it plays out and see how people change their engagement over time. I feel it would run into issues with scaling across thousands of users. On the back end, it might be more scalable to create hidden groups based on yours and others preferences and lump people into those buckets.
No matter what solution you employ, there's one aspect that will never go away: Some people will go through a topic, vote on comments, then never return. Even if you do return multiple times, at some point you won't come back again and there will probably be more comments that you end up missing. Thus, any good comment that is made earlier will naturally have a greater potential pool of votes to draw from than a newer comment.
Let's take the simplest example where a single person views a topic at any given time, votes on all comments, then leaves a comment of their own, and never returns. In this scenario, you'll see something like this occur:
If you repeat this pattern, you eventually get:
The way we can explain this generally in plain English is that the total potential voting impact of any one user is equal to the number of comments on the topic during that user's most recent visit. The corollary to this is that the more recent a comment is submitted, the smaller the impact a vote from a single user is because that comment will receive only 1/N of that total potential voting impact from that user.
The only way to combat this is by adding some multiplier to that 1/N, but that would be incredibly difficult to get right and you would have to account for e.g. unvoting and revoting a comment or what to do in the case of revisiting a topic, which could effectively reset the weight of the original vote, allowing for vote manipulation (a solvable problem, but the point is that it's a tricky one). Such a factor would also be unfair to early voters as it would "punish" them for viewing content and contributing to discussions earlier on. A temporal factor would have the problem of e.g. waiting for topic activity to die down then using your votes when they're valued more. No matter what vote fuzzing strategy you employ, it's only ever going to be a heuristic measure, someone's voting power is going to be significantly affected, and you will probably just open yourself up to voting manipulation strategies.
There's a reason this problem is so pervasive.
For completeness' sake, I feel I should note that at any instant where our variable N is some constant k (meaning no new comments have been made), every returning user increases the collective total potential voting impact toward N/N = 1. A collective total potential voting impact of 1 would mean that each and every comment could potentially have N votes, whereas 1/N (the case where a new comment was submitted and no users return to potentially see it) would mean that only 1 out of the N comments could potentially have N votes. Additionally, the total potential voting impact is only a guideline to help understand the potential maximum votes a comment could receive. Varying voting behaviors and other miscellaneous factors (like comment visibility) will have a significant impact on the actual results.
In other words, it's just a thought experiment to help consider the limitations of voting as a mechanism in general.