Reputation systems, "engagement" vs. participation, and the "first post!" effect
I'm mostly very appreciative of everything @Deimos has accomplished here; so far, it's been a very smooth and interesting alpha experience. I'm seeking some clarity on how the eventual reputation and trust system he proposes might eventually materialize, and would like to start a discussion among other users as to what mechanic they're seeking. [My apologies if this has been addressed previously - search functions are also anxiously awaited.]
There are multiple social sites (Slashdot, HN, Reddit, etc.) which use new/active/upvoted categorization for ranking front-page comments. This seems to be reproduced here, and generally, I don't have a problem with it as long as the permitted posts don't become just a reproduction of inflammatory click-bait available elsewhere, dank memes, etc.
However, on a per-user-basis, the first reasonably-well composed comment on a thread collects most of the votes. My observation is that in an active post thread, the best-reasoned/researched posts may occur after dozens/hundreds of comments, as people who don't spend their entire lives camping on a social site (highly-engaged!) join, read through prior material, and comment. These users don't garner the votes and reputation points which highly-engaged users might, even though they're working harder as quality participants. Threads die, potentially prematurely, because there's no reward for late arrivals or continuing disputation.
While this phenomenon hasn't become egregiously manifest on Tildes yet, there's certainly potential for it to arise. Would it make sense to age out the votes on the "first post!" comment, so that there's some encouragement for deeper or longer posts to continue on an active thread?
Since Tildes is ostensibly built to discourage the "engagement" tactics required to optimize for maximum ad views (e.g. https://www.bitcatcha.com/blog/instagram-tools-strategies-no-ones-talking/), is there another system which might further encourage participants to engage in thoughtful discussion and high-quality posts instead?
This is something I've been thinking a decent amount about recently. Regardless of how we want them to be used, the reality is that people use votes on comments for a lot of purposes, often just for "acknowledgement". I do this myself constantly—someone replies to me, and I just vote on their comment as basically a way of saying "I've seen this, thanks" when I don't really want or need to reply.
I think for reputation/vote-weight/etc. systems to work, we probably need a way to separate "I'm voting as acknowledgement" from "I'm voting because this is a really good post". Maybe we can just use the comment tagging system for that, but it's very possible that there will need to be more, and I'm not sure exactly what it is yet.
I know Slashdot had "insightful", "redundant", "interesting" tags for a while, and Amazon allows tagging for "helpful". It's possible that a multi-axis system is needed.
Yeah, that was my suggestion as well. A 2-click, radial menu based, contextual vote+tagging system:
https://i.imgur.com/d2cZ2GJ.png
I know for me, a 2 click vote system is too much effort and I’d be less inclined to vote unless it was above and beyond or I was in a generous mood
That isn't necessarily a bad thing IMO.
I agree, "effortful" voting, like putting the comment box at the bottom, seems like a good thing.
Slashdot doesn't even let you vote at all unless you've been given your semi-random allotment of mod points, and haven't already commented in the thread (another thing that might be worth considering).
That is definitely being considered. @deimos brought up the idea of "exemplary votes" ages ago, it just hasn't made its way into the documentation yet. But I am rewriting the site documentation as we speak and the potential mechanics of the exemplary votes are covered in it. It would most likely function very similar to what you're suggesting but it will also likely be tied into the trust/reputation system at some point too so that users with more trust in a group may have more weight to their exemplary vote in that group.
To counteract the "first
postcomment gets most votes" syndrome that the OP is worried about, couldn't you just change the default sort for threads to "newest first"?EDIT: I meant comment, not post.
It would help some things for sure, but it would also make voting mostly pointless overall. I think ideally we'd like to have a balance where voting is important, but also need to try to make it so that the first comments don't almost always end up with the highest vote counts as well.
Why not random sorting, like Reddit's contest mode? :)
That again tends to make voting meaningless. Voting either sorts the page or it does nothing except stroke the ego of the person receiving the votes.
I like to think the ideal balance is some algorithm that takes votes, freshness, and interactions into account when sorting, but every attempt at this has had its own inherent problems.
I believe that, in the longer term, votes will contribute to the "reputation" measure of "trusted users".
It won't be the only measure for deciding trusted users. I said "votes will contribute to the 'reputation' measure", not "votes will be the 'reputation' measure".
Read this: https://docs.tildes.net/mechanics-future#trust-based-on-consistency-and-accountability
And this: https://tildes.net/~tildes/2tk/suggestion_use_sortition_for_moderation#comment-v0w
Ah. Very good point.
Though, I still disagree with a fully random or fully chronological default sort. The community needs to be able to reinforce high quality content and draw more eyes to it. A fully random sort with enough people involved ends like looking at this: https://old.reddit.com/r/all/new/
I don't necessarily agree with a fully random sort, myself. Go back and read my earlier comment - and note the smiley face at the end. I was half-joking.
I think some form of mixing might help, particularly in the busiest threads.
There's no reason we need to see the same comments in the same order all the time on every page refresh. New comments can be mixed into the thread list alongside the old/popular ones. Top level comments could bump-up if there are a lot of replies being made to them. I'd like to do something to help get people's eyes spread around to more of the comments when they come into the thread.
We've also talked about splitting up threads a bit. Archive the 'previous' day's threads at the top-level into a 'yesterday' box and have a new one for 'today', or even let mods help splitting thread responses into categories based on the needs of the discussion - we could even get tags to do this and leave it up to the taggers. It's pretty clear we need to do something that goes beyond pancaking several thousand responses into one massive, unindexed multi-page wall of text. A further level of organization is needed, something beyond what prior websites have done.
So much this. Fuzz is a critical missing element in moderation /ranking systems.
We must be mindful of the danger of personal bias, malicious or otherwise. As soon as you introduce human intervention as a step for determining which posts gain visibility we open ourselves up to this risk. Regardless of good intentions, if we are going to allow mods the rights to affect what the masses see upon loading a thread then there needs to be extremely strict controls in place. If it is possible through rules-based automation, much better. In any case, the inner machinations of this would need to be made public if we are to avoid any adverse scenarios down the line where affected individuals start to call out favouritism or hail corporate-esque manipulation.
Just to feed the fire, while Metafilter had a vibrant discussion community at one time, its disavowal of a reputation system, and failure to gain consistent monetization via advertising, have relegated it to a backwater of the late 2010's web. Moderation without reputation may be a recipe for a long, slow decline: https://matt.haughey.com/on-the-future-of-metafilter-941d15ec96f0
I'll also include this, which I stumbled on in the search for the latest and greatest analysis of reputation systems in general: https://sloanreview.mit.edu/article/online-reputation-systems-how-to-design-one-that-does-what-you-need/
Metafilter's a site I've always wanted to look into more, and never quite have.
It had the highest general relevance score of any social network, indicated by the "FP:KK" score, by miles, though a small fraction of the content.
A thought I've been toying with generally across systems is some way to impose an appropriate cost to actions.
A key instigator of much negative behaviour is impunity. If an action has no cost, or carries no risk, or the person carrying them out thinks this, you'll see certain actions carried out. Immunity, impunity, indifference, ignorance, can all be superpowers, in the sense that those influenced by them have no concern of failure. Many evil masterminds are far less geniuses than simply uninhibited.
In an information and messaging / discussion context, each message carries an assessment cost: is this even worth my time? When channel saturation is low, so to is the cost. But as the channel fills or becomes oversaturated, the cost increases. If that cost isn't imposed on the message's originator, you get overprovision of low-value messages. Tragedy of the commons
Large-scale discussion needs some way of balancing this cost asymmetry.
The details are tricky, and I'm unsure of best solutions.
Negative points are an option. This plays poorly with brigading type abuse.
Initial cost is another: a post costs so many time-replenished points. Here, sock-puppets and identity (to bypass limits) become issues. Metafilter's nominal monetary fee is one approach, reputation vetting such as Advogato's trust root is another.
Limiting initial exposure to some randomly selected cohort is another. Effectively, if that subpopulation votes up (vets) content (posts, comments), that becomes more widely visible.
Subcommunity nomination is another approach: distinct groups source content tthaat is syndicated to largeer groups. Sizes being somewhat Dunbar's Number-ish. Path tried this, somewhat, it worked poorly, though might be improved.
There's also the question of what's too much information. I don't know of hard numbers, but suspect about 30 topics and discussions of fewer than 500 comments (and likely closer to 30-100) are probably a sweet spot. Tweeking to maintain this scale-independently of site growth may well be a success trick.
Some numbers:
Content rating, moderation, and ranking systems: some non-brief thoughts ... a long compilation.
Somehow, I knew you'd have a canonical answer.
Nonetheless, I want to make careful distinctions among:
It's considered. I don't ... erm .. consider it to be canonical.
That's some damn good reading, thanks for sharing it.
There is a very good short essay (?) recently written by the developer of HN that quite brilliantly explain how you can directly relate the length of a comment with its quality.
Of course it cannot be an absolute indicator, but I think that the ranking for "best" comments should take that into account as well.
Also: length, by itself, isn't a necessary signifier. A short response can be entirely sufficient: "yes", "no", "true", "false". A long response can be utter tosh.
Length commensurate to essential complexity, and more directly: a response appropriate to the discussion, should be the true measures. Assessing quality is a hard problem.
Absolutely, in fact I've wrote that it cannot be the unique identifier of a good comment. But certainly can filter out all the one-word, quirky and edgy comments just to begin with.
I've advocated for this sort of thing as well. Length and basic writing grade-level analysis can provide a lot of cues for a moderation system to use, and there are open source libraries and tools out there to perform these tasks. I trust them far more to 'clear out cruft' than I do to 'identify high quality' though. Junk comments (think youtube-level) can be reliably identified for automated action using this method.
I'm not really worried about teaching algorithms to spot the best comments, because that's pretty subjective and it won't be able to operate in-context of the thread/discussion. I don't think that's the right approach - eyes and votes (and replies) are what find the best stuff over time. I am worried about cleaning up the cruft, though - the less of that people have to slog through, the more attention and time they can give to the better content.
As long as it's one factor being taken into account.
I've seen some long comments that take a lot of words to say something that could have been said far more concisely and effectively in a comment a third of the length.
This happens when someone starts writing before being sure of what they want to say. Too often people don't take the time to distil their thoughts into a coherent, well made comment, and end up posting a long rambling "stream of consciousness", which is annoying to read, or scroll past.
My 2 cent on this is basically the reason I'm interested in Tildes.
I didn't join mainly for the "discussion" even if I absolutely would like @Deimos vision to come true, but because I strongly believe that it could be a very good chance at having machine learning applied "for good" in a community.
My idea is to have a first set timespan in which the algorithm should observe and learn by what the users votes, comments and tag. Without telling the community how long this time should be or when it will start. This mostly to prevent brigading the voting system and thus making the algorithm learn "bad" behaviour.
After the initial time, the algorithm can of course begin to identify what should be "best" and in parallel keep itself updated by still observing voting/commenting/tagging patterns.
I honestly think that the technologies available today can help a lot in enhancing the environment we live and something like this is just an example of what they could be used to.
The trust system itself, if you think about it, could rely heavily on machine learning to determine how much the users should gain/los trust, given some rules.
URL?
Paul Graham, Sam Altman, or Dan Gackle? Other?
Here: http://www.paulgraham.com/hackernews.html#f5n
Thanks! From 2009 ;-)
Sorry i can't find it but if I remember it well, it was shared here by @Deimos himself.
Perhaps a time/score weighted comment sort should be the default sorting method, that way those insightful late-comer comments aren't buried at the bottom? I suppose that would then make an issue of "how are those late comments supposed to get any points except by reverse time sort?" (barring the odd few people who choose to navigate to the bottom manually).
Perhaps set a random comment sort of "time weighted", "top", "best", and "reverse time sort" so that all options are seen and represented and can help bolster the visibility of those types of late "quality comments" that people generally hope to find?