56
votes
Activision and Call of Duty have published a paper detailing skill based matchmaking and how its presence or absence affects enjoyment of games
Link information
This data is scraped automatically and may be incorrect.
- Title
- Call of Duty Matchmaking Intel - White Paper #2
- Published
- Jul 26 2024
- Word count
- 1148 words
Activision and Call of Duty have published a paper detailing skill based matchmaking and how it affects enjoyment of a game, returning player counts, and the number of blowout wins/losses that occur. The link in the title is to their article and the paper can be found here: https://www.activision.com/cdn/research/CallofDuty_Matchmaking_Series_2.pdf
It's nice to have some hard proof with a large sample size (Call of Duty's player population) showing the benefits and drawbacks of SBMM. It appears that for all except the very top players of a game, the presence of it increases "fun", decreases the number of blowout matches, decreases mid-match quits, and results in a higher number of players that return to the game. Turning off SBMM resulted in more players returning in the highest skill bracket, but fewer players returning in any other skill bracket and significantly more mid-match quits from lower skill brackets.
To me, the results of this are unsurprising. When I play CoD, matches seem to be much more consistent for me than they were back in the day, but there are still a contingent of people arguing that SBMM is a negative influence on a game. Personally, I heavily disagree as I think people arguing against it just want to get 100 kills per game and not actually have a fair match.
I don't think it's as black and white as you make it. Public servers back in the golden era of first person shooters provided something that multiplayer games at that time were lacking; a sense of progression. If you played a game consistently for a few months, you could tangibly see and feel yourself getting better. You would go from getting trickshotted across the map consistently, unable to maintain a positive K:D ratio and feeling like you were losing all the time, to top fragging every game and feeling like a virtuoso, as long as you stuck with it.
That intuitive sense of progression, and the sense of community that a small group of strangers that regularly play together are the biggest things we lost with the widespread move to SBMM.
Yes, there are many other advantages, so I don't mean to suggest that public lobbies are necessarily better, and games have come up with ways to artificially build in a sense of progression (publically visible skill rankings, weapon and cosmetic unlocks, etc), but it will never be as intuitive as being able to feel yourself getting better at the game, since you always sort of feel like you're treading water and winning exactly as much as you're losing.
I think for the average gamer that doesn't want to make a specific competitive video game their hobby, SBMM is superior in almost every way, and for game companies, the financial incentives definitely make SBMM the clear choice, but for the type of gamer I was when I was in my teens; someone who wasn't necessarily great at games, but who loved that feeling that the constant feedback loop of getting better at a skill provided, and who loved seeing the same people trying to do the same thing every time I logged on, there's something lost in the cold mechanical balance of always maintaining a 50% winrate, and always feeling like the people I'm playing with and against are ephemeral ghosts that I'll never see after this one game is over.
A big factor with public servers is you can also find a place you can call home. You encountered the same players, who would greet you, who you’d trade blows with and even could become friends with.
With SBMM there is a severe lack of community inside the game.
That's not really a matter of SBMM, that's a matter of matchmaking.
I don't understand how SBMM prevents personal skill development or progress. If anything, it should make that a smoother progression.
It's not about the progression itself, it's about feeling that progression. You'll get better at anything automatically just by doing it over and over. Without SBMM, you can easily see the fruits of that progression. You win games more, you're at the top of the scoreboard, people will accuse you of hacking.
If the skill level of the people you're playing against is constantly increasing it just feels like you're treading water.
A SBMM game where you don't see a rank badge or score rating would quickly lose players because they wouldn't be able to see themselves climbing a ladder, it would feel like every game was pontless. Games without SBMM didn't need a rank badge to retain players. Players knew they were getting better because they were winning more.
It's a much more natural feedback loop.
This seems more like an issue with how matchmaking can be implemented and designed rather than an issue with SBMM itself. I think a lot of this boils down to how Call of Duty designs its multiplayer formats and engagement because SBMM has become something of a scapegoat for its other issues. SBMM is something that is largely invisible when implemented correctly and can improve player experiences (including in regard to skill development). I remember some games back in the day had had pub servers that toyed with a sort of "handicap" system that approximated SBMM to some extent to try to keep things level by giving newer players a chance and more skilled players a more worthwhile challenge. And the situation changes greatly with 1v1 games like RTS games. StarCraft BW had a SBMM ladder system back in the day and it was fantastic, and really lent to natural skill progression a lot more than matching with randoms in a lobby.
The way I've seen it (relatively well) implemented in many competitive games, you still rise up the scoreboard as you improve. Then maybe you drop as SBMM puts you with more skilled players but then you rise up the scoreboards more as you get better with these new players. Rinse and repeat. And most competitive games these days do have a ladder of some sort that allows players to easily chart their progress. Players like having this sort of laddering rank, as Halo 2 showed back in the day in the golden era of public servers.
COD is exceptional in both these regards. The scoreboard rarely accurate to the most valuable players since many COD modes are tracked in terms of objective scores, and COD is the last game without a clear laddering system since they are locked to their time-based unlock-ladder Prestige system (which started as a counter to Halo 2's skill-based ranking system).
There's a few other aspects here in how I've experienced the COD playerbase engaging in the game in terms of objectives and playstyles but that's another conversation—suffice to say, I'm fairly sure this is a big factor in why there's so much friction with COD's implementation of SBMM compared to any other game. Additionally, and anecdotally, the COD playerbase is not one I've seen prioritize improvement of skills compared to players of other competitive multiplayer games (I'm pretty sure this is why COD never really took off as an esport and why Rainbow 6 Siege found more success in that regard) which further poisoned the SBMM conversation as they lay much blame at its feet even though it's just a small aspect of COD's greater matchmaking and multiplayer systems.
One last thing I wanted to mention was accessibility. I've mentioned Counter-Strike in another comment but I was only able to start getting competitive and better after matchmaking and SBMM was introduced with CSGO because I was unable to find a pub server that really allowed for that well in CS 1.6 or Source due to various factors like region, timezones, server availability, or just finding a community on the internet who would give you access to their quality pub. Finding individuals who were willing to help or test mechanics or maps on empty servers was much easier and that's something that's sorely lost with the absence of pub servers.
Ultimately it does suck that pub servers are basically gone. There were a lot of advantages to them, especially for smaller micro-communities in gaming (whose absence is even more sorely felt). But I don't think it's fair to say that SBMM inherently stymies or frustrates personal skill development. Ultimately skill development is up to each individual and their capacity and willingness to want to improve.
Counterpoint though: most everyone who played competitive StarCraft (BW or 2) got considerably better from private matches or KOTHs against much better skilled players than any ladder. Ladder was mostly for bragging rights and gaining experience. At a certain point it's better to see or experience the highest level of the game vs playing only ever with people near your skill level. That is no guarantee of skill progression, as people can fall into local maxima or the random variance could end up deciding matches vs concrete improvements.
SBMM is not new nor is Activision's implementation of it, or Microsoft's trueskill2. I think it's fine but what particularly gets me is when the casual and competitive playlists both use SBMM I feel like I cannot escape the meta game, and I cease to understand the differences or purposes in the playlists. That said I'm a high skill player, so they are aware of my complaints, I guess. I don't super care about stomping people or not, I like higher quality games, but it's disconcerting to not have any opportunity to play relaxed or casually unless I smurf, which I'd like to avoid.
Very much agreed on how seeing or experiencing the highest level was a big boon to skill development. It's important to see what the endgame should look like, and learning from more experienced players the smaller minute mechanics that build towards that endgame. Watching pro matches and then hearing Day9 explain all the little things behind the play, for example. What's more important is going out and applying it. Whether that's in private or ladder matches, it's the practice that's important, and that kind of goes back to my ultimate point of skill development being up to the individual and their capacity and willingness.
Better players in private games definitely provide a more direct comparison for where you were too slow or inefficient, but ladder is also a very good place to practice things and keep getting better and better at it without reaching an end goal of sorts. I don't think it's possible to stall out on a ladder due to random variance unless there's something fundamentally wrong with the game, though. And in both, if you're not analyzing the replays, you're skipping out on crucial learning.
Well it's kinda like, I wasn't really thinking of StarCraft specifically, but as an example Brood War Zerg vs Zerg can be pretty coinflip-y and very insta-loss, that doesn't mean the better player can't win, but if you compare it to TvP the "better" player wins less of the time in ZvZ, and none of the pros have a great win rate, which isn't true in the other matchups. Or in Halo where lower than Onyx people are just bad mechanically, and win or lose their fights whether they happened to miss that time or not, and the same for their opponents. In Splatoon as well as an example I'd talk to players that were hard stuck before X rank despite having a thousand hours of ladder, and I'd definitely say that splatoon's matchmaking is almost anti-skill development, and essentially 100% of competitive players play customs/scrims or are in discords for +1 in order to actually improve, and then it becomes a problem of a walled garden and you must be this good to even get an invite. At a certain point I think I prefer just a single matchmaking queue that everyone is thrown into, like Cod4, which was insanely popular even without SBMM (or certainly not as aggressive because that game was pub stomp central for me). I think if you make a fun game people of all skill levels can enjoy it win or lose. Hell I feel that way about like bowling or golf or laser tag.
Just to note, COD4 did indeed have SBMM. I think people were just figuring out the playstyle at the time. It has really been around forever.
Yeah I couldn't recall if it did but I assumed it had some implementation of it, but it was baaaaad, I've never dropped as many hundred kill games in any other CoD, including ones where tac nuke ends the game lol. It could have just been having such a massive player base.
It's about the missing sense of progression and not about the progression itself.
I still don't see it, not enough for this to be a given.
Going up against similarly skilled opponents with SBMM allows for more transparent incremental skill progression as players will see basically the "next" skill to learn as they master one and get moved up to tougher opponents.
Playing in a pub with players miles ahead is being thrown into the deep end and skills can develop quick but it's also not as clear that skills are improving as the gulf between the best and the rest are much wider, and that difference could be the result of multiple advanced skills or mechanical understandings compounded together.
My big example here is Counter-Strike Global Offensive, which had matchmaking and SBMM with the competitive mode but the practice deathmatch modes did not. As skills went up through gradual progression in competitive mode, the DM performance saw huge boosts as well. No amount of only playing DM would get a player up to that level as there were other skills that had to be taught more gradually or granularly through practice understanding the mechanics of things like sound, timing, and understanding of where players look on maps.
As long as I wasn’t getting stomped every game, I had fun and learned so much more, much more quickly from the Pub Stars compared to around-my-skill SBMM players. With the stars, it was obvious what the right calls, the best paths, the effective strategies, and such were. I might not have been able to pull them off, but I had something Good to practice towards. With skill peers, it’s the blind leading the blind. Even if something stood out slightly, was that actually a good more, luck, good in low skill matches but a bad habit in the long run?
It’s a bit like learning an instrument. It’s great to practice with your peers so you don’t get totally discouraged, but it’s important to hear first-hand how it’s supposed to sound regularly so you’re practicing towards the right thing.
Plus it forces you to get creative - both individually and working with a team - to try and counter someone significantly better. And makes it easier to say “well, we’re getting wrecked already, might as well try X, Y, and Z” compared to a SBMM game where trying things could cost you and your team a loss.
Doesn't this also mean bad habits that work against worse players would be encouraged with the regular matchmaking? Since statistically once someone is above the median skill level, most encountered opponents would be below their level. And I also don't think it's that helpful to play against much better players because their decision making can become completely intransparent. Furthermore sbmm isn't perfect. Natural variations should still allow players to learn new things from "better" players but in a more controlled manner.
Either way, for most games there is gameplay footage available. Sometimes even with commentary including explanations. So the advantage for beginners having to struggle against much better opponents isn't that clear as they could just as well play against their peers, and watch gameplay footage to improve at a faster rate.
Kinda an extreme case, but I can't stop imagining this scenario where a pro tennis player is teaching a kid by going all out. And when told to turn it down a notch, the player just exclaims, "git gud".
A well designed SBMM system is going to be throwing you into games which are at, below, and above your skill based on the existing set of players and how confident it is in your current level of skill. The idea that you'll be winning as much as you're losing just doesn't hold up in the short term, only in the long term. More than half the paper talks about how to deal with the players which are online, skill brackets, grouping, similarities, disparities, and so on.
While I understand where you are coming from with a lack of public lobbies, with a lack of the same kind of communities we used to see (the only way for other players to not be ephemeral ghosts in modern gaming is to either be at the very top/bottom of players worldwide or to be in a clan), I think perhaps it's mostly about learning to adapt to a different system and how to interpret said system. I think there's perhaps a bit of nostalgia at play here with how you normally interact with and how you used to see skill progression vs. how it'll appear when you're more regularly being balanced.
As an aside, I want to acknowledge that I'm coming from a very different viewpoint. I've been top ranked in the world on more than a half dozen games over my lifetime, I've lead #1 clans/guilds/etc. and generally find myself in some of the top brackets. I don't play games as much as I used to, but when I do get into games I'm often pinged by friends where some famous twitch streamer is mad at me, accusing me of cheating, or otherwise had some kinda cute/funny moment with them as I regularly end up in their skill brackets.
I personally don't find win streaks very interesting or fun. I don't like stomping on people and often try and find inventive ways to play or challenge my skills because winning and losing is not what I find interesting or engaging about a game. I find exploration, learning new skills, being social/making friends, and doing funny things much more interesting, but that likely also comes from a place of privilege (the skill that I do have, and the fact that I do regularly get win streaks especially if grouped up with other high skill player friends).
I recognize, however, that finding people and becoming a 'regular' is a much more conscious thing today. Since I have a history of organizing and finding other good players, I almost always find a community, but I do remember falling into these communities much more easily back in the day. Nowadays you just don't run into the same people very often because the player pool is so big. You need to friend people and reach out, you need to jump on voice or chat with people and actively communicate in order to start forming those bonds to find a community. Many groups are also fairly insular because they persist across games or consist of friends which have a real-life presence. But I think a lot of this is about the design of space and how one interacts with a game. While matchmaking is a part of that, it's not the only way to smartly design for social connection and games (which aren't MMOs) just in general don't seem to prioritize this kind of design.
I think it's kind of funny how these people who oppose SBMM think it's going to work out in the long run. And the article addresses that:
I know that in my personal experience, as a low-skilled player, I wrote off these competitive multiplayer franchises many, many years ago. I can't imagine that I'm alone. Why pay for a game when the only experience you're going to get out of it is getting stopped constantly?
It seems like Activision is taking a very stats-based approach to this, which makes sense since that's the numbers they have. It's also in their best interests to see as many players return to the game as possible.
I can also see the argument though that it's a very different experience for players. Skill-based matchmaking has a certain homogeneity to it that can make games feel more predictable, and less exciting. If I play ten matches of a game, I know I'm going to win roughly five of them. I know this because I've already been balanced into a specific bracket. If I start doing better or worse, I'll be adjusted to the new bracket ensuring that I remain at that average at all times.
When I think of games before this system though, I remember a lot more variety. Going on winning and losing streaks. Having one or two extremely good players in the lobby that everybody learns to watch out for. It changes the game and makes it feel more dynamic. Instead of every player and every game feeling kind of the same, it's a different experience every time.
I don't think I'd say either approach is necessarily better or worse. A perfectly balanced game every time lets you hone your skill, while the uncertainty of a random lobby keeps the excitement fresh. It seems to me there ought to be room for both approaches.
However, I think the new breed of gamer has adapted to skill-based matchmaking, and server browsers today are largely going by the wayside. Even Valve's Source games have largely hidden that feature away. I'm a little sad that we'll no longer see little communities or rivalries forming like we used to, but I will always remember that period fondly. It wasn't always fair, but it was always fun.
Agreed. I find Call of Duty boring now because I know that I'm going to be punished for a good match.
You have two or three good matches and then your next three feel like you hit a brick wall.
Not saying sbmm isn't good in general, but the implementation has felt poor in COD.
Yeah I noticed it in MW2 remake, they pair you above and below your skill level so you can feel like you had good matches where you pulled your weight and sweaty matches where it's a struggle to stay afloat, instead of a monotone semi-sweaty matches.
It feels artificial, but avoids that losing streak that would make me put down past call of duty games, where I feel like I'm tilting. Problem is their+- for skill discrepancy is quite large I think, so the differences between matches are very pronounced, vs something more subtle with the same effect
That you feel like you're being 'punished' for your ELO increasing with your skill seems strange. Why do you think you deserve to be placed with 'lower ranked' players relative to your own skill? This seems to highlight that you feel the meta-game is as important as the basic mechanics, and if that's so maybe you should feel a higher rank is something to aspire to?
I mean, it sounds like the problem is the elo increasing in too high a step to be fun (think going from 500 to 1200, when you're around 750). I also don't remember many games which shows you your elo straight up, usually the closest is your rank which can map out pretty well to certain ranges... to being completely meaningless.
Another thing not usually taken into account is that low skill players often are very inconsistent and/or have huge gaps in skill sets. Which makes balancing even more fun I'm sure.
I think this graph is useful to visualise the system, and at lower levels perhaps the random/volatile element you're talking about has a broader, smoothing statistical effect.
But, as a counter-point, your argument referring to inconsistent skill gaps seems to be the same argument for having such a system in the first place. The point of the system is to narrow the skill gap, and if there are improvements that can reduce variability/lower skill gaps at lower levels then I'd consider that an improvement as well.
The graph helps, assuming most games use an elo-adjacent system.
The skill set gaps I was thinking more of things like tech vs ice in Trackmania, where even something as simple as doing a turn is so radically different it can't be counted as the same thing (even to a lesser degree tech vs fullspeed share the same bases, but their "advanced" skills don't overlap).
In that case the solution is to have multiple rankings per player. Works well enough, even if it can lead to some weirdness for a little time when you go from a high-level playlist to a low-level one (but as your graph shows, this is self correcting).
And the inconsistent skills is for in-match skills, where in the span of 10 seconds you go can't hit shit > headshots across the map > can't hit shit again (yes it's an exaggeration, but having a big personal best only to completely screw up next round is common enough in Trackmania to be called a curse).
The thing is, as far as I know (which isn't much tbh), no ranking systems handles that and basically hope that it (alongside across match inconsistency) gets smoothed out over time.
The Glicko rating system tries to account for the uncertainty in player ratings, but it only uses the amount of time that has passed to place a lower weight on older ratings or brand new players. I wonder if it could be further modified to be useful in video games.
Glicko-2 actually has a volatility variable, which is supposed to measure inconsistency. I think it would probably catch you up more quickly if you go on a massive winning or losing streak, but it can't really deal with match-to-match inconsistency. That's the big issue which no ranking system or matchmaking algorithm can really account for. Players, especially low-to-average skilled players, can swing wildly in performance from one game to the next, based on factors that are absolutely impossible to measure.
I only mean that I'll have a good game and then be thrown into several games in a row where I'm getting absolutely destroyed. I don't mind playing tons of sbmm matches in a row that feel fair, even if I lose most or all of them. As long as I'm actually being matched against people at my skill level. I also don't mind totally random matches that have varying difficulty. Sometimes I get stomped, sometimes half the team sucks and half are great, whatever. But when I can predict with 100% accuracy that I am going to get wrecked for the next 4 matches because I won a round and there is no possible way to change that, it sucks.
In other words, I want better implementation of sbmm or no sbmm at all.
Interesting, do you mind sharing which game this happens to you in?
Call of Duty.
Last one I played was Modern Warfare. I stopped playing (as did my friends) because we felt this was just a nonstop issue.
Yeah, Call of Duty has pretty bad slippery slope mechanics that reward good players with power-ups etc for consecutive kills. That doesn't surprise me lol.
That's an interesting point that I totally forgot about. When I play competitive shooters now, I'm never "scared" of a single person. I remember in the CS 1.6 days, you'd quickly realize if a person on another team was amazing, and develop an innate fear of them. When you encountered them you'd panic. You'd try to avoid big pushes at the site they were guarding, and if against all odds you managed to take down Goliath the feeling was incredible.
Some of this may have to do with me being old, but I just don't get that feeling in games anymore. Everyone is the same faceless threat, and I fight all of them in roughly the same way.
I find it funny how star players are a terrifying presence for you in cs, meanwhile back when I played tf2 the Dominating! symbol acted more as a magnet that progressively attracted more of the team's firepower as said star kept dominating more players.
Dunno how much is on permadeath for each round or on the giant "KILL THIS GUY" sign floating above whoever's dominating you.
Ah, how I too long for the days before monthly subscription cheats.
The anti sbmm movement has always been the funniest thing to me, because I've only ever seen fps players push it. Never seen someone who plays a 1 v 1 game be against it. The paper really just feels like a confirmation of common sense.
I've seen Master Duel (Yu-Gi-Oh) and League of Legends players say they prefer making new accounts and smurfing. It's fun to win, so it's not too surprising that people enjoy winning more than 50% of the time. None of them really argue against skill-based matchmaking though.
My anedoctal impression from a friend who plays and the r/FallGuysGame subreddit is that SBMM isn't well liked in Fall Guys either.
The explanation I've heard is that a lot of the casual fun of Fall Guys breaks down when you end up in a lobby filled with people of relatively equal skill (e.g. everyone trying to follow the optimal route through an obstacle course), instead of a random sampling of the userbase.
I think that also makes sense from the perspective of game win expectations — in a game where there are only two competing sides, you’re looking at 50% win rate long term. But in a game with 10 competing sides, you’re only expecting 10% win rate long term, and that feeling sucks for people who care about winning.
Just in my friends group, when we play 4-player games, a number of players in the group are a little bit sad if they only win 25% of their games, even though that’s the balanced expected value. I don’t care one way or the other, so I’ll often see that I’m able to win, and then make a subtle misplay to allow someone else to take the win because I know they’ll be happier to win than I would be.
What real fun is there in getting an easy win?
Every time? Perhaps not much at some point. After three 40 minute slogfests leading to a stressful defeat? A lot of real fun.
On the flip side, for newer or less-skilled players, it sucks to get trampled.
This way, everyone is least upset.
Style points, YouTube combos, overwhelming victory, etc.
For Master Duel, the main point to making a new account is that you get a lot of in-game currency and crafting materials as a new player, which lets you unlock decks much faster.
Otherwise once you made 2-3 decks and exhausted the solo mode, you need to grind for gems which is tedious.
Or masochist/master-pack runs if you're feeling spicy.
I get why people do it, but it's still a bit annoying to face brand new accounts running full-power snake eyes etc.
Oh god, it would hurt my soul to use all my gems on master packs, but it does sound like a fun challenge.
Yeah, it's annoying to face them when you're low-ranked because you're playing a casual/non-meta deck, but at least I get satisfied when they make a new account, made the most broken deck available, and then proceed to misplay enough for my Ninjas to beat them.
I don't have any solid evidence for this, but I suspect a lot of anti-SBMM sentiment is coming from streamers, whose viewership might depend on them "popping off" on lower skill players.
Not to mention streamers are more likely to be at the higher end of the skill range and the paper did show that the top 10% of players had more fun when SBMM was turned off. So from streamers points of view, I can see why they don’t want SBMM, but they’re unable to think past themselves and about the other 90% of players, yeah.
It's a cult of outrage manufactured by grifters masquerading as "content creators". They won't accept the facts presented here the same way any other cult of personality type won't publicly admit when they're wrong. Angry content like this is incredibly easy to make, gets people riled up, and keeps the ad and merch revenue flowing.
I've never heard of a movement called "the anti-SBMM movement" but I'm anti-SBMM, in the sense that I think everyone is overly focused on SBMM, and it comes down to a simple reason: community servers. Specifically, the community aspect. In 1v1 games this makes no sense, so it's not surprising.
This paper has absolutely nothing to do with community servers (which have features such as moderators and the ability to customize server settings), and proves absolutely nothing about SBMM in comparison to community servers, so my objections remain.
That's not anti skill based match making though, that's anti match making, which is a completely different argument all together. The paper is about whether or not sbmm is a good system for a game that has matchmaking, and it shows evidence that it is.
So why not offer both at the same time?
I think there may be some bias here towards matchmaking since it gives developers more control over the player experience. There's also some financial rewards for exposing players to more purchasable content when they are playing against a wider pool vs smaller server oriented communities.
That sounds like splitting the player base, but then putting that aside I’m curious as to your recommended implementation — are you giving players an option to choose SBMM or non-SBMM queues? In which case, you’d end up with the same result — players not in the top 10% would have a better experience in the SBMM queue, and the top 10% of players choose the non-SBMM queue and are only matched against themselves or newbies who don’t know what the different queues are?
I was taking about matchmaking and the older player created lobby systems.
If you're set on only providing matchmaking then of course it makes sense to have as narrow a range on skill gaps as the size of the player base allows. Only reason to widen or drop SB is to lower long queue times which also has a pretty massive effect on player retention.
They address this:
This doesn't happen to be in response to my earlier comment, does it? The timing is just too perfect.