Activision Secretly Turned Off Skill-Based Matchmaking in Call of Duty as an Experiment; Players Hated It [IGN]

In a 25-page white paper published last week, Activision reveals that it actually conducted an experiment in 2023’s Call of Duty: Modern Warfare 3 where it reduced SBMM for some of its players, resulting in more players quitting games or not coming back to the game for more sessions.

Matches Made in…Heaven?

Currently, Activision says the way matchmaking works in Call of Duty is by taking into account a number of factors. Top priority is player connection quality and the time it takes to enter a match, but other factors including skill, platform, recent maps and modes, and other things are considered too. And skill itself is a complex equation that takes into account individual match total kills, kill/death ratios, and kills/deaths by enemy ratios (to ensure players can’t drop their skill levels on purpose by self-killing). Skill levels are recalculated after every match, and Activision says it’s constantly trying to find a happy balance of ensuring players aren’t seeing wild swings in where the game thinks they’re at, but also to adjust quickly if a player’s having an off day or trying out a new loadout. In short, it’s skill calculations and matchmaking are finnicky processes, but Activision is doing the best it can.

In the study earlier this year, Activision ran a “deprioritize skill test” where the developers decreased skill’s importance in matchmaking in the algorithm, but did not shut it off entirely. It ran the test for 50% of its North American playerbase for a two-week period in early 2024, and the results are… pretty conclusive!

In Figure 3 we can observe the percent difference in the number of players returning after 14 days between the treatment and control groups. With deprioritized skill, returning player rate was down significantly for 90% of players. The 10% of highest skilled players came back in increased numbers, but in aggregate, we see meaningfully fewer players coming back to the game. This effect may appear small, but this change was observable within the duration of the test. This will compound over time, just like interest, and will have a meaningful impact on our player population. This is a concern for all players, including the top 10%, as if this pattern is allowed to continue, players will exit the game in increased numbers. Eventually a top 10% player will become a top 20% player, and eventually a top 30% player, until only the very best players remain playing the game. Those original top players will become increasingly likely to not return to the game. Ultimately, this will result in a worse experience for all players, as there will be fewer and fewer players available to play with.

While single-digit percentages might not seem like a lot, given the sheer volume of participants over a relatively short period of time, this is a fairly significant difference in drop-off. What’s more, Activision expects that based on other studies it has done over time, this would compound. If left unchecked, the lowest-skilled players would disappear at higher rates, meaning only players of higher skill levels would remain. Maybe that sounds great for the higher-skilled players, but in the smaller pool of players, those who had previously been mid-level would now be the lowest-skilled, and would themselves start to drop off at higher rates. Basically, it’s bad news for player retention all around, and retention is necessary to keep an online game alive.

Skill Issues

The paper details other data as well, including a similar survey where Activision tightened the skill constraints in matchmaking and saw inverse results: higher-skilled players dropped off at higher rates, but lower-skilled players stuck around more frequently. It also saw a higher rate of “blowouts” (when one team wins a match with a significantly higher score than the losing team) when skill was deprioritized, which players find less fun, and lower kill-per-minute rates with lower-skilled players. Basically, when SSBM was turned down low, the vast majority of players had less fun, and didn’t come back to play the game more as frequently as when SSBM was on.

In response, players have expressed skepticism about the study. Some point out that the algorithm is too aggressive and shouldn’t recalculate skill after every single game, while others point out that a matchmaking system based on wins/losses would be better than the numerical system Activision uses. It’s a controversial subject, one that online gamers and Call of Duty players in particular have been debating for years. Much of it boils down to, it’s fun to win and less fun to lose, and losing a lot feels exceptionally bad. Should players who are very good at the game be rewarded by being allowed to win a lot against random pools of players, or should games like Call of Duty try to balance the experience by pitting players against others of relatively equal skill so everyone wins and loses a roughly equal amount over time? Activision, at any rate, is erring on the side of keeping more players playing Call of Duty for longer.

You can read the entire white paper here for some fascinating data on how Call of Duty matches players against one another, and there was another paper from back in April that examines the impact of matchmaking based on ping.

In our review, we gave Call of Duty: Modern Warfare 3’s multiplayer a 6/10, saying it “reheats and reserves the same multiplayer from Modern Warfare 2 with some extra maps and tweaks as garnishes.” And Activision just dropped its Season 5 update last week, with a long list of balance change and improvements for that multiplayer mode.

Rebekah Valentine is a senior reporter for IGN. Got a story tip? Send it to rvalentine@ign.com.