The original post is . Below is a copy paste of the text.
The design proof of SBMM and how it's even worse than we imagined.
Hello Reddit,
Let me preface this with some background about myself, I am an indie game developer with several years of experience in planning and designing systems for use in games. I spend a lot of time not only creating games, but playing them for enjoyment and getting inspiration to use in my own projects. I picked up MW a few weeks after launch and have both enjoyed and hated many aspects of the game. Infinity Ward in particular got my attention recently because they have been increasingly silent towards the community about the development of the game, so much so, that it really had me suspicious behind the motivation of this silence. I'll provide some other background information such as how I got this information, as well as outline a few other aspects that are interesting but not necessary to read. You are free to skip down to the actual proof of concept, under the header of "Practical Application of a Virtual Coaching System… Based Upon a Determined Playstyle of a Player"
How I got this information
A general trend I've noticed for any AAA studio, any game system they create is often designed with the assumption of patenting the system. We've already seen some examples on this subreddit as well as other places about some of the predatory patents companies like Activision or EA have published. Patent applications are generally published 18 months after they are filed, and the best way to gauge the intent of many of these larger companies is to look at the patents they have developed and published. The patent system allows for some level of transparency because it shows where the business is dedicating it's resources towards. However with the publication date being 18 months out from the filing date, it can sometimes make it difficult to figure out what current systems are designed in a predatory way or an unintended design flaw. There is a separate database that only stores applications for patents, I searched this separate database to see if I could find anything pertaining to some of the systems I've observed in the game, and yea, the patent for what the community refers to as "SBMM" is in there.
Other Notes and Patents
Interestingly, many of Activision's patents are riddled with references and acute details of SBMM and other game design systems that players have complained about. They use them as examples to explain how their patents would function, but in doing so, they reveal some of the designs they use to implement it in their own games. Surely, what better way is there to describe their own invention than by explaining how they have used the ideas in their own games?
System and Method for Validating Video Gaming Data
()
- Abstract: The present specification describes systems and methods for filtering a video game user's match performance data or loadout data through validation mechanisms. For the performance data, the validated, signed performance data are written to a leaderboard service of the video gaming system. For the loadout data, the validated, signed performance loadout data are transmitted back to the client device and used when playing a game. Free computing and/or networking resources of the client game device are used as an intermediate between the client devices, validation services, and/or leaderboard services.
In general, this patent seems to touch upon issues related to cheating within multiplayer games. Specifically, in games like Call of Duty (which this one is clearly based on since it has a Call of Duty Ghosts leaderboard image as an attachment), it signifies how a server can validate custom user loadouts to ensure that they are not modified outside of the parameters the game developers have set. For example, being unable to modify the damage of your bullets. Or when they add store shop items like skins for your weapons, it will compare your loadout to your purchase history to ensure you actually own the skin. This all seems entirely reasonable and I don't consider any of this to be an issue. But, they also reference ways this can be used for leaderboard purposes as well, ensuring users cannot modify their post match results to send to their leaderboard service, guaranteeing that player data is authentic. They even describe what kinds of data can be gathered for the leaderboard:
a) "Data related to a plurality of scoring events that occur during a match. For example, high scores, kills or captures, fastest time periods to achieve certain scores, clearing or achieving specific game levels and/or win specific matches by a set of 'N' number of top performing users, ranking of the user with reference to the top 'N' users; and,
b) Data associated with in-game events such as, but not limited to, movement of the user throughout the virtual environment or topographical map of the game, interaction of the user's avatar with various virtual characters or elements in the game, virtual elements or items used and/or won, damage taken, perks acquired."
Now, on it's own none of this is surprising, as nearly all games try to gather player performance data and is usually just displayed back to the player when they request it. The second description is a little more interesting since it involves gathering more specific behavioral data, but still seems rather harmless. I am including this section because it seems that the next patent is an extension of this system to ensure leaderboard data is authentic.
Practical Application of a Virtual Coaching System… Based Upon a Determined Playstyle of a Player (SBMM Patent)
()
- "Abstract: The present specification provides methods and systems for determining a player's playstyle based on a plurality of traits, extracted and determined from gaming parameters, and using the playstyle to present recommendations to a player via a virtual coaching system to help the player improve or modify the player's gaming skills for multiplayer video game play."
From my understanding, the intent of this patent is to provide a system for allowing players to find similar people of skill so the player can try to understand, learn, and increase their own skill. The game would offer coaching advice through an in game analysis of performance data between the player and others within the match. It collects various data about the player during matches, analyzes the data, and is able to provide constructive feedback to help the player get better at the game. Now this all sounds good on paper, right?
Figure 2 indicates a generalized overview of how this whole system comes together.
Figure 3 shows a flowchart on how they process this data, but what is really concerning to me is the last few parts, "Stored statistics are analyzed to determine one or more of the player's traits. Determined traits are used to determine the player's playstyle." They infer the ability to derive specific player traits from the data they gather, which only seems possible if they were tracking a large variety of individual player data.
"The claimed inventions herein represent a practical application of analyzing videogame data to generate a specific categorization of a player, identify corresponding other players, and generate and present areas of improvement based on other player data in a manner that is tailored to a multiplayer videogame environment."
"… may receive player performance data regarding the player's level in the video game, number of kills, frequency of deaths, points scored, treasure obtained, geographical location in a virtual world corresponding to a level in the video game, materials used, weapons used, frequency of game play, player speed, player movement, player success at specific challenges, player reaction to specific challenges, causes of player death, player selected teams, divisions, or other groupings, among other data (collectively, "Player Performance Data")"
"… processes one or more portions of the Player Performance Data in order to derive numerous outputs related to the player's playstyle, what causes the player to die, what are the player's weaknesses, what are the player's strengths, the overall performance of the player, changes in play strategy or tactics that could result in improving the player's performance, among other outputs."
So here they define "Player Performance Data." In general, it's how they define the data that is used to determine what a player's skill level is in relation to everyone else. They define some of the factors as to what this data would include, because it encompasses two main factors, player behaviors and quantifiable match performance data.
"For example, for a first person shooter game, statistics such as, but not limited to, weapon kills, headshots, grenade kills, accuracy, and deaths tend to be strongly correlated, either positively or negatively, with the player's scoring rate and are therefore stored. Personal statistics such as, but not limited to, knife kills, unmanned aerial vehicle (UAV) or drone kills, and mine kills tend to not be strongly correlated, either positively or negatively, with the player's scoring rate and are therefore not stored."
"… the term "playstyle" comprises a combination of player traits which are indicative of certain behaviors, such as, but not limited to, how the player prefers to engage opposing players or how the player prefers to move in the game, and where each of the individual traits are determined from gaming parameters that quantify the player's performance in the game, such as, but not limited to, kill/death ratio, average kill distance, loadout/weapons/armor used, distance travelled, average speed, linearity of movement, or use of crouch, jump, or strafe."
"In an embodiment, the server determines a player's set of traits, and therefore playstyle, on the basis of one or more gaming parameters that are associated with that player. The one or more gaming parameters may be used to identify one or more traits that indicate the playstyle. In an example, a player trait of how the player engages in combat is partially indicative of a playstyle and may be identified using multiple gaming parameters such as, and not limited to, kill/death ratio, average kill distance, loadout/weapons/armor used, self-identification, or any other data. Similarly, another exemplary player trait partially indicating playstyle is how player moves during a game, which may be identified using multiple gaming parameters such as, and not limited to, distance travelled, average speed, linearity of movement, use of crouch, jump, or strafe, among others. In embodiments, the one or more traits that indicate a playstyle are combined to determine the overall playstyle of the player."
In essence, they track almost everything you do in the game and give you a performance score related to such factors. The way you play, from what paths you take and how fast you get there, to other metrics like K/D and average kill distance, are all tracked and stored in a database tied to your player profile. The entire way you play is tracked and stored in their databases.
"In embodiments, a player's playstyle is determined by a query from a database of the [server]. For example, in an embodiment, the [server] queries the average kill distance, total distance travelled, and average speed of a player in a first person shooter game to determine that player's playstyle. In an embodiment, a record representing the player's playstyle may be: avg_k_dist, total_dist_travelled, avg_speed…"
"The best players may be determined by referencing certain game statistics that are indicative of the player's performance, such as kill/death ratios, points scored, tokens earned, ranking, or other Player Performance Data, and comparing such data for all players to determine a set of players that are better than the player requesting improvement advice."
"Embodiments of the present specification seek the best players with the same classification of the playstyle, as the first player. In some embodiments, playstyles are evaluated and defined by metrics associated with a game type, for example, an FPS, and include, but are not limited to, average kills distance (defined as the average distance from a first player to a second player killed by the first player when the kill occurs), total distance travelled, average speed, kill-to-death ratio, score-per-minute, and player level. In another embodiment, the best players are sought that exhibit a similar playstyle as the first player. For example, the best players are searched who exhibit engagement and movement patterns (playstyle based on one or more traits) within some similarity threshold."
"The best players are identified based on one or more factors, which may vary based on a type of game or the context of the game. In an example of a FPS game, players with a high kill-to-death ratio, high score-per-minute, high level, or any other factor indicative of skill or performance may be identified as best players. In one embodiment, a specific percentage of the players with the highest performance are identified as the best players. In some embodiments, the player traits for identifying the best players is weighted depending on the game or the context of the game. For example, the player traits, or underlying player gaming parameters, may be weighted on the basis of total wins, total kills, win to loss ratio, kill-to-death ratio, experience, and level. In some embodiments, the best player(s) is determined by calculating averages and standard deviations of a particular metric among a certain number of players in one or more matches and identifying players scoring at least one standard deviation above the average as the best player(s). For example, in some embodiments, the averages and standard deviations are calculated from the metrics of players from 500 matches. Those players scoring at least one standard deviation above the average for a metric are considered the best players."
"… the first player is compared with the best players of similar playstyle, which were identified at 204. For a particular playstyle and/or context, embodiments of the present specification define the most important traits and/or statistics for success. These are the statistics that correlate most closely with a player being among best players. The statistics and/or traits to identify the best players may be combed through a machine learning algorithm, through human intervention, or through a combination of both. The machine can mine large and evolving datasets to derive and learn patterns to continuously improve its understanding of which statistics correlate highly with besting a best player. Programmers (humans) can manually define the statistics to compare."
The data they hold are thrown into a machine learning algorithm to create datasets of what factors into making a good player "good".
The entirety of the system maybe not be fully implemented. It seems that a very basic version of this system is incorporated into the game at this time, and are actively updating it as time goes by. Additionally, this system only works if there is enough data to support how the player interacts with the game. Early on, the matchmaking system felt very aggressive and I'd assume this is because they wanted to force the players into new undesirable situations to gather data since they had almost none. (Beta could have also been used to obtain some data as well) Now that they have accumulated more data, they can start placing players who have similar playstyles against each other in the same match. This process is even defined below.
Now figure 4 is the last piece of the puzzle in what they plan to do with this data.
"The result is the machine will have various models for different games/levels/modes/contexts of what statistics/traits are important to being successful within each playstyle. For example, for a "sniper" playstyle, it may be determined that shooting accuracy is a statistic that best correlates with overall player success. For a "run and gunner" playstyle, it may be determined that movement speed combined with total number of kills is a statistic that best correlates with overall player success. Accordingly, the correlation process identifies a subset of gaming parameters or statistics (of a larger total number of gaming parameters or statistics) that most strongly correlates with overall player success in a game."
"The best players may be determined by referencing certain game statistics that are indicative of the player's performance, such as kill/death ratios, points scored, tokens earned, ranking, or other Player Performance Data, and comparing such data for all players to determine a set of players that are better than the player requesting improvement advice."
"[software] for comparing the player with the best player(s) that have a playstyle similar to the player, determining improvements based on the discrepancies identified in the playstyles, presenting recommendations to the player so that the discrepancies are reduced/removed/minimized, thereby improving the player's gaming skills,"
Another concern I have is, at some point this data can become a universal tool tied to overall player accounts. These metrics may end up being tied to your Activision/Blizzard account which would determine what kind of players you play against in future unreleased games.
Unfortunately, I can assume that SBMM is here to stay, because judging by the rest of the patent, they have not even finished rolling out the entirety of what they have planned behind the scenes. The first step is to gather data by forcing matchmaking between similarly skilled players. They can then analyze this data in some computer model that can spit out what different players did to perform better. They seem to be using this data in machine learning to determine what makes a good player "good" and what makes a bad player "bad". I can only assume the next phase is going to introduce a coaching tool that would display what players can do to improve in a particular situation. They explain how that would work as well,
"The information may be in the form of recommendations that enable the player to improve gaming skills. The player may request or ask for recommendations or advice during, after, and/or between game play sessions. In some embodiments, the player requests information related to their performance, such as for example, how their performance can be improved. The player may ask for recommendations or request information either verbally or through one or more options provided by means of a graphical user interface, by the gaming system…. such as "How did I do in the last match?", "How do I improve my skills?", "What strategy should I use?", "What role should I play?". "What division should I use?", or "Why did I lose?", among other questions."
I am very concerned with the direction of game development these days and I hope by bringing attention to issues like this will in some way allow for greater oversight on these practices. I may be a game developer, but I am a gamer first, and these large studios have clearly lost their "gamer" identity by making ideas that sound good on paper, but are not practical. In this case, they took their original concept, matchmaking, gathered data that supported players generally aren't getting better, or do so at a slower rate, and want to provide a faster, automated, and convenient way of increasing the player's skill level, which is referenced within the patent. This system may be designed towards players who do not have the time to commit to researching, practicing, and improving their skill level on their own, or those who find it daunting to do so. In general I am all for giving players tools and resources to promote a healthier gameplay style that is suited and fitted to what they enjoy, but this is reaching a new territory in competitive gaming. I am not a fan of this system, and I have many reservations about implementing such a system in any game because of some of the consequences it has produced in this game, among other issues that will present itself as they roll out more of the features. More than anything, I am not comfortable with such data being recorded, this feels as though it's breaching numerous privacy barriers and it's unsettling how far this can go. This kind of service should be completely opt-in, so that the player understands all of their actions in the game are going to be tracked to spit out some performance number. It seems that with Activision's hard push towards dominating eSports, they are figuring out more intricate ways to determine player skill level in competitive games which I assume they will use for their own gain.
Modern Warfare is a proof of concept of how they envision the future of eSports.
Lastly, the question I have to you all is, do you believe this level of tracking is acceptable? Do you think that every single aspect of your gameplay should be recorded, analyzed, and stored just so that the game can pair you up with similar people or provide "coaching" solutions to you? I would even argue this detailed tracking of player performance can be used nefariously as well. If this is the direction the game is heading towards, I can certainly see why the development team has been completely silent on the issue.
Edit:
After reading some comments, there are some that are confused as to why this is a big deal. First let me address why I find these findings troubling: the data they gather can be used for more than just the purpose of skill based matchmaking. They have already admitted that conventional match making systems are sub optimal for business purposes, and are figuring out ways to extend this system in other areas like enticing the player to make in-game purchases.
Matchmaking System and Method for Multiplayer Video Games
()
- "Furthermore, conventional systems fail to assess a quality of gameplay used to tune matchmaking processes to optimize player combinations. Conventional systems also fail to reserve gameplay sessions for players in a way that minimizes the time that a player must wait to be matched. Conventional systems further fail to leverage matchmaking processes in other contexts, such as influencing game-related purchases, suggesting group formations, training/identifying non-player characters, and/or otherwise extending the use of the matchmaking process. These and other drawbacks exist with current matchmaking processes utilized in multiplayer video games."
They are spending a lot of resources into trying to create a fun and engaging match making system, but one that can also be used to influence the player into making decisions. However, with the general reception of the current implementation, they failed to accomplish the first goal of making it an enjoyable experience. Additionally, with so many variables being tracked, they can create a uniquely identifiable fingerprint in the same way advertising companies are trying to identify users based on mouse movements on a web page, browser choice, typing speed, etc. Ultiamtely with the goal of identifying who the user is, to display an ad relevant to them, and entice the user to buy it. Now translating this back to game design, the developers can use this data to identify a subset of players that would make purchases based on SBMM. Just look at the battle pass, they have conveniently given everyone the ability to earn free guns. In the future they will find a way to lock something behind a paywall, and I can imagine the game will try to pair you up with players who are better than you who own such items to make you think you need to buy it. It may not work on you, but it could work on someone else, and that's all they care about.
On the surface the intent is to gather this data on how to improve your skill in the game, but they can also do that without SBMM. The outcomes would be the same, the data would just take longer to produce accurate results because in the traditional matchmaking system, lobbies would have far more variety. Judging by what I've seen, read, and experienced, SBMM was very aggressive after release, whereby players where constantly yoyo-ing between good and bad matches. And I suspect it was due to gaining as much data as possible while there was still a huge player base as machine learning models require a lot of data to be useful at all. The proposed coaching system they included in the patent is a great idea, but it can also be accomplished without the SBMM they defined along side it. Again, it would have just taken longer to build the necessary models for the player.
