An Agent Based Model to look at Gwent Pro Ladder (results)

With the Agent Based Model (ABM) from the previous post we can simulate any number of Gwent players, with known skill and play rate, playing throughout a season. Using this model we’ll try to get a firmer grasp on the ladder system and test whether this rewards skill or grinding games…

Climbing Pro Ladder Skill vs Grind

The ABM mimics a population of agents playing Gwent for an entire season with different innate skill and time to play. Though unlike with actual people we can change the parameters of the model to examine what we want to test. So in the first model we’ll disable any kind of learning, where agents would get better as they play more games, to see only the impact of playing more games on the peak MMR score (of a single faction).

To do this a model was run with these parameters:

8000 players
100 steps resulting in players playing 25-170 games per season
no learning players don’t get better by playing more games
initial skill ranges from 1200-2700 distributed similarly as chess players

After running the model with these parameters, we can group players by skill (in bins of 200 ELO) and amount of games played (in 25 percentile bins). Then we can plot out their peak MMR and see how well they did. The resulting graph (shown below) immediately reveals two flaws with the current system. The spread within a group of players with similar skill, that played a similar number of games is quite large. Simple luck of when you play your games and who you queue into can make a difference of up to 70 MMR! Furthermore, players that play more tend to acquire a higher peak MMR than players with similar skill that play fewer games. Therefore players able to put in more games each season have a distinct advantage over those with less time on their hands.

When looking at actual data, it became obvious that players in the higher ranks tend to play more than players further down. When recreating this plot with the simulated data, this pattern doesn’t appear. This will be discussed further down in the article.

Adding Learning to the Model

We can reasonably assume that as a player plays more games their familiarity with the deck they are playing and the meta are increasing. Simply put, they get better at the games by playing more games. Perhaps this can explain the increase in players with a lot of games under their belt at the higher positions? To test this an experience factor is added to the model which gives players an ELO bonus as they play more games defined. A new model was run with this learning factor set to 20, as the ELO bonus is calculated as experience_factor * sqrt(games_played) this would give a player that played 100 games a 200 ELO increase over a player that played none even if their innate skill was the same.

As other parameters were left the same, for this simulation these settings were used:

experience factor 20
8000 players
100 steps as before
initial skill ranges from 1200-2700 with the same distribution as the previous

Now the same plot can be created showing the peak MMR of the agents grouped by innate skill and the number of games they played. As agents now learn by playing games the difference in peak MMR between players that play little and those that play a lot, despite starting at a similar skill level, increases.

To check if with agents learning, as you would expect from actual players, those that play more frequently shift to higher ranks the same ridgeline plot was created. There is a minor shift, but it is barely perceptible and nowhere near the clear shift that is observed in real data.

Even with the experience factor pushed to 100 (data in repository), which is far beyond the skill increase you can expect in real life, this pattern cannot be replicated. There is another factor at play here, one that is not included in our model…

Discussion

Using a model we can check a number of things which cannot be easily tested in real life, and this reveals a few things about how the ladder system works. With an Agent Based Model we know exactly what the innate skill is of each agent and we can check how high up the ladder they can get in a number of games. So we can quantify how well ladder is at ranking individuals based on their actual skill. Also the impact of playing more games could be uncovered both with players getting better by playing more games in the absence of any learning.

Will Playing More Games Get You Higher Up?

Answer: Yes, but not much

Playing more games gives you a relatively small advantage over players that play fewer games, even in the absence of a player getting better by playing more games. However, this advantage isn’t big enough that an average player will suddenly be able to get into the Top 200 or even the Top 500 simply by playing a ton of games.

As a matter of fact, there is a lot of variance between innate skill and peak MMR reached. By playing 30-50 games and being lucky with who you queue into, you could end up at 50 MMR higher than an equally skilled player that plays two or three times more games but has worse luck.

Why Are Players Higher Up Playing More ?

Answer: Human psychology ?

Players that play more don’t simply end up higher in the ranks because they play more. No amount of learning can account for the shift that is observed in real data. So human players exhibit behaviour that doesn’t emerge from the model. I’ll go out on a limb and give a few hypotheses which could explain this discrepancy between the model and observation.

Players that were playing a lot in previous seasons will be (i) more proficient at the game due to experience and (ii) will continue to do play the game more than average.
While the gains are relatively small, the way ladder works gives a small advantage to frequent players. The higher up ladder you get the more competitive it gets and more players are willing to play additional games to leverage this minor edge over their competition.
Players in higher ranks notice that other players at their level play more. Therefore there is a push to keep up the pace.
A combination of all of the above … there probably isn’t a single explanation that applies to all players

Is There a Need to Grind ?

Answer: Yes, at higher ranks

Playing more games gives you an advantage and this advantage becomes bigger within the most skilled groups of players. That taken together with the fact players in those groups tend to play quite a few games each season, you lose that edge unless you play even more … This is a feedback loop where players might feel compelled to play more than they would otherwise. Even though that edge isn’t big, a small difference in MMR in that relatively small pool of top players could mean the difference between a top 200 spot, crown points and a spot in an official tournament or missing out on all that.

Is Peak MMR a Good Metric ?

Answer: It can be improved

Peak MMR as a metric is reasonably well correlated with the innate skill of a player, so overall it does a fair job capturing the skill of a player. Though there are two main issues that emerged from this analysis:

There is a lot of variance, similarly skilled players that play roughly the same number of games can end up with peak MMR scores that are up to ~70 point apart. This is simply based on luck and who you queue into.
Playing more leads to higher peak MMR scores, even in the absence of learning. Having a winning streak at the right time can push the peak MMR score up. Playing more games increases the chance of such a streak happening at the right moment.

Because the simulation assumes players are playing with a single faction, and Gwent sums up the MMR scores for your four best factions this reduces variance as having luck with one will be cancelled out by having bad luck with another. This could be further improved by recalculating MMR each 5-10 games, similar to how calculations are done to adjust ELO scores for Chess players at the end of a tournament.

The impact of playing more games is very easy to get rid of … by ranking based on the current MMR instead of the peak MMR. As you can see in the graph below, in the absence of learning, when considering current MMR there is no difference between players with comparable skill based on the number of games they play.

Peak MMR creates an incentive to play as many games as possible, and gives players the option to play more leisurely once they no longer wish to climb with a certain faction. Therefore this cultivates a more active player base. The current MMR would not allow players to start playing more casually in ranked as this would lower their score again. Once you reach an MMR that feels like it would be a peak for you, it would actually be better to stop playing that faction for the remainder of the season. This would mean fewer people are active and queue times would increase making that part of the game worse.

Conclusion

No, you can’t grind your way to a top 200 spot unless you have the skills to get there. Though playing more games than your competitors with similar skills is an advantage. In the top brackets where players play fairly frequently this would mean playing even more games to use that as a competitive advantage.

Should peak MMR be replaced by current MMR as the metric? As a game developer CDPR likely wants as many people online playing as possible, and using peak MMR to measure ladder performance aligns well with that. However, this incentivises players to play more and competitive players to not only compete to play their best, but also as much as possible. In an age where digital wellbeing is starting to show up on the agenda of companies like Google, Samsung and Apple, this is something CDPR might need to have a look at sooner or later.