Simulated Ladder Seeding

[HappyLeaves]

There's been a lot of talk about seeding lately, especially for The Grand Melee. One alternative seeding method could be as follows:

1. Select all matches played between qualified players (ranked and unranked, only 1v1, only RM/EW)
2. Simulate a ladder by replaying the matches in chronological order using Elo, trueskill, etc
3. Use rankings as seeds

This method has some useful properties:
- Blends ladder and tournament matches
- Excludes irrelevant matchups (ladder vs lower players, orthogonal tournaments, etc)
- Doesn't rely on subjective point awards or multipliers
- Reduces likelihood of manipulation
- Resistant to inactivity in ladder or tournaments

And some downsides:
- Only helpful for top-50ish since it exploits the right side of the skill bell-curve resulting in lots of common matchups
- Relatively opaque - can't look at a spreadsheet to easily see how a player got their seeding

Using the prediction scoring proposed by felixferoc we can compare seeding methods for a few recent tournaments:

- "Sim" is the simulated ladder seeding
- "Actual" is the seeding that was actually used in the tournament
- "ATP" and "aoe-elo" are using *current* snapshots - so they overperform a bit since they include adjustments based on the outcome of these events whereas "Sim" only includes matches prior to tournament start

Warlords

Player        Sim    Actual    ATP    aoe-elo    Curr RM 1v1    Max RM 1v1
----------  -----  --------  -----  ---------  -------------  ------------
hera            1         7      7          1              1             1
tatoh           2         2      1          4             16            13
theviper        3         1      3          3              4             3
accm            4        12      6          5              8            11
nicov           5        11     11          9              2             5
yo              6         6      2          6              3             7
liereyy         7         4     10          2              6             6
villese         8         5      5          7              5             4
vinchester      9         3      4          8              7             9
mbl            10        10     16         10             10             2
jordanaoe      11         8      9         12             12             8
valas          12        14     15         11              9            12
classicpro     13        16     13         16             11            14
capoch         14        13     12         14             14            15
daut           15         9      8         13             15            16
miguel         16        15     14         15             13            10

Method         Total    Ideal    Accuracy
-----------  -------  -------  ----------
Sim              580      860         67%
Actual           550      860         64%
ATP              480      860         56%
aoe-elo          530      860         62%
Curr RM 1v1      530      860         62%
Max RM 1v1       570      860         66%

RBW

Player       Sim    Actual    ATP    aoe-elo    Curr RM 1v1    Max RM 1v1
---------  -----  --------  -----  ---------  -------------  ------------
hera           1         6      6          1              1             1
tatoh          2         7      1          4             16            12
theviper       3         1      3          3              4             3
villese        4         5      4          7              5             4
yo             5         3      2          6              3             6
mbl            6         9     14          9             10             2
liereyy        7         2      9          2              7             5
accm           8        10      5          5              8            10
dogao          9        13     11         13              6             9
sitaux        10        11     13          8              2             8
jordanaoe     11         8      8         11             11             7
valas         12        14     12         10              9            11
daniel        13        15     15         15             14            16
capoch        14        12     10         14             12            14
daut          15         4      7         12             13            15
kingstone     16        16     16         16             15            13

Method         Total    Ideal    Accuracy
-----------  -------  -------  ----------
Sim              460      700         66%
Actual           310      700         44%
ATP              480      700         69%
aoe-elo          360      700         51%
Curr RM 1v1      200      700         29%
Max RM 1v1       240      700         34%

TTL

Player           Sim    Actual    ATP    aoe-elo    Curr RM 1v1    Max RM 1v1
-------------  -----  --------  -----  ---------  -------------  ------------
tatoh              1         4      1          4             23            14
hera               2        10      7          1              1             1
theviper           3         1      3          3              5             4
jordanaoe          4         9      9         11             14             9
vinchester         5         3      4          8             10            10
villese            6         7      5          7              6             5
mbl                7        20     19         10             12             2
yo                 8         2      2          6              4             8
liereyy            9         6     10          2              9             7
nicov             10        13     11          9              3             6
daut              11         8      8         12             17            21
accm              12         5      6          5             11            13
barles            13        18     14         13              2             3
classicpro        14        16     13         17             13            17
dogao             15        12     17         14              7            12
bact              16        14     18         18             24            20
capoch            17        11     12         15             18            18
saymyname         18        24     23         23             21            24
slam              19        19     21         20             15            19
miguel            20        17     16         16             16            11
badboy            21        21     20         22             19            16
songsong          22        22     22         21              8            15
thedragonstar     23        15     15         19             20            22
bruh              24        23     24         24             22            23

Method         Total    Ideal    Accuracy
-----------  -------  -------  ----------
Sim             1960     2670         73%
Actual          1850     2670         69%
ATP             1880     2670         70%
aoe-elo         1680     2670         63%
Curr RM 1v1     1320     2670         49%
Max RM 1v1      1410     2670         53%

Posting this to generate some discussion on how to improve, pitfalls, etc. Not suggesting to use this for a real tournament in this state.

 

[siestes]

how to improve

hard to say since the thing is rather opaque but it looks good anyway

How come RBW actual seed was so bad? less than 50%?

 

[nimanoe]

hard to say since the thing is rather opaque but it looks good anyway

How come RBW actual seed was so bad? less than 50%?

Because they seeded the 4 previous winners as top 4 seeds, which made it so Viper was 1st seed, Mr Yo 3rd and DauT 4th seed, and all of them got eliminated in QF or earlier.
Besides that it was ATP S-Tier rankings, so rest was seeded pretty accurately

 

[Influenza]

Because they seeded the 4 previous winners as top 4 seeds, which made it so Viper was 1st seed, Liereyy 2nd and DauT 4th seed, and all of them got eliminated in QF or earlier.
Besides that it was ATP S-Tier rankings, so rest was seeded pretty accurately

Liereyy was in the finals, I guess you mean to say Mr yo instead

 

[nimanoe]

1. Select all matches played between qualified players (ranked and unranked, only 1v1, only RM/EW)

Does this include practice games? I don't think those should be included, but since they're private, I'm not sure if they're part of your data

Besides that, I think this is an interesting method.
It seems that this would only work for the top 16 or so though, it wouldn't really be a solution for the qualifiers.
Do you know how many rated games are taken into consideration compared to the amount of tournament games?
Also would love to see how the ATP S-Tier rankings would stack up, my feeling would be that it's the most accurate seeding method for the top 20 or so, just falls off below that.

Liereyy was in the finals men

My bad, meant Mr Yo

 

[SouFire]

I think that using alternative methods for seeding has caused those questionable seedings, they should(tournament organizers) stick with ranked ladder for seeding for several big reasons, it promotes activity which should give an actual meaning to the ladder, the activity of a player matters cause to get a good seeding he needs to be active and high rather than keep living on his 2018 voobly winnings(like the sheep method), and lastly the community would get more amazing games if players that currently only train with each others to actual face other players that wouldn't face on their group stages, we all win by using ladder as main tool with just few manual intervention regarding clowns to not occupy the spot of an actual top 16 player.

 

[LowEloNobody]

I think that using alternative methods for seeding has caused those questionable seedings, they should(tournament organizers) stick with ranked ladder for seeding for several big reasons, it promotes activity which should give an actual meaning to the ladder, the activity of a player matters cause to get a good seeding he needs to be active and high rather than keep living on his 2018 voobly winnings(like the sheep method), and lastly the community would get more amazing games if players that currently only train with each others to actual face other players that wouldn't face on their group stages, we all win by using ladder as main tool with just few manual intervention regarding clowns to not occupy the spot of an actual top 16 player.

If the ladder turned into 100% sweaty tryhard games 100% of the time, content would suffer for players. "SWEATY TRYHARD HUSSAR SPAM" doesn't get nearly as many views as "CUMANS DOUCHE MASTERPIECE!"

 

[The Bloodless]

I think that using alternative methods for seeding has caused those questionable seedings, they should(tournament organizers) stick with ranked ladder for seeding for several big reasons, it promotes activity which should give an actual meaning to the ladder, the activity of a player matters cause to get a good seeding he needs to be active and high rather than keep living on his 2018 voobly winnings(like the sheep method), and lastly the community would get more amazing games if players that currently only train with each others to actual face other players that wouldn't face on their group stages, we all win by using ladder as main tool with just few manual intervention regarding clowns to not occupy the spot of an actual top 16 player.

I can't wait for the full-lame, spec cheating, only strong civs, dodge all your weak maps ladder!

 

[The Bloodless]

There's been a lot of talk about seeding lately, especially for The Grand Melee. One alternative seeding method could be as follows:

1. Select all matches played between qualified players (ranked and unranked, only 1v1, only RM/EW)
2. Simulate a ladder by replaying the matches in chronological order using Elo, trueskill, etc
3. Use rankings as seeds

This method has some useful properties:
- Blends ladder and tournament matches
- Excludes irrelevant matchups (ladder vs lower players, orthogonal tournaments, etc)
- Doesn't rely on subjective point awards or multipliers
- Reduces likelihood of manipulation
- Resistant to inactivity in ladder or tournaments

And some downsides:
- Only helpful for top-50ish since it exploits the right side of the skill bell-curve resulting in lots of common matchups
- Relatively opaque - can't look at a spreadsheet to easily see how a player got their seeding

Using the prediction scoring proposed by felixferoc we can compare seeding methods for a few recent tournaments:

- "Sim" is the simulated ladder seeding
- "Actual" is the seeding that was actually used in the tournament
- "ATP" and "aoe-elo" are using *current* snapshots - so they overperform a bit since they include adjustments based on the outcome of these events whereas "Sim" only includes matches prior to tournament start

Warlords

Player        Sim    Actual    ATP    aoe-elo    Curr RM 1v1    Max RM 1v1
----------  -----  --------  -----  ---------  -------------  ------------
hera            1         7      7          1              1             1
tatoh           2         2      1          4             16            13
theviper        3         1      3          3              4             3
accm            4        12      6          5              8            11
nicov           5        11     11          9              2             5
yo              6         6      2          6              3             7
liereyy         7         4     10          2              6             6
villese         8         5      5          7              5             4
vinchester      9         3      4          8              7             9
mbl            10        10     16         10             10             2
jordanaoe      11         8      9         12             12             8
valas          12        14     15         11              9            12
classicpro     13        16     13         16             11            14
capoch         14        13     12         14             14            15
daut           15         9      8         13             15            16
miguel         16        15     14         15             13            10

Method         Total    Ideal    Accuracy
-----------  -------  -------  ----------
Sim              580      860         67%
Actual           550      860         64%
ATP              480      860         56%
aoe-elo          530      860         62%
Curr RM 1v1      530      860         62%
Max RM 1v1       570      860         66%

RBW

Player       Sim    Actual    ATP    aoe-elo    Curr RM 1v1    Max RM 1v1
---------  -----  --------  -----  ---------  -------------  ------------
hera           1         6      6          1              1             1
tatoh          2         7      1          4             16            12
theviper       3         1      3          3              4             3
villese        4         5      4          7              5             4
yo             5         3      2          6              3             6
mbl            6         9     14          9             10             2
liereyy        7         2      9          2              7             5
accm           8        10      5          5              8            10
dogao          9        13     11         13              6             9
sitaux        10        11     13          8              2             8
jordanaoe     11         8      8         11             11             7
valas         12        14     12         10              9            11
daniel        13        15     15         15             14            16
capoch        14        12     10         14             12            14
daut          15         4      7         12             13            15
kingstone     16        16     16         16             15            13

Method         Total    Ideal    Accuracy
-----------  -------  -------  ----------
Sim              460      700         66%
Actual           310      700         44%
ATP              480      700         69%
aoe-elo          360      700         51%
Curr RM 1v1      200      700         29%
Max RM 1v1       240      700         34%

TTL

Player           Sim    Actual    ATP    aoe-elo    Curr RM 1v1    Max RM 1v1
-------------  -----  --------  -----  ---------  -------------  ------------
tatoh              1         4      1          4             23            14
hera               2        10      7          1              1             1
theviper           3         1      3          3              5             4
jordanaoe          4         9      9         11             14             9
vinchester         5         3      4          8             10            10
villese            6         7      5          7              6             5
mbl                7        20     19         10             12             2
yo                 8         2      2          6              4             8
liereyy            9         6     10          2              9             7
nicov             10        13     11          9              3             6
daut              11         8      8         12             17            21
accm              12         5      6          5             11            13
barles            13        18     14         13              2             3
classicpro        14        16     13         17             13            17
dogao             15        12     17         14              7            12
bact              16        14     18         18             24            20
capoch            17        11     12         15             18            18
saymyname         18        24     23         23             21            24
slam              19        19     21         20             15            19
miguel            20        17     16         16             16            11
badboy            21        21     20         22             19            16
songsong          22        22     22         21              8            15
thedragonstar     23        15     15         19             20            22
bruh              24        23     24         24             22            23

Method         Total    Ideal    Accuracy
-----------  -------  -------  ----------
Sim             1960     2670         73%
Actual          1850     2670         69%
ATP             1880     2670         70%
aoe-elo         1680     2670         63%
Curr RM 1v1     1320     2670         49%
Max RM 1v1      1410     2670         53%

Posting this to generate some discussion on how to improve, pitfalls, etc. Not suggesting to use this for a real tournament in this state.

Rbw: max rm elo predicted worse than flipping a coin?

 

[shazzzm]

This is really interesting Happy Leaves, accuracy is measured as correct prediction vs not right? Do the numbers change much if you take into account how wrong the prediction is? As surely a prediction of 3rd against a finish of 4th is better than a prediction of 3rd against a finish of 12th?

 

[HappyLeaves]

Does this include practice games? I don't think those should be included, but since they're private, I'm not sure if they're part of your data

No private matches are included but there are practice matches that were played in public.

It seems that this would only work for the top 16 or so though, it wouldn't really be a solution for the qualifiers.

I think it can easily support the top 100, maybe more. So long as there is transitory overlap between matchups, the ladder will shake out accurately.

Do you know how many rated games are taken into consideration compared to the amount of tournament games?

Varies by player. Hera has a ton of rated games compared to Liereyy, for example.

Also would love to see how the ATP S-Tier rankings would stack up, my feeling would be that it's the most accurate seeding method for the top 20 or so, just falls off below that.

I'll add that next time I generate the data.

Rbw: max rm elo predicted worse than flipping a coin?

Yeah. It's because it chose TatoH a low seed (12), when he actually won. Largely because he hasn't played 1v1 since the RM 1v1 rating inflation occurred. A good reason to not weight max rating too highly.

This is really interesting Happy Leaves, accuracy is measured as correct prediction vs not right? Do the numbers change much if you take into account how wrong the prediction is? As surely a prediction of 3rd against a finish of 4th is better than a prediction of 3rd against a finish of 12th?

Accuracy measures binary correctness, yes. Points are awarded based on the round, like this. For the purpose of this calculation, group stage is considered Round 1. Could certainly tune the accuracy algorithm but I wanted to be consistent with felixferoc's prior work.

 

[nimanoe]

I think it can easily support the top 100, maybe more. So long as there is transitory overlap between matchups, the ladder will shake out accurately.

I think if you include a lot of extra players the downside is that your rating can get boosted if you played a lot of games against someone who's much lower rated.
I think the reason your simulation works so well is that it's similar to aoe-elo, but has more datapoints by looking at the rated games as well. However, if you expand the amount of players you'll look at, the tournament games become less important, as the amount of rated games against other players grows.

 

[SouFire]

I can't wait for the full-lame, spec cheating, only strong civs, dodge all your weak maps ladder!

With the exception of mbl, i don't see any top 16 player going full lame and there isn't a valid reason to start doing it from the rest of the players or any other behavior you mentioned, your complain would make sense talking of players with less elo, but they are already out from the top ladder, there is a limit you can reach even with dirty tricks.

 

[LowEloNobody]

I can't wait for the full-lame, spec cheating, only strong civs, dodge all your weak maps ladder!

This is just the 1v1 Arabia ladder

 

[The Bloodless]

This is just the 1v1 Arabia ladder

What do u mean