Zhangqi's "36% of Fox 8d are AI cheaters; 22% for 7d"

Once again thanks to Alexandre Amaro 8d Fox, recently I was introduced to Zhangqi, yet another Go platform for AI analysis, similar to Golaxy or AI Sensei. But Zhangqi offers a feature which truly sets it apart: it can estimate your rank from the quality of your moves. AI Sensei, much like KaTrain, does offer a "Performance Report", but it lacks a reference point, using percentages for the ratings, which, to me, feel meaningless, a type of number that doesn't feel actionable at all.

AI Sensei's "Performance Report" for the historic first game between Shin Jinseo 신진서 9p and KataGo, on a 2-stone handicap. The "Student level" slider does change the evaluation from "Good move" downwards, but not the "AI Move".

The same historic match on Zhangqi's report this time, with an auto-translation from Chinese. Shin Jinseo 9p is estimated as 11.2d Fox, while KataGo receives a 12d rating. Shin Jinseo 9p is often officially rated as a Fox 10d, perhaps he gets Fox 11d due to the inflation mentioned down below.

And that's not even mentioning that the more fighting a game has, the more likely humans will make egregious mistakes, which would sink the percentages down by a lot. So players who fight a lot tend to take lower percentages, without that meaning much since those percentages didn't take into account the difficulty behind each move. Zhangqi offers a solution to that problem with the "fit" and "difficulty" columns, and the variance on the first column, denoted by the orange number after "±".

It's unfortunate Zhangqi seems to estimate ranks by sections, it isn't able to give each move a rank yet. It would be amazing to see a rank graph for each move, much like what we get for points and percentages. And there are some constraints a game needs to meet in order to get rank estimates. For example, each game needs at least 100 moves to receive a rank; and the endgame demands 100 moves past move 100 for a rating.

On another note, it's worth mentioning that Zhangqi's rank estimates are perhaps unintentionally inflated. It seems they trained their AI to estimate each player's strength from a large database of blitz games. So, if you're playing with 30s or 60s byo-yomi on Fox, your moves will be of higher quality than the AI's reference points.

Anyhow, getting back to the main topic, Zhangqi has used its ranking tool to then estimate how many players are cheating with AI. According to their research, it seems like these are the percentages for AI cheaters:

• 36% of Fox 8d's
• 22% of Fox 7d's

So maybe around 1 in 3 Fox 8d's are cheaters, while 1 in 5 for Fox 7d's.

And who knows about Fox 9d... During my stay in Korea, most Fox 9d's didn't even accept games from other Fox 9d's unless they had a badge proving they were at some point certified to a similar rank in real life.

What I usually do and recommend, and that's something I was doing even before the AI era — yes, I'm really that old — is to check the current window's win/loss record on my opponent's card. If there are way too many losses, either that player is really that strong, or maybe we've found ourselves a cheater — do you wanna take a chance?

Shin Jinseo 9p's Fox card.

Shin Jinseo 9p's badge.

Going More In Depth On The Statistics

The following information is all thanks to Alexandre Amaro 8d Fox, who kindly translated the Chinese content in the videos, and much more.

Let's take a look at how exactly Zhangqi managed to get to those conclusions and statistics.

For both the 7d and 8d trials, they used 50 accounts on 20 games each, totalling 1,000 games. That generated 2,000 data points because each game features 2 players. It isn't that much data, but it's already something.

On the 8d trial, they found that 18% of players played beyond top pro level (world champion actually), and 18% played just like AIs. Playing beyond top pro level might mean the cheater is just not following all the AI recommendations, which would lead the average evaluation down a bit.

18% of 8d's are beyond top pro (world champion); and 18% are AIs.

The strength distribution follows a bell curve around 8d but with a sharp spike at 12d, which is where current AIs live. Also, a considerable number of opponents accumulate past 10d. It's a pity we don't have this sort of graph for the pre-AI era.

A sharp spike on the 8d distribution graph, at 12d.

Their way of calculating the percentage of cheaters doesn't come from the distribution graph, they seem to have done it by labeling a percentage of the accounts as cheaters from a minimum number of cheated games.

If you take into account that ranking up on Fox is quite demanding on that range — you need to win 15 games in a 20-game window —, having 3-4 games out of those 20 be virtually unwinnable makes leveling up borderline impossible. That's why you should check your opponent's current window before continuing the game past the first 10 moves — Fox annuls games with 10 or fewer moves.

By the way, Fox's way of ranking up and down is very harsh compared to typical Elo rating. In normal Elo, a one-rank difference (100 points) should mean a 65% win rate for the stronger player, not 15/20 or 75%. A 3:4 rate would mean a 2-stone difference in Elo (200 points).

Here's the full video for the 8d analysis:

Applying the same methods for 7d's, they found that 22% of those are likely cheaters. And the bell curve does again register a considerable number of players beyond 9-10d.

Another sharp spike at 12d.

The reason we also might have perceived strengths from 4d to 6d is likely because the account's owner has lent it to a weaker player. Some strong players follow the same pattern seen in MMORPGs: rank up an account in order to sell it to a weaker player. The buyer would then be able to experience and train with very strong players, a privilege one would typically need to pay for.