Executive Summary

The purpose of this document is to teach a learner who has finished "Building an AI Draughts Game for the Web" how to learn from the results of real players: rating every player and every computer level from the games already saved, matching each player to the level most likely to give them an even game, and testing a change fairly with an A/B experiment. It also covers handling data about people responsibly. The scope covers what the saved results can and cannot show, Elo ratings, rating the computer levels, matching, A/B testing, data retention and removal, tests and deployment. User accounts, databases and personal data beyond a chosen name are out of scope.

All code in this course was built and tested before the course was written. Because the true skill of real players is unknown, the rating system and the experiment were tested on simulated players whose relative strength was set in advance, playing the real game engine over twenty simulated days. The matters assessed are whether the ratings place simulated players in roughly their designed order, whether matching produces more even games than choosing levels at random, and whether an A/B experiment reaches the right conclusion under two different designs. Sources are listed in the appendix.

Module 0: Before You Start

What you will build

Every finished game in Course 1 was saved to a daily file, but the game only ever used today's file, for the scoreboard. This course puts all of that history to work.

• Ratings. Every player, and each of the three computer levels, gets an Elo rating, rebuilt from every game ever saved

• Matching. A new "Match me" choice picks the computer level that should give the player an even game

• An A/B test. Half the players get a slightly different Medium level, and the results show which version is tougher

• Privacy tools. Commands to delete old results and to remove a player on request

This is learning from data, not from a model: the system gets better at matching players as more games are played.

What you need

• The finished Course 1 project, working on your computer and your server

• Node.js, as used for the tests in Course 1

Nothing new needs installing. Make a copy of your Course 1 folder called draughts-adapt and open it in VS Code, and create a folder inside it called tools.

Your copy includes the venv folder, but a virtual environment cannot be copied: the copy still points at the folder where it was first made, so anything you install would go there instead. In the copy, delete the venv folder, then create and activate a new one exactly as in Course 1, and install the packages with pip install -r requirements.txt.

Module 1: What the Scoreboard Data Can and Cannot Show

What we have

Each saved game records the name of the player with Black, the name of the player with White, and the winner. Games against the computer use names such as "Computer (Medium)". The files are named by date, and within a file the games are in the order they finished. In this course each game will also record which computer level was played and which A/B group the player was in.

What it can show

• How often each person wins, draws and loses, and against whom

• How strong each computer level is against the people who actually play it

• Whether a change to the game alters how often people win

What it cannot show

• Who someone really is. Two people called Sam share one record, and one person can play under several names

• Whether results are genuine. As Course 1 noted, anyone could send a made-up result

• Why a game was won. A loss on a phone on a train and a loss after careful thought look the same

Module 2: Elo Ratings Explained

The idea

The Elo system, devised by the physicist Arpad Elo and used for chess ratings, gives each player a number. The gap between two players' numbers predicts how a game between them should go. After each game, both numbers move according to how surprising the result was.

Expected score

expected score = 1 / (1 + 10 ^ ((their rating - my rating) / 400))

Figure 1 shows this curve. Equal ratings expect 0.5, which is to say an even game. A player 200 points ahead expects 0.76, and one 400 points ahead about 0.91.

Updating after a game

new rating = old rating + K x (actual score - expected score)

A win scores 1, a draw a half and a loss 0. K controls how far one game can move a rating; we use 32. Whatever one player gains, the other loses, so the total never changes.

A worked example: Ann and Bob both start at 1200, so each expects 0.5. Ann wins. She gains 32 x (1 - 0.5) = 16 points, rising to 1216, and Bob falls to 1184. If Ann then beat a player rated 400 below her, she would gain only about 3 points, because the result was expected.

Provisional ratings

A rating based on one or two games means very little. We call a rating provisional until the player has five games, mark it with a star on the page, and do not use it for matching until then.

The listing

Create ratings.py next to app.py. load_games reads every daily file in date order, compute replays every game through the Elo formula, choose_level is explained in Module 4, and leaderboard prepares the table for the page.

File: ratings.py

"""
ratings.py - learning from real players' results (Course 4).

Every finished game is already saved in data/scores-YYYY-MM-DD.json. This
module reads all of those files in date order and gives every player - people
and the three computer levels alike - an Elo rating:

  * Everyone starts at 1200.
  * Before a game, Elo predicts each side's EXPECTED score from the gap in
    ratings:  expected = 1 / (1 + 10 ** ((their rating - my rating) / 400))
    Equal ratings expect 0.5; 400 points ahead expects about 0.91.
  * After the game, each rating moves by K x (actual score - expected score),
    where a win scores 1, a draw 0.5 and a loss 0.
So a surprising result moves ratings a lot, and an expected one hardly at all.

Because the computer levels are rated from the same games, we learn how
strong each level really is against these players, instead of guessing.
"""

import glob
import json
import os
import re

START = 1200
K = 32                       # how far one game can move a rating
PROVISIONAL_GAMES = 5        # fewer games than this: we do not trust the rating yet
LEVEL_PLAYERS = {"easy": "Computer (Low)", "medium": "Computer (Medium)",
                 "hard": "Computer (High)"}


def load_games(data_dir):
    """All saved games, oldest first. File names sort by date, games by the order saved."""
    games = []
    for path in sorted(glob.glob(os.path.join(data_dir, "scores-*.json"))):
        day = re.search(r"scores-(\d{4}-\d{2}-\d{2})\.json$", path).group(1)
        with open(path, encoding="utf-8") as f:
            for g in json.load(f):
                games.append({**g, "date": day})
    return games


def expected(my_rating, their_rating):
    return 1 / (1 + 10 ** ((their_rating - my_rating) / 400))


def compute(games):
    """Replay every game in order and return {name: {"rating": r, "games": n}}."""
    table = {}
    for g in games:
        black = table.setdefault(g["black"], {"rating": START, "games": 0})
        white = table.setdefault(g["white"], {"rating": START, "games": 0})
        score_black = {"b": 1.0, "w": 0.0, "draw": 0.5}[g["winner"]]
        e_black = expected(black["rating"], white["rating"])
        change = K * (score_black - e_black)
        black["rating"] += change
        white["rating"] -= change          # what one side gains, the other loses
        black["games"] += 1
        white["games"] += 1
    return table


def choose_level(name, table):
    """
    Pick the computer level that gives this player the most even game: the one
    whose rating is closest to theirs, so the expected score is nearest 0.5.
    New players, and levels with no games yet, fall back to Medium.
    """
    me = table.get(name)
    if me is None or me["games"] < PROVISIONAL_GAMES:
        return {"level": "medium", "provisional": True,
                "rating": round(me["rating"]) if me else START}
    best = None
    for level, player in LEVEL_PLAYERS.items():
        if player not in table:
            continue
        e = expected(me["rating"], table[player]["rating"])
        if best is None or abs(e - 0.5) < abs(best[1] - 0.5):
            best = (level, e)
    if best is None:
        return {"level": "medium", "provisional": True, "rating": round(me["rating"])}
    return {"level": best[0], "provisional": False, "rating": round(me["rating"]),
            "expected_score": round(best[1], 2)}


def leaderboard(table, min_games=1):
    rows = [{"name": n, "rating": round(v["rating"]), "games": v["games"],
             "provisional": v["games"] < PROVISIONAL_GAMES}
            for n, v in table.items() if v["games"] >= min_games and n != "(removed)"]
    return sorted(rows, key=lambda r: (-r["rating"], r["name"].lower()))

Check your progress

In a Python session, import ratings and then ratings.expected(1600, 1200) should give about 0.909.

Module 3: Rating the Computer Levels from Real Games

The computer is a player too

compute treats "Computer (Low)", "Computer (Medium)" and "Computer (High)" exactly like people. That gives us something we could not get any other way: a measure of how strong each level really is against the people who play it, rather than our guess when we wrote it.

How do we know the ratings are right?

With real players we cannot check, because nobody knows their true strength. So we test the rating system on simulated players whose strength we set ourselves: each is a computer player weakened on purpose, with a shallow search and a habit of making careless random moves. We named them Ada to Hal and intended them to be in that order, strongest first.

tools/play_game.js plays one game between any two computer players, using the real game engine. One small change to ai.js lets it accept a settings object as well as a level name; make it in chooseMove.

From static/js/ai.js

  // Pick a move for the side to move. `level` is a level name such as 'medium',
  // or (Course 4) a settings object such as { depth: 4, randomChance: 0.05 }.
  function chooseMove(state, level) {
    const settings = typeof level === 'string' ? (LEVELS[level] || LEVELS.medium) : level;

File: tools/play_game.js

/*
 * tools/play_game.js - play one game between two computer players, for simulations.
 *
 * Reads JSON on standard input:
 *   { "black": {"depth": 2, "randomChance": 0.2}, "white": {"level": "medium"},
 *     "medium_random_chance": 0.05 }
 * A player is either a computer level by name, or a depth and random chance
 * that imitate a person of a certain skill. Prints "b", "w" or "draw".
 */

const Rules = require('../static/js/rules.js');
const AI = require('../static/js/ai.js');

function settingsFor(p, mediumRandom) {
  if (p.level) {
    const s = { ...AI.LEVELS[p.level] };
    if (p.level === 'medium' && mediumRandom !== undefined) s.randomChance = mediumRandom;
    return s;
  }
  return { depth: p.depth, randomChance: p.randomChance };
}

function play(request) {
  const black = settingsFor(request.black, request.medium_random_chance);
  const white = settingsFor(request.white, request.medium_random_chance);
  let state = Rules.newGame();
  for (let ply = 0; ply < 200; ply++) {
    const outcome = Rules.result(state);
    if (outcome) return outcome.winner;
    state = Rules.applyMove(state, AI.chooseMove(state, state.turn === 'b' ? black : white));
  }
  return 'draw';
}

let input = '';
process.stdin.on('data', c => { input += c; });
process.stdin.on('end', () => process.stdout.write(play(JSON.parse(input))));

tools/simulate.py runs twenty simulated days. For the first ten, each player plays the computer at randomly chosen levels, plus one game against another player. For the last ten, each player plays the level chosen by matching (Module 4). Everything is written to a separate sim_data folder, never to the real data folder.

File: tools/simulate.py

"""
tools/simulate.py - test the rating system on players whose skill we already know.

Real players' true strength is unknown, so we cannot tell whether ratings are
right. Simulated players solve that: each one is a computer player made weak
on purpose (a shallow search and some random moves), so we know their order of
strength in advance. If the ratings put them in roughly that order, the rating
system works.

The simulation runs for 20 days:
  days 1-10   each player plays the computer at RANDOM levels, plus one other player
  days 11-20  each player plays the level chosen by ratings.choose_level()
Then it reports whether matching made games more even, and runs the A/B analysis.

Everything is written to sim_data/, never to the real data/ folder.
Run:  python tools/simulate.py
"""

import json
import os
import random
import shutil
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
from datetime import date, timedelta

ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
import experiment  # noqa: E402
import ratings     # noqa: E402

SIM_DIR = os.path.join(ROOT, "sim_data")
PLAY = os.path.join(ROOT, "tools", "play_game.js")

# Simulated people, strongest first. "depth" is how far ahead they look,
# "randomChance" how often they make a careless move.
PLAYERS = [
    ("Sim Ada", 4, 0.05), ("Sim Ben", 3, 0.10), ("Sim Cat", 3, 0.20), ("Sim Dan", 2, 0.15),
    ("Sim Eve", 2, 0.30), ("Sim Fin", 1, 0.30), ("Sim Gus", 1, 0.50), ("Sim Hal", 1, 0.80),
]
SKILL = {name: {"depth": d, "randomChance": r} for name, d, r in PLAYERS}
LEVELS = ["easy", "medium", "hard"]
rng = random.Random(2026)


def play(black, white, medium_random=None):
    request = {"black": black, "white": white}
    if medium_random is not None:
        request["medium_random_chance"] = medium_random
    done = subprocess.run(["node", PLAY], input=json.dumps(request),
                          capture_output=True, text=True, check=True)
    return done.stdout.strip()


def game_against_level(name, level):
    """One game between a simulated person and a computer level, colours at random."""
    variant = experiment.assign(name) if level == "medium" else None
    medium_random = (experiment.VARIANTS[variant]["medium_random_chance"] if variant else None)
    computer = ratings.LEVEL_PLAYERS[level]
    if rng.random() < 0.5:
        winner = play(SKILL[name], {"level": level}, medium_random)
        g = {"black": name, "white": computer, "winner": winner}
    else:
        winner = play({"level": level}, SKILL[name], medium_random)
        g = {"black": computer, "white": name, "winner": winner}
    g["level"] = level
    if variant:
        g["variant"] = variant
    return g


def game_between(a, b):
    return {"black": a, "white": b, "winner": play(SKILL[a], SKILL[b])}


def run_day(day, jobs):
    with ThreadPoolExecutor(max_workers=os.cpu_count() or 2) as pool:
        games = list(pool.map(lambda job: job[0](*job[1:]), jobs))
    path = os.path.join(SIM_DIR, f"scores-{day.isoformat()}.json")
    with open(path, "w") as f:
        json.dump(games, f, indent=1)
    return games


def player_scores(games):
    """Each simulated player's average score in a list of games."""
    totals = {}
    for g in games:
        for colour, name in (("b", g["black"]), ("w", g["white"])):
            if name in SKILL:
                s = 0.5 if g["winner"] == "draw" else (1.0 if g["winner"] == colour else 0.0)
                totals.setdefault(name, []).append(s)
    return {n: sum(v) / len(v) for n, v in totals.items()}


def main():
    shutil.rmtree(SIM_DIR, ignore_errors=True)
    os.makedirs(SIM_DIR)
    first_day = date(2026, 9, 1)
    random_phase, matched_phase = [], []
    names = list(SKILL)

    for d in range(20):
        day = first_day + timedelta(days=d)
        jobs = []
        if d < 10:
            for name in names:
                jobs += [(game_against_level, name, rng.choice(LEVELS)) for _ in range(3)]
                jobs.append((game_between, name, rng.choice([n for n in names if n != name])))
            random_phase += run_day(day, jobs)
        else:
            table = ratings.compute(ratings.load_games(SIM_DIR))
            for name in names:
                level = ratings.choose_level(name, table)["level"]
                jobs += [(game_against_level, name, level) for _ in range(3)]
            matched_phase += run_day(day, jobs)
        print(f"day {d + 1:2d} done")

    table = ratings.compute(ratings.load_games(SIM_DIR))
    print("\nRatings after 20 days (true order of skill: Ada strongest ... Hal weakest):")
    for row in ratings.leaderboard(table):
        print(f"  {row['name']:<18} {row['rating']:5d}  ({row['games']} games)")

    def evenness(games):
        only_levels = [g for g in games if "level" in g]
        scores = player_scores(only_levels)
        return sum(abs(s - 0.5) for s in scores.values()) / len(scores)

    print(f"\nHow far from an even game, on average (0 = perfectly even, 0.5 = one-sided):")
    print(f"  days 1-10, random levels : {evenness(random_phase):.3f}")
    print(f"  days 11-20, matched level: {evenness(matched_phase):.3f}")
    print("\nA/B test on all simulated games against Medium:")
    print(json.dumps(experiment.analyse(ratings.load_games(SIM_DIR)), indent=1))


if __name__ == "__main__":
    main()
python tools/simulate.py

On the computer used for this course the simulation took about twenty minutes.

What happened

Figure 2 shows the ratings after twenty days.

The ratings got the big picture right. Ada, intended to be strongest, rated highest of the simulated players at 1433. The three weakest, Fin, Gus and Hal, ended between 804 and 830, far below the rest. Within the groups the order was not exact. Dan, at 1323, rated above Ben and Cat, and Fin and Gus were the wrong way round, by 10 points.

Is the rating system wrong, or was our intended order wrong? Dan searches less deeply than Cat, but makes fewer careless moves, so it was never certain which was stronger. And 10 points between Fin and Gus is a gap that a few games of luck could easily reverse. That is the honest answer: the ratings agree with what we know for certain, and are uncertain where we were uncertain too.

The computer levels were rated too: Low at 1135, Medium at 1524 and High at 1634, all measured from games rather than assumed.

Module 4: Matching Players to a Level

Choosing the most even game

choose_level in ratings.py works out the player's expected score against each computer level, and picks the level where that expected score is closest to 0.5. New players, with provisional ratings, get Medium.

The new routes

The server gains two routes, and the results route accepts two new optional fields, level and variant, which it checks like everything else. The complete app.py is below; the Course 4 parts are marked in comments.

File: app.py

"""
app.py - the small Python web application behind the game.

It does two jobs:
  1. Sends the game files (HTML, CSS, JavaScript) to the browser.
  2. Keeps today's scoreboard in a JSON file. A new file starts each day,
     so the scoreboard resets at midnight (server time). No database is needed.
  3. (Course 4) Rates every player from all saved games, matches each player
     to a computer level, and runs a small A/B test.
"""

import json
import os
import re
from datetime import date

from flask import Flask, jsonify, request

import experiment
import ratings

# fcntl provides file locking, so two results saved at the same moment cannot
# overwrite each other. It exists on Linux and macOS (and so on the Apache
# server) but not on Windows, where we simply skip the lock during development.
try:
    import fcntl
except ImportError:
    fcntl = None

BASE_DIR = os.path.dirname(os.path.abspath(__file__))
DATA_DIR = os.path.join(BASE_DIR, "data")

# static_url_path="" serves static/index.html at "/" and static/js/... at "/js/...".
app = Flask(__name__, static_folder="static", static_url_path="")

# Names may contain letters, numbers, spaces and a few common symbols.
NAME_PATTERN = re.compile(r"^[\w .'()\-]{1,20}$", re.UNICODE)


def today_file():
    """Path of the JSON file holding today's games, e.g. data/scores-2026-10-27.json"""
    return os.path.join(DATA_DIR, f"scores-{date.today().isoformat()}.json")


def read_games():
    path = today_file()
    if not os.path.exists(path):
        return []
    with open(path, encoding="utf-8") as f:
        return json.load(f)


def build_table(games):
    """Turn the list of games into a league table: win 3 points, draw 1."""
    table = {}
    for g in games:
        for colour, name in (("b", g["black"]), ("w", g["white"])):
            row = table.setdefault(name, {"name": name, "played": 0, "won": 0,
                                          "drawn": 0, "lost": 0, "points": 0})
            row["played"] += 1
            if g["winner"] == "draw":
                row["drawn"] += 1
                row["points"] += 1
            elif g["winner"] == colour:
                row["won"] += 1
                row["points"] += 3
            else:
                row["lost"] += 1
    # Highest points first, then most wins, then name in alphabetical order.
    return sorted(table.values(), key=lambda r: (-r["points"], -r["won"], r["name"].lower()))


@app.route("/")
def index():
    return app.send_static_file("index.html")


@app.get("/api/scores")
def get_scores():
    return jsonify({"date": date.today().isoformat(), "table": build_table(read_games())})


@app.post("/api/scores")
def add_score():
    data = request.get_json(silent=True) or {}
    black = str(data.get("black", "")).strip()
    white = str(data.get("white", "")).strip()
    winner = data.get("winner")
    level = data.get("level")          # Course 4: which computer level, if any
    variant = data.get("variant")      # Course 4: which A/B group, if any

    # Never trust what the browser sends: check every value.
    if not NAME_PATTERN.match(black) or not NAME_PATTERN.match(white):
        return jsonify({"error": "Names must be 1-20 letters, numbers or spaces."}), 400
    if winner not in ("b", "w", "draw"):
        return jsonify({"error": "winner must be 'b', 'w' or 'draw'."}), 400
    if level not in (None, "easy", "medium", "hard") or variant not in (None, "A", "B"):
        return jsonify({"error": "Unknown level or variant."}), 400

    os.makedirs(DATA_DIR, exist_ok=True)
    path = today_file()
    # Open (or create) the file and lock it while we read, add and write.
    with open(path, "a+", encoding="utf-8") as f:
        if fcntl:
            fcntl.flock(f, fcntl.LOCK_EX)
        f.seek(0)
        text = f.read()
        games = json.loads(text) if text.strip() else []
        game = {"black": black, "white": white, "winner": winner}
        if level:
            game["level"] = level
        if variant:
            game["variant"] = variant
        games.append(game)
        f.seek(0)
        f.truncate()
        json.dump(games, f, indent=1)
        if fcntl:
            fcntl.flock(f, fcntl.LOCK_UN)

    return jsonify({"ok": True, "table": build_table(games)}), 201


# ---------- Course 4: ratings, matching and the A/B test ----------
@app.get("/api/ratings")
def get_ratings():
    table = ratings.compute(ratings.load_games(DATA_DIR))
    return jsonify({"table": ratings.leaderboard(table)})


@app.get("/api/match")
def get_match():
    name = str(request.args.get("name", "")).strip()
    if not NAME_PATTERN.match(name):
        return jsonify({"error": "Names must be 1-20 letters, numbers or spaces."}), 400
    table = ratings.compute(ratings.load_games(DATA_DIR))
    choice = ratings.choose_level(name, table)
    choice["variant"] = experiment.assign(name)
    choice["variant_settings"] = experiment.VARIANTS[choice["variant"]]
    return jsonify(choice)


if __name__ == "__main__":
    # Development server for your own computer only. Apache runs the app in production.
    app.run(host="127.0.0.1", port=5000, debug=True)

The browser side

Figure 3 shows what happens when a player chooses "Match me".

The setup dialog gains a "Match me" option, and the page gains a ratings table under the scoreboard.

From static/index.html

        <label><input type="radio" name="level" value="hard"> High</label>
        <label><input type="radio" name="level" value="auto"> Match me</label>

From static/index.html

    <!-- Course 4: Elo ratings from every game ever saved. -->
    <section class="scores" aria-label="Ratings">
      <h2>Ratings</h2>
      <table>
        <thead><tr><th>Player</th><th>Rating</th><th>Games</th></tr></thead>
        <tbody id="ratingRows">
          <tr><td colspan="3" class="muted">No rated games yet.</td></tr>
        </tbody>
      </table>
      <p class="muted small">Everyone starts at 1200. * means fewer than 5 games.</p>
    </section>

main.js changes in four places: the setup handler asks the server for a match before starting, the AI uses a settings object so the A/B variant can change Medium, the saved result includes the level and variant, and a new function draws the ratings table. The complete file is below.

File: static/js/main.js

/*
 * main.js - connects the rules and the AI to the web page.
 *
 * It draws the board, reacts to taps and clicks, asks the AI for its move,
 * and sends finished results to the server's scoreboard.
 */

// ---------- Game settings and state ----------
const LEVEL_NAMES = { easy: 'Low', medium: 'Medium', hard: 'High' };

let game = null;         // the current position, from Rules.newGame()
let players = null;      // { b: {name, human}, w: {name, human}, level }
let selected = null;     // { from, steps } while a move is being chosen
let lastMove = null;     // the previous move, highlighted on the board
let finished = false;    // true once the game has a result
let aiThinking = false;  // true while the computer is choosing

// ---------- Page elements ----------
const boardEl = document.getElementById('board');
const statusEl = document.getElementById('status');
const setupDialog = document.getElementById('setup');
const setupForm = document.getElementById('setupForm');

// ---------- Starting a game ----------
function showSetup() {
  setupDialog.showModal();
}

// Show or hide the second-name box and difficulty choice when the mode changes.
setupForm.addEventListener('change', () => {
  const twoPlayer = setupForm.mode.value === '2';
  document.getElementById('whiteField').hidden = !twoPlayer;
  document.getElementById('levelField').hidden = twoPlayer;
  document.getElementById('nameWhite').required = twoPlayer;
});

setupForm.addEventListener('submit', async () => {
  const twoPlayer = setupForm.mode.value === '2';
  let level = setupForm.level.value;
  const blackName = setupForm.nameBlack.value.trim();

  // Course 4: ask the server for this player's rating, a matched level and
  // their A/B group. If the server cannot be reached, fall back to Medium.
  let match = null;
  if (!twoPlayer) {
    try {
      const response = await fetch('api/match?name=' + encodeURIComponent(blackName));
      if (response.ok) match = await response.json();
    } catch (e) { /* play on without matching */ }
    if (level === 'auto') level = match ? match.level : 'medium';
  }

  const whiteName = twoPlayer ? setupForm.nameWhite.value.trim()
                              : 'Computer (' + LEVEL_NAMES[level] + ')';
  // Remember the name on this device so it is filled in next time.
  try {
    localStorage.setItem('draughtsName', blackName);
  } catch (e) { /* storage may be blocked */ }

  players = {
    b: { name: blackName, human: true },
    w: { name: whiteName, human: twoPlayer },
    level: level,
    mode: twoPlayer ? '2p' : '1p',
    variant: (!twoPlayer && level === 'medium' && match) ? match.variant : null,
    settings: aiSettings(level, match)
  };
  startGame();
  if (match && setupForm.level.value === 'auto') {
    setStatus('Matched to ' + LEVEL_NAMES[level] + ' (your rating ' + match.rating +
              (match.provisional ? ', still settling' : '') + '). ' + statusEl.textContent);
  }
});

function startGame() {
  game = Rules.newGame();
  selected = null;
  lastMove = null;
  finished = false;
  aiThinking = false;
  document.getElementById('blackName').textContent = players.b.name;
  document.getElementById('whiteName').textContent = players.w.name;
  render();
  nextTurn();
}

// ---------- Turns ----------
function nextTurn() {
  const outcome = Rules.result(game);
  if (outcome) return endGame(outcome);

  const mover = players[game.turn];
  if (mover.human) {
    const mustCapture = Rules.legalMoves(game)[0].captures.length > 0;
    setStatus(mover.name + ' to move' + (mustCapture ? ' - you must capture.' : '.'));
  } else {
    aiThinking = true;
    setStatus(mover.name + ' is thinking...');
    // Wait a moment so the page can show the message before the AI starts working.
    setTimeout(() => {
      const move = AI.chooseMove(game, players.settings);
      aiThinking = false;
      playMove(move);
    }, 350);
  }
}

function playMove(move) {
  game = Rules.applyMove(game, move);
  lastMove = move;
  selected = null;
  render();
  nextTurn();
}

function endGame(outcome) {
  finished = true;
  let text;
  if (outcome.winner === 'draw') text = 'Draw: ' + outcome.reason + '.';
  else text = players[outcome.winner].name + ' wins (' + outcome.reason + ').';
  setStatus(text + ' Tap New game to play again.');
  saveResult(outcome.winner);
}

// ---------- Choosing a move by tapping ----------
// Moves that match the piece and the squares chosen so far.
function candidateMoves() {
  if (!selected) return [];
  return Rules.legalMoves(game).filter(m => {
    if (!same(m.path[0], selected.from)) return false;
    for (let i = 0; i < selected.steps.length; i++) {
      if (!m.path[i + 1] || !same(m.path[i + 1], selected.steps[i])) return false;
    }
    return true;
  });
}

function onSquareTap(r, c) {
  if (finished || aiThinking || !players || !players[game.turn].human) return;
  const piece = game.board[r][c];

  // Tapping one of your own pieces (before a multi-jump has begun) selects it.
  if (Rules.ownerOf(piece) === game.turn && (!selected || selected.steps.length === 0)) {
    const hasMoves = Rules.legalMoves(game).some(m => same(m.path[0], [r, c]));
    if (hasMoves) {
      selected = { from: [r, c], steps: [] };
    } else {
      selected = null;
      const mustCapture = Rules.legalMoves(game)[0].captures.length > 0;
      setStatus(mustCapture ? 'You must capture with another piece.'
                            : 'That piece cannot move.');
    }
    return render();
  }

  if (!selected) return;

  // Is the tapped square the next square of any legal move?
  const depth = selected.steps.length + 1;
  const matching = candidateMoves().filter(m => m.path[depth] && same(m.path[depth], [r, c]));
  if (matching.length === 0) {
    // During a multi-jump the player must continue; otherwise cancel the selection.
    if (selected.steps.length === 0) selected = null;
    return render();
  }

  selected.steps.push([r, c]);
  const complete = matching.find(m => m.path.length === depth + 1);
  if (complete) return playMove(complete);
  setStatus('Keep jumping.');
  render();
}

// ---------- Drawing the board ----------
// While a multi-jump is part-way through, show the piece where it has got to.
function displayBoard() {
  if (!selected || selected.steps.length === 0) return game.board;
  const board = Rules.copyBoard(game.board);
  const move = candidateMoves()[0];
  const [fr, fc] = selected.from;
  const [tr, tc] = selected.steps[selected.steps.length - 1];
  board[tr][tc] = board[fr][fc];
  board[fr][fc] = '.';
  for (let i = 0; i < selected.steps.length; i++) {
    const [cr, cc] = move.captures[i];
    board[cr][cc] = '.';
  }
  return board;
}

function render() {
  const board = displayBoard();
  const targets = new Set();
  if (selected) {
    const depth = selected.steps.length + 1;
    for (const m of candidateMoves()) if (m.path[depth]) targets.add(key(m.path[depth]));
  }
  const lastSquares = new Set(lastMove ? lastMove.path.map(key) : []);
  const pieceAt = selected
    ? key(selected.steps.length ? selected.steps[selected.steps.length - 1] : selected.from)
    : null;

  boardEl.innerHTML = '';
  for (let r = 0; r < 8; r++) {
    for (let c = 0; c < 8; c++) {
      const sq = document.createElement('div');
      sq.className = 'sq' + ((r + c) % 2 === 1 ? ' dark' : '');
      if (targets.has(key([r, c]))) sq.classList.add('target');
      if (lastSquares.has(key([r, c]))) sq.classList.add('last');
      const p = board[r][c];
      if (p !== '.') {
        const pieceEl = document.createElement('div');
        pieceEl.className = 'piece ' + Rules.ownerOf(p) + (Rules.isKing(p) ? ' king' : '');
        if (pieceAt === key([r, c])) pieceEl.classList.add('selected');
        sq.appendChild(pieceEl);
      }
      sq.addEventListener('click', () => onSquareTap(r, c));
      boardEl.appendChild(sq);
    }
  }

  const n = Rules.countPieces(game.board);
  document.getElementById('blackCount').textContent = (n.b + n.B) + ' pieces';
  document.getElementById('whiteCount').textContent = (n.w + n.W) + ' pieces';
}

// ---------- Talking to the server ----------
// The URLs have no leading slash, so the game works in any folder on the server.
async function saveResult(winner) {
  try {
    await fetch('api/scores', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ black: players.b.name, white: players.w.name, winner: winner,
                             level: players.mode === '1p' ? players.level : undefined,
                             variant: players.variant || undefined })
    });
  } catch (e) {
    console.warn('Could not save the result', e);
  }
  loadScores();
  loadRatings();
}

// Course 4: the A/B test changes how often Medium plays a random move.
function aiSettings(level, match) {
  const settings = { ...AI.LEVELS[level] };
  if (level === 'medium' && match && match.variant_settings) {
    settings.randomChance = match.variant_settings.medium_random_chance;
  }
  return settings;
}

async function loadRatings() {
  const rowsEl = document.getElementById('ratingRows');
  try {
    const data = await (await fetch('api/ratings')).json();
    if (data.table.length === 0) return;
    rowsEl.innerHTML = '';
    for (const row of data.table) {
      const tr = document.createElement('tr');
      for (const value of [row.name + (row.provisional ? ' *' : ''), row.rating, row.games]) {
        const td = document.createElement('td');
        td.textContent = value;
        tr.appendChild(td);
      }
      rowsEl.appendChild(tr);
    }
  } catch (e) {
    rowsEl.innerHTML = '<tr><td colspan="3" class="muted">Ratings unavailable.</td></tr>';
  }
}

async function loadScores() {
  const rowsEl = document.getElementById('scoreRows');
  try {
    const response = await fetch('api/scores');
    const data = await response.json();
    if (data.table.length === 0) return;
    rowsEl.innerHTML = '';
    for (const row of data.table) {
      const tr = document.createElement('tr');
      for (const value of [row.name, row.played, row.won, row.drawn, row.lost, row.points]) {
        const td = document.createElement('td');
        td.textContent = value; // textContent is safe: names are never treated as HTML
        tr.appendChild(td);
      }
      rowsEl.appendChild(tr);
    }
  } catch (e) {
    rowsEl.innerHTML = '<tr><td colspan="6" class="muted">Scoreboard unavailable.</td></tr>';
  }
}

// ---------- Small helpers ----------
function same(a, b) { return a[0] === b[0] && a[1] === b[1]; }
function key(sq) { return sq[0] + ',' + sq[1]; }
function setStatus(text) { statusEl.textContent = text; }

// ---------- Start up ----------
document.getElementById('newGameBtn').addEventListener('click', showSetup);
try {
  setupForm.nameBlack.value = localStorage.getItem('draughtsName') || '';
} catch (e) { /* ignore */ }
game = Rules.newGame();
render();
loadScores();
loadRatings();
showSetup();

Add the style rules at the end of style.css so that, on a wide screen, both tables sit to the right of the board.

From static/css/style.css

/* ---------- Course 4: room for the ratings table ---------- */
@media (min-width: 820px) {
  .layout { grid-template-rows: auto 1fr; }
  .game { grid-row: span 2; }
  .scores { align-self: start; }

Did matching help?

The simulation measured how far each simulated player's average score was from an even 0.5. With levels chosen at random, the average distance was 0.296. With matched levels it fell to 0.223. Matching made games more even, though it could not make them perfectly even with only three levels to choose from.

Check your progress

Play a game with "Match me" selected. The status line names your level and rating, and after the game your name appears in the ratings table with a star.

Module 5: A/B Testing a Change

The question

Does the Medium level become tougher if it makes fewer random moves? We could guess, or we could test it with real players. An A/B test gives one group of players the current version (A, random moves 10% of the time) and another group the new version (B, 5%), and compares the results.

Assigning groups

experiment.assign puts each player in a group using a hash of their name: a calculation that turns the name into a number that looks random, but is always the same for the same name. So a player always gets the same version, and no list of players has to be stored.

experiment.analyse compares the players' average score against Medium in each group, works out the difference and its 95% range, and refuses to give a verdict until each group has at least 30 games. It also runs a balance check, warning when one player supplies more than a quarter of a group's games.

File: experiment.py

"""
experiment.py - a simple A/B test (Course 4).

Question: does making the Medium computer less random make it a tougher
opponent for real players?

  Variant A: Medium plays a random move 10% of the time (as in Course 1)
  Variant B: Medium plays a random move 5% of the time

Each player is put in group A or B by their name, so the same player always
gets the same variant and no list of players has to be stored. After enough
games, analyse() compares how well players scored against Medium in each group.
"""

import hashlib
import math

EXPERIMENT = "medium-randomness"
VARIANTS = {"A": {"medium_random_chance": 0.10}, "B": {"medium_random_chance": 0.05}}
MEDIUM = "Computer (Medium)"


def assign(name):
    """Always the same answer for the same name: 'A' or 'B', about half each."""
    digest = hashlib.sha256((EXPERIMENT + ":" + name.strip().lower()).encode("utf-8")).digest()
    return "A" if digest[0] % 2 == 0 else "B"


def analyse(games):
    """Compare the players' average score against Medium in each variant."""
    groups = {"A": [], "B": []}
    players = {"A": {}, "B": {}}
    for g in games:
        if g.get("variant") not in groups or MEDIUM not in (g["black"], g["white"]):
            continue
        human_colour = "w" if g["black"] == MEDIUM else "b"
        score = 0.5 if g["winner"] == "draw" else (1.0 if g["winner"] == human_colour else 0.0)
        groups[g["variant"]].append(score)
        human = g["white"] if human_colour == "w" else g["black"]
        players[g["variant"]][human] = players[g["variant"]].get(human, 0) + 1

    result = {}
    for v, scores in groups.items():
        n = len(scores)
        result[v] = {"games": n, "player_score": round(sum(scores) / n, 3) if n else None,
                     "players": len(players[v])}
    # Balance check: if one person plays most of a group's games, the groups may
    # differ in skill, and that - not the variant - could explain the result.
    warnings = []
    for v, counts in players.items():
        if counts and max(counts.values()) / sum(counts.values()) > 0.25:
            top = max(counts, key=counts.get)
            share = round(100 * counts[top] / sum(counts.values()))
            warnings.append(f"{top} played {share}% of group {v}'s games, so the groups "
                            f"may differ in skill. Treat the result with caution.")
    result["warnings"] = warnings
    a, b = groups["A"], groups["B"]
    if min(len(a), len(b)) < 30:
        result["verdict"] = "Not enough games yet: wait for at least 30 in each group."
        return result
    pa, pb = sum(a) / len(a), sum(b) / len(b)
    diff = pb - pa
    se = math.sqrt(pa * (1 - pa) / len(a) + pb * (1 - pb) / len(b))
    low, high = diff - 1.96 * se, diff + 1.96 * se
    result["difference_B_minus_A"] = round(diff, 3)
    result["range_95"] = [round(low, 3), round(high, 3)]
    if high < 0:
        result["verdict"] = "B is tougher: players score less against it, beyond what chance explains."
    elif low > 0:
        result["verdict"] = "B is easier: players score more against it, beyond what chance explains."
    else:
        result["verdict"] = "No clear difference: the 95% range includes zero."
    return result

tools/ab_report.py prints the current result.

File: tools/ab_report.py

"""
tools/ab_report.py - print the current result of the A/B test (Course 4).
Run:  python tools/ab_report.py           (uses data/)
      python tools/ab_report.py sim_data  (uses the simulation's data)
"""
import json
import os
import sys

ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
import experiment  # noqa: E402
import ratings     # noqa: E402

folder = os.path.join(ROOT, sys.argv[1] if len(sys.argv) > 1 else "data")
print(json.dumps(experiment.analyse(ratings.load_games(folder)), indent=1))

The first result was wrong

We ran the analysis on the twenty simulated days. Group A's players scored 0.147 against Medium in 51 games, and group B's 0.308 in 73 games. The difference was +16.1 points, with a 95% range from +1.7 to +30.5, so the analysis concluded that B, the less random version, was easier. That makes no sense: fewer careless moves cannot make a player weaker.

The balance check explains it. Ada, the strongest simulated player, was in group B and played 58% of B's games against Medium, because matching kept sending her there. Group B looked easier because it contained the best player, not because of the change. When something other than the change differs between the groups, it is called a confounding factor. With only eight players, assigning whole players to groups could not make the groups alike.

Asking the question fairly

tools/fair_ab.py asks the same question with a better design: every simulated player plays Medium under both versions, the same number of times each, alternating game by game. Now each player is compared with themselves, so skill cannot differ between the groups.

File: tools/fair_ab.py

"""
tools/fair_ab.py - the same A/B question, asked fairly.

In the main simulation each player always gets the same variant, chosen by
name. With only eight players, chance put the strongest player in group B,
so group B looked "easier" for the wrong reason.

Here every simulated player plays Medium under BOTH variants, alternating game
by game, and the same number of times each. Any difference between A and B can
then only come from the variant itself (or from chance, which the 95% range
measures). Results go to sim_data_fair/.

Run:  python tools/fair_ab.py
"""

import json
import os
import sys
from concurrent.futures import ThreadPoolExecutor

ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, ROOT)
sys.path.insert(0, os.path.join(ROOT, "tools"))
import experiment                    # noqa: E402
from simulate import SKILL, play     # noqa: E402

GAMES_EACH = 16          # per player per variant, half as Black and half as White
OUT_DIR = os.path.join(ROOT, "sim_data_fair")


def one(job):
    name, variant, as_black = job
    rate = experiment.VARIANTS[variant]["medium_random_chance"]
    if as_black:
        g = {"black": name, "white": "Computer (Medium)",
             "winner": play(SKILL[name], {"level": "medium"}, rate)}
    else:
        g = {"black": "Computer (Medium)", "white": name,
             "winner": play({"level": "medium"}, SKILL[name], rate)}
    g.update({"level": "medium", "variant": variant})
    return g


def main():
    jobs = [(name, variant, i % 2 == 0)
            for name in SKILL for i in range(GAMES_EACH) for variant in ("A", "B")]
    with ThreadPoolExecutor(max_workers=os.cpu_count() or 2) as pool:
        games = list(pool.map(one, jobs))
    os.makedirs(OUT_DIR, exist_ok=True)
    with open(os.path.join(OUT_DIR, "scores-2026-10-01.json"), "w") as f:
        json.dump(games, f, indent=1)
    print(json.dumps(experiment.analyse(games), indent=1))


if __name__ == "__main__":
    main()

Figure 4 compares the two designs.

With the fair design, players scored 0.199 against version A and 0.109 against version B, 128 games each. B was tougher by 9.0 points, with a 95% range from 17.8 to 0.2 points tougher, and no balance warnings. That is the result common sense expected, now backed by evidence.

What to take from this

• Randomly splitting people only balances the groups when there are many people. With few, check the balance, or let every person try both versions

• A result that makes no sense is a reason to look for a confounding factor, not to publish it

• Run the analysis only after the planned number of games, not every day until it says what you hoped

Module 6: Handling Data About People Responsibly

A player's chosen name, with the games they played and when, is information about a person. The United Kingdom's data protection law sets out principles for handling personal data, and the Information Commissioner's Office (ICO) publishes guidance on them (see the appendix). This course is not legal advice, but three habits from those principles are easy to build in.

• Keep only what you need. The game stores a chosen name and results, nothing more. Resist adding email addresses or locations "in case they are useful"

• Do not keep it for ever. Decide how long results are useful and delete older ones automatically

• Act on requests. If someone asks to be removed, be able to do it quickly and completely

tools/privacy.py provides the last two.

File: tools/privacy.py

"""
tools/privacy.py - looking after the data we keep about people (Course 4).

A player's name, with the games they played and when, is information about
a person. Three habits keep that information under control:

  * keep only what the game needs (we already store just names and results);
  * do not keep it for ever:      python tools/privacy.py expire 90
    deletes daily files older than 90 days;
  * remove someone on request:    python tools/privacy.py forget "Ann"
    replaces that name with "(removed)" in every saved game.

Both commands change ratings, because ratings are rebuilt from the saved games.
That is the honest consequence of deleting data, and the course discusses it.
"""

import glob
import json
import os
import re
import sys
from datetime import date, timedelta

DATA_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "data")
REMOVED = "(removed)"


def expire(days, data_dir=DATA_DIR, today=None):
    """Delete daily files older than `days` days. Returns the files deleted."""
    cutoff = (today or date.today()) - timedelta(days=days)
    deleted = []
    for path in glob.glob(os.path.join(data_dir, "scores-*.json")):
        day = date.fromisoformat(re.search(r"(\d{4}-\d{2}-\d{2})", path).group(1))
        if day < cutoff:
            os.remove(path)
            deleted.append(os.path.basename(path))
    return sorted(deleted)


def forget(name, data_dir=DATA_DIR):
    """Replace a player's name everywhere. Returns how many games were changed."""
    wanted = name.strip().lower()
    changed = 0
    for path in glob.glob(os.path.join(data_dir, "scores-*.json")):
        with open(path, encoding="utf-8") as f:
            games = json.load(f)
        touched = False
        for g in games:
            sides = [s for s in ("black", "white") if g[s].strip().lower() == wanted]
            for side in sides:
                g[side] = REMOVED
            if sides:
                changed += 1
                touched = True
        if touched:
            with open(path, "w", encoding="utf-8") as f:
                json.dump(games, f, indent=1)
    return changed


if __name__ == "__main__":
    if len(sys.argv) == 3 and sys.argv[1] == "expire":
        print("Deleted:", expire(int(sys.argv[2])) or "nothing")
    elif len(sys.argv) == 3 and sys.argv[1] == "forget":
        print("Games changed:", forget(sys.argv[2]))
    else:
        print(__doc__)
python tools/privacy.py expire 90
python tools/privacy.py forget "Ann"

Notice the honest consequence: ratings are rebuilt from saved games, so deleting games changes everyone's ratings, including the computer levels'. Removed players become "(removed)", which keeps the history consistent for the others, and is never shown in the ratings table.

Module 7: Testing

The new tests check the Elo formula against values worked out by hand, matching, group assignment, the A/B analysis in both its "too early" and "clear" cases, the privacy commands on a temporary folder, and the new routes.

File: tests/test_ratings.py

"""Run with:  python tests/test_ratings.py"""
import json
import os
import sys
import tempfile
from datetime import date

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import experiment          # noqa: E402
import ratings             # noqa: E402
from tools import privacy  # noqa: E402


def test_expected_scores():
    assert ratings.expected(1200, 1200) == 0.5
    assert round(ratings.expected(1600, 1200), 3) == 0.909      # 400 points ahead
    assert abs(ratings.expected(1300, 1200) + ratings.expected(1200, 1300) - 1) < 1e-9


def test_a_win_moves_points_from_loser_to_winner():
    table = ratings.compute([{"black": "Ann", "white": "Bob", "winner": "b"}])
    assert table["Ann"]["rating"] == 1216 and table["Bob"]["rating"] == 1184   # K/2 each way
    total = sum(v["rating"] for v in table.values())
    assert abs(total - 2 * ratings.START) < 1e-9


def test_new_players_get_medium():
    choice = ratings.choose_level("Newcomer", {})
    assert choice == {"level": "medium", "provisional": True, "rating": 1200}


def test_level_closest_in_strength_is_chosen():
    table = {"Ann": {"rating": 1450, "games": 10},
             "Computer (Low)": {"rating": 1000, "games": 50},
             "Computer (Medium)": {"rating": 1300, "games": 50},
             "Computer (High)": {"rating": 1500, "games": 50}}
    assert ratings.choose_level("Ann", table)["level"] == "hard"


def test_ab_groups_are_stable_and_roughly_even():
    assert experiment.assign("Ann") == experiment.assign(" ann ")
    groups = [experiment.assign(f"Player {i}") for i in range(1000)]
    assert 400 < groups.count("A") < 600


def test_ab_analysis_waits_for_enough_games():
    games = [{"black": "Ann", "white": "Computer (Medium)", "winner": "b", "variant": "A"}] * 5
    assert "Not enough games" in experiment.analyse(games)["verdict"]


def test_ab_analysis_spots_a_clear_difference():
    a = [{"black": "P", "white": "Computer (Medium)", "winner": "b", "variant": "A"}] * 60
    b = [{"black": "P", "white": "Computer (Medium)", "winner": "w", "variant": "B"}] * 60
    assert experiment.analyse(a + b)["verdict"].startswith("B is tougher")


def test_forget_and_expire():
    folder = tempfile.mkdtemp()
    old, new = "scores-2020-01-01.json", f"scores-{date.today().isoformat()}.json"
    for name in (old, new):
        with open(os.path.join(folder, name), "w") as f:
            json.dump([{"black": "Ann", "white": "Bob", "winner": "b"}], f)
    assert privacy.forget("ANN", folder) == 2
    games = ratings.load_games(folder)
    assert all(g["black"] == "(removed)" for g in games)
    assert privacy.expire(90, folder) == [old]
    assert len(ratings.load_games(folder)) == 1


def test_routes():
    import app as server
    server.DATA_DIR = tempfile.mkdtemp()
    client = server.app.test_client()
    r = client.post("/api/scores", json={"black": "Ann", "white": "Computer (Medium)",
                                         "winner": "b", "level": "medium", "variant": "A"})
    assert r.status_code == 201
    bad = client.post("/api/scores", json={"black": "Ann", "white": "Bob", "winner": "b",
                                           "variant": "C"})
    assert bad.status_code == 400
    table = client.get("/api/ratings").get_json()["table"]
    assert table[0]["name"] == "Ann" and table[0]["provisional"]
    match = client.get("/api/match?name=Ann").get_json()
    assert match["level"] == "medium" and match["variant"] in ("A", "B")
    assert client.get("/api/match?name=<x>").status_code == 400


if __name__ == "__main__":
    for name, fn in list(globals().items()):
        if name.startswith("test_"):
            fn()
            print("passed", name)
python tests/test_ratings.py
python tests/test_app.py
node tests/test_rules.js

Check your progress

Every line from the Python test files starts with passed, and the Course 1 rules tests still pass.

Module 8: Deploying the Adaptive Opponent

No new software is needed on the server. Copy the new and changed files from your project folder.

scp -r app.py ratings.py experiment.py tools static you@your-server:/var/www/draughts/

Do not copy sim_data or sim_data_fair: they hold simulated games, and mixing them with real results would make the real ratings meaningless.

Reload Apache as in Course 1, then play a game with "Match me".

sudo systemctl reload apache2

Deleting old results automatically

A cron job runs a command on a schedule. To delete results older than 90 days at 3 a.m. every day, open the schedule for the www-data user, which owns the data folder.

sudo crontab -u www-data -e

Add this line at the end, then save.

0 3 * * * /var/www/draughts/venv/bin/python /var/www/draughts/tools/privacy.py expire 90

Module 9: Where to Go Next

• More levels. With only three levels, matching cannot make every game even. Add levels between Low and High by varying depth and random moves, and rate them

• Anchor the scale. Every rating floats at present. Fixing Low at 1000, for example, makes ratings comparable over months

• Per-game assignment for real players. Change assign to choose A or B for each game rather than each player, as in the fair simulation, and record the variant with every Medium game

• Plan the sample size. Before starting an experiment, work out how many games are needed to detect the difference you care about, and stop exactly there

Known limitations

• Ratings are only as good as the names. Shared and changed names blur them, and faked results distort them

• Ratings are rebuilt on every request. That is simple and fine for thousands of games, but would need caching for many more

• The simulation is not real people. It tests that the system behaves sensibly, not how real players will behave

Glossary of Terms

Appendix: Sources