The larger evaluation is still pending. These small exploratory runs are not the Lab’s 100,000-hand studies.
No clear winner emerged in this batch. An interval that crosses zero means this test has not established which policy comes out ahead. It does not mean they are equivalent.
Start with your rules.
Hong Kong New Style, Chinese Official (MCR), Japanese Riichi and Taiwanese Club are tested separately. Taiwanese Classic is not part of this batch; Club results do not carry over to Classic.
10 completed comparisons shown. Points use each ruleset’s own units; do not rank different rulesets by these numbers.
Does calling more often help?
A check of our earlier calling experiments. The same advanced player makes every other decision; only its willingness to take legal build claims changes.
Hong Kong
No clear result
Change in net points / hand-0.59
Approximate 95% interval-1.76 to +0.58
Matched deals256
Treatment minus control. Positive values favour the tested change; negative values favour the control.
The exact comparison & supporting numbers
Control
Advanced AI calling decisions
Tested change
Take the highest-priority legal build claim; same advanced AI elsewhere
Mean net points per hand
0.32 control → -0.27 treatment
Win rate
26.6% → 24.6%
Discarder of a winning tile
14.5% → 11.3%
Treatment behaviour
665 eligible decisions; 540 differed from the default AI. These count decisions, not hands.
In Riichi, we test keeping a tile that at least two opponents have discarded. The reserve is released when the hand is near ready or someone declares riichi.
Japanese Riichi
Model diagnostic
Change in net points / hand-42.97
Approximate 95% interval-140.84 to +54.9
Matched deals256
Treatment minus control. Positive values favour the tested change; negative values favour the control.
The exact comparison & supporting numbers
Control
Normal advanced discards
Tested change
After six own discards, retain the only tile discarded by at least two opponents; release near ready or on a declaration
Mean net points per hand
-29.3 control → -72.27 treatment
Win rate
19.1% → 18.4%
Discarder of a winning tile
17.2% → 17.6%
Treatment behaviour
117 eligible decisions; 117 differed from the default AI. These count decisions, not hands.
We test an early discard preference with a strong starting suit. This changes discards only; it does not force every call into a flush.
Hong Kong
No clear result
Change in net points / hand+1.03
Approximate 95% interval-0.94 to +3
Matched deals256
Treatment minus control. Positive values favour the tested change; negative values favour the control.
The exact comparison & supporting numbers
Control
Flexible advanced discards
Tested change
With at least 8 tiles of one suit at first decision, prefer discarding another suit with at most one extra distance unit; stop on exposure or a declaration
Mean net points per hand
-0.69 control → 0.34 treatment
Win rate
16.8% → 16.4%
Discarder of a winning tile
16.8% → 16.0%
Treatment behaviour
46 eligible decisions; 46 differed from the default AI. These count decisions, not hands.
Treatment minus control. Positive values favour the tested change; negative values favour the control.
The exact comparison & supporting numbers
Control
Flexible advanced discards
Tested change
With at least 10 tiles of one suit at first decision, prefer discarding another suit with at most one extra distance unit; stop on exposure or a declaration
Mean net points per hand
-0.49 control → -0.45 treatment
Win rate
14.1% → 14.1%
Discarder of a winning tile
13.3% → 13.3%
Treatment behaviour
10 eligible decisions; 10 differed from the default AI. These count decisions, not hands.
While the hand is concealed, take an eligible dragon, seat-wind or round-wind pung—or pass that opportunity. Other decisions stay with the same advanced player.
Japanese Riichi
Model diagnostic
Change in net points / hand-50.39
Approximate 95% interval-153.33 to +52.55
Matched deals256
Treatment minus control. Positive values favour the tested change; negative values favour the control.
The exact comparison & supporting numbers
Control
Pass eligible value-honor pungs while concealed
Tested change
Accept an eligible dragon, seat-wind or round-wind pung while concealed; same advanced AI elsewhere
Mean net points per hand
41.02 control → -9.38 treatment
Win rate
24.6% → 23.8%
Discarder of a winning tile
12.1% → 13.7%
Treatment behaviour
41 eligible decisions; 9 differed from the default AI. These count decisions, not hands.
Each pair uses the same deal seed, focal seat and round wind. The focal seat rotates through all four positions, facing three unchanged advanced opponents. Policies start fresh each hand. Taiwanese Club uses balanced East, South, West and North rounds; the other tests use East rounds.
The outcome is the focal player’s net settled points per hand, including automatic transfers and riichi deposits. These are independent hands: full-match placement, honba, future unawarded sticks and manual table payments are not modelled.
Sample sizes were fixed before the campaign: 256 paired deals per comparison, or 128 for Taiwanese Club. The approximate intervals use the spread of paired point differences. They are exploratory, use a normal approximation, and are not adjusted for the ten comparisons. Rare expensive hands can make small samples unstable.
Four runs started with an older test runner while the working tree was being reorganised. Those runs were excluded because their recorded source version was wrong, then rerun with the same planned seeds. The original records were retained; no result was dropped for being unfavourable.
