paper-to-game

Home Judges — design memo

This is a rebuild. The previous game from this paper, One of Us (an identification detective game), is in previous/ with its memo.

1. Paper

Fuchs-Schündeln, N. & Holub, F. Homophily in Grant Evaluations. Manuscript, 65 pages including Supplementary Information. WZB Berlin Social Science Center and Goethe University Frankfurt. The paper gives no DOI; the code and pseudo data are at doi:10.6084/m9.figshare.30789395 (not used). Available: the PDF only, already in this folder. The ERC data are confidential. Every number comes from the text, Tables 1–2, Figures 1–4 and Supplementary Tables 1–25.

2. What the paper hands over

Primary: an estimate. Two-way fixed-effects estimates (eq. 1) of how much one ERC panel reviewer’s score rises when reviewer and applicant share a trait. Project scores, Step 1 (Supp. Table 2): +0.164 when country of work and citizenship both match, +0.075 for citizenship only, +0.056 for country of work only. Same gender: +0.024 (Supp. Table 1). These are causal within the design: same proposal, same reviewer. Secondary: an exposure table. Table 2 gives the share of panel members who match an applicant, split by where the applicant works. For the five biggest countries (Germany, UK, France, Italy, Spain) the shares are 4.6% (both), 4.5% (country only) and 3.8% (citizenship only). For everyone else they are 1.5%, 1.8% and 2.6%. Panel-level funding associations (Table 1, Table 2 lower block) are correlational.

3. The discovery, and its kind

A mechanism made intelligible, with a counterintuitive corner. The micro rule is small: a judge who shares your passport and your workplace gives the same proposal about a sixth of a point more. Two to three times more than shared gender, which is what most people worry about. The macro pattern comes from where the judges come from. Applicants from big research systems meet such judges far more often (Table 2), so the same small tilt falls on the same side, again and again. The counterintuitive corner: a panel that looks fair because it mirrors who applies is exactly the panel that hands the biggest countries the biggest home bonus. The paper says this in its Discussion (“matches are unevenly distributed … systematic advantages for applicants from large or well-represented research systems”) and names more diverse panels as a lever.

4. The core narrative

A big science prize is judged by a table of fifteen experts from all over Europe. Every idea is read by about four of them. The researchers found that a judge gives a slightly better mark when the applicant comes from the judge’s own country and works there too. It is a small nudge, like an extra sixth of a point. But you get to choose who sits at the table. And the table that looks fairest, with lots of people from the countries that send the most ideas, turns out to give those countries the most nudges.

5. Who lives in the paper

6. Precedent

The last three memos (One of Us, Say It Plainly, The Original or the Copy), plus The Mixed Street, which stays in the table:

Game Position Format Feedback device Chassis
One of Us (this paper, replaced) experimenter discovery gap meter + up/down bars cream review form, system sans, orange accent
Say It Plainly agent (press officer) challenge, bets marker on a ruler cream press release, Georgia serif, blue/yellow/green
The Original or the Copy forecaster, then controller (two dials) bet + challenge before/after bars newsprint, system sans, red/blue
The Mixed Street experimenter challenge + discovery two-bar race cream paper, system sans, teal/ochre

What this game does differently.

7. The aim and the verdict

Aim (start screen, play screen, How to play): “Seat 15 judges. Give small countries as much home advantage as big ones.” Round 1 says “Seat the panel you think is fair”; the jars stay covered until the 15th seat is filled. Round 2: “Get the two jars within half a star of each other.” Four swaps. Yardstick: the two jars, in the aim’s units: home stars per 10 juries for big-five applicants and for everyone else. Round 2 succeeds when the jars are within half a star of each other. Verdict lines (end screen):

  1. (what you did) “Your own panel gave the big five about 2.6 times the home advantage of everyone else. Then you levelled the ERC-style panel in 3 swaps.” (numbers are the player’s)
  2. (what the paper says) “ERC judges give the same proposal about a sixth of a point more when they share the applicant’s passport and workplace. Sharing just one of the two is worth two to three times what shared gender is worth. Real ERC panels hand big-five applicants about two and a half times as many home stars.”
  3. (what that means) “One judge’s nudge is small. But on real panels it falls on the same side again and again. The study links it to a funding edge of about 1.7 percentage points for the big five against 0.8 for everyone else. That last part is a correlation, not a proven cause.”

8. Three concepts

A. The Seating Plan (picked). Position: controller (the ERC appointing a panel). Format: blind build, then challenge. Controls: tap a judge to seat or unseat them; in round 2, tap a seated judge, then a judge on the bench, to swap them (four swaps). Before pressing: “Who is this judge a home judge for? How many applicants is that? If I take her off, which jar drops?” After: the two jars fill or drain, with stars, at once. Discovery moment: the covers come off the jars after round 1. The player’s own fair-looking panel, mirroring Europe, leans heavily to the big five. Then in round 2 the player finds that seating a judge from a small country barely moves the small jar, because each small country sends few applicants. Taking a home judge from a big country off drains the big jar fast. Fun: anticipation (the covered jars), agency (every swap moves both jars), discovery (the mirror panel leans; passports travel: an Italian in London still fills the Italians’ jar), mastery (round 2 replayed takes fewer wrong swaps). Second play: a different first panel (diverse instead of mirrored) gives a different reveal. Round 2 has several solutions: removing big-country home judges, or seating judges who work abroad.

B. The Jury Draft (strategic; rejected). Position: agent (an applicant). The player and a rival applicant from Germany take turns drafting judges for their juries from a panel of 15. Both proposals are identical; marks decide who passes. Before pressing: grab my compatriot or deny the rival theirs? Moment: the rival has three compatriots to draft and you have none. Rejected for three reasons. At the paper’s size (0.164 against a mark spread of about 0.66 within a proposal) noise decides every duel, so the player cannot learn the bonus from outcomes. Showing expected marks would turn it into arithmetic. And drafting judges casts homophily as lobbying, which the paper does not claim; it does not say whether the bonus is deliberate.

C. The Sixth (reckless; rejected). Position: agent, and the agent is the bonus itself, a sixth of a point. Format: real time, no words on the play screen. Marks stream down towards the Step 1 line. The player is a small glowing wedge that can only ride on a compatriot’s mark, and it must land where it tips a proposal over the line. Moment: the wedge is almost useless anywhere except right at the line. A big-country applicant has three times as many compatriot marks to ride. Fun: timing, mastery, a gleeful villain role. Rejected because its central choice, where the bonus lands, is not a variable in the paper. The bonus lands wherever a match is, systematically, with no aim. Walkthrough: at 0:20 the player is darting between orange marks, at 1:00 they notice only marks near the line matter, and at 2:00 they feel the big country’s advantage in sheer supply. All of that agency is invented. What was given up: the bodily feeling that the bonus matters only at the line. Round 2’s jars say it in a calmer way, and the end screen says it in words.

9. The first five minutes

Pick (A).

Runner-up (B, the Jury Draft).

10. The pick, and why

A. The player builds the very variable the paper studies at panel level, the panel, and every action moves both jars, visibly, by an amount that follows from the paper’s estimates. The deliberation is real (“who is this judge a home judge for, and how many applicants is that?”). The discovery lands at minute 1:30 (the mirror panel leans) and again at 3:00 (small-country judges help few people; big-country home judges carry the most). Author test: the authors see their eq. 1 estimates and their Table 2 exposure gap turned into a table they can seat. The headline comparison is their Table 2, and the levers are the ones their Discussion proposes. Their correlational funding numbers appear only in the verdict, labelled. Runner-up: B. Switch if the authors want the game to be about one applicant’s fate rather than the institution; it would need a noise-free display, which costs it its tension. What the reckless concept would have given: the feeling that the bonus bites only at the line. The game says this in words instead.

11. The shape

Screens: start → rules card → round 1 (blind seating) → round 1 reveal → round 2 (level the ERC-like panel, four swaps, reset allowed) → round 2 result → end (verdict, then how the researchers know) → About. How to play is reachable from the start screen and from a ? on every play screen; returning lands where the player was. Presets, all from the paper: the bonus sizes (Supp. Tables 1–2); the ERC-like panel in round 2, calibrated to Table 2’s exposure shares; the big-five split is Table 2’s own grouping. Order: the player’s own panel comes first, so the reveal is about their choice, and the ERC comparison arrives inside the same reveal. The fix comes second, on the ERC-like panel, so everyone gets the same puzzle. Score: none across rounds. Round 2 says LEVEL or NOT YET, plus swaps used. Length: about four minutes. It is not padded.

12. Controls, mechanism, feedback, moment

13. The look

Register: a board game. It comes from the subject’s own picture, a round table of judges with name cards. A seating chart is the most board-game object in academic life, and a board game lets the panel be chunky tokens the player moves.

14. What is cut

Step 2 (similar sizes, Supp. Table 2); PI scores (smaller); gender from the play (it appears on the rules card only, because a gender bonus spread across both groups does not move the two jars, and the paper finds no robust gender effect at panel level); scheme and domain splits; the null same-rank result; chair effects; the tails test and the shared-taste controls (explained in words on the end screen as how the researchers know); and the gate itself (the Step 1 A/B/C decision is correlational and appears only as the paper’s funding numbers in the verdict).

15. Divergences


16. Mechanism kind and claim boundaries

Reconstructed, on a toy. The bonus sizes are the paper’s eq. 1 estimates. The ERC-like panel is reconstructed so that its exposure shares match Table 2. Everything that turns a panel into jars is a toy: random assignment of about 4.2 judges per proposal, every judge carrying the average bonus, a constructed applicant population.

17. Calibration plan

calibration.js, run against mechanism.js:

  1. The bonuses equal Supp. Tables 1–2; the star weights and the rules card’s words (“about half”, “about a third”, “about a seventh”; single matches 2–3× gender, p. 7) follow from them.
  2. The applicant population sums to 100 and puts 53.8% in the big five (Table 2 N). The ERC-like panel reproduces each of Table 2’s six exposure shares within ±1.2 p.p. Its stars are within ±0.4 of what Table 2 implies (3.31 and 1.39). Its ratio is in 1.8–3.4 (Table 2: 2.38). It starts not level.
  3. The paper’s own Table 2 arithmetic (APE × share = 1.7 and 0.8 p.p.), quoted in the verdict, reproduces.
  4. A Monte Carlo of random assignment (4 or 5 judges, mean 4.2) matches the analytic jar values on three seeds.
  5. Balance: a mirror first panel leans at least as much as the ERC; a diverse one leans less. No 1- or 2-swap plan levels round 2. Four-swap plans that unseat big-five home judges level more than half the time. Plans that unseat only judges without big-five ties never do. Random plans level under 2% of the time. The largest single swap moves the big jar more than 3× what any swap moves the small jar.
  6. Robustness: discoveries (1)–(2) and solvability under Step 2 bonuses, Step 1 PI bonuses, half and 1.5× applicant mobility, and exactly 4 judges per proposal.
  7. The correlational funding edge is never computed from the player’s panel.

18. Calibration result

node calibration.js → calibration-result.txt: ALL PASS, 35 checks. Highlights:

19. Build log

  1. Mechanism first. The applicant population and the ERC-like panel were tuned by hand until Table 2’s six shares and the implied stars were close. A single 15-seat panel cannot hit all six shares; the misses are listed above.
  2. Loop (loop.js, loop-log.txt). The naive move (swap a small-country judge for another) visibly does nothing; unseating a big home judge visibly does a lot. The first balance check showed three swaps were too few (2% of sensible plans succeed), so the budget went to four.
  3. Skin. Indigo board, two-ring tokens, Bungee and Rubik embedded as subset woff2 (43 KB together, OFL licences in fonts/). The first screenshots showed the jars scrolling out of view while the player tapped the bench, so the jars became a sticky strip at the top of both rounds. The ERC label was clipped and moved; the US judge lost a misleading star; the “teal jar barely moves” hint now appears only after a swap that moved neither jar much.
  4. Self-review from the screenshots. Aim from the start screen alone: “seat a panel that gives small countries as much home advantage as big ones.” Discovery from the top of the end screen alone: “judges give their own a sixth of a point; big countries have more judges, so they collect more of it.” Does it look like an earlier game? No: every earlier game is cream paper with a system font and a chart; this is a dark board game with tokens and jars.
  5. Browser session (session-test.js, session-log.txt, screenshot-*.png) at 380 px. A first-timer mirrors who applies, opens How to play mid-round and returns with seven judges still seated, sees the reveal (3½× vs the ERC’s 2½×), tries a naive swap, resets, levels in three swaps and reads the verdict and About. A replay seats a diverse panel (1½×), presses DONE without levelling and gets NOT YET. No console errors and no horizontal scroll on any screen. The session length, about four minutes, is an estimate; no person has timed it.

20. Revision 1: Felix Holub’s feedback

Felix, one of the two co-authors, sent four changes. All four are in.

21. Iteration 2: tokens, the authors’ data, a fixed phone screen

This iteration replaces parts of §§11–19 and §20; where they disagree, this section holds.