Why individual variability is the signal, not the noise_

The standard way to analyse a prediction experiment is to pool every session, compute a hit rate, and test it against chance. It is a clean design and it answers a clean question — does the group as a whole beat chance? — but it is the wrong question if the effect you are looking for is not evenly distributed across people.

Across our first 10,000 blinded sessions, the pooled effect is small. Split by participant, it stops being uniform: a minority of contributors carry almost all of the deviation, and they do so consistently across months, target pools and task formats.

Calibration beats confidence

Self-reported confidence, taken at face value, is a weak predictor of accuracy. Calibrated per participant — mapping each person's confidence scale onto their own historical accuracy — it becomes one of the strongest features in the model.

That is a mundane statistical point with an interesting consequence: the useful unit of analysis is the individual, not the trial. A MetaModel that learns a personal response curve for every contributor extracts information that a pooled analysis destroys.

What we are changing

Every session now carries a participant-level calibration record. Group statistics are still published, unchanged and pre-registered, but they are no longer the primary read-out.

Null results remain null results. Nothing here suggests the pooled effect is larger than reported — only that the pooled effect was never the most informative thing in the dataset.