ELO Ratings Questions

Opublikowano: 2025-09-18

Key Argument Thesis: Using ELO for AI agent evaluation = measuring noiseProblem: Wrong evaluators, wrong metrics, wrong assumptions Solution: Quantitative assessment frameworksThe Comparison (00:00-02:00) Chess ELO FIDE arbiters: 120hr trainingBinary outcome: win/lossTest-retest: r=0.95Cohen's κ=0.92AI Agent ELO Random users: Google engineer? CS student? 10-year-old?Undefined dimensions: accuracy? style? speed?Test-retest: r=0.31 (coin flip)Cohen's κ=0.42Cognitive Bias Cascade (02:00-03:30) Anchoring: 34% rating variance in first 3 secondsConfirmation: 78% selective attention to preferred featuresDunning-Kruger: d=1.24 effect sizeResult: Circular preferences (A>B>C>A)The Quantitative Alternative (03:30-05:00) Objective Metrics McCabe co...