Ranked Probability Scores

import penaltyblog as pb

Examples

Examples below taken from Solving the problem of inadequate scoring rules for assessing probabilistic football forecast models

predictions = [
    [1, 0, 0],
    [0.9, 0.1, 0],
    [0.8, 0.1, 0.1],
    [0.5, 0.25, 0.25],
    [0.35, 0.3, 0.35],
    [0.6, 0.3, 0.1],
    [0.6, 0.25, 0.15],
    [0.6, 0.15, 0.25],
    [0.57, 0.33, 0.1],
    [0.6, 0.2, 0.2],
]

observed = [0, 0, 0, 0, 1, 1, 0, 0, 0, 0]

pb.metrics.rps_array(predictions, observed)

array([0.     , 0.005  , 0.025  , 0.15625, 0.1225 , 0.185  , 0.09125,
       0.11125, 0.09745, 0.1    ])

pb.metrics.rps_average(predictions, observed)

0.08937