{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ranked Probability Scores\n",
    "\n",
    "The Ranked Probability Score (RPS) is a metric used to evaluate the accuracy of probabilistic forecasts, particularly valuable for assessing football predictions due to its suitability for outcomes with an inherent order (such as win, draw, and loss). \n",
    "\n",
    "It quantifies how closely predicted probabilities align with actual results, rewarding forecasts that assign probabilities accurately across all possible outcomes. \n",
    "\n",
    "A lower RPS indicates more accurate and reliable predictions, making it an excellent metric for assessing and comparing the performance of football forecasting models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import penaltyblog as pb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `rps_average`\n",
    "\n",
    "The `rps_average` function takes one or more sets of probabilites and observed outcomes and calculates the average ranked probability scores across all the sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13833333333333334"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = [[0.8, 0.1, 0.1], [0.2, 0.1, 0.7], [0.1, 0.1, 0.8]]\n",
    "observed = [0, 2, 1]\n",
    "rps_score = pb.metrics.rps_average(predictions, observed)\n",
    "\n",
    "rps_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `rps_array`\n",
    "\n",
    "The `rps_array` function takes one or more sets of probabilites and observed outcomes and returns the individual ranked probability scores across all the sets.\n",
    "\n",
    "Examples below taken from [Solving the problem of inadequate scoring rules for assessing probabilistic football forecast models](http://constantinou.info/downloads/papers/solvingtheproblem.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.     , 0.005  , 0.025  , 0.15625, 0.1225 , 0.185  , 0.09125,\n",
       "       0.11125, 0.09745, 0.1    ])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = [\n",
    "    [1, 0, 0],\n",
    "    [0.9, 0.1, 0],\n",
    "    [0.8, 0.1, 0.1],\n",
    "    [0.5, 0.25, 0.25],\n",
    "    [0.35, 0.3, 0.35],\n",
    "    [0.6, 0.3, 0.1],\n",
    "    [0.6, 0.25, 0.15],\n",
    "    [0.6, 0.15, 0.25],\n",
    "    [0.57, 0.33, 0.1],\n",
    "    [0.6, 0.2, 0.2],\n",
    "]\n",
    "\n",
    "observed = [0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0]\n",
    "\n",
    "pb.metrics.rps_array(predictions, observed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}