Scoring¶

Proper scoring rules for posterior predictive evaluation.

CRPS and coverage treat each row as a univariate case: predictions are (*cases, members) and truth is (*cases,). Energy instead treats each case as a multivariate vector: predictions are (*cases, members, variables) and truth is (*cases, variables). A single jointly scored vector therefore uses a (members, variables) ensemble. The Bayesian tutorial stores predictions as (test_location, member) for marginal CRPS/coverage and transposes them for joint energy scoring. RMSE/MAE compare aligned point predictions with truth.

`trade_study.score(metric, predictions, truth, *, alpha=None, level=0.95)` ¶

Compute a scalar scoring rule.

Parameters:

Name	Type	Description	Default
`metric`	`str`	One of "crps", "wis", "interval", "energy", "rmse", "mae", "coverage", "brier".	required
`predictions`	`NDArray[floating[Any]]`	For CRPS/coverage, `(cases, members)` with truth `(cases,)`. For energy, `(cases, members, variables)` with truth `(cases, variables)`. RMSE/MAE use arrays aligned with truth; interval uses a trailing lower/upper pair. WIS uses trailing interval-level and lower/median/upper dimensions.	required
`truth`	`NDArray[floating[Any]]`	Known ground truth values.	required
`alpha`	`float \| NDArray[floating[Any]] \| None`	Significance level for interval-based scores.	`None`
`level`	`float`	Nominal coverage level for coverage metric.	`0.95`

Returns:

Type	Description
`float`	Scalar score value.

Raises:

Type	Description
`ValueError`	If the metric name is not recognized.

Examples:

>>> predictions = np.array([1.0, 3.0])
>>> score("rmse", predictions, np.array([0.0, 2.0]))
1.0
>>> members = np.array([[0.0, 0.0], [2.0, 0.0]])
>>> score("energy", members, np.zeros(2))
0.5

Source code in src/trade_study/_scoring.py

def score(
    metric: str,
    predictions: NDArray[np.floating[Any]],
    truth: NDArray[np.floating[Any]],
    *,
    alpha: float | NDArray[np.floating[Any]] | None = None,
    level: float = 0.95,
) -> float:
    """Compute a scalar scoring rule.

    Args:
        metric: One of "crps", "wis", "interval", "energy",
            "rmse", "mae", "coverage", "brier".
        predictions: For CRPS/coverage, ``(*cases, members)`` with truth
            ``(*cases,)``. For energy, ``(*cases, members, variables)`` with
            truth ``(*cases, variables)``. RMSE/MAE use arrays aligned with
            truth; interval uses a trailing lower/upper pair. WIS uses
            trailing interval-level and lower/median/upper dimensions.
        truth: Known ground truth values.
        alpha: Significance level for interval-based scores.
        level: Nominal coverage level for coverage metric.

    Returns:
        Scalar score value.

    Raises:
        ValueError: If the metric name is not recognized.

    Examples:
        >>> predictions = np.array([1.0, 3.0])
        >>> score("rmse", predictions, np.array([0.0, 2.0]))
        1.0
        >>> members = np.array([[0.0, 0.0], [2.0, 0.0]])
        >>> score("energy", members, np.zeros(2))
        0.5
    """
    simple = {
        "crps": _crps,
        "energy": _energy,
        "brier": _brier,
        "rmse": _rmse,
        "mae": _mae,
    }
    if metric in simple:
        return simple[metric](predictions, truth)
    if metric == "wis":
        return _wis(predictions, truth, alpha=alpha)
    if metric == "interval":
        return _interval(predictions, truth, alpha=alpha)
    if metric == "coverage":
        return _coverage(predictions, truth, level=level)
    msg = f"Unknown metric: {metric!r}"
    raise ValueError(msg)

`trade_study.coverage_curve(posteriors, truth, levels=None)` ¶

Compute empirical coverage across nominal levels.