Confusion Tables

A replicated recovery or discrimination study generates independent cohorts from each candidate structure and reads every cohort with a prespecified rule, for example "favour A", "favour B", "indistinguishable" or "failed". confusion_table tabulates the verdicts by generating structure and gives a fixed-sample proportion interval for every cell, so the probability of the right verdict is reported beside the wrong-structure and no-verdict rates.

table = confusion_table(truth, verdict, rows=["A", "B"],
                        columns=["A", "B", "none", "failed"],
                        adjustment="bonferroni")
table.cell("A", "A")   # probability of favouring A when A generated the cohort
table.row_total("A")   # cohorts generated from A, failed trials included

Every cohort is in a declared row and column: undeclared labels raise, a declared row without a cohort raises, and a failed trial is a column of its own, never a dropped cohort. The intervals assume independent cohorts and a prespecified number per row; the caller establishes both. Without adjustment each interval is marginal; adjustment="bonferroni" uses level 1 - (1 - confidence) / len(columns) so that the statements about the columns of one row hold simultaneously at approximately the nominal level (Wilson coverage is approximate, and different rows are separate statements that are not adjusted). Labels that compare equal (1, True, 1.0) are one label. power_curve reads one 0/1 indicator per design; the confusion table reports all verdicts of a design together.

trade_study.ConfusionTable(rows, columns, counts, intervals, confidence, family_confidence, adjustment, method) dataclass

Counts and row-wise proportion intervals of verdicts by truth.

Attributes:

Name	Type	Description
rows	tuple[Hashable, ...]	Row labels (the truth), in the order declared by the caller.
columns	tuple[Hashable, ...]	Column labels (the verdicts), in declared order.
counts	tuple[tuple[int, ...], ...]	counts[i][j] is the number of cohorts of row i with verdict j.
intervals	tuple[tuple[ProportionInterval, ...], ...]	intervals[i][j] is the fixed-sample interval for the probability of verdict j given row i; its n is the row total.
confidence	float	Two-sided level of each reported interval (after any adjustment).
family_confidence	float	Nominal simultaneous level across the columns of a row (approximate for Wilson intervals); equals confidence when adjustment is None.
adjustment	str | None	None (marginal intervals) or "bonferroni".
method	str	Interval method passed to proportion_interval.

cell(row, column)

Interval for the probability of column given row.

Returns:

Type	Description
ProportionInterval	The fixed-sample interval of that cell.

Source code in src/trade_study/confusion.py

def cell(self, row: Hashable, column: Hashable) -> ProportionInterval:
    """Interval for the probability of ``column`` given ``row``.

    Returns:
        The fixed-sample interval of that cell.
    """
    return self.intervals[_index(self.rows, row, "row")][
        _index(self.columns, column, "column")
    ]

row_total(row)

Number of cohorts of row, including failed trials.

Returns:

Type	Description
int	The row total.

Source code in src/trade_study/confusion.py

def row_total(self, row: Hashable) -> int:
    """Number of cohorts of ``row``, including failed trials.

    Returns:
        The row total.
    """
    return sum(self.counts[_index(self.rows, row, "row")])

trade_study.confusion_table(truth, verdict, rows, columns, *, confidence=0.95, method='wilson', adjustment=None)

Tabulate one verdict per independent cohort against its truth.

Each position of truth and verdict is one independent cohort. Every label must be declared in rows or columns and every declared row must have at least one cohort, so no cohort is dropped and a failed trial is a column of its own. Each cell interval is a fixed-sample interval for independent cohorts of a prespecified number per row; the caller establishes the independence and the number (see proportion_interval). With adjustment="bonferroni" the intervals of a row use level 1 - (1 - confidence) / len(columns), so that the statements about the columns of one row hold simultaneously at approximately the nominal level (Wilson coverage is approximate; the rows are separate statements and are not adjusted). Labels that compare equal (1, True and 1.0) are the same label.

Parameters:

Name	Type	Description	Default
truth	Sequence[Hashable]	Generating label of each cohort.	required
verdict	Sequence[Hashable]	Verdict label of each cohort, aligned with truth.	required
rows	Sequence[Hashable]	Declared truth labels.	required
columns	Sequence[Hashable]	Declared verdict labels, including a failure label if the study can fail.	required
confidence	float	Two-sided family level in (0, 1).	0.95
method	str	"wilson" or "clopper-pearson".	'wilson'
adjustment	Literal['bonferroni'] | None	None for marginal intervals or "bonferroni".	None

Returns:

Type	Description
ConfusionTable	The counts and the interval of every cell.

Raises:

Type	Description
ValueError	If lengths differ, labels are repeated or undeclared, a declared row has no cohort, or adjustment is unsupported.

Examples:

>>> table = confusion_table(
...     ["A", "A", "A", "B"],
...     ["A", "A", "none", "B"],
...     rows=["A", "B"],
...     columns=["A", "B", "none"],
... )
>>> table.counts
((2, 0, 1), (0, 1, 0))
>>> table.cell("A", "A").estimate
0.6666666666666666

Source code in src/trade_study/confusion.py

def confusion_table(
    truth: Sequence[Hashable],
    verdict: Sequence[Hashable],
    rows: Sequence[Hashable],
    columns: Sequence[Hashable],
    *,
    confidence: float = 0.95,
    method: str = "wilson",
    adjustment: Literal["bonferroni"] | None = None,
) -> ConfusionTable:
    """Tabulate one verdict per independent cohort against its truth.

    Each position of ``truth`` and ``verdict`` is one independent cohort.
    Every label must be declared in ``rows`` or ``columns`` and every declared
    row must have at least one cohort, so no cohort is dropped and a failed
    trial is a column of its own. Each cell interval is a fixed-sample
    interval for independent cohorts of a prespecified number per row; the
    caller establishes the independence and the number (see
    ``proportion_interval``). With ``adjustment="bonferroni"`` the intervals of
    a row use level ``1 - (1 - confidence) / len(columns)``, so that the
    statements about the columns of one row hold simultaneously at
    approximately the nominal level (Wilson coverage is approximate; the
    rows are separate statements and are not adjusted). Labels that compare
    equal (``1``, ``True`` and ``1.0``) are the same label.

    Args:
        truth: Generating label of each cohort.
        verdict: Verdict label of each cohort, aligned with ``truth``.
        rows: Declared truth labels.
        columns: Declared verdict labels, including a failure label if the
            study can fail.
        confidence: Two-sided family level in (0, 1).
        method: ``"wilson"`` or ``"clopper-pearson"``.
        adjustment: ``None`` for marginal intervals or ``"bonferroni"``.

    Returns:
        The counts and the interval of every cell.

    Raises:
        ValueError: If lengths differ, labels are repeated or undeclared, a
            declared row has no cohort, or ``adjustment`` is unsupported.

    Examples:
        >>> table = confusion_table(
        ...     ["A", "A", "A", "B"],
        ...     ["A", "A", "none", "B"],
        ...     rows=["A", "B"],
        ...     columns=["A", "B", "none"],
        ... )
        >>> table.counts
        ((2, 0, 1), (0, 1, 0))
        >>> table.cell("A", "A").estimate
        0.6666666666666666
    """
    if adjustment not in _ADJUSTMENTS:
        msg = f"adjustment must be one of {_ADJUSTMENTS}"
        raise ValueError(msg)
    if len(truth) != len(verdict):
        msg = "truth and verdict must have the same length"
        raise ValueError(msg)
    row_labels = _unique(rows, "rows")
    column_labels = _unique(columns, "columns")
    try:
        undeclared = (set(truth) - set(row_labels)) | (
            set(verdict) - set(column_labels)
        )
    except TypeError as error:
        msg = "truth and verdict must contain hashable labels"
        raise ValueError(msg) from error
    if undeclared:
        msg = f"labels not declared in rows or columns: {sorted(map(repr, undeclared))}"
        raise ValueError(msg)
    row_at = {label: i for i, label in enumerate(row_labels)}
    column_at = {label: j for j, label in enumerate(column_labels)}
    tally = [[0] * len(column_labels) for _ in row_labels]
    for t, v in zip(truth, verdict, strict=True):
        tally[row_at[t]][column_at[v]] += 1
    empty = [
        row for row, counts in zip(row_labels, tally, strict=True) if not sum(counts)
    ]
    if empty:
        msg = f"declared rows without a cohort: {sorted(map(repr, empty))}"
        raise ValueError(msg)
    level = (
        1 - (1 - confidence) / len(column_labels)
        if adjustment == "bonferroni"
        else confidence
    )
    intervals = tuple(
        tuple(
            proportion_interval(k, sum(counts), confidence=level, method=method)
            for k in counts
        )
        for counts in tally
    )
    return ConfusionTable(
        rows=row_labels,
        columns=column_labels,
        counts=tuple(tuple(counts) for counts in tally),
        intervals=intervals,
        confidence=float(level),
        family_confidence=float(confidence),
        adjustment=adjustment,
        method=method,
    )