Model Information¶
A discrimination study generates independent cohorts from each candidate
structure and lets a reader (a trained classifier, a Bayes-factor computation,
a stacking weight) give every structure a posterior probability for every
cohort. confusion_table reports how often a prespecified
verdict rule is right; model_information reports how much the reader's
posterior resolves the structure at all, as one number that can compare designs.
For any distribution q(m | y) that the reader reports,
I(M; Y) >= H(M) + E[log q(M | Y)]
because the difference is E_Y KL(p(M | Y) || q(M | Y)) >= 0. The estimate is
the stratified mean H(pi) + sum_m pi_m * mean log q_i(m) over the replicates of
each structure m, in nats (bound_bits divides by log 2), with a
normal-approximation interval and the score of each structure.
info = model_information(posterior, truth, labels=["A", "B"],
failed="prior")
info.bound, info.low, info.high # nats; at most info.prior_entropy
info.fraction_resolved # share of the prior entropy resolved
info.by_truth # which structure the reader recognises worst
Conditions that the caller establishes:
- Held out. The posteriors are for replicates the reader was not fitted on. On training replicates the bound is optimistic.
- Independent replicates drawn from the structures with the frequencies
prior(uniform by default); the number of replicates per structure may differ fromprior, and at least two are needed for every structure with positive prior. - Calibration is not required for validity: for any reader, calibrated or
not, the bound does not exceed
I(M; Y); miscalibration lowers it. A calibrated reader is what makes the bound close toI(M; Y). - Failures are never dropped: an all-NaN row raises unless
failed="prior", which replaces it by the prior (zero information). - Failed strata. If every replicate of a structure fails and
failed="prior"is used, that structure contributes no information and a standard error of zero; checkn_failedbeside the interval. - Floor. A probability for the true structure below
flooris raised to it;n_flooredcounts them and any is a warning, because a probability of zero would make the bound-inf. - The bound describes the simulated generating structures. It says nothing about the real data when none of the structures generated them.
Along a design size¶
A design study needs the bound as a function of a size (units per cohort,
sampling days, follow-up) and a rule that picks the smallest adequate size, as
power_curve and minimum_adequate do for success indicators.
information_curve reads raw replicate rows in which the generating structure
is a configuration factor and the reader's log probability of it on a held-out
replicate, log q(truth | y), is an observable; minimum_adequate_information
selects the smallest size that reaches a target in nats (or as a fraction of
H(M)).
curves = information_curve(rows, "log_score", truth="model", along="n_units",
labels=["A", "B"], adjustment="bonferroni")
minimum_adequate_information(curves, 0.5, fraction=True)
The conditions are those of model_information, per size: held-out
replicates, independent, drawn with the declared frequencies, at least two per
structure at every size. Scores must be finite and at most 0: a failed
replicate is scored by the caller (for example with the log of the prior
probability of its structure, zero information), never dropped. The probability
floor is the caller's responsibility because the score is computed outside the
library. As for power curves, the lower end of a marginal interval holds at the
selected size only; adjustment="bonferroni" makes the statements across the
scanned sizes simultaneous (approximately), and monotone_above reports a
curve that dips below the target after the selection. A selected size is a
statement about the simulated structures and the scanned grid; confirm it with
fresh replicates at the selected size.
trade_study.ModelInformation(labels, prior, n, n_failed, n_floored, floor, prior_entropy, bound, standard_error, low, high, confidence, fraction_resolved, by_truth)
dataclass
¶
Lower bound on I(M; Y) and its uncertainty, in nats.
Attributes:
| Name | Type | Description |
|---|---|---|
labels |
tuple[Hashable, ...]
|
Declared structures, in column order of the posterior. |
prior |
tuple[float, ...]
|
Generating frequency of each structure. |
n |
int
|
Total number of replicates. |
n_failed |
int
|
Replicates whose posterior was missing and was replaced by
the prior ( |
n_floored |
int
|
Replicates whose probability for the true structure was
below |
floor |
float
|
The probability floor used inside the logarithm. |
prior_entropy |
float
|
|
bound |
float
|
|
standard_error |
float
|
Standard error of |
low |
float
|
Lower end of the normal-approximation interval for |
high |
float
|
Upper end of that interval. |
confidence |
float
|
Two-sided level of the interval. |
fraction_resolved |
float
|
|
by_truth |
tuple[TruthScore, ...]
|
Per-structure scores, in declared order. |
bound_bits
property
¶
The bound in bits.
Returns:
| Type | Description |
|---|---|
float
|
|
trade_study.TruthScore(label, n, mean_log_probability, standard_error, mean_probability)
dataclass
¶
Held-out log score of the reader for the cohorts of one structure.
Attributes:
| Name | Type | Description |
|---|---|---|
label |
Hashable
|
The generating structure. |
n |
int
|
Number of replicates generated from |
mean_log_probability |
float
|
Mean of |
standard_error |
float
|
Standard error of that mean (sample standard
deviation over |
mean_probability |
float
|
Mean probability the reader gives the true
structure ( |
trade_study.model_information(posterior, truth, labels, *, prior=None, failed='raise', confidence=0.95, floor=1e-12)
¶
Bound I(M; Y) from below with held-out posterior model probabilities.
Row i of posterior is the reader's probability of every declared
structure for replicate i, whose generating structure is truth[i].
With pi the generating frequency of the structures and n_m the
number of replicates of structure m the estimator is the stratified
mean H(pi) + sum_m pi_m * mean_{i : truth_i = m} log q_i(m), so the
number of replicates per structure may differ from pi and only the
prior pi enters the target. Its standard error is
sqrt(sum_m pi_m^2 s_m^2 / n_m) and the interval is the normal
approximation.
The bound is valid for any reader, calibrated or not, provided the
posteriors are for replicates the reader was not fitted on; otherwise it
is optimistic. A miscalibrated reader lowers the bound; it cannot raise it
above I(M; Y). The replicates must be independent, and the generating
frequencies prior must be those used to draw them (and, for a trained
classifier, to train it). A failed replicate is never dropped: with
failed="prior" a missing (all-NaN) row is replaced by the prior, which
counts the failure as zero information. If every replicate of a structure
fails, its scores all equal log pi_m and its standard error is zero:
it contributes no information and no sampling uncertainty, so inspect
n_failed beside the interval.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
posterior
|
ArrayLike
|
Array of shape |
required |
truth
|
Sequence[Hashable]
|
Generating structure of each replicate. |
required |
labels
|
Sequence[Hashable]
|
Declared structures, in the column order of |
required |
prior
|
Sequence[float] | None
|
Generating frequency of each structure; uniform if |
None
|
failed
|
Literal['raise', 'prior']
|
|
'raise'
|
confidence
|
float
|
Two-sided level of the interval in (0, 1). |
0.95
|
floor
|
float
|
Probability floor inside the logarithm; positive. |
1e-12
|
Returns:
| Type | Description |
|---|---|
ModelInformation
|
The bound, its interval and the per-structure scores. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If there is no replicate, the shapes or labels disagree,
probabilities are invalid, a structure with positive prior has
fewer than two replicates, a replicate has a structure of zero
prior, |
Examples:
An uninformative reader (the prior) resolves nothing and a reader that is certain and right resolves the whole prior entropy.
>>> import math
>>> truth = ["A", "A", "B", "B"]
>>> flat = [[0.5, 0.5]] * 4
>>> model_information(flat, truth, ["A", "B"]).bound
0.0
>>> sure = [[1.0, 0.0], [1.0, 0.0], [0.0, 1.0], [0.0, 1.0]]
>>> info = model_information(sure, truth, ["A", "B"])
>>> math.isclose(info.bound, math.log(2))
True
>>> info.fraction_resolved
1.0
Source code in src/trade_study/information.py
243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 | |
trade_study.InformationPoint(value, n, bound, standard_error, low, high, fraction_resolved, by_truth)
dataclass
¶
The bound at one value of the size factor.
Attributes:
| Name | Type | Description |
|---|---|---|
value |
Any
|
Value of the size factor. |
n |
int
|
Number of replicates at this value, over all structures. |
bound |
float
|
Stratified lower bound on |
standard_error |
float
|
Standard error of |
low |
float
|
Lower end of the interval. |
high |
float
|
Upper end of the interval. |
fraction_resolved |
float
|
|
by_truth |
tuple[TruthScore, ...]
|
Per-structure scores ( |
trade_study.InformationCurve(log_score, truth, along, stratum, labels, prior, prior_entropy, values, points, confidence, family_confidence, adjustment)
dataclass
¶
Bounds on I(M; Y) for one stratum along an ordered size factor.
Attributes:
| Name | Type | Description |
|---|---|---|
log_score |
str
|
Name of the observable holding |
truth |
str
|
Name of the factor holding the generating structure. |
along |
str
|
Name of the ordered size factor. |
stratum |
dict[str, Any]
|
Values of every other configuration factor. |
labels |
tuple[Hashable, ...]
|
Declared structures. |
prior |
tuple[float, ...]
|
Generating frequency of each structure. |
prior_entropy |
float
|
|
values |
tuple[Any, ...]
|
Sorted values of |
points |
tuple[InformationPoint, ...]
|
Bound at each value, in |
confidence |
float
|
Two-sided level used for each interval. |
family_confidence |
float
|
Nominal simultaneous level across the values;
equals |
adjustment |
str | None
|
|
trade_study.AdequateInformation(stratum, along, value, point, target, fraction, rule, monotone_above)
dataclass
¶
Smallest size along an information curve that meets a target.
Attributes:
| Name | Type | Description |
|---|---|---|
stratum |
dict[str, Any]
|
Values of every other configuration factor. |
along |
str
|
Name of the ordered size factor. |
value |
Any
|
Selected size, or |
point |
InformationPoint | None
|
The curve point at the selected size, or |
target |
float
|
Required information, in nats or as a fraction of |
fraction |
bool
|
Whether |
rule |
str
|
|
monotone_above |
bool
|
Whether every larger scanned size also qualifies;
always |
trade_study.information_curve(results, log_score, truth, along, labels, *, prior=None, confidence=0.95, adjustment=None)
¶
Build per-stratum curves of the held-out bound I(M; Y) along a size.
Each row of results is one independent replicate (a cohort drawn from
the structure in factor truth at the size in factor along) whose
observable log_score is the reader's log probability of that
structure on that held-out cohort, log q(truth | y). Rows are grouped
by the remaining factors and by the size, and each group gives the
stratified bound of :func:model_information with its interval. The
conditions of that function apply: the reader was not fitted on these
replicates, the replicates are independent and drawn with the declared
frequencies prior (uniform by default), and every structure with
positive prior has at least two replicates at every size. A failed
replicate must be scored by the caller, for example with the log of the
prior probability of its structure (zero information); non-finite scores
are refused. The probability floor is the caller's responsibility, since
the score is computed outside this function. With
adjustment="bonferroni" each stratum's intervals use level
1 - (1 - confidence) / m for its m scanned sizes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
results
|
ResultsTable
|
Raw replicate rows (for example from |
required |
log_score
|
str
|
Observable holding |
required |
truth
|
str
|
Configuration factor holding the generating structure. |
required |
along
|
str
|
Configuration factor that orders the curve; values must be mutually orderable. |
required |
labels
|
Sequence[Hashable]
|
Declared structures. |
required |
prior
|
Sequence[float] | None
|
Generating frequency of each structure; uniform if |
None
|
confidence
|
float
|
Two-sided family level in (0, 1). |
0.95
|
adjustment
|
Literal['bonferroni'] | None
|
|
None
|
Returns:
| Type | Description |
|---|---|
list[InformationCurve]
|
One curve per combination of the remaining factors, in first-seen |
list[InformationCurve]
|
order. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a name is missing, scores are non-finite or positive,
values of |
Examples:
Two structures, a reader that is certain and right at the larger size only.
>>> import math
>>> import numpy as np
>>> from trade_study import ResultsTable
>>> rows = [(m, n) for n in (1, 2) for m in "AB" for _ in range(3)]
>>> table = ResultsTable(
... [{"model": m, "n": n} for m, n in rows],
... np.array([[math.log(0.5) if n == 1 else 0.0] for _, n in rows]),
... ["log_score"],
... )
>>> curve = information_curve(table, "log_score", "model", "n", ["A", "B"])[
... 0
... ]
>>> [round(p.bound, 6) for p in curve.points]
[0.0, 0.693147]
Source code in src/trade_study/information.py
493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 | |
trade_study.minimum_adequate_information(curves, target, *, rule='lower_bound', fraction=False)
¶
Select the smallest size on each curve whose information meets target.
"lower_bound" requires the interval's lower end to reach the target;
"estimate" uses the point estimate. A curve that dips below the
target above the selection is reported through monotone_above=False
rather than smoothed. Unless the curves were built with
adjustment="bonferroni", the lower end is marginal at the selected
size only, and a size chosen as the first of several scanned is subject
to selection.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
curves
|
list[InformationCurve]
|
Output of :func: |
required |
target
|
float
|
Required information in nats, or in |
required |
rule
|
Literal['lower_bound', 'estimate']
|
|
'lower_bound'
|
fraction
|
bool
|
Read |
False
|
Returns:
| Type | Description |
|---|---|
list[AdequateInformation]
|
One selection per curve, in the same order. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Examples:
>>> import math
>>> import numpy as np
>>> from trade_study import ResultsTable
>>> rows = [(m, n) for n in (1, 2) for m in "AB" for _ in range(3)]
>>> table = ResultsTable(
... [{"model": m, "n": n} for m, n in rows],
... np.array([[math.log(0.5) if n == 1 else 0.0] for _, n in rows]),
... ["log_score"],
... )
>>> curve = information_curve(table, "log_score", "model", "n", ["A", "B"])
>>> minimum_adequate_information(curve, 0.5)[0].value
2