Runner

Execute simulations across experimental grids.

For expensive grids, pass checkpoint_path="run.sqlite" and a caller-managed checkpoint_key="model-v1-data-v3". Successful evaluations are committed one at a time, including by parallel workers. Resume with the same ordered grid, replicate count, observable/annotation definitions, and revision. Completed tasks are loaded; only unfinished tasks are evaluated. Configurations with identical values at different grid positions remain separate design points. Callbacks run for all rows, including recovered rows, in original task order.

max_retries=2 permits two additional attempts for each unfinished task during that invocation. The default is zero and propagates the first exception. Simulator and scorer exceptions are retried with the same configuration and replicate id. Interrupts are not retried. Metadata records total attempts across invocations and marks recovered rows with recovered=True.

Checkpoint identity checks inspectable class code and structural definitions. Change the revision when instance settings, external data, or randomness semantics change. Use a new path for an incompatible definition. Run one grid invocation at a time against a ledger; its workers may run in parallel. A crash between an external side effect and saving its result may require repeating that evaluation. Make side effects safe to repeat. Retry limits do not bound the number of explicit resume invocations.

Reusing identified evaluations

Reuse is opt-in through an explicit cache:

cache = EvaluationCache(
    "evaluations.sqlite",
    revision="simulator-v2-scorer-v1-data-v3-annotations-v1",
    replicate_namespace="experiment-2026-seed-17",
    fidelity="high",
)
results = run_grid(world, scorer, grid, observables, n_reps=3, cache=cache)

The key includes a typed canonical configuration, replicate id, caller revision, replicate namespace, fidelity, inspectable simulator/scorer class code, objective definitions and annotation semantics. Grid position and total replicate count are excluded: reordering designs or requesting more replicates can reuse identical existing evaluations. Raw result rows still retain the current grid's design-point ids. Different replicate ids and independent experiment namespaces remain distinct even when their configurations match.

The caller must update the revision when instance settings, external data, globals or opaque callable behavior changes. The replicate namespace must identify the actual randomness/seed convention. Use a fresh namespace for an independent experiment; matching configuration alone never establishes reuse. Supported scalar/container configurations preserve types; opaque values are refused. Cached evaluators must not mutate their configuration.

cache_bypass=True skips reads and writes, leaving stored evidence untouched. cache.clear() invalidates all contexts in the cache file. Changing identity creates a new context. Conflicting scores for one identity are refused. Metadata exposes cache_hit, cache_key, and the revision/namespace/fidelity context. Cached wall time is the original evaluation's time. Completed grid checkpoints can populate the cache without re-evaluation; bypassing the cache does not bypass checkpoints. Concurrent workers may evaluate an absent identity more than once before either saves; reuse does not promise exactly-once effects.

trade_study.run_grid(world, scorer, grid, observables, *, annotations=None, n_jobs=1, n_reps=1, callback=None, checkpoint_path=None, checkpoint_key=None, max_retries=0, cache=None, cache_bypass=False)

Run all configurations in a grid.

Parameters:

Name	Type	Description	Default
world	Simulator	Simulator that generates (truth, observations).	required
scorer	Scorer	Scorer that evaluates observables.	required
grid	list[dict[str, Any]]	List of config dicts to evaluate.	required
observables	list[Observable]	Observable definitions (for column ordering).	required
annotations	list[Annotation] | None	Optional external annotations (costs, etc.).	None
n_jobs	int	Number of parallel workers (-1 for all CPUs).	1
n_reps	int	Number of times to evaluate each design point. When greater than 1 and world.generate accepts a rep keyword argument, each replicate is called with its 0-indexed rep so the simulator can vary its own randomness; a simulator without a rep parameter is called unchanged, which reproduces its n_reps=1 behavior on every replicate. The returned table has n_reps * len(grid) rows; each row's metadata carries design_point (index into grid) and rep so replicates can be grouped, e.g. via :meth:~trade_study.protocols.ResultsTable.aggregate_replicates.	1
callback	ProgressCallback | None	Optional progress callback invoked after each trial with (trial_index, total_trials, trial_result), where total_trials is n_reps * len(grid). Recovered rows also invoke it, in original task order.	None
checkpoint_path	str | Path | None	Optional SQLite ledger. Each completed evaluation is committed before callbacks and retained across interruptions. Reopening skips completed tasks with the same definition.	None
checkpoint_key	str | None	Caller-managed model/data revision. Change this when instance state, external data, or other behavior changes; class source alone cannot detect those changes.	None
max_retries	int	Additional attempts per unfinished task per invocation. Zero (default) propagates the first evaluation exception. Retries retain the same config and replicate id.	0
cache	EvaluationCache | None	Opt-in revision/replicate/fidelity/schema-aware evaluation reuse. Independent experiments must use different replicate namespaces.	None
cache_bypass	bool	Skip cache reads and writes. Completed checkpoint tasks still resume; use a new checkpoint path to re-evaluate them.	False

Returns:

Type	Description
ResultsTable	ResultsTable with scored results.

Raises:

Type	Description
ValueError	If n_reps is not positive, max_retries is negative, or a checkpoint definition is incompatible or unverifiable.

Source code in src/trade_study/runner.py

def run_grid(  # ruff: ignore[too-many-arguments]
    world: Simulator,
    scorer: Scorer,
    grid: list[dict[str, Any]],
    observables: list[Observable],
    *,
    annotations: list[Annotation] | None = None,
    n_jobs: int = 1,
    n_reps: int = 1,
    callback: ProgressCallback | None = None,
    checkpoint_path: str | Path | None = None,
    checkpoint_key: str | None = None,
    max_retries: int = 0,
    cache: EvaluationCache | None = None,
    cache_bypass: bool = False,
) -> ResultsTable:
    """Run all configurations in a grid.

    Args:
        world: Simulator that generates (truth, observations).
        scorer: Scorer that evaluates observables.
        grid: List of config dicts to evaluate.
        observables: Observable definitions (for column ordering).
        annotations: Optional external annotations (costs, etc.).
        n_jobs: Number of parallel workers (-1 for all CPUs).
        n_reps: Number of times to evaluate each design point.
            When greater than 1 and ``world.generate`` accepts a ``rep``
            keyword argument, each replicate is called with its 0-indexed
            ``rep`` so the simulator can vary its own randomness; a
            simulator without a ``rep`` parameter is called unchanged,
            which reproduces its ``n_reps=1`` behavior on every replicate.
            The returned table has ``n_reps * len(grid)`` rows; each row's
            metadata carries ``design_point`` (index into ``grid``) and
            ``rep`` so replicates can be grouped, e.g. via
            :meth:`~trade_study.protocols.ResultsTable.aggregate_replicates`.
        callback: Optional progress callback invoked after each trial
            with ``(trial_index, total_trials, trial_result)``, where
            ``total_trials`` is ``n_reps * len(grid)``.
            Recovered rows also invoke it, in original task order.
        checkpoint_path: Optional SQLite ledger. Each completed evaluation
            is committed before callbacks and retained across interruptions.
            Reopening skips completed tasks with the same definition.
        checkpoint_key: Caller-managed model/data revision. Change this when
            instance state, external data, or other behavior changes; class
            source alone cannot detect those changes.
        max_retries: Additional attempts per unfinished task per invocation.
            Zero (default) propagates the first evaluation exception.
            Retries retain the same config and replicate id.
        cache: Opt-in revision/replicate/fidelity/schema-aware evaluation reuse.
            Independent experiments must use different replicate namespaces.
        cache_bypass: Skip cache reads and writes. Completed checkpoint tasks
            still resume; use a new checkpoint path to re-evaluate them.

    Returns:
        ResultsTable with scored results.

    Raises:
        ValueError: If ``n_reps`` is not positive, ``max_retries`` is negative,
            or a checkpoint definition is incompatible or unverifiable.
    """
    if n_reps < 1:
        msg = f"run_grid: n_reps must be positive, got {n_reps}"
        raise ValueError(msg)
    if max_retries < 0:
        msg = "max_retries must be >= 0"
        raise ValueError(msg)
    ledger = (
        _GridLedger(
            checkpoint_path,
            _grid_identity(
                world, scorer, grid, observables, annotations, n_reps, checkpoint_key
            ),
        )
        if checkpoint_path is not None
        else None
    )

    supports_rep = _generate_accepts_rep(world)
    bound_cache = (
        _bind_cache(cache, world, scorer, observables, annotations)
        if cache is not None and not cache_bypass
        else None
    )
    tasks = [
        (design_point, cfg, rep)
        for design_point, cfg in enumerate(grid)
        for rep in range(n_reps)
    ]
    total = len(tasks)

    if n_jobs == 1:
        results: list[TrialResult] = []
        for i, task in enumerate(tasks):
            r = _run_grid_task(
                world,
                scorer,
                task,
                supports_rep=supports_rep,
                ledger=ledger,
                max_retries=max_retries,
                cache=bound_cache,
            )
            results.append(r)
            if callback is not None:
                callback(i, total, r)
    else:
        from joblib import Parallel, delayed  # type: ignore[import-untyped]

        results = Parallel(n_jobs=n_jobs)(
            delayed(_run_grid_task)(
                world,
                scorer,
                task,
                supports_rep=supports_rep,
                ledger=ledger,
                max_retries=max_retries,
                cache=bound_cache,
            )
            for task in tasks
        )
        if callback is not None:
            for i, r in enumerate(results):
                callback(i, total, r)

    obs_names = [o.name for o in observables]
    score_matrix = np.array(
        [[r.scores.get(name, np.nan) for name in obs_names] for r in results],
        dtype=float,
    ).reshape(len(results), len(obs_names))

    ann_matrix = None
    ann_names: list[str] = []
    if annotations:
        ann_names = [a.name for a in annotations]
        ann_matrix = np.array(
            [[a.resolve(r.config) for a in annotations] for r in results],
            dtype=float,
        ).reshape(len(results), len(ann_names))

    return ResultsTable(
        configs=[r.config for r in results],
        scores=score_matrix,
        observable_names=obs_names,
        annotations=ann_matrix,
        annotation_names=ann_names,
        metadata=[{"wall_seconds": r.wall_seconds, **r.metadata} for r in results],
    )

trade_study.run_adaptive(world, scorer, factors, observables, *, n_trials=100, n_reps=1, seed=42)

Run adaptive multi-objective optimization via optuna.

Parameters:

Name	Type	Description	Default
world	Simulator	Simulator.	required
scorer	Scorer	Scorer for observables.	required
factors	list[Factor]	Factor definitions (from design module).	required
observables	list[Observable]	Observable definitions.	required
n_trials	int	Number of optuna trials.	100
n_reps	int	Replicate draws averaged into each trial's objective values before Optuna sees them, the same convention run_grid(..., n_reps>1) uses -- a simulator opts in via an optional keyword-only rep parameter on generate, detected via introspection. Without it, NSGA-II can select a "best" config that's just a lucky draw for one data realization rather than one that's robust across draws. Simulators that don't accept rep see identical draws repeated n_reps times, which only adds redundant compute. Default 1 (a single draw per trial, today's behavior).	1
seed	int	Random seed.	42

Returns:

Type	Description
ResultsTable	ResultsTable with scored results, one row per optuna trial (each
ResultsTable	row the weighted mean of n_reps replicate draws). Metadata
ResultsTable	retains trial ids, raw means, standard errors and replicate counts.

Raises:

Type	Description
ValueError	If n_reps is less than 1.

Source code in src/trade_study/runner.py

def run_adaptive(
    world: Simulator,
    scorer: Scorer,
    factors: list[Factor],
    observables: list[Observable],
    *,
    n_trials: int = 100,
    n_reps: int = 1,
    seed: int = 42,
) -> ResultsTable:
    """Run adaptive multi-objective optimization via optuna.

    Args:
        world: Simulator.
        scorer: Scorer for observables.
        factors: Factor definitions (from design module).
        observables: Observable definitions.
        n_trials: Number of optuna trials.
        n_reps: Replicate draws averaged into each trial's objective
            values before Optuna sees them, the same convention
            ``run_grid(..., n_reps>1)`` uses -- a simulator opts in
            via an optional keyword-only ``rep`` parameter on
            ``generate``, detected via introspection. Without it, NSGA-II
            can select a "best" config that's just a lucky draw for one
            data realization rather than one that's robust across draws.
            Simulators that don't accept ``rep`` see identical draws
            repeated ``n_reps`` times, which only adds redundant compute.
            Default 1 (a single draw per trial, today's behavior).
        seed: Random seed.

    Returns:
        ResultsTable with scored results, one row per optuna trial (each
        row the weighted mean of ``n_reps`` replicate draws). Metadata
        retains trial ids, raw means, standard errors and replicate counts.

    Raises:
        ValueError: If ``n_reps`` is less than 1.
    """
    import optuna as _optuna

    from .session import AdaptiveSession

    if n_reps < 1:
        msg = f"n_reps must be >= 1; got {n_reps}"
        raise ValueError(msg)

    supports_rep = _generate_accepts_rep(world)
    session = AdaptiveSession(factors, observables, seed=seed)
    _optuna.logging.set_verbosity(_optuna.logging.WARNING)
    for _ in range(n_trials):
        ((trial_id, config),) = session.ask(1)
        rep_scores: list[dict[str, float]] = []
        for rep in range(n_reps):
            if supports_rep:
                truth, observations = world.generate(config, rep=rep)  # type: ignore[call-arg]
            else:
                truth, observations = world.generate(config)
            rep_scores.append(scorer.score(truth, observations, config))
        session.tell(
            trial_id,
            {
                o.name: [s.get(o.name, float("nan")) for s in rep_scores]
                for o in observables
            },
        )

    return session.results()

Sequential replication

run_sequential(..., policy=ReplicationPolicy(...)) starts every design with min_reps, then allocates additional replicates among unresolved designs, preferring the least sampled design with grid-order tie breaking. It retires confidently infeasible or dominated designs and stops when it resolves a single choice, a Pareto trade-off, or practical equivalence. Otherwise it reports max_reps or budget_exhausted; a budget stop does not establish a winner.

Specify known almost-sure score_bounds for every observable, min_reps, max_reps, max_evaluations, and joint confidence. Optional equivalence tolerances use raw observable units. Directions guide comparisons; observable weights do not transform scores. Monotone constraints use the simultaneous intervals and policy confidence instead of normal standard-error bounds.

For D designs, M observables, horizon H=max_reps, and alpha=1-confidence, the interval radius after n replicates is (upper_bound - lower_bound) * sqrt(log(2*D*M*H/alpha)/(2*n)), clipped to the declared bounds. Hoeffding's inequality bounds each failure by alpha/(DMH); a union bound gives simultaneous coverage over every design, observable, and allowed sample size. This permits data-dependent allocation and stopping under independent replicates with a fixed mean within each design. Common random numbers may correlate designs; the within-design replicate sequences must still satisfy those assumptions. Bounds must be known before inspecting data, not estimated from the observed minimum and maximum. Out-of-bounds/non-finite scores are errors. The bounds are conservative and may spend the entire budget on small gaps, even for a deterministic simulator with broad score bounds. For context on time-uniform inference and sharper approaches, see Kuchibhotla and Zheng (2021).

SequentialResult.results preserves raw scores and design-point/replicate ids; summary contains design means and exportable allocation/stopping metadata. Use intervals and feasibility for the sequential decision. Ordinary ResultsTable.feasible() normal bounds and post-selection paired bootstrap/t intervals do not inherit sequential coverage. Allocations may have unmatched tail replicates; explicitly select a common prefix for descriptive paired comparisons. Existing run_grid(n_reps=...) behavior is unchanged.

Fixed plans and recorded policy

Set ReplicationPolicy(..., allocation="fixed") to collect exactly max_reps for every design, even when an early winner or infeasible design is apparent. The total budget must cover the entire plan before any simulator call. Feasibility is assessed only after completion, stopping reasons are fixed_plan_complete, and no winner is automatically selected. This mode uses the same bounded-mean Hoeffding method with one final look: replace H by 1 in the radius above. It does not repeatedly inspect fixed-sample Wilson/Clopper–Pearson intervals. allocation="adaptive" preserves the existing default behavior.

The runner snapshots bounds/tolerances and the outer candidate/observable lists before evaluation; caller-owned mapping changes cannot narrow the used bounds after results are seen. Configuration contents, scoring definitions, means and scientific units must still remain fixed. Recorded assumptions are declarations, not verification of the simulator or authentication of a saved artifact.

Each summary row retains the existing interval/stopping fields and adds a primitive-JSON replication_contract with schema_version=1:

Recorded field	Interpretation
Allocation, bounds, objectives, constraints, confidence	The actual used policy; constraint requests use the joint policy intervals.
Planned minimum, exact fixed-plan count or null, maximum and actual replicates	Counts of simulator/scorer evaluations, not individual observations inside one evaluation.
Total budget and actual evaluations	All completed evaluations across the declared design family.
Look horizon, multiplicity and per-look error	The prespecified inference family; adaptive mode includes even unvisited sample counts through the horizon.
Interval half-widths	Observed interval precision in raw score units, not gate power.
Gate power / gate false-failure rate	Explicitly not_assessed, with null values. Interval coverage does not estimate either quantity.
Decision status	unresolved for an exhausted adaptive budget/horizon, resolved for the existing retirement rules, or not_assessed for a fixed plan. This does not certify a statistical gate.
Failure policy / evaluability	abort; available completed runs have zero failed replicates. A failed run returns no successful summary.

replicate_unit="independent_cluster" can declare one independent cluster score per simulator/scorer evaluation. Members may be dependent inside that cluster, but they never inflate the replicate count. The estimand is the mean declared replicate/cluster score; it is not automatically an individual-weighted population mean. General serial or across-cluster dependence remains unsupported. The default is independent_replicate. Cross-design common random numbers remain allowed because the union bound needs no independence across designs. Neither mode supplies a paired-contrast interval; ordinary paired post-selection inference does not acquire sequential coverage from this metadata.

Missing, non-finite or out-of-bound scores abort instead of producing a pass with a reduced denominator. Simulator/scorer exceptions also propagate by default. Opt-in abort capture, described below, preserves diagnostic rows and failed attempt identities for the existing bundle writer. It supplies no automatic persistence/resume or gate-specific power/false-failure thresholds. An aborted run remains unavailable, and unresolved precision is not relabeled as assessed power. Statistical qualification and gate operating characteristics remain separate obligations.

Howard et al. (2021) and the authors' confseq implementations describe mixture, stitching and betting alternatives for tighter or unbounded-horizon inference. They need separate domain/tuning/numerical qualification; the current known finite horizon and generic bounded scores do not require replacing this NumPy-only baseline or introducing a compiled dependency.

Aborted-run artifacts

By default, evaluator and declared-score errors retain their existing exception behavior. With capture_failure=True, an ordinary evaluation or score-validation exception raises SequentialEvaluationError, with the original error in __cause__ and a detached SequentialFailure in .failure. The runner snapshots the declared definitions before evaluation; completed raw rows are copied only when capturing a failure. Successful runs keep the same scores and allocation.

The failure record separates completed, attempted and failed replicate counts, retains the failed design/replicate identity, stage and error, and distinguishes fixed planned counts from adaptive capacity. Raw rows include their original evaluation_index, even though table rows are grouped by design. All captured rows are marked run_status="aborted" and decision_status="unavailable". The record supplies no intervals, feasibility pass or completed sequential result.

Given the caller's simulator, scorer, grid, objectives and policy:

from trade_study import SequentialEvaluationError, run_sequential, save_result_bundle

try:
    result = run_sequential(
        world, scorer, grid, observables,
        policy=policy, capture_failure=True,
    )
except SequentialEvaluationError as error:
    abort = error.failure
    save_result_bundle(
        abort.results, "new-aborted-run-directory",
        observables=abort.observables,
        provenance={
            "model_revision": "caller-supplied-revision",
            "random_stream": "caller-supplied-stream-identity",
            "sequential_failure": abort.to_metadata(),
        },
    )

The existing bundle writer preserves diagnostic metadata even when the first attempt fails and there are zero completed rows. Configuration/provenance values must satisfy its primitive-JSON contract; it refuses existing destinations. These artifacts support inspection and preservation of work. They supply no automatic retries/resume, process-kill recovery or inference conditional on successful attempts. Do not treat the completed subset as a complete fixed plan or an available sequential decision.

trade_study.ReplicationPolicy(score_bounds, min_reps=2, max_reps=100, max_evaluations=1000, confidence=0.95, equivalence=dict(), allocation='adaptive', replicate_unit='independent_replicate') dataclass

Explicit budget and bounded-score assumptions for sequential replication.

Attributes:

Name	Type	Description
score_bounds	dict[str, tuple[float, float]]	Known almost-sure bounds in raw observable units.
min_reps	int	Initial adaptive batch per design; must not exceed max_reps.
max_reps	int	Adaptive per-design maximum/horizon, or exact fixed-plan count.
max_evaluations	int	Total simulator/scorer evaluations allowed.
confidence	float	Joint coverage across all designs, observables and looks.
equivalence	dict[str, float]	Practical-equivalence tolerances in raw observable units; omitted observables use zero.
allocation	Literal['adaptive', 'fixed']	Adaptive retirement/allocation, or a fixed plan collecting exactly max_reps for every design before any inference look.
replicate_unit	Literal['independent_replicate', 'independent_cluster']	Independent simulator/scorer evaluations, or independent cluster scores. Cluster members are not additional replicates.

trade_study.SequentialResult(results, summary, intervals, allocations, stopping_reasons, feasibility, policy) dataclass

Raw replicates, allocation decisions, and simultaneous mean intervals.

trade_study.SequentialFailure(results, grid, observables, constraints, policy, completed_reps, failed_design, failed_rep, stage, exception_type, exception_message) dataclass

Detached diagnostics from an aborted run, without inferential results.

Attributes:

Name	Type	Description
results	ResultsTable	Completed raw rows, each marked as belonging to an aborted run.
grid	list[dict[str, Any]]	Declared designs captured before evaluation.
observables	list[Observable]	Declared objectives captured before evaluation.
constraints	list[Constraint]	Declared constraints captured before evaluation.
policy	ReplicationPolicy	Detached policy used by the failed run.
completed_reps	tuple[int, ...]	Completed counts in declared design order.
failed_design	int	Zero-based design index of the failed attempt.
failed_rep	int	Zero-based replicate index of the failed attempt.
stage	Literal['evaluation', 'score_validation']	Evaluation or declared-score validation stage.
exception_type	str	Qualified type of the original evaluation error.
exception_message	str	Original error message.

to_metadata()

Describe the abort for caller provenance in a result bundle.

Returns:

Type	Description
dict[str, Any]	Versioned diagnostic metadata. Designs must also satisfy the bundle
dict[str, Any]	writer's primitive-JSON contract when persisted. Counts distinguish
dict[str, Any]	failed attempts from completed rows and unattempted capacity.

Source code in src/trade_study/sequential.py

def to_metadata(self) -> dict[str, Any]:
    """Describe the abort for caller provenance in a result bundle.

    Returns:
        Versioned diagnostic metadata. Designs must also satisfy the bundle
        writer's primitive-JSON contract when persisted. Counts distinguish
        failed attempts from completed rows and unattempted capacity.
    """
    failed = [int(index == self.failed_design) for index in range(len(self.grid))]
    attempted = [
        count + failures
        for count, failures in zip(self.completed_reps, failed, strict=True)
    ]
    return {
        "schema_version": 1,
        "kind": "sequential_failure",
        "status": "aborted",
        "evaluability": "unavailable",
        "decision_status": "unavailable",
        "inference": "not_supplied",
        "failure_policy": "abort",
        "designs": deepcopy(self.grid),
        "completed_reps": list(self.completed_reps),
        "attempted_reps": attempted,
        "failed_reps": failed,
        "actual_evaluations": sum(attempted),
        "completed_evaluations": sum(self.completed_reps),
        "failed_design": self.failed_design,
        "failed_rep": self.failed_rep,
        "failed_evaluation_index": sum(self.completed_reps),
        "stage": self.stage,
        "error_type": self.exception_type,
        "error_message": self.exception_message,
        "allocation_policy": self.policy.allocation,
        "replicate_unit": self.policy.replicate_unit,
        "planned_min_reps": self.policy.min_reps,
        "planned_reps": self.policy.max_reps
        if self.policy.allocation == "fixed"
        else None,
        "max_reps": self.policy.max_reps,
        "max_evaluations": self.policy.max_evaluations,
        "unattempted_capacity": [self.policy.max_reps - n for n in attempted],
        "requested_confidence": self.policy.confidence,
        "score_bounds": {
            name: [lower, upper]
            for name, (lower, upper) in self.policy.score_bounds.items()
        },
        "equivalence": dict(self.policy.equivalence),
        "observables": [
            {
                "name": item.name,
                "direction": item.direction.value,
                "weight": float(item.weight),
            }
            for item in self.observables
        ],
        "constraints": [
            {
                "name": item.name,
                "observable": item.observable,
                "op": item.op,
                "threshold": float(item.threshold),
                "requested_confidence": None
                if item.confidence is None
                else float(item.confidence),
            }
            for item in self.constraints
        ],
    }

trade_study.SequentialEvaluationError(failure)

Bases: RuntimeError

An opt-in aborted-run error carrying diagnostics and its original cause.

Attributes:

Name	Type	Description
failure		Detached raw rows and failure metadata, without intervals, summary feasibility or a completed sequential result.

Store diagnostics for an explicitly requested failure capture.

Source code in src/trade_study/sequential.py

def __init__(self, failure: SequentialFailure) -> None:
    """Store diagnostics for an explicitly requested failure capture."""
    self.failure = failure
    super().__init__(
        f"Sequential evaluation aborted at design {failure.failed_design}, "
        f"replicate {failure.failed_rep}: {failure.exception_type}: "
        f"{failure.exception_message}"
    )

trade_study.run_sequential(world, scorer, grid, observables, *, policy, constraints=None, capture_failure=False)

Allocate extra replicates to unresolved comparisons and constraints.

Parameters:

Name	Type	Description	Default
world	Simulator	Simulator, with optional replicate-id support as in run_grid.	required
scorer	Scorer	Scorer returning finite values within declared score bounds.	required
grid	list[dict[str, Any]]	Fixed candidate designs; adaptive proposal selection is excluded.	required
observables	list[Observable]	Objectives with directions; scores and bounds use raw units.	required
policy	ReplicationPolicy	Explicit budgets, bounds, confidence and adaptive/fixed allocation. The validated used policy is snapshotted before simulator calls.	required
constraints	list[Constraint] | None	Monotone feasibility constraints on scored observables. Simultaneous policy intervals replace normal standard-error bounds.	None
capture_failure	bool	On ordinary evaluation/score-validation exceptions, raise SequentialEvaluationError with detached raw diagnostics and the original exception as its cause. No successful result is returned.	False

Returns:

Type	Description
SequentialResult	Raw per-replicate results, per-design means, simultaneous mean intervals,
SequentialResult	allocations and stopping reasons. Unresolved cases retain unknown
SequentialResult	feasibility and stop at max_reps or the total evaluation budget.

Raises:

Type	Description
ValueError	If definitions/budgets are invalid or a score violates bounds.
TypeError	If capture_failure is not boolean.
SequentialEvaluationError	If explicitly requested failure capture aborts an evaluation. Its failure record supplies no inferential result.

Notes

Coverage requires independent replicates with a fixed mean within each design and known almost-sure score bounds. Dependence across designs through common random numbers is allowed. Adaptive intervals use Hoeffding bounds and a union bound over all designs, observables and sample sizes through max_reps, permitting data-dependent allocation and stopping. Fixed mode collects every planned replicate and uses one final look; those intervals supply no interim-look guarantee. Independent clusters can supply one score per replicate, without counting members separately. Gate power and false-failure rates are not assessed by these intervals. Missing/non-finite/out-of-bound scores abort rather than reduce the denominator. Returned metadata records the used policy and actual counts; ordinary post-selection paired intervals gain no sequential coverage.

Source code in src/trade_study/sequential.py

def run_sequential(
    world: Simulator,
    scorer: Scorer,
    grid: list[dict[str, Any]],
    observables: list[Observable],
    *,
    policy: ReplicationPolicy,
    constraints: list[Constraint] | None = None,
    capture_failure: bool = False,
) -> SequentialResult:
    """Allocate extra replicates to unresolved comparisons and constraints.

    Args:
        world: Simulator, with optional replicate-id support as in run_grid.
        scorer: Scorer returning finite values within declared score bounds.
        grid: Fixed candidate designs; adaptive proposal selection is excluded.
        observables: Objectives with directions; scores and bounds use raw units.
        policy: Explicit budgets, bounds, confidence and adaptive/fixed allocation.
            The validated used policy is snapshotted before simulator calls.
        constraints: Monotone feasibility constraints on scored observables.
            Simultaneous policy intervals replace normal standard-error bounds.
        capture_failure: On ordinary evaluation/score-validation exceptions, raise
            SequentialEvaluationError with detached raw diagnostics and the
            original exception as its cause. No successful result is returned.

    Returns:
        Raw per-replicate results, per-design means, simultaneous mean intervals,
        allocations and stopping reasons. Unresolved cases retain unknown
        feasibility and stop at max_reps or the total evaluation budget.

    Raises:
        ValueError: If definitions/budgets are invalid or a score violates bounds.
        TypeError: If capture_failure is not boolean.
        SequentialEvaluationError: If explicitly requested failure capture aborts
            an evaluation. Its failure record supplies no inferential result.

    Notes:
        Coverage requires independent replicates with a fixed mean within each
        design and known almost-sure score bounds. Dependence across designs
        through common random numbers is allowed. Adaptive intervals use Hoeffding
        bounds and a union bound over all designs, observables and sample sizes
        through max_reps, permitting data-dependent allocation and stopping.
        Fixed mode collects every planned replicate and uses one final look;
        those intervals supply no interim-look guarantee. Independent clusters
        can supply one score per replicate, without counting members separately.
        Gate power and false-failure rates are not assessed by these intervals.
        Missing/non-finite/out-of-bound scores abort rather than reduce the
        denominator. Returned metadata records the used policy and actual counts;
        ordinary post-selection paired intervals gain no sequential coverage.
    """  # ruff: ignore[docstring-extraneous-exception] -- helper propagates error
    if not isinstance(capture_failure, bool):
        msg = "capture_failure must be boolean"
        raise TypeError(msg)
    grid, observables = list(grid), list(observables)
    if not grid:
        msg = "Sequential replication requires at least one design"
        raise ValueError(msg)
    constraints = list(constraints or [])
    _validate(policy, observables, constraints, len(grid))
    policy = _snapshot_policy(policy)
    capture = (
        _FailureCapture(deepcopy(grid), deepcopy(observables), deepcopy(constraints))
        if capture_failure
        else None
    )
    if policy.allocation == "fixed":
        return _run_fixed(
            world, scorer, grid, observables, policy, constraints, capture
        )
    allocation = _Allocation(
        [[] for _ in grid],
        [""] * len(grid),
        ["unknown"] * len(grid),
        failure_capture=capture,
    )
    for _rep in range(policy.min_reps):
        for index, config in enumerate(grid):
            _evaluate(allocation, world, scorer, config, index, policy)
    intervals = _intervals(allocation, observables, policy)
    active = _classify(allocation, intervals, observables, constraints, policy)
    while active:
        eligible = [i for i in active if len(allocation.trials[i]) < policy.max_reps]
        if not eligible or allocation.evaluations >= policy.max_evaluations:
            for index in active:
                allocation.reasons[index] = (
                    "max_reps"
                    if len(allocation.trials[index]) >= policy.max_reps
                    else "budget_exhausted"
                )
            break
        index = min(eligible, key=lambda i: (len(allocation.trials[i]), i))
        _evaluate(allocation, world, scorer, grid[index], index, policy)
        intervals = _intervals(allocation, observables, policy)
        active = _classify(allocation, intervals, observables, constraints, policy)
    return _result(allocation, intervals, observables, policy, constraints)

trade_study.run_successive_halving(trials, sim, *, rungs, eta=3.0, metric, mode='min')

Successive-halving multi-fidelity early-stopping.

Evaluates every trial at the lowest rung, keeps the top 1/eta by metric (according to mode), promotes survivors to the next budget, and repeats until the highest rung. Every (trial, rung) evaluation is recorded as one row in the returned :class:ResultsTable, with rung index and budget stored in per-row metadata.

Parameters:

Name	Type	Description	Default
trials	list[dict[str, Any]]	Candidate configurations to evaluate.	required
sim	PartialEvaluator	A :class:PartialEvaluator whose evaluate(config, budget) returns observables including metric.	required
rungs	list[float]	Strictly ascending list of budgets (e.g. epochs, iterations). Length determines the number of halving rounds.	required
eta	float	Reduction factor between rungs (>1). Each rung keeps ceil(n_prev / eta) survivors. Defaults to 3 per Li et al. (2017).	3.0
metric	str	Observable name used to rank trials at each rung.	required
mode	str	"min" (lower is better) or "max".	'min'

Returns:

Type	Description
ResultsTable	class:ResultsTable whose rows are (trial, rung) evaluations.
ResultsTable	Per-row metadata contains rung (0-indexed), budget,
ResultsTable	trial_index (position in the input trials list),
ResultsTable	promoted (whether this trial advanced past this rung), and
ResultsTable	wall_seconds. Propagates :class:ValueError from the input
ResultsTable	validator when arguments are invalid.

Raises:

Type	Description
KeyError	If metric is missing from a returned observables dict.

Source code in src/trade_study/runner.py

def run_successive_halving(
    trials: list[dict[str, Any]],
    sim: PartialEvaluator,
    *,
    rungs: list[float],
    eta: float = 3.0,
    metric: str,
    mode: str = "min",
) -> ResultsTable:
    """Successive-halving multi-fidelity early-stopping.

    Evaluates every trial at the lowest rung, keeps the top ``1/eta`` by
    ``metric`` (according to ``mode``), promotes survivors to the next
    budget, and repeats until the highest rung. Every (trial, rung)
    evaluation is recorded as one row in the returned :class:`ResultsTable`,
    with ``rung`` index and ``budget`` stored in per-row metadata.

    Args:
        trials: Candidate configurations to evaluate.
        sim: A :class:`PartialEvaluator` whose ``evaluate(config, budget)``
            returns observables including ``metric``.
        rungs: Strictly ascending list of budgets (e.g. epochs, iterations).
            Length determines the number of halving rounds.
        eta: Reduction factor between rungs (>1). Each rung keeps
            ``ceil(n_prev / eta)`` survivors. Defaults to 3 per Li et al.
            (2017).
        metric: Observable name used to rank trials at each rung.
        mode: ``"min"`` (lower is better) or ``"max"``.

    Returns:
        :class:`ResultsTable` whose rows are (trial, rung) evaluations.
        Per-row metadata contains ``rung`` (0-indexed), ``budget``,
        ``trial_index`` (position in the input ``trials`` list),
        ``promoted`` (whether this trial advanced past this rung), and
        ``wall_seconds``. Propagates :class:`ValueError` from the input
        validator when arguments are invalid.

    Raises:
        KeyError: If ``metric`` is missing from a returned observables
            dict.
    """
    _sh_validate_inputs(trials, rungs, eta, metric, mode)

    # rung_records[r] = list of (trial_idx, budget, observables) at rung r
    rung_records: list[list[tuple[int, float, dict[str, float], float]]] = [
        [] for _ in rungs
    ]
    survivors: list[int] = list(range(len(trials)))

    for r, budget in enumerate(rungs):
        for trial_idx in survivors:
            t0 = time.perf_counter()
            obs = sim.evaluate(trials[trial_idx], budget)
            wall = time.perf_counter() - t0
            if metric not in obs:
                msg = (
                    f"run_successive_halving: PartialEvaluator did not return "
                    f"metric {metric!r} at rung {r} for trial {trial_idx}"
                )
                raise KeyError(msg)
            rung_records[r].append((trial_idx, budget, obs, wall))

        if r < len(rungs) - 1:
            ranked = sorted(
                rung_records[r],
                key=lambda row: row[2][metric],
                reverse=(mode == "max"),
            )
            n_keep = max(1, int(np.ceil(len(ranked) / eta)))
            survivors = [row[0] for row in ranked[:n_keep]]

    obs_names = _sh_collect_observables([
        [(idx, b, o) for idx, b, o, _w in rung] for rung in rung_records
    ])

    promoted_at_rung: list[set[int]] = [set() for _ in rungs]
    for r in range(len(rungs) - 1):
        ranked = sorted(
            rung_records[r],
            key=lambda row: row[2][metric],
            reverse=(mode == "max"),
        )
        n_keep = max(1, int(np.ceil(len(ranked) / eta)))
        promoted_at_rung[r] = {row[0] for row in ranked[:n_keep]}

    configs: list[dict[str, Any]] = []
    score_rows: list[list[float]] = []
    metadata: list[dict[str, Any]] = []
    for r, rung in enumerate(rung_records):
        for trial_idx, budget, obs, wall in rung:
            configs.append(trials[trial_idx])
            score_rows.append([obs.get(name, float("nan")) for name in obs_names])
            metadata.append({
                "rung": r,
                "budget": budget,
                "trial_index": trial_idx,
                "promoted": trial_idx in promoted_at_rung[r],
                "wall_seconds": wall,
            })

    return ResultsTable(
        configs=configs,
        scores=np.array(score_rows) if score_rows else np.zeros((0, len(obs_names))),
        observable_names=obs_names,
        metadata=metadata,
    )

trade_study.run_hyperband(trial_factory, sim, *, max_budget, eta=3.0, metric, mode='min')

Hyperband: multi-bracket successive-halving.

Wraps :func:run_successive_halving with the bracket schedule from Li et al. (2017). Each bracket trades off the number of initial trials against the minimum budget per trial; together they hedge against picking either ratio wrong.

Parameters:

Name	Type	Description	Default
trial_factory	Callable[[int, int], list[dict[str, Any]]]	Callable (bracket_index, n_trials) -> trials that returns a fresh list of candidate configs for each bracket. Typically this wraps :func:trade_study.build_grid with a bracket-derived seed so brackets sample different points.	required
sim	PartialEvaluator	A :class:PartialEvaluator.	required
max_budget	float	Maximum resource R per trial.	required
eta	float	Reduction factor (>1). Defaults to 3.	3.0
metric	str	Observable used for ranking within each bracket.	required
mode	str	"min" or "max".	'min'

Returns:

Name	Type	Description
Concatenated	ResultsTable	class:ResultsTable across all brackets, with an
	ResultsTable	additional bracket field in each row's metadata.

Raises:

Type	Description
ValueError	If max_budget <= 0 or eta <= 1.

Source code in src/trade_study/runner.py

def run_hyperband(
    trial_factory: Callable[[int, int], list[dict[str, Any]]],
    sim: PartialEvaluator,
    *,
    max_budget: float,
    eta: float = 3.0,
    metric: str,
    mode: str = "min",
) -> ResultsTable:
    """Hyperband: multi-bracket successive-halving.

    Wraps :func:`run_successive_halving` with the bracket schedule from
    Li et al. (2017). Each bracket trades off the number of initial
    trials against the minimum budget per trial; together they hedge
    against picking either ratio wrong.

    Args:
        trial_factory: Callable ``(bracket_index, n_trials) -> trials`` that
            returns a fresh list of candidate configs for each bracket.
            Typically this wraps :func:`trade_study.build_grid` with a
            bracket-derived seed so brackets sample different points.
        sim: A :class:`PartialEvaluator`.
        max_budget: Maximum resource ``R`` per trial.
        eta: Reduction factor (>1). Defaults to 3.
        metric: Observable used for ranking within each bracket.
        mode: ``"min"`` or ``"max"``.

    Returns:
        Concatenated :class:`ResultsTable` across all brackets, with an
        additional ``bracket`` field in each row's metadata.

    Raises:
        ValueError: If ``max_budget <= 0`` or ``eta <= 1``.
    """
    if max_budget <= 0:
        msg = "run_hyperband: max_budget must be positive"
        raise ValueError(msg)
    if eta <= 1:
        msg = "run_hyperband: eta must be > 1"
        raise ValueError(msg)

    brackets = _hyperband_brackets(max_budget, eta)

    all_configs: list[dict[str, Any]] = []
    all_scores: list[list[float]] = []
    all_metadata: list[dict[str, Any]] = []
    obs_names: list[str] = []

    for bracket_idx, (n_initial, r_min) in enumerate(brackets):
        trials = trial_factory(bracket_idx, n_initial)
        if not trials:
            continue
        s = len(brackets) - 1 - bracket_idx
        rungs = [r_min * eta**i for i in range(s + 1)]
        bracket_results = run_successive_halving(
            trials,
            sim,
            rungs=rungs,
            eta=eta,
            metric=metric,
            mode=mode,
        )
        if not obs_names:
            obs_names = bracket_results.observable_names
        elif obs_names != bracket_results.observable_names:
            # Pad / reorder so all brackets share the column layout.
            union = sorted(set(obs_names) | set(bracket_results.observable_names))
            obs_names = union

        for i, cfg in enumerate(bracket_results.configs):
            all_configs.append(cfg)
            row = bracket_results.scores[i]
            all_scores.append([
                float(row[bracket_results.observable_names.index(n)])
                if n in bracket_results.observable_names
                else float("nan")
                for n in obs_names
            ])
            meta = dict(bracket_results.metadata[i])
            meta["bracket"] = bracket_idx
            all_metadata.append(meta)

    return ResultsTable(
        configs=all_configs,
        scores=np.array(all_scores) if all_scores else np.zeros((0, len(obs_names))),
        observable_names=obs_names,
        metadata=all_metadata,
    )