Hyperparameter Trade Study

This tutorial runs a multi-objective hyperparameter sweep over scikit-learn's GradientBoostingRegressor, balancing prediction accuracy against training cost and model complexity.

Run it yourself

The full runnable script is at examples/sklearn_study.py.

pip install scikit-learn
uv run python examples/sklearn_study.py

The problem

This example compares prediction error, training time, and model size. The preferred balance depends on the application's computing and accuracy requirements.

This example treats hyperparameter selection as a multi-objective design-of-experiments problem with three competing objectives:

Objective	Direction	What it measures
RMSE	minimize	Root mean squared error on a held-out test set
Training time	minimize	Wall-clock seconds to fit() the model
Complexity	minimize	Total number of leaf nodes across all trees

Why the objectives conflict

• More estimators increase the number of trees and training work. Their effect on held-out error depends on the data and other settings.
• Deeper trees permit more leaves and can change both fit and generalization.
• Learning rate controls each tree's contribution to the prediction.
• Subsampling changes the training data used for each tree and the variability of gradient estimates.

Pareto analysis identifies configurations that are nondominated across the three measured objectives.

Dataset

We use scikit-learn's make_friedman1, a standard synthetic regression benchmark. The true function is:

\[y = 10\sin(\pi x_1 x_2) + 20(x_3 - 0.5)^2 + 10 x_4 + 5 x_5 + \varepsilon\]

where \(\varepsilon \sim \mathcal{N}(0, 1)\) and \(x_6, \dots, x_{10}\) are noise features. We generate 800 samples and hold out 25 % for testing:

X, y = make_friedman1(n_samples=800, noise=1.0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=42,
)

Simulator and scorer

In trade-study, every experiment needs a Simulator and a Scorer:

• The Simulator wraps model training. Its generate() method receives a hyperparameter config, fits a GradientBoostingRegressor, and returns the test-set ground truth plus predictions and metadata (training time, leaf count).
• The Scorer extracts the three objective values from each trial.

class GBSimulator:
    """Simulator that trains a GradientBoostingRegressor.

    The 'truth' is the test-set ground truth; 'observations' are the
    model's test-set predictions plus training metadata.
    """

    def generate(self, config: dict[str, Any]) -> tuple[Any, Any]:
        """Train a model and return predictions on the test set.

        Args:
            config: Hyperparameter dict with n_estimators, max_depth,
                learning_rate, and subsample.

        Returns:
            Tuple of (y_test array, observation dict with predictions,
            wall time, and number of leaves).
        """
        model = GradientBoostingRegressor(
            n_estimators=config["n_estimators"],
            max_depth=config["max_depth"],
            learning_rate=config["learning_rate"],
            subsample=config["subsample"],
            random_state=42,
        )
        t0 = time.perf_counter()
        model.fit(X_train, y_train)
        wall = time.perf_counter() - t0

        preds = model.predict(X_test)
        n_leaves = sum(tree[0].tree_.n_leaves for tree in model.estimators_)
        return y_test, {"predictions": preds, "wall": wall, "n_leaves": n_leaves}


class GBScorer:
    """Score GradientBoosting results for three objectives."""

    def score(
        self,
        truth: Any,
        observations: Any,
        config: dict[str, Any],
    ) -> dict[str, float]:
        """Compute RMSE, training time, and complexity.

        Args:
            truth: True target values (y_test).
            observations: Dict with predictions, wall time, n_leaves.
            config: Hyperparameter dict (unused).

        Returns:
            Scores for rmse, train_time, and complexity.
        """
        return {
            "rmse": root_mean_squared_error(truth, observations["predictions"]),
            "train_time": observations["wall"],
            "complexity": float(observations["n_leaves"]),
        }

Define observables and factors

Observables tell trade-study what you're measuring and which direction is better:

observables = [
    Observable("rmse", Direction.MINIMIZE),
    Observable("train_time", Direction.MINIMIZE),
    Observable("complexity", Direction.MINIMIZE),
]

Factors define the hyperparameter search space. We use discrete levels so a full factorial grid is tractable (\(4 \times 4 \times 4 \times 3 = 192\) combinations):

factors = [
    Factor("n_estimators", FactorType.DISCRETE, levels=[50, 100, 200, 400]),
    Factor("max_depth", FactorType.DISCRETE, levels=[2, 3, 4, 5]),
    Factor("learning_rate", FactorType.DISCRETE, levels=[0.01, 0.05, 0.1, 0.2]),
    Factor("subsample", FactorType.DISCRETE, levels=[0.6, 0.8, 1.0]),
]

Run the sweep

build_grid with method="full" generates every combination. run_grid evaluates them all and returns a ResultsTable:

    grid = build_grid(factors, method="full")
    print(f"Full factorial grid: {len(grid)} configurations")

    results = run_grid(
        world=GBSimulator(),
        scorer=GBScorer(),
        grid=grid,
        observables=observables,
    )

Hyperparameter landscape

Before looking at the Pareto front, it helps to see how RMSE varies across two hyperparameters. The heatmap below shows the best RMSE achieved at each (n_estimators, max_depth) combination, minimised over learning rate and subsample fraction:

Compare this error surface with training time and leaf count. Additional trees or depth do not ensure lower held-out error.

Inspect the Pareto front

extract_front retains configurations for which no other evaluated design is at least as good on every objective and strictly better on one. The front is defined by this grid and the measured outcomes.

    # Pareto front
    front_idx = extract_front(
        results.scores,
        [o.direction for o in observables],
    )
    print(f"Pareto front: {len(front_idx)} / {len(grid)} designs\n")

    print(
        f"{'n_est':>6s}  {'depth':>5s}  {'lr':>6s}  {'sub':>5s}  "
        f"{'RMSE':>6s}  {'Time':>6s}  {'Leaves':>6s}"
    )
    print("-" * 52)
    for i in front_idx:
        cfg = results.configs[i]
        rmse, t, leaves = results.scores[i]
        print(
            f"{cfg['n_estimators']:6d}  {cfg['max_depth']:5d}  "
            f"{cfg['learning_rate']:6.2f}  {cfg['subsample']:5.1f}  "
            f"{rmse:6.3f}  {t:6.3f}  {leaves:6.0f}"
        )

    # Best RMSE on the front
    front_rmse = results.scores[front_idx, 0]
    best = front_idx[np.argmin(front_rmse)]
    print(f"\nLowest-RMSE Pareto design: {results.configs[best]}")
    print(
        f"  RMSE={results.scores[best, 0]:.4f}  "
        f"time={results.scores[best, 1]:.3f}s  "
        f"leaves={results.scores[best, 2]:.0f}"
    )

Front membership can change with the dataset, seed, and measured training time. A practitioner can choose based on priorities, such as selecting the lowest-RMSE design if accuracy is paramount, or the fastest design that still meets an RMSE threshold.

Pareto front scatter matrix

Each dot is one hyperparameter combination; highlighted points lie on the Pareto front. The off-diagonal panels expose pairwise trade-offs between RMSE, training time, and leaf count.

Parallel coordinates

Every line is one design, coloured by Pareto membership. Compare the hyperparameter ranges occupied by the evaluated front.

RMSE strip plot

Scores for each design, split by Pareto membership. A design may remain on the front because it is faster or smaller despite having higher RMSE.

What to try next

• Use build_grid(factors, method="lhs", n_samples=50) for a Latin hypercube design with continuous hyperparameters.
• Wrap the sweep in a Study with multiple phases. Screen first, then refine the promising region.
• Add an Annotation for dollar cost (e.g., cloud compute pricing per second) to include cost as a non-simulated objective.
• Use save_results() / load_results() to persist results across sessions.