Surrogate#

Layer 3: Extend Components

saealib restricts the responsibility of the predictive model approximating the objective function to Surrogate, a swappable component. Surrogate knows only fit/predict — it knows nothing about how a prediction is converted into a score (AcquisitionFunction), or where training data comes from (TrainingSet, via SurrogateManager).

Surrogate’s role#

Surrogate requires two methods to be implemented.

fit(train_x, train_y) -> None: Trains the model on inputs of shape (n_samples, n_features) and outputs of shape (n_samples, n_obj) (or (n_samples,) for single-objective).

predict(test_x) -> SurrogatePrediction: Returns predictions for inputs of shape (n_samples, n_features).

The class attribute provides_uncertainty: bool = False indicates whether predictions carry uncertainty (standard deviation). The default is False; only Gaussian Process implementations override it to True.

There are two marker base classes between Surrogate and predict().

RegressionSurrogate: A marker for regression surrogates where train_y is a real-valued objective function output.

ComparisonSurrogate: For comparison surrogates where train_y is a {0, 1}-valued binary comparison label. predict_proba(test_x) -> SurrogatePrediction (values are win probabilities in [0, 1]) is the primary interface, with predict() delegating to predict_proba() by default.

SurrogatePrediction#

predict()’s return value is SurrogatePrediction, a unified dataclass.

Field	Content
`value`	The predicted value. Shape `(n_samples, n_obj)`
`std`	Uncertainty (standard deviation). `None` for surrogates that don’t provide it
`label`	Class labels, only present for classification models
`metadata`	A `dict` holding implementation-specific auxiliary information, such as SHAP values

value and std are convenience properties for the objective channel. When a surrogate’s prediction represents a quantity that isn’t the objective value (such as a novelty score), this mechanism prevents contaminating things like pbest. Archive-based criteria such as NoveltyAcquisition instead receive candidate and archive design points directly.

Built-in Surrogates#

RBFSurrogate(kernel, polynomial_degree="auto", solver="lstsq", alpha=1e-8): A surrogate using RBF interpolation [16, 17, 18] (the origin of RBF interpolation itself is Hardy, 1971). kernel is a required RBFKernel object with no default (built-in kernels are listed below); a custom kernel is a subclass implementing evaluate(r). predict() explicitly returns std=None (RBF interpolation provides no uncertainty).

Parameter	Values	Role
`polynomial_degree`	`"auto"` (default) / `None` / `0` / `1`	Augments the system with a polynomial term (from `kernel.auto_polynomial_degree` / none / constant / linear). A conditionally positive definite kernel such as `ThinPlateSplineKernel` requires one; a strictly positive definite kernel such as `GaussianKernel` does not, but resolves `"auto"` to a constant term so that predictions stay unchanged when a constant is added to every objective value.
`solver`	`"lstsq"` (default) / `"solve"` / `"tikhonov"`	How the (possibly augmented) linear system is solved: least squares (tolerant of rank-deficient systems) / direct / ridge-regularized via `alpha`.
`alpha`	`1e-8` (default)	Ridge regularization strength, used only when `solver="tikhonov"`.

kernel and polynomial_degree are the values as configured; resolved_kernel and resolved_polynomial_degree expose what a fit() call actually resolves and uses, keeping the configured RBFKernel instance itself immutable. Reassigning kernel/polynomial_degree/solver/alpha invalidates any fitted state; predict() then raises RuntimeError until the next fit().

Kernels carrying a length_scale field (in the same distance units as the input data) resolve it automatically to the median pairwise training-point distance when left None.

Each kernel declares two independent degrees. min_polynomial_degree is the lowest degree the kernel accepts — passing anything below it, None included, raises ValidationError. auto_polynomial_degree is what polynomial_degree="auto" resolves to, which may be higher than the minimum.

Kernel	Formula	`min_polynomial_degree`	`"auto"` resolves to
`GaussianKernel`	\(\exp\left(-\dfrac{r^2}{2\,\ell^2}\right)\)	none	`0` (constant)
`ThinPlateSplineKernel`	\(r^2 \log r\) (no `length_scale`)	`1` (linear)	`1` (linear)
`LinearKernel`	\(r\) (no `length_scale`)	`0` (constant)	`0` (constant)
`CubicKernel`	\(r^3\) (no `length_scale`)	`1` (linear)	`1` (linear)
`MultiquadricKernel`	\(\sqrt{r^2 + \ell^2}\)	`0` (constant)	`0` (constant)
`MaternKernel` (`nu=0.5/1.5/2.5`)	Matérn covariance, `length_scale` = \(\ell\)	none	`0` (constant)

All six kernels and all three solver values are supported: each is covered by tests and usable as shipped. Passing polynomial_degree=None to GaussianKernel or MaternKernel is likewise supported, and drops the constant term that "auto" would have added.

PerObjectiveSurrogate(surrogates): A RegressionSurrogate subclass, a composite class that assigns a different surrogate per objective. Raises ValueError at fit time if train_y’s column count doesn’t match len(surrogates). provides_uncertainty is a composite judgment, returning True only if every constituent surrogate is True.

External library adapters#

Regression surrogates via a scikit-learn-compatible API carry a Sklearn prefix.

Class	Model
`SklearnGPRSurrogate`	Gaussian Process [18, 19]. The sole implementation with `provides_uncertainty=True`
`SklearnRFRSurrogate`	Random Forest regression
`SklearnSVMSurrogate`	SVM
`SklearnNNSurrogate`	MLP
`SklearnXGBSurrogate`	XGBoost (the `xgboost` extra)
`SklearnLGBMSurrogate`	LightGBM (the `lightgbm` extra)
`TorchSurrogate`	PyTorch-based models (the `torch` extra)

SklearnGPRSurrogate computes standard deviation from the GP kernel via return_std=True, and returns provides_uncertainty=True.

The following classes are classification surrogates aimed at feasibility prediction.

Class	Model
`SklearnClassificationSurrogate`	Classification models compatible with scikit-learn in general
`SklearnRFCClassificationSurrogate`	Random Forest classification
`SklearnSVCClassificationSurrogate`	SVM classification

See TrainingSet’s FeasibilityClassificationSet for how training data is extracted for these classification surrogates. For pairwise comparison, use the dedicated ComparisonSurrogate-family implementations rather than these classification surrogates, paired with SurrogateManager’s PairwiseSurrogateManager and PairwiseComparisonSet.

See Installation for how to install each extra.

Note

Adapters for BoTorch/SMT surrogates, beyond scikit-learn/XGBoost/LightGBM/PyTorch, are not currently implemented in saealib. pyproject.toml has no corresponding extra either. pymoo has adapters, but at the Problem/Crossover/Mutation/Algorithm level, not as a Surrogate — pymoo doesn’t provide surrogate models.

Extension hooks#

If you just want to add post-fit processing, you can add it to an existing Surrogate instance with with_post_fit(fn) instead of creating a new subclass. with_post_fit doesn’t modify the original instance — it returns a copy with fn added.

from saealib import GaussianKernel, RBFSurrogate

def log_fit(train_x, train_y, ctx=None):
    print(f"fit on {len(train_x)} samples")

base = RBFSurrogate(kernel=GaussianKernel())
logged = base.with_post_fit(log_fit)

fn’s signature is fn(train_x, train_y, ctx) -> None.

Implementing a custom Surrogate#

If you need a custom predictive model, subclass Surrogate and implement fit()/predict(). Choose RegressionSurrogate for regression, or ComparisonSurrogate (implementing predict_proba()) for comparison, as the class to subclass.

The following example is a simple regression surrogate that returns the nearest training point’s objective value directly as the prediction.

import numpy as np
from saealib import RegressionSurrogate, SurrogatePrediction


class NearestNeighborSurrogate(RegressionSurrogate):
    """A simple surrogate that returns the nearest point's objective value directly as the prediction."""

    def fit(self, train_x, train_y):
        self.train_x = np.asarray(train_x, dtype=float)
        self.train_y = np.asarray(train_y, dtype=float)

    def predict(self, test_x):
        test_x = np.atleast_2d(test_x)
        dists = np.linalg.norm(self.train_x[None, :, :] - test_x[:, None, :], axis=2)
        nearest = dists.argmin(axis=1)
        value = self.train_y[nearest]
        return SurrogatePrediction(value=value)

Uncertainty support table#

To use an uncertainty-based AcquisitionFunction, Surrogate needs to return std.

Class	`provides_uncertainty`
`SklearnGPRSurrogate`	`True`
`RBFSurrogate` / `SklearnRFRSurrogate` / `SklearnSVMSurrogate` / `SklearnNNSurrogate` / `SklearnXGBSurrogate` / `SklearnLGBMSurrogate` / `TorchSurrogate`	`False`
`PerObjectiveSurrogate`	`True` only if every constituent surrogate is `True`

Optimizer.validate() detects and warns about a mismatch between AcquisitionFunction’s requires_uncertainty and Surrogate’s provides_uncertainty.