Initial models, costs and evaluation¶

Initial-model contract¶

The initial model represents the model already available before the first decision. Its historical training cost is outside the planning horizon. Choose one of these forms:

Pass initial_model=artifact to plan(). The model adapter evaluates it on every evaluation block.
Set PlannerConfig(initial_loss=...) for an explicit constant baseline loss at every timestep. This works even when training data exist at step 0.
Leave both unset to use the compatibility bootstrap: fit a model on step 0 using initial_model_samples or the full first block. Its fit is outside the schedule’s objective. Prefer an explicit artifact or baseline when comparing algorithms or accounting for all work.

initial_loss and an explicit initial_model are mutually exclusive. The bootstrap needs training data at step 0. A timeline with no retrain candidates can still produce an all-keep plan when an initial model or baseline is valid. Result metadata records initial_model_source: provided_initial_model, configured_initial_loss, trained_on_step0, or provider_supplied for a custom provider without its own source label. Initial-model records carry zero training cost; they do not account for bootstrap elapsed time.

Units and accounting¶

The default evaluator measures ModelAdapter.fit() with a wall clock. Sampling and later evaluations are outside that fit-time measurement. Timing depends on hardware, process load, synchronization and the adapter’s work; fixed seeds alone do not make it deterministic.

Value	Meaning
`CandidateEvaluation.training_seconds`	Raw seconds measured for a candidate’s fit.
`CandidateEvaluation.training_cost`	Training cost in objective units; the default is `time_cost_weight * training_seconds`.
`CandidateEvaluation.future_losses[t]`	Raw modeling loss at evaluation time `t`.
`PlanResult.training_cost`	Sum of selected retraining actions’ costs in objective units.
`PlanResult.loss_cost`	Sum of selected models’ raw losses over the timeline.
`PlanResult.weighted_loss_cost`	`loss_cost_weight * loss_cost`.
`PlanResult.total_cost`	`training_cost + weighted_loss_cost`.
`PlanResult.estimated_total_cost`	For Bayes, the returned schedule’s cost using the table from before final measurement.

The plan’s training cost is not the expense of finding the plan. Exact search trains all candidates; approximate search trains a subset and may perform extra fits when measuring the final schedule or collecting diagnostic history. record_bayes_true_history=True can add substantial diagnostic work. There is no guarantee that a small candidate-selection setting caps total fit time.

After measuring the estimated proposal, Bayes reoptimizes over all measured candidates. The returned plan can therefore differ from that proposal, and every selected candidate has measured costs. This final optimization performs no fits. surrogate_evaluated_candidates counts candidates observed before final measurement; evaluated_candidates includes final measurement, while finalization_new_candidates lists its additions. These counts exclude extra diagnostic fits. Diagnostic history is collected after selection and cannot change the returned plan or add candidates to the search cache.

Custom training costs¶

Pass training_cost_fn to RetrainingPlanner, EvaluationEngine, collect_online_training_data, or OnlineController to change training-cost accounting while retaining the built-in fitting and loss evaluation. The TrainingCostFunction protocol accepts these keyword arguments:

def training_cost_fn(*, step, n_samples, training_seconds, model) -> float:
    ...

Return a finite, nonnegative number in final objective units. This replaces time_cost_weight * training_seconds; the time weight is not applied again. loss_cost_weight still applies to raw losses. With no callback, the existing wall-clock calculation remains the default. For example, a fixed cost per sample can make a small experiment independent of timing noise:

>>> from retrainopt import PlannerConfig, RetrainingPlanner
>>> from retrainopt.adapters import ArrayTimelineAdapter, PrototypeModelAdapter
>>> def sample_cost(*, step, n_samples, training_seconds, model):
...     return 0.125 * n_samples
>>> rows = [((0.0,), 0), ((1.0,), 1)]
>>> timeline = ArrayTimelineAdapter([rows], [rows])
>>> config = PlannerConfig(n_choices=(1.0,), initial_loss=1.0,
...                        time_cost_weight=999.0, loss_cost_weight=2.0)
>>> result = RetrainingPlanner(timeline, PrototypeModelAdapter(), config,
...                            training_cost_fn=sample_cost).plan("exact")
>>> (result.training_cost, result.weighted_loss_cost, result.total_cost)
(0.25, 0.0, 0.25)

The callback receives the artifact returned by fit, so it may read energy or billing measurements stored by the adapter. It must not mutate that artifact. It receives no timeline or future evaluation data. Historical tuning and deployment should use the same cost definition; callbacks must not read future data through external state or depend on the previously deployed model. The offline recurrence assumes a candidate’s training cost is independent of that previous model.

The callback runs after a new candidate fit. Returning a cached evaluation does not call it again; additional diagnostic fits can. KEEP decisions and initial models, including compatibility bootstrap fits, have zero training cost and do not invoke it. training_seconds continues to report fit wall-clock seconds, whatever units the callback returns. A callback failure or invalid return value raises an error; online controller state is committed only after the whole step succeeds. Applications are responsible for any external callback side effects when retrying.

Custom evaluators¶

For precomputed costs, controlled timing, or external training code, implement the public EvaluationProvider protocol and inject it:

planner = RetrainingPlanner(
    timeline, model_adapter, config,
    evaluator=my_evaluator,
)

An explicit evaluator and training_cost_fn are mutually exclusive: a custom provider already supplies the final training cost.

The provider exposes:

evaluate_initial_model(initial_model=None) -> CandidateEvaluation
evaluate_candidate(step, n_samples) -> CandidateEvaluation
evaluate_candidate_uncached(step, n_samples) -> CandidateEvaluation
iter_cached_candidates() -> Iterable[CandidateEvaluation]

The initial record has ID (-1, 0), zero training cost, and losses for the entire timeline. A candidate (s, n) has losses for every timestep from s through the end. Losses and measured costs must be finite and nonnegative. The provider returns training cost in final objective units and raw losses; RetrainOpt applies the loss weight. iter_cached_candidates() lists observed candidate records, excluding the initial model. The uncached method supports diagnostics without adding a new observation to the search cache. If initial_loss is configured, the provider’s initial losses must match that baseline at every timestep.

Model IDs must be pairs of integers (excluding booleans), including on records returned for a requested candidate. An equal-valued float is not an integer ID. Every provider record must set observed=True as a boolean; strings such as "false" and numeric flags are rejected.

See examples/verify_exact_against_bruteforce.py for a complete fixed-table provider and independent exhaustive verification. No subclass of the default evaluation engine is needed.

Lifetime and errors¶

A planner uses a configuration snapshot and a cache scoped to that planner. Treat the timeline, adapter, initial-model semantics, cost function, and custom provider as unchanged during its lifetime. Create a new planner for a different experiment. This in-memory cache is not a persistent cross-experiment result store.

Serialized results use schema_version=2; metadata uses metadata_schema_version=1 and algorithm_version="0.3". The algorithm version identifies planning semantics independently of the package version; 0.3 adds the final optimization over measured candidates. Approximate runs record acquisition_semantics="complete_stage_independent_posteriors".

Invalid input or a failed fit/evaluation raises an error. Do not encode a failed training run as a zero loss, an empty successful result, or an arbitrary cost.