Backtesting

This section explains the concept of Backtesting

Backtests

Backtesting enables users to generate Model predictions for a historical period of time. This allow users to replicate the behavior of a Model that is deployed as part of a Network of Time Series (NoTS), using the exact same Model fit and predict schedule. Users can analyze the results of a Backtest flexibly in our client library to help them decide whether a Model is meeting their needs.

A Backtest in the Myst Platform has the following parameters:

ParameterDescription
TitleThe title of the Backtest
ProjectThe Project in which the user wants to create the Backtest
ModelThe Model for which a user wants to create a Backtest
Test Start TimeThe start time of the test period
Test End TimeThe end time of the test period
Fit Start TimingThe start of each Model fit period (Absolute Timing or Relative Timing)
Fit End TimingThe end of each Model fit period (Absolute Timing or Relative Timing)
Fit Reference TimingSchedule or frequency at which Model fits occur (Cron Timing)
Predict Start TimingThe start of each Model predict period (Absolute Timing or Relative Timing)
Predict End TimingThe start of each Model predict period (Absolute Timing or Relative Timing)
Predict Reference TimingSchedule or frequency at which Model predictions occur (Cron Timing)

Cross-validation

These parameters determine the Backtest’s cross-validation configuration, which contains a fold for each Model fit in the test period. The frequency of fits is determined by the Fit Reference Timing. Each fold contains one or more predictions, whose frequency is determined by the Predict Reference Timing.

In each fold, the Fit Reference Time is used to determine the start and end time of the Model fit. The Predict Reference Times are used to determine the start and end times of the Model predictions.

Example

In this example, we'll consider the following parameters:

ParameterValue
Test Start Time2021-03-01T00:00:00Z
Test End Time2021-03-14T00:00:00Z
Fit Start Timing-P1M
Fit End TimingPT0H
Fit Reference Timing0 0 * * 1
Predict Start TimingPT0H
Predict End TimingPT24H
Predict Reference Timing0 0 * * *

This configuration would result in weekly Model fits with daily Model predictions. Since the test period spans 14 days, this would lead to two one-week folds with seven predictions sharing the same Model fit. Each Model would be fitted once a week on Monday (0 0 * * 1) on the most recent month of data, and generates a forecast once daily (0 0 * * *). If the Model uses an hourly sample period, each forecast contains a prediction for each hour over the following 24 hours.

The figure below shows the two folds in this Backtest, each containing one fit and seven predictions. In addition to the fit, we are highlighting the final prediction of each fold.

27082708

The two folds in the backtest