Backtesting

Backtests

Backtesting enables users to generate Model predictions for a historical period of time. This allow users to replicate the behavior of a Model that is deployed as part of a Network of Time Series (NoTS), using the exact same Model fit and predict schedule. Users can analyze the results of a Backtest flexibly in our client library to help them decide whether a Model is meeting their needs.

A Backtest in the Myst Platform has the following parameters:

Parameter	Description
Title	The title of the Backtest
Project	The Project in which the user wants to create the Backtest
Model	The Model for which a user wants to create a Backtest
Test Start Time	The start time of the test period
Test End Time	The end time of the test period
Fit Start Timing	The start of each Model fit period (Absolute Timing or Relative Timing)
Fit End Timing	The end of each Model fit period (Absolute Timing or Relative Timing)
Fit Reference Timing	Schedule or frequency at which Model fits occur (Cron Timing)
Predict Start Timing	The start of each Model predict period (Absolute Timing or Relative Timing)
Predict End Timing	The start of each Model predict period (Absolute Timing or Relative Timing)
Predict Reference Timing	Schedule or frequency at which Model predictions occur (Cron Timing)

Cross-validation

These parameters determine the Backtest’s cross-validation configuration, which contains a fold for each Model fit in the test period. The frequency of fits is determined by the Fit Reference Timing. Each fold contains one or more predictions, whose frequency is determined by the Predict Reference Timing.

In each fold, the Fit Reference Time is used to determine the start and end time of the Model fit. The Predict Reference Times are used to determine the start and end times of the Model predictions.

Example

In this example, we'll consider the following parameters:

Parameter	Value
Test Start Time	`2021-03-01T00:00:00Z`
Test End Time	`2021-03-14T00:00:00Z`
Fit Start Timing	`-P1M`
Fit End Timing	`PT0H`
Fit Reference Timing	`0 0 * * 1`
Predict Start Timing	`PT0H`
Predict End Timing	`PT24H`
Predict Reference Timing	`0 0 * * *`

This configuration would result in weekly Model fits with daily Model predictions. Since the test period spans 14 days, this would lead to two one-week folds with seven predictions sharing the same Model fit. Each Model would be fitted once a week on Monday (0 0 * * 1) on the most recent month of data, and generates a forecast once daily (0 0 * * *). If the Model uses an hourly sample period, each forecast contains a prediction for each hour over the following 24 hours.

The figure below shows the two folds in this Backtest, each containing one fit and seven predictions. In addition to the fit, we are highlighting the final prediction of each fold.