This section explains the concept of Backtesting
Backtesting enables users to generate Model predictions for a historical period of time. This allow users to replicate the behavior of a Model that is deployed as part of a Network of Time Series (NoTS), using the exact same Model fit and predict schedule. Users can analyze the results of a Backtest flexibly in our client library to help them decide whether a Model is meeting their needs.
A Backtest in the Myst Platform has the following parameters:
|Title||The title of the Backtest|
|Project||The Project in which the user wants to create the Backtest|
|Model||The Model for which a user wants to create a Backtest|
|Test Start Time||The start time of the test period|
|Test End Time||The end time of the test period|
|Fit Start Timing||The start of each Model fit period (Absolute Timing or Relative Timing)|
|Fit End Timing||The end of each Model fit period (Absolute Timing or Relative Timing)|
|Fit Reference Timing||Schedule or frequency at which Model fits occur (Cron Timing)|
|Predict Start Timing||The start of each Model predict period (Absolute Timing or Relative Timing)|
|Predict End Timing||The start of each Model predict period (Absolute Timing or Relative Timing)|
|Predict Reference Timing||Schedule or frequency at which Model predictions occur (Cron Timing)|
These parameters determine the Backtest’s cross-validation configuration, which contains a fold for each Model fit in the test period. The frequency of fits is determined by the Fit Reference Timing. Each fold contains one or more predictions, whose frequency is determined by the Predict Reference Timing.
In each fold, the Fit Reference Time is used to determine the start and end time of the Model fit. The Predict Reference Times are used to determine the start and end times of the Model predictions.
In this example, we'll consider the following parameters:
|Test Start Time|
|Test End Time|
|Fit Start Timing|
|Fit End Timing|
|Fit Reference Timing|
|Predict Start Timing|
|Predict End Timing|
|Predict Reference Timing|
This configuration would result in weekly Model fits with daily Model predictions. Since the test period spans 14 days, this would lead to two one-week folds with seven predictions sharing the same Model fit. Each Model would be fitted once a week on Monday (
0 0 * * 1) on the most recent month of data, and generates a forecast once daily (
0 0 * * *). If the Model uses an hourly sample period, each forecast contains a prediction for each hour over the following 24 hours.
The figure below shows the two folds in this Backtest, each containing one fit and seven predictions. In addition to the fit, we are highlighting the final prediction of each fold.
Updated 9 months ago