Hyperparameter Optimization
This page describes the concept of Hyperparameter Optimization
HPOs
Hyperparameter Optimization (HPO) optimizes model hyperparameters to ensure models are performing at their best. HPOs run a set of trials where each trial is a full Backtest and optimize hyperparameters for a given model based on the results of the collection of trials run. You can analyze results of HPO runs in our Web Application and client library to confirm your models have the best set of hyperparameters.
With the exception of the Number of Trials, Max Concurrent Trials, and Search Space, HPO shares all parameters with Backtesting. An HPO in the Myst platform therefore has the following parameters:
| Parameter | Description |
|---|---|
| Title | The title of the HPO |
| Model | The Model for which a user wants to create an HPO |
| Metric | The metric for determining the optimal hyperparameters. Today, MSE is the only available metric. |
| Number of Trials | The number of trials in the HPO |
| Max Concurrent Trials | The maximum number of trials running at any one time |
| Test Start Time | The start time of each trial test period |
| Test End Time | The end time of each trial test period |
| Fit Start Timing | The start of each Model fit period (Absolute Timing or Relative Timing) |
| Fit End Timing | The end of each Model fit period (Absolute Timing or Relative Timing) |
| Fit Reference Timing | Schedule or frequency at which Model fits occur (Cron Timing) |
| Predict Start Timing | The start of each Model predict period (Absolute Timing or Relative Timing) |
| Predict End Timing | The start of each Model predict period (Absolute Timing or Relative Timing) |
| Predict Reference Timing | Schedule or frequency at which Model predictions occur (Cron Timing) |
| Search Space | The parameters to optimize along with their search space |
Example
The tables below include the parameters for an HPO with 10 different trials, where each trial runs sequentially. Each trial in the HPO is a full backtest and the parameters are the same as our example in our Backtesting page. See that page for a detailed description of the backtest and its parameters.
Parameters
| Parameter | Value |
|---|---|
| Number of Trials | 10 |
| Max Concurrent Trials | 1 |
| Test Start Time | 2021-03-01T00:00:00Z |
| Test End Time | 2021-03-14T00:00:00Z |
| Fit Start Timing | -P1M |
| Fit End Timing | PT0H |
| Fit Reference Timing | 0 0 * * 1 |
| Predict Start Timing | PT0H |
| Predict End Timing | PT24H |
| Predict Reference Timing | 0 0 * * * |
Search Space
The parameters below are an example of a search space for an XGBoost model.
| Parameter | Sampler | Parameters |
|---|---|---|
| Boosting Rounds | LogUniform | Lower: 100Upper: 1000Base: 10 |
| Max Depth | QUniform | Lower: 1Upper: 12q: 1 |
| Learning Rate | LogUniform | Lower: 0.005Upper: 0.2Base: 10 |
| Min Child Weight | QUniform | Lower: 0Upper: 100q: 5 |
Updated over 1 year ago