Optuna Examples

This page contains a list of example codes written with Optuna.

Simplest Codeblock

import optuna


def objective(trial):
    x = trial.suggest_float("x", -100, 100)
    return x ** 2


if __name__ == "__main__":
    study = optuna.create_study()
    # The optimization finishes after evaluating 1000 times or 3 seconds.
    study.optimize(objective, n_trials=1000, timeout=3)
    print(f"Best params is {study.best_params} with value {study.best_value}")

Note

If you are interested in a quick start of Optuna Dashboard with in-memory storage, please take a look at this example.

Tip

Couldn't find your usecase? FAQ might be helpful for you to implement what you want. In this example repository, you can also find the examples for the following scenarios:

1. Objective function with additional arguments, which is useful when you would like to pass arguments besides trial to your objective function.

2. Manually provide trials with sampler, which is useful when you would like to force certain parameters to be sampled.

3. Callback to control the termination criterion of study, which is useful when you would like to define your own termination criterion other than n_trials or timeout.

Examples for Diverse Problem Setups

Here are the URLs to the example codeblocks to the corresponding setups.

Simple Black-box Optimization

• Quadratic Function
• Quadratic Multi-Objective Function
• Quadratic Function with Constraints

Multi-Objective Optimization

• Optimization with BoTorch
• Optimization of Multi-Layer Perceptron with PyTorch

Machine Learning (Incl. LightGBMTuner and OptunaSearchCV)

• AllenNLP
• AllenNLP (Jsonnet)
• Catalyst
• CatBoost
• Chainer
• ChainerMN
• Dask-ML
• FastAI
• Haiku
• Keras
• LightGBM
• LightGBM Tuner
• PyTorch
• PyTorch Ignite
• PyTorch Lightning
• PyTorch Lightning (DDP)
• RAPIDS
• Scikit-learn
• Scikit-learn OptunaSearchCV
• Scikit-image
• SKORCH
• Tensorflow
• Tensorflow (eager)
• XGBoost

If you are looking for an example of reinforcement learning, please take a look at the following:

• Optimization of Hyperparameters for Stable-Baslines Agent

Pruning

The following example demonstrates how to implement pruning logic with Optuna.

• Simple pruning (scikit-learn)

In addition, integration modules are available for the following libraries, providing simpler interfaces to utilize pruning.

• Pruning with Catalyst Integration Module
• Pruning with CatBoost Integration Module
• Pruning with Chainer Integration Module
• Pruning with ChainerMN Integration Module
• Pruning with FastAI Integration Module
• Pruning with Keras Integration Module
• Pruning with LightGBM Integration Module
• Pruning with PyTorch Integration Module
• Pruning with PyTorch Ignite Integration Module
• Pruning with PyTorch Lightning Integration Module
• Pruning with PyTorch Lightning Integration Module (DDP)
• Pruning with Tensorflow Integration Module
• Pruning with XGBoost Integration Module
• Pruning with XGBoost Integration Module (Cross Validation Version)

Samplers

• Warm Starting CMA-ES

If you are interested in defining a user-defined sampler, here is an example:

• SimulatedAnnealingSampler

Terminator

• Optuna Terminator
• OptunaSearchCV with Terminator

Visualization

• Visualizing Study
• Visualizing Study with HiPlot

Distributed Optimization

• Optimizing on Dask Cluster
• Optimizing on Kubernetes
• Optimizing with Ray's Joblib Backend

MLOps Platform

• Tracking Optimization Process with aim
• Tracking Optimization Process with MLflow
• Tracking Optimization Process with Weights & Biases
• Optimization with Hydra

External Projects Using Optuna

• Hugging Face Trainer's Hyperparameter Search
• Allegro Trains
• BBO-Rietveld: Automated Crystal Structure Refinement
• Catalyst
• CuPy
• Hydra's Optuna Sweeper Plugin
• Mozilla Voice STT
• neptune.ai
• OptGBM: A scikit-learn Compatible LightGBM Estimator with Optuna
• Optuna-distributed
• PyKEEN
• RL Baselines Zoo
• Hyperparameter Optimization for Machine Learning, Code Repository for Online Course

Important

PRs to add additional real-world examples or projects are welcome!

Running with Optuna's Docker images?

Our Docker images for most examples are available with the tag ending with -dev. For example, PyTorch Simple can be run via:

$ docker run --rm -v $(pwd):/prj -w /prj optuna/optuna:py3.7-dev python pytorch/pytorch_simple.py

Additionally, our visualization example can also be run on Jupyter Notebook by opening localhost:8888 in your browser after executing the following:

$ docker run -p 8888:8888 --rm optuna/optuna:py3.7-dev jupyter notebook --allow-root --no-browser --port 8888 --ip 0.0.0.0 --NotebookApp.token='' --NotebookApp.password=''