Distributed hyperparameter optimization at scale

Ayush Chaurasia, Contributor
28
Aug
2020

Weights & Biases 💜 Ray Tune

Weights & Biases helps your ML team unlock their productivity by optimizing, visualizing, collaborating on, and standardizing their model and data pipelines – regardless of framework, environment, or workflow. Used by the likes of OpenAI, Toyota and Github, W&B is part of the new standard of best practices for machine learning. By saving everything you need to track and compare models — architecture, hyperparameters, weights, model predictions, GPU usage, git commits, and even datasets – W&B makes your ML workflows reproducible.

image.png

Today we're announcing an integration with a tool our community adores – Ray/Tune is one of first and most respected libraries for scalable hyperparameter optimization. With just a few lines of code Ray/Tune helps researchers optimize their models with state-of-the-art algorithms and scale their hyperparameter optimization process to hundreds of nodes and GPUs.

image.png

Why we chose Ray Tune – Delivering model development and hyperparameter optimization at scale

We're especially excited about the possibilities this collaboration with our friends at Ray/Tune opens up. Both Weights and Biases and Ray/Tune are built for scale and handle millions of models every month for teams doing some of the most cutting-edge deep learning research.

Whereas W&B is a centralized repository for everything you need to track, reproduce and gain insights from your models easily; Ray/Tune provides a simple interface for scaling and running distributed experiments. A few reasons why our community likes Ray/Tune –

  1. Simple Distributed execution: Ray Tune makes it easy to scale from a single node, to multiple GPUs, and further multiple nodes
  2. Large number of algorithms: Ray Tune has a huge number of algorithms including Population Based Training, ASHA, and HyperBand
  3. Framework agnostic: Ray Tune works across frameworks including PyTorch, Keras, Tensorflow, XGBoost, and PyTorchLightning.
  4. Fault-tolerance: Ray Tune is built on top of Ray, providing fault tolerance out of the box.

Getting Started

There are 2 ways you can use the wandb integration with Ray Tune.

1. The WandbLogger

tune.run(
   train,
    loggers=[WandbLogger],
    config={
     "wandb": {"project": "rayTune", "monitor_gym": True}},        
   })

WandbLogger automatically logs the metrics reported to the W&B dashboard of the project.

2. The wandb_mixin

You can also use wandb_mixin function decorator when you need to log any custom metrics, charts and other visualizations

@wandb_mixin
def train(...):
           ...
    wandb.log({...})
    tune.report(metric = score)
       ...

Read the full post →

Join our mailing list to get the latest machine learning updates.