Stacey Svetlichnaya, Deep Learning Engineer

A random walk embedding of Neurips 2019

Neurips 2019 was once again an overwhelming festival of all flavors and patterns of machine learning. It’s impossible to see or capture in its entirety, and the late-night conversations and coffee break encounters are more magical than the talks anyway.

As a field, I think we’re shifting from breakneck growth and hype towards a starting semblance of maturity[1], or at least self-awareness, with the rising themes of ethics, robustness, and interpretability. Big data, big compute, and big ad-driven attention-usurping tech are out, and active learning, application-specific hardware and urgent real-world applications like social good and climate change are in.

The niches are proliferating— I noticed way more biotech (protein sequences as a language[2]), experimental cognitive psychology and neuroscience, and of course graphs. Classic math and computational theory are still relevant, but you can be effective without them—fewer equations and more straightforward step-by-step instructions make your work more reproducible, quoth Edward Raff, the hero who implemented two hundred and fifty-five machine learning papers without looking at any supporting code [3].

I particularly liked this table from his poster. It’s surprising that there is no relation between reproduction and the year a paper was published, and that papers with detailed pseudo code and no pseudo code were equally reproducible.

This year’s posters included a breath of fresh air— large TL;DR takeaways visible from a distance.

The reproducibility exercise demonstrates the necessity of the update step in the Earth-scale metalearning program: the more insights, workflows, and code libraries we can reuse, the faster the machine learning community can converge on better practices and better models. W&B strives to support this with tools for automatic logging, easy visualization, and flexible mind-mapping of your workflow—from data to model to training loop to hyperparameter sweeps and so on up the meta-ladder, all organized in one place.

The Weights & Biases team enjoyed the conference, chatting with hundreds of users about what we’re doing well and what to fix next. Internet allowing, we ran a fun interactive demo raffle at our booth where users could configure one guess at—or randomly roll—the best hyperparameters for a small convnet (a toy version of a serious benchmark on drought prediction from satellite). We successfully nerd-sniped, improved val_acc, and delighted folks with real-time training plots.

We’re excited to keep simplifying and enhancing collaboration in deep learning— please message Carey (carey@wandb.com) if you have ideas of how to make W&B more useful for you! Stay tuned for deeper dives into Neurips, more benchmarks on meaningful problems, and perhaps more speculative forays into consciousness and AGI.


[1] Celeste Kidd, How to Know. [link] This talk yields the most generalizable gradient updates of the conference, imo.

[2] Rao, R. et al. Evaluating Protein Transfer Learning with TAPE. [link]

[3] Raff, E. A Step Toward Quantifying Independently Reproducible Machine Learning Research [link]

[4]  Yoshua Bengio, Andrew Ng, Carla Gomes, Lester Mackey, Jeff Dean: Panel - Climate Change: A Grand Challenge for ML [link]


Enter your email to get updates about new features and blog posts.

Weights & Biases

We're building lightweight, flexible experiment tracking tools for deep learning. Add a couple of lines to your python script, and we'll keep track of your hyperparameters and output metrics, making it easy to compare runs and see the whole history of your progress. Think of us like GitHub for deep learning.

Partner Program

We are building our library of deep learning articles, and we're delighted to feature the work of community members. Contact Lavanya to learn about opportunities to share your research and insights.

Try our free tools for experiment tracking →