20
Dec
2019
/
Stacey Svetlichnaya, Deep Learning Engineer

A random walk embedding of Neurips 2019

Neurips 2019 was once again an overwhelming festival of all flavors and patterns of machine learning. It’s impossible to see or capture in its entirety, and the late-night conversations and coffee break encounters are more magical than the talks anyway.

As a field, I think we’re shifting from breakneck growth and hype towards a starting semblance of maturity[1], or at least self-awareness, with the rising themes of ethics, robustness, and interpretability. Big data, big compute, and big ad-driven attention-usurping tech are out, and active learning, application-specific hardware and urgent real-world applications like social good and climate change are in.

The niches are proliferating— I noticed way more biotech (protein sequences as a language[2]), experimental cognitive psychology and neuroscience, and of course graphs. Classic math and computational theory are still relevant, but you can be effective without them—fewer equations and more straightforward step-by-step instructions make your work more reproducible, quoth Edward Raff, the hero who implemented two hundred and fifty-five machine learning papers without looking at any supporting code [3].

I particularly liked this table from his poster. It’s surprising that there is no relation between reproduction and the year a paper was published, and that papers with detailed pseudo code and no pseudo code were equally reproducible.

This year’s posters included a breath of fresh air— large TL;DR takeaways visible from a distance.

The reproducibility exercise demonstrates the necessity of the update step in the Earth-scale metalearning program: the more insights, workflows, and code libraries we can reuse, the faster the machine learning community can converge on better practices and better models. W&B strives to support this with tools for automatic logging, easy visualization, and flexible mind-mapping of your workflow—from data to model to training loop to hyperparameter sweeps and so on up the meta-ladder, all organized in one place.

The Weights & Biases team enjoyed the conference, chatting with hundreds of users about what we’re doing well and what to fix next. Internet allowing, we ran a fun interactive demo raffle at our booth where users could configure one guess at—or randomly roll—the best hyperparameters for a small convnet (a toy version of a serious benchmark on drought prediction from satellite). We successfully nerd-sniped, improved val_acc, and delighted folks with real-time training plots.

We’re excited to keep simplifying and enhancing collaboration in deep learning— please message Carey (carey@wandb.com) if you have ideas of how to make W&B more useful for you! Stay tuned for deeper dives into Neurips, more benchmarks on meaningful problems, and perhaps more speculative forays into consciousness and AGI.

References

[1] Celeste Kidd, How to Know. [link] This talk yields the most generalizable gradient updates of the conference, imo.

[2] Rao, R. et al. Evaluating Protein Transfer Learning with TAPE. [link]

[3] Raff, E. A Step Toward Quantifying Independently Reproducible Machine Learning Research [link]

[4]  Yoshua Bengio, Andrew Ng, Carla Gomes, Lester Mackey, Jeff Dean: Panel - Climate Change: A Grand Challenge for ML [link]

Join our mailing list to get the latest machine learning updates.