Nolan Dey

I am a Research Scientist at Cerebras Systems where I research methods for more compute-efficient training of large-scale neural networks.

I received my MASc in Systems Design Engineering at the University of Waterloo, where I was supervised by Bryan Tripp, Graham Taylor, and Alexander Wong. My research focused on developing techniques to understand and explain the activation space of deep neural networks. I also ran a Deep Learning Reading Club.

I also received my BASc in Systems Design Engineering at the University of Waterloo, where I completed machine learning internships at Mind Foundry, Apple, and Capital One and software engineering interships at Parabol, Connected, and Kik.

LinkedIn / Scholar / GitHub / Twitter
Research
The practitioner's guide to the maximal update parameterization
Blog & open-source code, 2024
Nolan Dey, Quentin Anthony, Joel Hestness
[Cerebras Blog][Eleuther AI Blog][nanoGPT-mup Code]
Sparse maximal update parameterization: A holistic approach to sparse training dynamics
NeurIPS, 2024
Nolan Dey, Shane Bergsma, Joel Hestness
[arXiv] [Code] [NeurIPS Poster]
Position Interpolation Improves ALiBi Extrapolation
Technical Report, 2023
Faisal Al-Khateeb, Nolan Dey, Daria Soboleva, Joel Hestness
[arXiv]
BTLM-3B-8K: 7B Performance in a 3 Billion Parameter Model
Efficient Natural Language and Speech Processing NeurIPS Workshop, 2023
Nolan Dey*, Daria Soboleva*, Faisal Al-Khateeb, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming (Charles) Chen, Bowen Yang, Siyun Li, Abhay Gupta, Shreyas Saxena, Robert Myers, Jacob Robert Steeves, Marvin Tom, Joel Hestness
[arXiv] [Workshop Paper] [Blog] [Hugging Face] [1.08M downloads and 10th most popular text generation model in first month]
SlimPajama: A 627B token cleaned and deduplicated version of RedPajama
Open-Source Dataset Release, 2023
Daria Soboleva*, Faisal Al-Khateeb*, Robert Myers, Jacob Robert Steeves, Joel Hestness, Nolan Dey
[Blog] [Hugging Face] [Data processing code] [693k downloads and 3rd most popular text dataset in first month]
Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster
Technical Report, 2023
Nolan Dey, Gurpreet Gosal, Zhiming (Charles) Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness
[arXiv] [Hugging Face] [Blog] [Podcast]
Identifying and interpreting tuning dimensions in deep networks
Shared Visual Representations in Human & Machine Intelligence NeurIPS Workshop, 2020
Nolan Dey, J. Eric Taylor, Bryan P. Tripp, Alexander Wong, Graham W. Taylor
[Paper] [Colab Demo]
37,000 Human-Planned Robotic Grasps With Six Degrees of Freedom
IEEE Robotics and Automation Letters, 2020
Victor R. Osorio, Rajan Iyengar, Xueyang Yao, Preshish Bhattachan, Adrian Ragobar, Nolan Dey, Bryan Tripp
[Paper]
Projects
Actor-Critic RL using Spiking Neurons
Nolan Dey

Released the only open-source implementation of “Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons” by Frémaux et al. using Nengo and OpenAI Gym

[GitHub] [Original paper]
Synthesizing Preferred Inputs for Deep Neurons via GANs
Nolan Dey*, Nick Torenvliet*, Austin Kothig*

Released an open-source PyTorch implementation (originally Caffe 1.X) of “Synthesizing the preferred inputs for neurons in neural networks via deep generator networks” by Nguyen et al.

[GitHub] [Original paper] [Original implementation]
Graph Convolutional Neural Network Explainability
Nolan Dey

Released the only open-source implementation of “Explainability Methods for Graph Convolutional Neural Networks” by Pope & Kolouri et al. using PyTorch-Geometric and RDKit

[GitHub] [Original paper]
SYDE 2019 Class Survey
Nolan Dey, Jason Manson-Hing

Surveyed 55 respondents from my undergraduate class with questions related to demographics, academics, internships, lifestyle, and post-graduation plans. Published a detailed analysis of the survey results with 109 graphs and open-sourced my code to help future classes conduct similar surveys

[SYDE 2019 Class Profile] [GitHub]
Carols for Cans
Nolan Dey

Founded and annually organized an event where students sing Christmas carols and ask for food donations. Between 2012 and 2019 we donated ~10,314 food items to GTA food banks since 2012, with 599 students participating over 9 years!

[WordPress] [Facebook]