Nolan Dey

I am a Research Scientist at Cerebras Systems where I research methods for more compute-efficient training of large-scale neural networks.

I received my MASc in Systems Design Engineering at the University of Waterloo, where I was supervised by Bryan Tripp, Graham Taylor, and Alexander Wong. My research focused on developing techniques to understand and explain the activation space of deep neural networks. I also ran a Deep Learning Reading Club.

I also received my BASc in Systems Design Engineering at the University of Waterloo, where I completed machine learning internships at Mind Foundry, Apple, and Capital One and software engineering interships at Parabol, Connected, and Kik.

Research
The practitioner's guide to the maximal update parameterization
Blog & open-source code, 2024
Nolan Dey, Quentin Anthony, Joel Hestness
Sparse maximal update parameterization: A holistic approach to sparse training dynamics
NeurIPS, 2024
Nolan Dey, Shane Bergsma, Joel Hestness
Position Interpolation Improves ALiBi Extrapolation
Technical Report, 2023
Faisal Al-Khateeb, Nolan Dey, Daria Soboleva, Joel Hestness
BTLM-3B-8K: 7B Performance in a 3 Billion Parameter Model
Efficient Natural Language and Speech Processing NeurIPS Workshop, 2023
Nolan Dey*, Daria Soboleva*, Faisal Al-Khateeb, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming (Charles) Chen, Bowen Yang, Siyun Li, Abhay Gupta, Shreyas Saxena, Robert Myers, Jacob Robert Steeves, Marvin Tom, Joel Hestness
SlimPajama: A 627B token cleaned and deduplicated version of RedPajama
Open-Source Dataset Release, 2023
Daria Soboleva*, Faisal Al-Khateeb*, Robert Myers, Jacob Robert Steeves, Joel Hestness, Nolan Dey
Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster
Technical Report, 2023
Nolan Dey, Gurpreet Gosal, Zhiming (Charles) Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness
Identifying and interpreting tuning dimensions in deep networks
Shared Visual Representations in Human & Machine Intelligence NeurIPS Workshop, 2020
Nolan Dey, J. Eric Taylor, Bryan P. Tripp, Alexander Wong, Graham W. Taylor
37,000 Human-Planned Robotic Grasps With Six Degrees of Freedom
IEEE Robotics and Automation Letters, 2020
Victor R. Osorio, Rajan Iyengar, Xueyang Yao, Preshish Bhattachan, Adrian Ragobar, Nolan Dey, Bryan Tripp
Projects
Actor-Critic RL using Spiking Neurons
Nolan Dey

Released the only open-source implementation of “Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons” by Frémaux et al. using Nengo and OpenAI Gym

Synthesizing Preferred Inputs for Deep Neurons via GANs
Nolan Dey*, Nick Torenvliet*, Austin Kothig*

Released an open-source PyTorch implementation (originally Caffe 1.X) of “Synthesizing the preferred inputs for neurons in neural networks via deep generator networks” by Nguyen et al.

Graph Convolutional Neural Network Explainability
Nolan Dey

Released the only open-source implementation of “Explainability Methods for Graph Convolutional Neural Networks” by Pope & Kolouri et al. using PyTorch-Geometric and RDKit

SYDE 2019 Class Survey
Nolan Dey, Jason Manson-Hing

Surveyed 55 respondents from my undergraduate class with questions related to demographics, academics, internships, lifestyle, and post-graduation plans. Published a detailed analysis of the survey results with 109 graphs and open-sourced my code to help future classes conduct similar surveys

Carols for Cans
Nolan Dey

Founded and annually organized an event where students sing Christmas carols and ask for food donations. Between 2012 and 2019 we donated ~10,314 food items to GTA food banks since 2012, with 599 students participating over 9 years!