Nolan Dey

I am a Research Scientist at Cerebras Systems where I research methods for more compute-efficient training of large-scale neural networks, supervised by Joel Hestness.

I received my MASc in Systems Design Engineering at the University of Waterloo, where I was supervised by Bryan Tripp, Graham Taylor, and Alexander Wong. My research focused on developing techniques to understand and explain the activation space of deep neural networks. I also ran a Deep Learning Reading Club.

I also received my BASc in Systems Design Engineering at the University of Waterloo, where I completed machine learning internships at Mind Foundry, Apple, and Capital One and software engineering interships at Parabol, Connected, and Kik.

LinkedIn / Scholar / GitHub / Twitter
Research
Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
arXiv, 2025
Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, Joel Hestness
[arXiv]
Don't be lazy: CompleteP enables compute-efficient deep transformers
arXiv, 2025
Nolan Dey*, Bin Claire Zhang*, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, Joel Hestness
[arXiv]
Neuron-based explanations of neural networks sacrifice completeness and interpretability
TMLR, 2025
Nolan Dey, Eric Taylor, Alexander Wong, Bryan Tripp, Graham Taylor
[OpenReview] [Project page with interactive demo] [arXiv] [Slides]
Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
ICLR, 2025
Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, Joel Hestness
[OpenReview] [arXiv]
The practitioner's guide to the maximal update parameterization
Blog & open-source code, 2024
Nolan Dey, Quentin Anthony, Joel Hestness
[Cerebras Blog] [Eleuther AI Blog] [nanoGPT-mup Code]
Sparse maximal update parameterization: A holistic approach to sparse training dynamics
NeurIPS, 2024
Nolan Dey, Shane Bergsma, Joel Hestness
[arXiv] [Code] [NeurIPS Poster]
Position Interpolation Improves ALiBi Extrapolation
Technical Report, 2023
Faisal Al-Khateeb, Nolan Dey, Daria Soboleva, Joel Hestness
[arXiv]
BTLM-3B-8K: 7B Performance in a 3 Billion Parameter Model
Efficient Natural Language and Speech Processing NeurIPS Workshop, 2023
Nolan Dey*, Daria Soboleva*, Faisal Al-Khateeb, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming (Charles) Chen, Bowen Yang, Siyun Li, Abhay Gupta, Shreyas Saxena, Robert Myers, Jacob Robert Steeves, Marvin Tom, Joel Hestness
[arXiv] [Workshop Paper] [Blog] [Hugging Face] [1.08M downloads and 10th most popular text generation model in first month]
SlimPajama: A 627B token cleaned and deduplicated version of RedPajama
Open-Source Dataset Release, 2023
Daria Soboleva*, Faisal Al-Khateeb*, Robert Myers, Jacob Robert Steeves, Joel Hestness, Nolan Dey
[Blog] [Hugging Face] [Data processing code] [693k downloads and 3rd most popular text dataset in first month]
Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster
Technical Report, 2023
Nolan Dey, Gurpreet Gosal, Zhiming (Charles) Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness
[arXiv] [Hugging Face] [Blog] [Podcast]
Identifying and interpreting tuning dimensions in deep networks
Shared Visual Representations in Human & Machine Intelligence NeurIPS Workshop, 2020
Nolan Dey, J. Eric Taylor, Bryan P. Tripp, Alexander Wong, Graham W. Taylor
[arXiv] [Colab Demo]
37,000 Human-Planned Robotic Grasps With Six Degrees of Freedom
IEEE Robotics and Automation Letters, 2020
Victor R. Osorio, Rajan Iyengar, Xueyang Yao, Preshish Bhattachan, Adrian Ragobar, Nolan Dey, Bryan Tripp
[Paper]
Projects
Actor-Critic RL using Spiking Neurons
2020
Nolan Dey

Released the only open-source implementation of “Reinforcement Learning Using a Continuous Time Actor-Critic Framework with Spiking Neurons” by Frémaux et al. using Nengo and OpenAI Gym

[GitHub] [Original paper]
Synthesizing Preferred Inputs for Deep Neurons via GANs
2020
Nolan Dey*, Nick Torenvliet*, Austin Kothig*

Released an open-source PyTorch implementation (originally Caffe 1.X) of “Synthesizing the preferred inputs for neurons in neural networks via deep generator networks” by Nguyen et al.

[GitHub] [Original paper] [Original implementation]
Graph Convolutional Neural Network Explainability
2019
Nolan Dey

Released the only open-source implementation of “Explainability Methods for Graph Convolutional Neural Networks” by Pope & Kolouri et al. using PyTorch-Geometric and RDKit

[GitHub] [Original paper]
SYDE 2019 Class Survey
2019
Nolan Dey, Jason Manson-Hing

Surveyed 55 respondents from my undergraduate class with questions related to demographics, academics, internships, lifestyle, and post-graduation plans. Published a detailed analysis of the survey results with 109 graphs and open-sourced my code to help future classes conduct similar surveys

[SYDE 2019 Class Profile] [GitHub]
Carols for Cans
2012 - 2019
Nolan Dey

Founded and annually organized an event where students sing Christmas carols and ask for food donations. Between 2012 and 2019 we donated ~10,314 food items to GTA food banks since 2012, with 599 students participating over 8 years!

[WordPress] [Facebook]