Deep Learning Training Tricks
Intro Here is a blog to summarize all the deep learning training tricks I encountered these days. AdamW Optimizer “W” means weight decay, which is the same concept of weight decay in L2 regulari...
Jinyu's Mind Garden
Secret Sauce of Distributed Deep Learning - Ring-All-Reduce
Here is a tech blog on Ring-All-Reduce. It is a very smart way to perform all-reduce under data parallel setup.
Parallelisms in Large Scale Deep Learning
Data Parallelism Model Parallelism Pipeline Parallelism Tensor Parallelism Data Parallelism Split non-overlapping batch of training data to multiple GPUs (GPU clusters if a model cannot...
Another 2 ways of regularization, Label Smoothing and Weight Tying
Intro I have been implementing GPT model myself these days, and I found 2 interesting regularization techniques. One is Label Smoothing mentioned in the transformer paper. The other is Weight Tyin...
Scaling in Dropout and Transformer Attention Block
Intro Dropout is a regularization method by randomly setting some of the elements output by a layer of neurons to zero during the training stage, but acts like an Identity layer during the inferen...
Machine Learning Notes
<?xml version=”1.0” encoding=”utf-8”?> <!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Strict//EN” “http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd”> ‎ ...
Algorithm Notes
<?xml version=”1.0” encoding=”utf-8”?> <!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Strict//EN” “http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd”> ‎ ...
Binary Heap
Definition Heap is a tree-based data structure that satisfies the heap property. Binary Heap is a binary tree with two properties: Shape Property: it must be a complete binary tree, whic...