Secret Sauce of Distributed Deep Learning - Ring-All-Reduce
Here is a tech blog on Ring-All-Reduce.
It is a very smart way to perform all-reduce under data parallel setup.
This post is licensed under CC BY 4.0 by the author.
Post
Here is a tech blog on Ring-All-Reduce.
It is a very smart way to perform all-reduce under data parallel setup.
Comments powered by Disqus.