DeepSpeed

DeepSpeed is an easy-to-use deep learning optimization software suite that powers unprecedented scale and speed for both training and inference. It is part of Microsoft AI at Scale and is a deep learning optimization library that makes distributed training easy, efficient, and effective.