Baidu Brain, a Chinese core AI technology engine, announces the release of PaddlePaddle 2.0. PaddlePaddle (PArallel Distributed Deep LEarning)) is an open-sourced AI platform released by Baidu Brain in 2016 to apply deep learning(DL) to many products at Baidu, such as NLP (Natural Language Processing), translation, and image processing.
PaddlePaddle’s latest version has features like dynamic (computational) graphs, a new API system, distributed training for trillion-parameter models, and better hardware support.
Earlier versions had already added the support of dynamic graphs, but it was optional. Compared with static graphs, dynamic graphs are found to be more flexible in programming and better in debug mechanisms. In the latest version 2.0, PaddlePaddle has now switched its default programming paradigm from static graphs to dynamic graphs.
- It allows developers to check the inputs and outputs of tensors dynamically and helps debug their codes quickly.
- Automatic mixed precision and quantization aware training are also supported in the new version.
- It also provides easy conversion of dynamic graphs to static graphs with uniform interfaces to accelerate model training and deployment. Static graphs are the graphs that are already defined, so they enable optimized training and inference performance. This can be done using paddle.enable_static().
New API upgrade
- Version 2.0 has improved the API namespaces for easy location of target API.
- It allows the developers to build their neural networks with stateful class APIs and stateless functional APIs.
- It has also added 217 new APIs to the system and re optimized 195 APIs.
- It also provides high-level APIs in data augmentation, data pipeline, batch training, and more tasks tailored for low-code programming.
- It provides a hierarchical design of high-level and low-level APIs leading to smooth programming. It means when both types of APIs are used, it allows developers to leverage both the API types and to switch between simple implementation and detailed customization promptly.
- It also provides ‘upgrade tools’ readily available to the developers to reduce upgrade and migration costs.
Large-scale model training
As the name ‘PArallel Distributed Deep Learning’ suggests, PaddlePaddle is designed to perform large-scale deep network training, with data sources distributed across multiple nodes. PaddlePaddle 2.0 introduces a new hybrid parallel mode that allows developers to combine data-parallel, model-parallel, and pipeline-parallel training. This allows each layer of a network’s parameters to distribute to multiple GPUs, leading to more efficient training of a trillion-parameter model.
General-purpose heterogeneous parameter server
PaddlePaddle 2.0 is the first to release general-purpose heterogeneous parameter server technology. It supports different hardware types – CPU, GPU, and AI chips – efficiently in a single training job. Usually, high IO usage leads to inefficient utilization of computing resources. This heterogeneous parameter server technology tackles such issues. It also trains large-scale sparse-feature models of search and recommendation applications efficiently. It allows developers to deploy distributed training tasks on heterogeneous hardware clusters quickly and efficiently leverage different types of chips. It has also been proven to be cost-effective. As observed, two CPUs and two GPUs can achieve a training speed similar to 4 GPUs, at approximately 65% of the original cost.
New Hardware Support
PaddlePaddle 2.0 is compatible with around 29 categories of processors from semiconductor manufacturers like Intel, NVIDIA, and ARM.
Dynamic graphs: https://github.com/PaddlePaddle/models/tree/develop/dygraph
PaddlePaddle API Documentation: https://www.paddlepaddle.org.cn/documentation/docs/en/api/index_en.html