A team of researchers from Google Brain introduces a new way of programming automated machine learning (AutoML) based on symbolic programming. The team also introduces a new python library called PyGlove that demonstrates the new paradigm’s promising results.
At present, Neural networks are extensively being adopted in various fields. However, determining the correct neural network architecture for a particular task becomes time and resource-consuming because of the increasing complexity in their architecture. Complicated architectures also lead to difficulty in hyperparameter selection. Automated Machine Learning (AutoML) is highly encouraged for automating these selection choices. In May 2017, Google coined AutoML for its neural architecture search (NAS). The approach is widely accepted in the ML research community to automate network hyperparameter and architecture choices.
However, current ML software libraries are quite inadequate in handling the dynamic interactions among the components of AutoML. For instance, popular NAS algorithms require implementation for coupling the AutoML search space and search algorithm. But, implementing complex search flows to handle the dynamic interactions between such AutoML components is challenging.
Therefore, the researchers at Google Brain have reformulated the program logic for AutoML as an automated process to symbolically manipulate ML programs.
In AutoML, selecting architectures and hyper-parameters is formulated as a search problem. The child program is one of the crucial components of AutoML. It refers to a performant program that could be a neural network architecture or a data augmentation strategy. A search space is defined to represent all possible selections. AutoML aims to identify performant child programs out of the search space. The search algorithm then iteratively samples multiple child programs to find and evaluate the most suitable options. A numeric standard for quality ad reward is provided to the search algorithm to improve the child program sampling in the future.
The symbolic programming approach for AutoML allows manipulation of symbolic tree-based representation that encodes a machine learning program’s critical elements. The ML programs, therefore, became mutable objects and can be cloned and modified after their creation. Thus, there is no need to repeatedly create ML programs’ objects as the existing objects can be cloned and modified into new programs. Symbolic programming makes AutoML programming more flexible as the child program’s interactions, search space, and search algorithms are no longer static.
The researchers also presented a general symbolic programming library for Python dubbed PyGlove to test their new reformulation’s potential. PyGlove enables AutoML techniques to be quickly released into preexisting ML pipelines. This allows easy modification of search spaces, search algorithms, and search flows with only a few code lines.
The researchers state that the new approach can benefit multiple research areas such as medicine design and video surveillance by simplifying search-based programs. The team believes that the proposed symbolic programming combined with PyGlove will make AutoML more productive and accessible to ML practitioners.