Learning in continuous action space for developing high dimensional potential energy models

S Manna and TD Loeffler and R Batra and S Banik and H Chan and B Varughese and K Sasikumar and M Sternberg and T Peterka and MJ Cherukara and SK Gray and BG Sumpter and SKRS Sankaranarayanan, NATURE COMMUNICATIONS, 13, 368 (2022).

DOI: 10.1038/s41467-021-27849-6

Reinforcement learning algorithms are emerging as powerful machine learning approaches. This paper introduces a novel machine-learning approach for learning in continuous action space and applies this strategy to the generation of high dimensional potential models for a wide variety of materials. Reinforcement learning (RL) approaches that combine a tree search with deep learning have found remarkable success in searching exorbitantly large, albeit discrete action spaces, as in chess, Shogi and Go. Many real-world materials discovery and design applications, however, involve multi-dimensional search problems and learning domains that have continuous action spaces. Exploring high- dimensional potential energy models of materials is an example. Traditionally, these searches are time consuming (often several years for a single bulk system) and driven by human intuition and/or expertise and more recently by global/local optimization searches that have issues with convergence and/or do not scale well with the search dimensionality. Here, in a departure from discrete action and other gradient-based approaches, we introduce a RL strategy based on decision trees that incorporates modified rewards for improved exploration, efficient sampling during playouts and a "window scaling scheme" for enhanced exploitation, to enable efficient and scalable search for continuous action space problems. Using high-dimensional artificial landscapes and control RL problems, we successfully benchmark our approach against popular global optimization schemes and state of the art policy gradient methods, respectively. We demonstrate its efficacy to parameterize potential models (physics based and high-dimensional neural networks) for 54 different elemental systems across the periodic table as well as alloys. We analyze error trends across different elements in the latent space and trace their origin to elemental structural diversity and the smoothness of the element energy surface. Broadly, our RL strategy will be applicable to many other physical science problems involving search over continuous action spaces.

Return to Publications page