Multiscale Modeling at the Interface of Molecular Mechanics and Natural Language through Attention Neural Networks

MJ Buehler, ACCOUNTS OF CHEMICAL RESEARCH, 55, 3387-3403 (2022).

DOI: 10.1021/acs.accounts.2c00330

CONSPECTUS: Humans are continually bombarded with massive amounts of data. To deal with this influx of information, we use the concept of attention in order to perceive the most relevant input from vision, hearing, touch, and others. Thereby, the complex ensemble of signals is used to generate output by querying the processed data in appropriate ways. Attention is also the hallmark of the development of scientific theories, where we elucidate which parts of a problem are critical, often expressed through differential equations. In this Account we review the emergence of attention-based neural networks as a class of approaches that offer many opportunities to describe materials across scales and modalities, including how universal building blocks interact to yield a set of material properties. In fact, the self-assembly of hierarchical, structurally complex, and multifunctional biomaterials remains a grand challenge in modeling, theory, and experiment. Expanding from the process by which material building blocks physically interact to form a type of material, in this Account we view self-assembly as both the functional emergence of properties from interacting building blocks as well as the physical process by which elementary building blocks interact and yield structure and, thereby, functions. This perspective, integrated through the theory of materiomics, allows us to solve multiscale problems with a first-principles-based computational approach based on attention-based neural networks that transform information to feature to property while providing a flexible modeling approach that can integrate theory, simulation, and experiment. Since these models are based on a natural language framework, they offer various benefits including incorporation of general domain knowledge via general-purpose pretraining, which can be accomplished without labeled data or large amounts of lower-quality data. Pretrained models then offer a general-purpose platform that can be fine-tuned to adapt these models to make specific predictions, often with relatively little labeled data. The transferrable power of the language-based modeling approach realizes a neural olog description, where mathematical categorization is learned by multiheaded attention, without domain knowledge in its formulation. It can hence be applied to a range of complex modeling tasks-such as physical field predictions, molecular properties, or structure predictions, all using an identical formulation. This offers a complementary modeling approach that is already finding numerous applications, with great potential to solve complex assembly problems, enabling us to learn, build, and utilize functional categorization of how building blocks yield a range of material functions. In this Account, we demonstrate the approach in various application areas, including protein secondary structure prediction and prediction of normal-mode frequencies as well as predicting mechanical fields near cracks. Unifying these diverse problem areas is the building block approach, where the models are based on a universally applicable platform that offers benefits ranging from transferability, interpretability, and cross-domain pollination of knowledge as exemplified through a transformer model applied to predict how musical compositions infer de novo protein structures. We discuss future potentialities of this approach for a variety of material phenomena across scales, including the use in multiparadigm modeling schemes.

Return to Publications page