Evaluating the predictive power of machine learning model for shear transformation in metallic glasses using metrics for an imbalanced dataset

J Lee and S Ryu, FRONTIERS IN MATERIALS, 9, 874339 (2022).

DOI: 10.3389/fmats.2022.874339

Plastic deformation of metallic glasses, which show no long-range structural order, proceeds by shear transformation of a local group of atoms referred to as the shear transformation zone (STZ). Unlike crystalline solids, it is difficult to identify STZs and predict the onset of plasticity from a random atomic configuration under a given loading. Recently, significant efforts have been made to predict the shear transformation with initial atomic properties using machine learning. However, despite the class imbalance, where the atoms participating in shear transformation is much rarer compared to the others, few studies have explored the issue of the proper predictive metric choice, with most studies considering widely used metrics such as Recall or AUC in the machine learning community. Therefore, here we train a graph neural network that predicts the initially activated STZ and evaluate its predictive power using various metrics considered to be proper for handling imbalanced datasets. We find that the AUC value is significantly overestimated due to the class imbalance and too many atoms are misclassified as initial STZ, so other metrics such as the precision, f1, MCC, and AP indicate very low predictive power close to zero. Additionally, we reveal that the predictive performance changes significantly over the threshold value of non-affine displacement, above which an atom is classified as the initially activated STZ, due to the change in the degree of class imbalance. Our study implies that it is crucial to use an identical threshold for this type of classification (i.e., the class ratio) for a fair assessment of ML models adapted in different studies and to holistically evaluate the predictive performance based on various metrics.

Return to Publications page