Metrics

Set of metrics that can be used for an enhanced understanding of your machine learning model.

First are the easily-interpretable error measures of the relative information capacity of feature space F with respect to feature space F’. The methods returns a value between 0 and 1, where 0 means that F and F’ are completey distinct in terms of linearly-decodable information, and where 1 means that F’ is contained in F. All methods are implemented as the root mean-square error for the regression of the feature matrix X_F’ (or sometimes called Y in the doc) from X_F (or sometimes called X in the doc) for transformations with different constraints (linear, orthogonal, locally-linear). By default a custom 2-fold cross-validation skosmo.linear_model.Ridge2FoldCV is used to ensure the generalization of the transformation and efficiency of the computation, since we deal with a multi-target regression problem. Methods were applied to compare different forms of featurizations through different hyperparameters and induced metrics and kernels [Goscinski2021] .

These reconstruction measures are available:

• Global Reconstruction Error (GRE) computes the amount of linearly-decodable information recovered through a global linear reconstruction.

• Global Reconstruction Distortion (GRD) computes the amount of distortion contained in a global linear reconstruction.

• Local Reconstruction Error (LRE) computes the amount of decodable information recovered through a local linear reconstruction for the k-nearest neighborhood of each sample.

Next, we offer a set of prediction rigidity metrics, which can be used to quantify the robustness of the local or component-wise predictions that the machine learning model has been trained to make, based on the training dataset composition.

These prediction rigidities are available:

• Local Prediction Rigidity (LPR) computes the local prediction rigidity of a linear or kernel model.

• Component-wise Prediction Rigidity (CPR) computes the component-wise prediction rigidity of a linear or kernel model.

There are also two distance metrics compatible with the periodic boundary conditions available.

Note

Currently only rectangular cells are supported. Cell format: [side_length_1, …, side_length_n]

• Pairwise Euclidean Distances computes the euclidean distance between two sets of points. It is compatible with the periodic boundary conditions. If the cell length is not provided, it will fall back to the scikit-learn version of the euclidean distance sklearn.metrics.pairwise.euclidean_distances().

• Pairwise Mahalanobis Distance computes the Mahalanobis distance between two sets of points. It is compatible with the periodic boundary conditions.

Global Reconstruction Error

skmatter.metrics.pointwise_global_reconstruction_error(X, Y, train_idx=None, test_idx=None, scaler=None, estimator=None)

Computes the pointwise global reconstruction error using the source X to reconstruct the features or samples of target Y based on a minimization by linear regression:

\[GRE^{(i)}(X,Y) = \min_W ||y_i - x_iW||\]

If used with X and Y of shape (n_samples, n_features) it computes the pointwise global reconstruction error of the features as defined in Ref. [Goscinski2021]. In this case the number of samples of X and Y should agree with each other, but the number of features can be different. The error is expressed per sample.

If used with X and Y of shape(n_features, n_samples) it computes the reconstruction error of the samples. In this case the number of features of X and Y should agree with each other, but the number of samples can be different. The error is expressed per feature.

The default parameters mimics the ones of Ref. [Goscinski2021].

Parameters:

• X (numpy.ndarray of shape (n_samples, X_n_features)) – Source data which reconstructs target Y. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• Y (numpy.ndarray of shape (n_samples, Y_n_targets)) – Target data which is reconstructed with X. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• train_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the test_idx is used. If train_size is also None, 2-fold split is taken.

• test_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the train_idx is used. If test_size is also None, 2-fold split is taken.

• scaler (object implementing fit/transfom) – Scales the X and Y before computing the reconstruction measure. The default value scales the features such that the reconstruction measure on the training set is upper bounded to 1.

• estimator (object implementing fit/predict, default=None) – Sklearn estimator used to reconstruct features/samples.

Returns:

pointwise_global_reconstruction_error (numpy.ndarray) – The global reconstruction error for each sample/point

skmatter.metrics.global_reconstruction_error(X, Y, test_idx=None, train_idx=None, scaler=None, estimator=None)

Computes the global reconstruction error using the source X to reconstruct the features or samples of target Y based on a minimization by linear regression:

\[GRE(X,Y) = \min_W ||Y - XW||\]

If used with X and Y of shape (n_samples, n_features) it computes the global reconstruction error of the features as defined in Ref. [Goscinski2021]. In this case the number of samples of X and Y should agree with each other, but the number of features can be different. The error is expressed per sample.

If used with X and Y of shape(n_features, n_samples) it computes the reconstruction error of the samples. In this case the number of features of X and Y should agree with each other, but the number of samples can be different. The error is expressed per feature.

The default parameters mimics the ones of Ref. [Goscinski2021].

Parameters:

• X (numpy.ndarray of shape (n_samples, X_n_features)) – Source data which reconstructs target Y. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• Y (numpy.ndarray of shape (n_samples, Y_n_targets)) – Target data which is reconstructed with X. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• train_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the test_idx is used. If train_size is also None, 2-fold split is taken.

• test_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the train_idx is used. If test_size is also None, 2-fold split is taken.

• scaler (object implementing fit/transfom) – Scales the X and Y before computing the reconstruction measure. The default value scales the features such that the reconstruction measure on the training set is upper bounded to 1.

• estimator (object implementing fit/predict, default=None) – Sklearn estimator used to reconstruct features/samples.

Returns:

global_reconstruction_error (ndarray) – The global reconstruction error

Global Reconstruction Distortion

skmatter.metrics.pointwise_global_reconstruction_distortion(X, Y, test_idx=None, train_idx=None, scaler=None, estimator=None)

Computes the pointwise global reconstruction distortion using the source X to reconstruct the features or samples of target Y based on a minimization by orthogonal regression:

\[GRD^{(i)}(X,Y) = \min_Q ||y_i - x_iQ\|| \quad\mathrm{subject\ to}\quad Q^TQ=I\]

If used with X and Y of shape (n_samples, n_features) it computes the pointwise global reconstruction distortion of the features as defined in Ref. [Goscinski2021]. In this case the number of samples of X and Y should agree with each other, but the number of features can be different. The distortion is expressed per sample.

If used with X and Y of shape(n_features, n_samples) it computes the reconstruction distortion of the samples. In this case the number of features of X and Y should agree with each other, but the number of samples can be different. The distortion is expressed per feature.

The default parameters mimics the ones of Ref. [Goscinski2021].

Parameters:

• X (numpy.ndarray of shape (n_samples, X_n_features)) – Source data which reconstructs target Y. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• Y (numpy.ndarray of shape (n_samples, Y_n_targets)) – Target data which is reconstructed with X. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• train_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the test_idx is used. If train_size is also None, 2-fold split is taken.

• test_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the train_idx is used. If test_size is also None, 2-fold split is taken.

• scaler (object implementing fit/transfom) – Scales the X and Y before computing the reconstruction measure. The default value scales the features such that the reconstruction measure on the training set is upper bounded to 1.

• estimator (object implementing fit/predict, default=None) – Sklearn estimator used to reconstruct features/samples.

Returns:

pointwise_global_reconstruction_distortion (ndarray) – The global reconstruction distortion for each sample/point

skmatter.metrics.global_reconstruction_distortion(X, Y, test_idx=None, train_idx=None, scaler=None, estimator=None)

Computes the global reconstruction distortion using the source X to reconstruct the features or samples of target Y based on a minimization by orthogonal regression:

\[GRD(X,Y) = \min_Q ||y - XQ\|| \quad\mathrm{subject\ to}\quad Q^TQ=I\]

If used with X and Y of shape (n_samples, n_features) it computes the global reconstruction distortion of the features as defined in Ref. [Goscinski2021]. In this case the number of samples of X and Y should agree with each other, but the number of features can be different. The distortion is expressed per sample.

If used with X and Y of shape(n_features, n_samples) it computes the reconstruction distortion of the samples. In this case the number of features of X and Y should agree with each other, but the number of samples can be different. The distortion is expressed per feature.

The default parameters mimics the ones of Ref. [Goscinski2021].

Parameters:

• X (numpy.ndarray of shape (n_samples, X_n_features)) – Source data which reconstructs target Y. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• Y (numpy.ndarray of shape (n_samples, Y_n_targets)) – Target data which is reconstructed with X. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• train_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the test_idx is used. If train_size is also None, 2-fold split is taken.

• test_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the train_idx is used. If test_size is also None, 2-fold split is taken.

• scaler (object implementing fit/transfom) – Scales the X and Y before computing the reconstruction measure. The default value scales the features such that the reconstruction measure on the training set is upper bounded to 1.

• estimator (object implementing fit/predict, default=None) – Sklearn estimator used to reconstruct features/samples.

Returns:

global_reconstruction_distortion (numpy.ndarray) – The global reconstruction distortion

Local Reconstruction Error

skmatter.metrics.pointwise_local_reconstruction_error(X, Y, n_local_points, test_idx=None, train_idx=None, scaler=None, estimator=None, n_jobs=None)

Computes the pointwise local reconstruction error using the source X to reconstruct the features or samples of target Y based on a minimization by linear regression:

\[\tilde{\mathbf{x}}'_i = \bar{\mathbf{x}} + (\mathbf{x}_i - \bar{\mathbf{x}})\mathbf{P}^{(i)}\]

\[LRE^{(i)}(X,Y) = \|\mathbf{x}'_i - \tilde{\mathbf{x}}'_i\|^2\]

If used with X and Y of shape (n_samples, n_features) it computes the pointwise local reconstruction error of the features as defined in Ref. [Goscinski2021]. In this case the number of samples of X and Y should agree with each other, but the number of features can be different. The error is expressed per sample.

If used with X and Y of shape(n_features, n_samples) it computes the reconstruction error of the samples. In this case the number of features of X and Y should agree with each other, but the number of samples can be different. The error is expressed per feature.

The default parameters mimics the ones of Ref. [Goscinski2021].

Parameters:

• X (numpy.ndarray of shape (n_samples, X_n_features)) – Source data which reconstructs target Y. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• Y (numpy.ndarray of shape (n_samples, Y_n_targets)) – Target data which is reconstructed with X. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• n_local_points (int,) – Number of neighbour points used to compute the local reconstruction weight for each sample/point.

• train_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the test_idx is used. If train_size is also None, 2-fold split is taken.

• test_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the train_idx is used. If test_size is also None, 2-fold split is taken.

• scaler (object implementing fit/transfom) – Scales the X and Y before computing the reconstruction measure. The default value scales the features such that the reconstruction measure on the training set is upper bounded to 1.

• estimator (object implementing fit/predict, default=None) – Sklearn estimator used to reconstruct features/samples.

Returns:

pointwise_local_reconstruction_error (numpy.ndarray) – The local reconstruction error for each sample/point

skmatter.metrics.local_reconstruction_error(X, Y, n_local_points, test_idx=None, train_idx=None, scaler=None, estimator=None, n_jobs=None)

Computes the local reconstruction error using the source X to reconstruct the features or samples of target Y based on a minimization by linear regression:

\[LRE(X,Y) = \sqrt{\sum_i LRE^{(i)}(X,Y)}/\sqrt{n_\text{test}}\]

If used with X and Y of shape (n_samples, n_features) it computes the local reconstruction error of the features as defined in Ref. [Goscinski2021]. In this case the number of samples of X and Y should agree with each other, but the number of features can be different. The error is expressed per sample.

If used with X and Y of shape(n_features, n_samples) it computes the reconstruction error of the samples. In this case the number of features of X and Y should agree with each other, but the number of samples can be different. The error is expressed per feature.

The default parameters mimics the ones of Ref. [Goscinski2021].

Parameters:

• X (numpy.ndarray of shape (n_samples, X_n_features)) – Source data which reconstructs target Y. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• Y (numpy.ndarray of shape (n_samples, Y_n_targets)) – Target data which is reconstructed with X. For feature reconstruction of Y using X use input shape (samples, features). For sample reconstruction of Y using X use input shape (features, samples).

• n_local_points (int,) – Number of neighbour points used to compute the local reconstruction weight for each sample/point.

• train_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the test_idx is used. If train_size is also None, 2-fold split is taken.

• test_idx (numpy.ndarray, dtype=int, default=None) – array of indices used for training, if None, If None, the complement of the train_idx is used. If test_size is also None, 2-fold split is taken.

• scaler (object implementing fit/transfom) – Scales the X and Y before computing the reconstruction measure. The default value scales the features such that the reconstruction measure on the training set is upper bounded to 1.

• estimator (object implementing fit/predict, default=None) – Sklearn estimator used to reconstruct features/samples.

Returns:

local_reconstruction_error (numpy.ndarray) – The local reconstruction error

Local Prediction Rigidity

skmatter.metrics.local_prediction_rigidity(X_train, X_test, alpha)

Computes the local prediction rigidity (LPR) of a linear or kernel model trained on a training dataset provided as input, on the local environments in the test set provided as a separate input. LPR is defined as follows:

\[LPR_{i} = \frac{1}{X_i (X^{T} X + \lambda I)^{-1} X_i^{T}}\]

The function assumes that the model training is undertaken in a manner where the global prediction targets are averaged over the number of atoms appearing in each training structure, and the average feature vector of each structure is hence used in the regression. This ensures that (1) regularization strength across structures with different number of atoms is kept constant per structure during model training, and (2) range of resulting LPR values are loosely kept between 0 and 1 for the ease of interpretation. This requires the user to provide the regularizer value that results from such training procedure. To guarantee valid comparison in the LPR across different models, feature vectors are scaled by a global factor based on standard deviation across atomic envs.

If the model is a kernel model, K_train and K_test can be provided in lieu of X_train and X_test, alnog with the appropriate regularizer for the trained model.

Parameters:

• X_train (list of numpy.ndarray of shape (n_atoms, n_features)) – Training dataset where each training set structure is stored as a separate ndarray.

• X_test (list of numpy.ndarray of shape (n_atoms, n_features)) – Test dataset where each training set structure is stored as a separate ndarray.

• alpha (float) – Regularizer value that the linear/kernel model has been optimized to.

Returns:

• LPR (list of numpy.array of shape (n_atoms)) – Local prediction rigidity (LPR) of the test set structures. LPR is separately stored for each test structure, and hence list length = n_test_strucs.

• rank_diff (int) – integer value of the difference between cov matrix dimension and rank

Component-wise Prediction Rigidity

skmatter.metrics.componentwise_prediction_rigidity(X_train, X_test, alpha, comp_dims)

Computes the component-wise prediction rigidity (CPR) and the local CPR (LCPR) of a linear or kernel model trained on a training dataset provided as input, on the local environments in the test set provided as a separate input. CPR and LCPR are defined as follows:

\[CPR_{A,c} = \frac{1}{X_{A,c} (X^{T} X + \lambda I)^{-1} X_{A,c}^{T}}\]

\[LCPR_{i,c} = \frac{1}{X_{i,c} (X^{T} X + \lambda I)^{-1} X_{i,c}^{T}}\]

The function assumes that the feature vectors for the local environments and structures are built by concatenating the descriptors of different prediction components together. It also assumes, like the case of LPR, that model training is undertaken in a manner where the global prediction targets are averaged over the number of atoms appearing in each training structure, and the average feature vector of each structure is hence used in the regression. Likewise, to guarantee valid comparison in the (L)CPR across different models, feature vectors are scaled by a global factor based on standard deviation across atomic envs.

If the model is a kernel model, K_train and K_test can be provided in lieu of X_train and X_test, alnog with the appropriate regularizer for the trained model. However, in computing the kernels, one must strictly keep the different components separate, and compute separate kernel blocks for different prediction components.

Parameters:

• X_train (list of numpy.ndarray of shape (n_atoms, n_features)) – Training dataset where each training set structure is stored as a separate ndarray.

• X_test (list of numpy.ndarray of shape (n_atoms, n_features)) – Test dataset where each training set structure is stored as a separate ndarray.

• alpha (float) – Regularizer value that the linear/kernel model has been optimized to.

• comp_dims (numpy.ndarray of int values) – Dimensions of the feature vectors pertaining to each prediction component.

Returns:

• CPR (numpy.ndarray of shape (n_test_strucs, n_comps)) – Component-wise prediction rigidity computed for each prediction component, pertaining to the entire test structure.

• LCPR (list of ndarrays of shape (n_atoms, n_comps)) – Local component-wise prediction rigidity of the test set structures. Values are separately stored for each test structure, and hence list length = n_test_strucs

• rank_diff (int) – value of the difference between cov matrix dimension and rank

Pairwise Euclidean Distances

skmatter.metrics.periodic_pairwise_euclidean_distances(X, Y=None, *, squared=False, cell_length=None)

Compute the pairwise distance matrix between each pair from a vector array X and Y.

\[\begin{split}d_{i, j} = \\sqrt{\\sum_{k=1}^n (x_{i, k} - y_{j, k})^2}\end{split}\]

For efficiency reasons, the euclidean distance between a pair of row vector x and y is computed as:

dist(x, y) = sqrt(dot(x, x) - 2 * dot(x, y) + dot(y, y))

This formulation has two advantages over other ways of computing distances. First, it is computationally efficient when dealing with sparse data. Second, if one argument varies but the other remains unchanged, then dot(x, x) and/or dot(y, y) can be pre-computed.

However, this is not the most precise way of doing this computation, because this equation potentially suffers from “catastrophic cancellation”. Also, the distance matrix returned by this function may not be exactly symmetric as required by, e.g., scipy.spatial.distance functions.

Read more in the User Guide.

Parameters:

• X ({array-like, sparse matrix} of shape (n_samples_X, n_components)) – An array where each row is a sample and each column is a component.

• Y ({array-like, sparse matrix} of shape (n_samples_Y, n_components), ) – default=None An array where each row is a sample and each column is a component. If None, method uses Y=X.

• cell_length (array-like of shape (n_components,), default=None) –

  The side length of rectangular cell used for periodic boundary conditions. None for non-periodic boundary conditions.

  Note

  Only side lengths of rectangular cells are supported. Cell format: [side_length_1, …, side_length_n]

Returns:

distances (ndarray of shape (n_samples_X, n_samples_Y)) – Returns the distances between the row vectors of X and the row vectors of Y.

Examples

>>> import numpy as np
>>> from skmatter.metrics import periodic_pairwise_euclidean_distances
>>> X = np.array([[0, 1], [1, 1]])
>>> origin = np.array([[0, 0]])
>>> # distance between rows of X
>>> periodic_pairwise_euclidean_distances(X, X)
array([[0., 1.],
       [1., 0.]])
>>> # get distance to origin
>>> periodic_pairwise_euclidean_distances(X, origin, cell_length=[0.5, 0.7])
array([[0.3],
       [0.3]])

Pairwise Mahalanobis Distance

skmatter.metrics.pairwise_mahalanobis_distances(X: ndarray, Y: ndarray, cov_inv: ndarray, cell_length: ndarray | None = None, squared: bool = False)

Calculate the pairwise Mahalanobis distance between two arrays.

This metric is used for calculating the distances between observations from Gaussian distributions. It is defined as:

\[d_{\Sigma}(x, y)^2 = (x - y)^T \Sigma^{-1} (x - y)\]

where \(\Sigma\) is the covariance matrix, \(x\) and \(y\) are observations from the same distribution.

Parameters:

• X (numpy.ndarray of shape (n_samples_X, n_components)) – An array where each row is a sample and each column is a component.

• Y (np.ndarray of shape (n_samples_Y, n_components)) – An array where each row is a sample and each column is a component.

• cov_inv (np.ndarray) – The inverse covariance matrix of shape (n_components, n_components).

• cell_length (np.ndarray, optinal, default=None) –

  The cell size for periodic boundary conditions. None for non-periodic boundary conditions.

  Note

  Only cubic cells are supported. Cell format: [side_length_1, …, side_length_n]

• squared (bool, default=False) – Whether to return the squared distance.

Returns:

np.ndarray – The pairwise Mahalanobis distance between the two input arrays, of shape (cov_inv.shape[0], x.shape[0], y.shape[0]).

Examples

>>> import numpy as np
>>> from skmatter.metrics import pairwise_mahalanobis_distances
>>> iv = np.array([[1, 0.5, 0.5], [0.5, 1, 0.5], [0.5, 0.5, 1]])
>>> X = np.array([[1, 0, 0], [0, 2, 0], [2, 0, 0]])
>>> Y = np.array([[0, 1, 0]])
>>> pairwise_mahalanobis_distances(X, Y, iv)
array([[[1.        ],
        [1.        ],
        [1.73205081]]])