Linear Models

Orthogonal Regression

class skmatter.linear_model.OrthogonalRegression(use_orthogonal_projector=True, linear_estimator=None)

Orthogonal regression by solving the Procrustes problem

Linear regression with the additional constraint that the weight matrix must be an orthogonal matrix/projection. It minimizes the Procrustes problem:

\[\min_\Omega ||y - X\Omega\||_F \quad\mathrm{subject\ to}\quad \Omega^T\Omega=I\]

Parameters:

• use_orthogonal_projector (bool, default=True) –

  Controls if orthogonal projectors are used to predict y fitting on X. If this parameter is set to False X and y are padded with zeros to the larger number of features of X and y. The projection method is similar to the procedure in the computation GFRD in the first version of Ref. [Goscinski2021]. The method has been adapted obtain a full weight matrix.

  The projection can introduce nonanalytic behavior with respect to changes in dimensions of X for cases where X n_features > y n_targets. See examples/OrthogonalRegressionNonAnalytic_no-doc.ipynb

• linear_estimator (object implementing fit/predict, default=None) – The linear estimator is used when use_orthogonal_projector is set to True, to compute the projection matrix

max_components_

The source X and target y are padded with zeros to match in feature/target dimension, when use_orthogonal_projector is set to False. This attribute is set to the maximum of the feature and target dimension.

Type:

int

coef_

Weight matrix. The shape (max_components, max_components) is used if use_orthogonal_projector is set to False.

Type:

numpy.ndarray of shape (n_features,) or (n_targets, n_features) or (max_components, max_components)

Ridge Regression with Two-fold Cross Validation

class skmatter.linear_model.Ridge2FoldCV(alphas=(0.1, 1.0, 10.0), alpha_type='absolute', regularization_method='tikhonov', cv=None, scoring=None, random_state=None, shuffle=True, n_jobs=None)

Ridge regression with an efficient 2-fold cross-validation method using the SVD solver.

Minimizes the objective function:

\[\|y - Xw\|^2_2 + \alpha \|w\|^2_2,\]

while the alpha value is determined with a 2-fold cross-validation from a list of alpha values. It is more efficient version than doing 2-fold cross-validation naively The algorithmic trick is to reuse the matrices obtained by SVD for each regularization paramater :param alpha: The 2-fold CV can be broken down to

\[\begin{split}\begin{align} &\mathbf{X}_1 = \mathbf{U}_1\mathbf{S}_1\mathbf{V}_1^T, \qquad\qquad\qquad\quad \textrm{feature matrix }\mathbf{X}\textrm{ for fold 1} \\ &\mathbf{W}_1(\lambda) = \mathbf{V}_1 \tilde{\mathbf{S}}_1(\lambda)^{-1} \mathbf{U}_1^T y_1, \qquad \textrm{weight matrix fitted on fold 1}\\ &\tilde{y}_2 = \mathbf{X}_2 \mathbf{W}_1, \qquad\qquad\qquad\qquad \textrm{ prediction of } y\textrm{ for fold 2} \end{align}\end{split}\]

where the matrices

\[\begin{align} &\mathbf{A}_1 = \mathbf{X}_2 \mathbf{V}_1, \quad \mathbf{B}_1 = \mathbf{U}_1^T y_1. \end{align}\]

are stored to not recompute the SVD.

It offers additional functionalities in comparison to sklearn.linear_model.RidgeCV: The regularization parameters can be chosen relative to the largest eigenvalue of the feature matrix using :param alpha_type: as well as type of regularization using :param regularization_method:. Details are explained in the Parameters section.

It does not offer :param fit_intercept: as sklearn linear models do. It only can fit with no intercept.

Parameters:

• alphas (numpy.ndarray of shape (n_alphas,), default=(0.1, 1.0, 10.0)) – Array of alpha values to try. Regularization strength; must be a positive float. Regularization improves the conditioning of the problem and reduces the variance of the estimates. Larger values specify stronger regularization.

• alpha_type (str, default="absolute") – “absolute” assumes that alphas are absolute values and does not scale them, “relative” assumes that alphas as relative values in the range [0,1] and scales them with respect to the largest eigenvalue.

• regularization_method (str, default="tikhonov") – “tikhonov” uses the alphas with standard Tihhonov regularization, “cutoff” uses the alphas as a cutoff for the eigenvalues of the kernel, The case "cutoff" with "relative" ``alphas has the same effect as the rcond parameter in e.g. numpy.linalg.lstsq. Be aware that for every case we always apply a small default cutoff dependend on the numerical accuracy of the data type of X in the fitting function.

• cv (cross-validation generator or an iterable, default=None) – The first yield of the generator is used do determine the two folds. If None, a 0.5 split of the two folds is used using the arguments :param shuffle: and :param random_state:

• shuffle (bool, default=True) – Whether or not to shuffle the data before splitting. If :param cv: is not None, this parameter is ignored.

• random_state (int or numpy.random.`RandomState instance, default=None) – Controls the shuffling applied to the data before applying the split. Pass an int for reproducible output across multiple function calls. See random_state glossary from sklearn (external link) parameter is ignored. If :param cv: is not None, this parameter is ignored.

• scoring (str, callable, default=None) – A string (see model evaluation documentation) or a scorer callable object / function with signature scorer(estimator, X, y). If None, the negative mean squared error is used.

• n_jobs (int, default=None) – The number of CPUs to use to do the computation. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See n_jobs glossary from sklearn (external link) for more details.

cv_values_

2-fold cross-validation values for each alpha. After fit() has been called, this attribute will contain the values out of score function

Type:

numpy.ndarray of shape (n_alphas)

coef_

Weight vector(s).

Type:

numpy.ndarray of shape (n_features) or (n_targets, n_features)

alpha_

Estimated regularization parameter.

Type:

float

best_score_

Score of base estimator with best alpha.

Type:

float

PCovR

Principal Covariates Regression is a linear model, see PCovR.