Regression¶
Ordinary Least Squares and Ridge Regression¶
-
class
mlpy.
RidgeRegression
(alpha=0.0)¶ Ridge Regression and Ordinary Least Squares (OLS).
Initialization.
Parameters: - alpha : float (>= 0.0)
regularization (0.0: OLS)
New in version 2.2.0.
-
beta
()¶ Return b_1, ..., b_p.
-
beta0
()¶ Return b_0.
-
learn
(x, y)¶ Compute the regression coefficients.
Parameters: - x : numpy 2d array (n x p)
matrix of regressors
- y : numpy 1d array (n)
response
-
pred
(x)¶ Compute the predicted response.
Parameters: - x : numpy 2d array (nxp)
matrix of regressors
Returns: - yp : 1d ndarray
predicted response
-
selected
()¶ Returns the regressors ranking.
Note
The predicted response is computed as:
\hat{y} = \beta_0 + X \boldsymbol\beta
Example (requires matplotlib module):
>>> import numpy as np
>>> import mlpy
>>> import matplotlib.pyplot as plt
>>> x = np.array([[1], [2], [3], [4], [5], [6]]) # p = 1
>>> y = np.array([0.13, 0.19, 0.31, 0.38, 0.49, 0.64])
>>> rr = mlpy.RidgeRegression(alpha=0.0) # OLS
>>> rr.learn(x, y)
>>> y_hat = rr.pred(x)
>>> plt.figure(1)
>>> plt.plot(x[:, 0], y, 'o') # show y
>>> plt.plot(x[:, 0], y_hat) # show y_hat
>>> plt.show()

>>> rr.beta0()
0.0046666666666667078
>>> rr.beta()
array([ 0.10057143])
Kernel Ridge Regression¶
-
class
mlpy.
KernelRidgeRegression
(kernel, alpha)¶ Ridge Regression and Ordinary Least Squares (OLS).
Initialization.
Parameters: alpha : float (> 0.0) New in version 2.2.0.
-
learn
(x, y)¶ Compute the regression coefficients.
Parameters: - x : numpy 2d array (n x p)
matrix of regressors
- y : numpy 1d array (n)
response
-
pred
(x)¶ Compute the predicted response.
Parameters: - x : numpy 2d array (n x p)
matrix of regressors
Returns: - yp : 1d ndarray
predicted response
-
Example (requires matplotlib module):
>>> import numpy as np
>>> import mlpy
>>> import matplotlib.pyplot as plt
>>> x = np.array([[1], [2], [3], [4], [5], [6]]) # p = 1
>>> y = np.array([0.13, 0.19, 0.31, 0.38, 0.49, 0.64])
>>> kernel = mlpy.KernelGaussian(sigma=0.01)
>>> krr = mlpy.KernelRidgeRegression(kernel=kernel, alpha=0.01)
>>> krr.learn(x,y)
>>> y_hat = krr.pred(x)
>>> plt.figure(1)
>>> plt.plot(x[:, 0], y, 'o') # show y
>>> plt.plot(x[:, 0], y_hat) # show y_hat
>>> plt.show()

Least Angle Regression (LAR)¶
Least Angle Regression is described in [Efron04].
Covariates should be standardized to have mean 0 and unit length, and the response should have mean 0:
\sum_{i=1}^n{x_{ij}} = 0, \hspace{1cm} \sum_{i=1}^n{x_{ij}^2} = 1, \hspace{1cm} \sum_{i=1}^n{y_i} = 0 \hspace{1cm} \mathrm{for} \hspace{0.2cm} j = 1, 2, \dots, p.
-
class
mlpy.
Lar
(m=None)¶ LAR.
Initialization.
Parameters: - m : int (> 0)
max number of steps (= number of features selected). If m=None -> m=x.shape[1] in .learn(x, y)
New in version 2.2.0.
-
beta
()¶ Return b_1, ..., b_p.
-
learn
(x, y)¶ Compute the regression coefficients.
Parameters: - x : numpy 2d array (nxp)
matrix of regressors
- y : numpy 1d array (n)
response
-
pred
(x)¶ Compute the predicted response.
Parameters: - x : numpy 2d array (nxp)
matrix of regressors
Returns: - yp : 1d ndarray
predicted response
-
selected
()¶ Returns the regressors ranking.
-
steps
()¶ Return the number of steps really performed.
LASSO (LARS implementation)¶
It implements simple modifications of the LARS algorithm that produces Lasso estimates. See [Efron04] and [Tibshirani96].
Covariates should be standardized to have mean 0 and unit length, and the response should have mean 0:
\sum_{i=1}^n{x_{ij}} = 0, \hspace{1cm} \sum_{i=1}^n{x_{ij}^2} = 1, \hspace{1cm} \sum_{i=1}^n{y_i} = 0 \hspace{1cm} \mathrm{for} \hspace{0.2cm} j = 1, 2, \dots, p.
-
class
mlpy.
Lasso
(m)¶ LASSO computed with LARS algoritm.
Initialization.
Parameters: - m : int (> 0)
max number of steps.
New in version 2.2.0.
-
beta
()¶ Return b_1, ..., b_p.
-
learn
(x, y)¶ Compute the regression coefficients.
Parameters: - x : numpy 2d array (nxp)
matrix of regressors
- y : numpy 1d array (n)
response
-
pred
(x)¶ Compute the predicted response.
Parameters: - x : numpy 2d array (nxp)
matrix of regressors
Returns: - yp : 1d ndarray
predicted response
-
selected
()¶ Returns the regressors ranking.
-
steps
()¶ Return the number of steps really performed.
Gradient Descent¶
-
class
mlpy.
GradientDescent
(kernel, t, stepsize)¶ Gradient Descent Method
Initialization.
Parameters: - kernel: kernel object
kernel
- t : int (> 0)
number of iterations
- stepsize: float
step size
New in version 2.2.0.
-
learn
(x, y)¶ Compute the regression coefficients.
Parameters: - x : numpy 2d array (n x p)
matrix of regressors
- y : numpy 1d array (n)
response
-
pred
(x)¶ Compute the predicted response.
Parameters: - x : numpy 2d array (n x p)
matrix of regressors
Returns: - yp : 1d ndarray
predicted response
[Efron04] | (1, 2) Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani. Least Angle Regression. Annals of Statistics, 2004, volume 32, pages 407-499. |
[Tibshirani96] | Robert Tibshirani. Regression shrinkage and selection via the lasso. J. Royal. Statist. Soc B., 1996, volume 58, number 1, pages 267-288. |