# Linear Ridge Regression¶

A linear ridge regression model can be defined as $$y_i = \bf{w}^\top\bf{x_i}$$ where $$y_i$$ is the predicted value, $$\bf{x_i}$$ is a feature vector and $$\bf{w}$$ is the weight vector. We aim to find the linear function that best explains the data, i.e. minimizes the squared loss plus a $$L_2$$ regularization term. One can show the solution can be written as:

${\bf w} = \left(\tau {\bf I}+ \sum_{i=1}^N{\bf x}_i{\bf x}_i^\top\right)^{-1}\left(\sum_{i=1}^N y_i{\bf x}_i\right)$

where $$N$$ is the number of training samples and $$\tau>0$$ scales the regularization term.

A bias term, which is the squared empirical error, can also be calculated.

For the special case when $$\tau = 0$$, a wrapper class CLeastSquaresRegression is available.

## Example¶

Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CRegressionLabels as

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = RegressionLabels(f_labels_train)
labels_test = RegressionLabels(f_labels_test)

features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = RegressionLabels(f_labels_train);
labels_test = RegressionLabels(f_labels_test);

RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
RegressionLabels labels_train = new RegressionLabels(f_labels_train);
RegressionLabels labels_test = new RegressionLabels(f_labels_test);

features_train = Shogun::RealFeatures.new f_feats_train
features_test = Shogun::RealFeatures.new f_feats_test
labels_train = Shogun::RegressionLabels.new f_labels_train
labels_test = Shogun::RegressionLabels.new f_labels_test

features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- RegressionLabels(f_labels_train)
labels_test <- RegressionLabels(f_labels_test)

features_train = shogun.RealFeatures(f_feats_train)
features_test = shogun.RealFeatures(f_feats_test)
labels_train = shogun.RegressionLabels(f_labels_train)
labels_test = shogun.RegressionLabels(f_labels_test)

RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
RegressionLabels labels_train = new RegressionLabels(f_labels_train);
RegressionLabels labels_test = new RegressionLabels(f_labels_test);

auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CRegressionLabels>(f_labels_train);
auto labels_test = some<CRegressionLabels>(f_labels_test);


We create an instance of CLinearRidgeRegression classifier, passing it $$\tau$$, training data and labels.

tau = 0.001
lrr = LinearRidgeRegression(tau, features_train, labels_train)

tau = 0.001;
lrr = LinearRidgeRegression(tau, features_train, labels_train);

double tau = 0.001;
LinearRidgeRegression lrr = new LinearRidgeRegression(tau, features_train, labels_train);

tau = 0.001
lrr = Shogun::LinearRidgeRegression.new tau, features_train, labels_train

tau <- 0.001
lrr <- LinearRidgeRegression(tau, features_train, labels_train)

tau = 0.001
lrr = shogun.LinearRidgeRegression(tau, features_train, labels_train)

double tau = 0.001;
LinearRidgeRegression lrr = new LinearRidgeRegression(tau, features_train, labels_train);

auto tau = 0.001;
auto lrr = some<CLinearRidgeRegression>(tau, features_train, labels_train);


Then we train the regression model and apply it to test data to get the predicted CRegressionLabels and bias.

lrr.train()
labels_predict = lrr.apply_regression(features_test)
bias = lrr.get_bias()

lrr.train();
labels_predict = lrr.apply_regression(features_test);
bias = lrr.get_bias();

lrr.train();
RegressionLabels labels_predict = lrr.apply_regression(features_test);
double bias = lrr.get_bias();

lrr.train
labels_predict = lrr.apply_regression features_test
bias = lrr.get_bias

lrr$train() labels_predict <- lrr$apply_regression(features_test)
bias <- lrr$get_bias()  lrr:train() labels_predict = lrr:apply_regression(features_test) bias = lrr:get_bias()  lrr.train(); RegressionLabels labels_predict = lrr.apply_regression(features_test); double bias = lrr.get_bias();  lrr->train(); auto labels_predict = lrr->apply_regression(features_test); auto bias = lrr->get_bias();  Optionally the bias can be disabled to avoid redundant computation. lrr.set_compute_bias(False)  lrr.set_compute_bias(false);  lrr.set_compute_bias(false);  lrr.set_compute_bias false  lrr$set_compute_bias(FALSE)

lrr:set_compute_bias(False)

lrr.set_compute_bias(false);

lrr->set_compute_bias(false);


Imagine, we know the bias term. We can set it as

lrr.set_bias(bias)

lrr.set_bias(bias);

lrr.set_bias(bias);

lrr.set_bias bias

lrr$set_bias(bias)  lrr:set_bias(bias)  lrr.set_bias(bias);  lrr->set_bias(bias);  After training, we can extract $${\bf w}$$. weights = lrr.get_w()  weights = lrr.get_w();  DoubleMatrix weights = lrr.get_w();  weights = lrr.get_w  weights <- lrr$get_w()

weights = lrr:get_w()

double[] weights = lrr.get_w();

auto weights = lrr->get_w();


Finally, we can evaluate the CMeanSquaredError.

eval = MeanSquaredError()
mse = eval.evaluate(labels_predict, labels_test)

eval = MeanSquaredError();
mse = eval.evaluate(labels_predict, labels_test);

MeanSquaredError eval = new MeanSquaredError();
double mse = eval.evaluate(labels_predict, labels_test);

eval = Shogun::MeanSquaredError.new
mse = eval.evaluate labels_predict, labels_test

eval <- MeanSquaredError()
mse <- eval\$evaluate(labels_predict, labels_test)

eval = shogun.MeanSquaredError()
mse = eval:evaluate(labels_predict, labels_test)

MeanSquaredError eval = new MeanSquaredError();
double mse = eval.evaluate(labels_predict, labels_test);

auto eval = some<CMeanSquaredError>();
auto mse = eval->evaluate(labels_predict, labels_test);