# Linear Discriminant Analysis¶

LDA learns a linear classifier via finding a projection matrix that maximally discriminates the provided classes. The learned linear classification rule is optimal under the assumption that both classes a gaussian distributed with equal co-variance. To find a linear separation \({\bf w}\) in training, the in-between class variance is maximized and the within class variance is minimized. The projection matrix is computed by maximizing the following objective:

where \({\bf S_B}\) is between class scatter matrix and \({\bf S_W}\) is within class scatter matrix. The above derivation of LDA requires the invertibility of the within class matrix. This condition however, is violated when there are fewer data-points than dimensions. In this case SVD is used to compute projection matrix using an orthonormal basis \({\bf Q}\)

See Chapter 16 in [Bar12] for a detailed introduction.

## Example¶

We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CBinaryLabels from files with training and test data.

```
features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = BinaryLabels(f_labels_train)
labels_test = BinaryLabels(f_labels_test)
```

```
features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = BinaryLabels(f_labels_train);
labels_test = BinaryLabels(f_labels_test);
```

```
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
BinaryLabels labels_train = new BinaryLabels(f_labels_train);
BinaryLabels labels_test = new BinaryLabels(f_labels_test);
```

```
features_train = Shogun::RealFeatures.new f_feats_train
features_test = Shogun::RealFeatures.new f_feats_test
labels_train = Shogun::BinaryLabels.new f_labels_train
labels_test = Shogun::BinaryLabels.new f_labels_test
```

```
features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- BinaryLabels(f_labels_train)
labels_test <- BinaryLabels(f_labels_test)
```

```
features_train = shogun.RealFeatures(f_feats_train)
features_test = shogun.RealFeatures(f_feats_test)
labels_train = shogun.BinaryLabels(f_labels_train)
labels_test = shogun.BinaryLabels(f_labels_test)
```

```
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
BinaryLabels labels_train = new BinaryLabels(f_labels_train);
BinaryLabels labels_test = new BinaryLabels(f_labels_test);
```

```
auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CBinaryLabels>(f_labels_train);
auto labels_test = some<CBinaryLabels>(f_labels_test);
```

We create an instance of the CLDA classifier and set features and labels. By default, Shogun automatically chooses the decomposition method based on \({N<=D}\) or \({N>D}\).

```
lda = LDA()
lda.set_features(features_train)
lda.set_labels(labels_train)
```

```
lda = LDA();
lda.set_features(features_train);
lda.set_labels(labels_train);
```

```
LDA lda = new LDA();
lda.set_features(features_train);
lda.set_labels(labels_train);
```

```
lda = Shogun::LDA.new
lda.set_features features_train
lda.set_labels labels_train
```

```
lda <- LDA()
lda$set_features(features_train)
lda$set_labels(labels_train)
```

```
lda = shogun.LDA()
lda:set_features(features_train)
lda:set_labels(labels_train)
```

```
LDA lda = new LDA();
lda.set_features(features_train);
lda.set_labels(labels_train);
```

```
auto lda = some<CLDA>();
lda->set_features(features_train);
lda->set_labels(labels_train);
```

Then we train and apply it to test data, which here gives CBinaryLabels.

```
lda.train()
labels_predict = lda.apply_binary(features_test)
```

```
lda.train();
labels_predict = lda.apply_binary(features_test);
```

```
lda.train();
BinaryLabels labels_predict = lda.apply_binary(features_test);
```

```
lda.train
labels_predict = lda.apply_binary features_test
```

```
lda$train()
labels_predict <- lda$apply_binary(features_test)
```

```
lda:train()
labels_predict = lda:apply_binary(features_test)
```

```
lda.train();
BinaryLabels labels_predict = lda.apply_binary(features_test);
```

```
lda->train();
auto labels_predict = lda->apply_binary(features_test);
```

We can extract weights \({\bf w}\).

```
w = lda.get_w()
```

```
w = lda.get_w();
```

```
DoubleMatrix w = lda.get_w();
```

```
w = lda.get_w
```

```
w <- lda$get_w()
```

```
w = lda:get_w()
```

```
double[] w = lda.get_w();
```

```
auto w = lda->get_w();
```

We can evaluate test performance via e.g. CAccuracyMeasure.

```
eval = AccuracyMeasure()
accuracy = eval.evaluate(labels_predict, labels_test)
```

```
eval = AccuracyMeasure();
accuracy = eval.evaluate(labels_predict, labels_test);
```

```
AccuracyMeasure eval = new AccuracyMeasure();
double accuracy = eval.evaluate(labels_predict, labels_test);
```

```
eval = Shogun::AccuracyMeasure.new
accuracy = eval.evaluate labels_predict, labels_test
```

```
eval <- AccuracyMeasure()
accuracy <- eval$evaluate(labels_predict, labels_test)
```

```
eval = shogun.AccuracyMeasure()
accuracy = eval:evaluate(labels_predict, labels_test)
```

```
AccuracyMeasure eval = new AccuracyMeasure();
double accuracy = eval.evaluate(labels_predict, labels_test);
```

```
auto eval = some<CAccuracyMeasure>();
auto accuracy = eval->evaluate(labels_predict, labels_test);
```

## References¶

Wikipedia: Linear_discriminant_analysis

- Bar12
D. Barber.

*Bayesian reasoning and machine learning*. Cambridge University Press, 2012.