# Multi-class Logistic Regression¶

Multinomial logistic regression assigns the sample $$\mathbf{x}_i$$ to class $$c$$ based on the probability for sample $$\mathbf{x}_i$$ to be in class $$c$$:

$P(Y_i = c | \mathbf{x}_i) = \frac{\exp(\mathbf{\theta}^\top_c\mathbf{x}_i)}{1+ \sum_{k=1}^{K}\exp(\mathbf{\theta}^\top_k\mathbf{x}_i)}$

in which $$K$$ is the number of classes.

The loss function that needs to be minimized is:

${\min_{\mathbf{\theta}}}\sum_{k=1}^{K}\sum_{i=1}^{m}w_{ik}\log(1+\exp(-y_{ik}(\mathbf{x}_k^\top\mathbf{a}_{ik} + c_k))) + \lambda\left \| \mathbf{x} \right \|_{l_1/l_q}$

where $$\mathbf{a}_{ik}$$ denotes the $$i$$-th sample for the $$k$$-th class, $$w_{ik}$$ is the weight for $$\mathbf{a}_{ik}^\top$$, $$y_{ik}$$ is the response of $$\mathbf{a}_{ik}$$, and $$c_k$$ is the intercept (scalar) for the $$k$$-th class. $$\lambda$$ is the $$l_1/l_q$$-norm regularization parameter.

## Example¶

Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CMulticlassLabels as

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = MulticlassLabels(f_labels_train)
labels_test = MulticlassLabels(f_labels_test)

features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = MulticlassLabels(f_labels_train);
labels_test = MulticlassLabels(f_labels_test);

RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
MulticlassLabels labels_train = new MulticlassLabels(f_labels_train);
MulticlassLabels labels_test = new MulticlassLabels(f_labels_test);

features_train = Shogun::RealFeatures.new f_feats_train
features_test = Shogun::RealFeatures.new f_feats_test
labels_train = Shogun::MulticlassLabels.new f_labels_train
labels_test = Shogun::MulticlassLabels.new f_labels_test

features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- MulticlassLabels(f_labels_train)
labels_test <- MulticlassLabels(f_labels_test)

features_train = shogun.RealFeatures(f_feats_train)
features_test = shogun.RealFeatures(f_feats_test)
labels_train = shogun.MulticlassLabels(f_labels_train)
labels_test = shogun.MulticlassLabels(f_labels_test)

RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
MulticlassLabels labels_train = new MulticlassLabels(f_labels_train);
MulticlassLabels labels_test = new MulticlassLabels(f_labels_test);

auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CMulticlassLabels>(f_labels_train);
auto labels_test = some<CMulticlassLabels>(f_labels_test);


We create an instance of the CMulticlassLogisticRegression classifier by passing it the dataset, lables, and specifying the regularization constant $$\lambda$$ for each machine

classifier = MulticlassLogisticRegression(1, features_train, labels_train)

classifier = MulticlassLogisticRegression(1, features_train, labels_train);

MulticlassLogisticRegression classifier = new MulticlassLogisticRegression(1, features_train, labels_train);

classifier = Shogun::MulticlassLogisticRegression.new 1, features_train, labels_train

classifier <- MulticlassLogisticRegression(1, features_train, labels_train)

classifier = shogun.MulticlassLogisticRegression(1, features_train, labels_train)

MulticlassLogisticRegression classifier = new MulticlassLogisticRegression(1, features_train, labels_train);

auto classifier = some<CMulticlassLogisticRegression>(1, features_train, labels_train);


Then we train and apply it to test data, which here gives CMulticlassLabels.

classifier.train()
labels_predict = classifier.apply_multiclass(features_test)

classifier.train();
labels_predict = classifier.apply_multiclass(features_test);

classifier.train();
MulticlassLabels labels_predict = classifier.apply_multiclass(features_test);

classifier.train
labels_predict = classifier.apply_multiclass features_test

classifier$train() labels_predict <- classifier$apply_multiclass(features_test)

classifier:train()
labels_predict = classifier:apply_multiclass(features_test)

classifier.train();
MulticlassLabels labels_predict = classifier.apply_multiclass(features_test);

classifier->train();
auto labels_predict = classifier->apply_multiclass(features_test);


We can evaluate test performance via e.g. CMulticlassAccuracy.

eval = MulticlassAccuracy()
accuracy = eval.evaluate(labels_predict, labels_test)

eval = MulticlassAccuracy();
accuracy = eval.evaluate(labels_predict, labels_test);

MulticlassAccuracy eval = new MulticlassAccuracy();
double accuracy = eval.evaluate(labels_predict, labels_test);

eval = Shogun::MulticlassAccuracy.new
accuracy = eval.evaluate labels_predict, labels_test

eval <- MulticlassAccuracy()
accuracy <- eval\$evaluate(labels_predict, labels_test)

eval = shogun.MulticlassAccuracy()
accuracy = eval:evaluate(labels_predict, labels_test)

MulticlassAccuracy eval = new MulticlassAccuracy();
double accuracy = eval.evaluate(labels_predict, labels_test);

auto eval = some<CMulticlassAccuracy>();
auto accuracy = eval->evaluate(labels_predict, labels_test);