Gaussian Process Classifier

Application of Gaussian processes in binary and multi-class classification. See Gaussian process regression cookbook and [RW06] for more information on Gaussian processes.

Example

Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and CMulticlassLabels as

features_train = RealFeatures(f_feats_train)
features_test = RealFeatures(f_feats_test)
labels_train = MulticlassLabels(f_labels_train)
labels_test = MulticlassLabels(f_labels_test)
features_train = RealFeatures(f_feats_train);
features_test = RealFeatures(f_feats_test);
labels_train = MulticlassLabels(f_labels_train);
labels_test = MulticlassLabels(f_labels_test);
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
MulticlassLabels labels_train = new MulticlassLabels(f_labels_train);
MulticlassLabels labels_test = new MulticlassLabels(f_labels_test);
features_train = Shogun::RealFeatures.new f_feats_train
features_test = Shogun::RealFeatures.new f_feats_test
labels_train = Shogun::MulticlassLabels.new f_labels_train
labels_test = Shogun::MulticlassLabels.new f_labels_test
features_train <- RealFeatures(f_feats_train)
features_test <- RealFeatures(f_feats_test)
labels_train <- MulticlassLabels(f_labels_train)
labels_test <- MulticlassLabels(f_labels_test)
features_train = shogun.RealFeatures(f_feats_train)
features_test = shogun.RealFeatures(f_feats_test)
labels_train = shogun.MulticlassLabels(f_labels_train)
labels_test = shogun.MulticlassLabels(f_labels_test)
RealFeatures features_train = new RealFeatures(f_feats_train);
RealFeatures features_test = new RealFeatures(f_feats_test);
MulticlassLabels labels_train = new MulticlassLabels(f_labels_train);
MulticlassLabels labels_test = new MulticlassLabels(f_labels_test);
auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);
auto features_test = some<CDenseFeatures<float64_t>>(f_feats_test);
auto labels_train = some<CMulticlassLabels>(f_labels_train);
auto labels_test = some<CMulticlassLabels>(f_labels_test);

To fit the input (training) data \(\mathbf{X}\), we have to choose appropriate CMeanFunction and CKernel. Here we use a basic CConstMean and a CGaussianKernel with chosen width parameter.

kernel = GaussianKernel(2.0)
mean_function = ConstMean()
kernel = GaussianKernel(2.0);
mean_function = ConstMean();
GaussianKernel kernel = new GaussianKernel(2.0);
ConstMean mean_function = new ConstMean();
kernel = Shogun::GaussianKernel.new 2.0
mean_function = Shogun::ConstMean.new 
kernel <- GaussianKernel(2.0)
mean_function <- ConstMean()
kernel = shogun.GaussianKernel(2.0)
mean_function = shogun.ConstMean()
GaussianKernel kernel = new GaussianKernel(2.0);
ConstMean mean_function = new ConstMean();
auto kernel = some<CGaussianKernel>(2.0);
auto mean_function = some<CConstMean>();

We need to specify the inference method to find the posterior distribution of the function values \(\mathbf{f}\). Here we choose to perform Laplace approximation inference method with an instance of CMultiLaplaceInferenceMethod (See Chapter 18.2 in [Bar12] for a detailed introduction) and pass it the chosen kernel, the training features, the mean function, the labels and an instance of CSoftMaxLikelihood, to specify the distribution of the targets/labels as above. Finally we create an instance of the CGaussianProcessClassification classifier.

gauss_likelihood = SoftMaxLikelihood()
inference_method = MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood)
gp_classifier = GaussianProcessClassification(inference_method)
gauss_likelihood = SoftMaxLikelihood();
inference_method = MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood);
gp_classifier = GaussianProcessClassification(inference_method);
SoftMaxLikelihood gauss_likelihood = new SoftMaxLikelihood();
MultiLaplaceInferenceMethod inference_method = new MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood);
GaussianProcessClassification gp_classifier = new GaussianProcessClassification(inference_method);
gauss_likelihood = Shogun::SoftMaxLikelihood.new 
inference_method = Shogun::MultiLaplaceInferenceMethod.new kernel, features_train, mean_function, labels_train, gauss_likelihood
gp_classifier = Shogun::GaussianProcessClassification.new inference_method
gauss_likelihood <- SoftMaxLikelihood()
inference_method <- MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood)
gp_classifier <- GaussianProcessClassification(inference_method)
gauss_likelihood = shogun.SoftMaxLikelihood()
inference_method = shogun.MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood)
gp_classifier = shogun.GaussianProcessClassification(inference_method)
SoftMaxLikelihood gauss_likelihood = new SoftMaxLikelihood();
MultiLaplaceInferenceMethod inference_method = new MultiLaplaceInferenceMethod(kernel, features_train, mean_function, labels_train, gauss_likelihood);
GaussianProcessClassification gp_classifier = new GaussianProcessClassification(inference_method);
auto gauss_likelihood = some<CSoftMaxLikelihood>();
auto inference_method = some<CMultiLaplaceInferenceMethod>(kernel, features_train, mean_function, labels_train, gauss_likelihood);
auto gp_classifier = some<CGaussianProcessClassification>(inference_method);

Then we can train the model and evaluate the predictive distribution. We get predicted CMulticlassLabels.

gp_classifier.train()
labels_predict = gp_classifier.apply_multiclass(features_test)
gp_classifier.train();
labels_predict = gp_classifier.apply_multiclass(features_test);
gp_classifier.train();
MulticlassLabels labels_predict = gp_classifier.apply_multiclass(features_test);
gp_classifier.train 
labels_predict = gp_classifier.apply_multiclass features_test
gp_classifier$train()
labels_predict <- gp_classifier$apply_multiclass(features_test)
gp_classifier:train()
labels_predict = gp_classifier:apply_multiclass(features_test)
gp_classifier.train();
MulticlassLabels labels_predict = gp_classifier.apply_multiclass(features_test);
gp_classifier->train();
auto labels_predict = gp_classifier->apply_multiclass(features_test);

We can extract the probabilities:

prob = gp_classifier.get_probabilities(features_train)
prob = gp_classifier.get_probabilities(features_train);
DoubleMatrix prob = gp_classifier.get_probabilities(features_train);
prob = gp_classifier.get_probabilities features_train
prob <- gp_classifier$get_probabilities(features_train)
prob = gp_classifier:get_probabilities(features_train)
double[] prob = gp_classifier.get_probabilities(features_train);
auto prob = gp_classifier->get_probabilities(features_train);

We can evaluate test performance via e.g. CMulticlassAccuracy.

evals = MulticlassAccuracy()
accuracy = evals.evaluate(labels_predict, labels_test)
evals = MulticlassAccuracy();
accuracy = evals.evaluate(labels_predict, labels_test);
MulticlassAccuracy evals = new MulticlassAccuracy();
double accuracy = evals.evaluate(labels_predict, labels_test);
evals = Shogun::MulticlassAccuracy.new 
accuracy = evals.evaluate labels_predict, labels_test
evals <- MulticlassAccuracy()
accuracy <- evals$evaluate(labels_predict, labels_test)
evals = shogun.MulticlassAccuracy()
accuracy = evals:evaluate(labels_predict, labels_test)
MulticlassAccuracy evals = new MulticlassAccuracy();
double accuracy = evals.evaluate(labels_predict, labels_test);
auto evals = some<CMulticlassAccuracy>();
auto accuracy = evals->evaluate(labels_predict, labels_test);

References

Wikipedia: Gaussian_process

Bar12

D. Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.

RW06

C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.