Gaussian Mixture Models

A Gaussian mixture model is a probabilistic model that assumes that data are generated from a finite mixture of Gaussians with unknown parameters. The model likelihood can be written as:

\[p(x|\theta) = \sum_{i=1}^{K}{\pi_i \mathcal{N}(x|\mu_i, \Sigma_i)}\]

where \(p(x|\theta)\) is probability distribution given \(\theta:=\{\pi_i, \mu_i, \Sigma_i\}_{i=1}^K\), \(K\) denotes number of mixture components, \(\pi_i\) denotes weight for \(i\)-th component, \(\mathcal{N}\) denotes a multivariate normal distribution with mean vector \(\mu_i\) and covariance matrix \(\Sigma_i\).

The expectation maximization (EM) algorithm is used to learn parameters of the model, via finding a local maximum of a lower bound on the likelihood.

See Chapter 20 in [Bar12] for a detailed introduction.

Example

We start by creating CDenseFeatures (here 64 bit floats aka RealFeatures) as

features_train = RealFeatures(f_feats_train)
features_train = RealFeatures(f_feats_train);
RealFeatures features_train = new RealFeatures(f_feats_train);
features_train = Shogun::RealFeatures.new f_feats_train
features_train <- RealFeatures(f_feats_train)
features_train = shogun.RealFeatures(f_feats_train)
RealFeatures features_train = new RealFeatures(f_feats_train);
auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);

We initialize CGMM, passing the desired number of mixture components.

num_components = 3
gmm = GMM(num_components)
num_components = 3;
gmm = GMM(num_components);
int num_components = 3;
GMM gmm = new GMM(num_components);
num_components = 3
gmm = Shogun::GMM.new num_components
num_components <- 3
gmm <- GMM(num_components)
num_components = 3
gmm = shogun.GMM(num_components)
int num_components = 3;
GMM gmm = new GMM(num_components);
auto num_components = 3;
auto gmm = some<CGMM>(num_components);

We provide training features to the CGMM object, train it by using EM algorithm and sample data-points from the trained model.

gmm.set_features(features_train)
gmm.train_em()
output = gmm.sample()
gmm.set_features(features_train);
gmm.train_em();
output = gmm.sample();
gmm.set_features(features_train);
gmm.train_em();
DoubleMatrix output = gmm.sample();
gmm.set_features features_train
gmm.train_em 
output = gmm.sample 
gmm$set_features(features_train)
gmm$train_em()
output <- gmm$sample()
gmm:set_features(features_train)
gmm:train_em()
output = gmm:sample()
gmm.set_features(features_train);
gmm.train_em();
double[] output = gmm.sample();
gmm->set_features(features_train);
gmm->train_em();
auto output = gmm->sample();

We extract parameters like \(\pi\), \(\mu_i\) and \(\Sigma_i\) for any componenet from the trained model.

component_num = 1
nth_mean = gmm.get_nth_mean(component_num)
nth_cov = gmm.get_nth_cov(component_num)
coef = gmm.get_coef()
component_num = 1;
nth_mean = gmm.get_nth_mean(component_num);
nth_cov = gmm.get_nth_cov(component_num);
coef = gmm.get_coef();
int component_num = 1;
DoubleMatrix nth_mean = gmm.get_nth_mean(component_num);
DoubleMatrix nth_cov = gmm.get_nth_cov(component_num);
DoubleMatrix coef = gmm.get_coef();
component_num = 1
nth_mean = gmm.get_nth_mean component_num
nth_cov = gmm.get_nth_cov component_num
coef = gmm.get_coef 
component_num <- 1
nth_mean <- gmm$get_nth_mean(component_num)
nth_cov <- gmm$get_nth_cov(component_num)
coef <- gmm$get_coef()
component_num = 1
nth_mean = gmm:get_nth_mean(component_num)
nth_cov = gmm:get_nth_cov(component_num)
coef = gmm:get_coef()
int component_num = 1;
double[] nth_mean = gmm.get_nth_mean(component_num);
double[,] nth_cov = gmm.get_nth_cov(component_num);
double[] coef = gmm.get_coef();
auto component_num = 1;
auto nth_mean = gmm->get_nth_mean(component_num);
auto nth_cov = gmm->get_nth_cov(component_num);
auto coef = gmm->get_coef();

We obtain log likelihood of belonging to clusters and being generated by this model.

log_likelihoods = gmm.cluster(nth_mean)
log_likelihoods = gmm.cluster(nth_mean);
DoubleMatrix log_likelihoods = gmm.cluster(nth_mean);
log_likelihoods = gmm.cluster nth_mean
log_likelihoods <- gmm$cluster(nth_mean)
log_likelihoods = gmm:cluster(nth_mean)
double[] log_likelihoods = gmm.cluster(nth_mean);
auto log_likelihoods = gmm->cluster(nth_mean);

We can also use Split-Merge Expectation-Maximization algorithm [UNGH00] for training.

gmm.train_smem()
gmm.train_smem();
gmm.train_smem();
gmm.train_smem 
gmm$train_smem()
gmm:train_smem()
gmm.train_smem();
gmm->train_smem();

References

Wikipedia: Mixture_model

Wikipedia: Expectation–maximization_algorithm

Bar12

D. Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.

UNGH00

N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton. Smem algorithm for mixture models. Neural Computation, 12(9):2109–2128, 2000.