March 30, 2015 03:24 PM

MLOSS workshop at ICML 2015: Open Ecosystems

MLOSS workshops are returning to ICML this summer!

Key dates:

  • Submission DL 28 April 2015
  • Workshop date 10 July 2015

We (Gaël Varoquaux, Cheng Soon Ong and Antti Honkela) are organising another MLOSS workshop at ICML 2015 in Lille, France this July. The theme for this edition is "Open Ecosystems" whereby we wish to invoke discussion on benefits (or drawbacks?) of multiple tools in the same ecosystem. Our invited speakers (John Myles White and Matthew Rocklin) will share some of their experiences on Julia and Python, and we would be happy to hear from others either on the same or different ecosystems through contributed talks. Usual demonstrations of new great software are naturally also welcome!

In addition to the talks, we have planned two more active sessions:

  • an open discussion with themes voted by workshop participants similar to MLOSS 2013; and
  • a hackathon for planning and starting to develop infrastructure for measuring software impact.

If you have any comments or suggestions regarding these, please add a comment here or email the organisers!

More details at the workshop website at

by Antti Honkela at March 30, 2015 03:24 PM

January 30, 2015 07:30 PM

Prepositions and Arguments

I had some crazy idea how to make C++ code more readable using prepositions. It would take some discipline to use and probably serious industrial people would consider me mad.

January 30, 2015 07:30 PM

January 28, 2015 10:02 PM

Mailcheck.js in Production

Mailcheck.js in Production

I was recently tasked with adding Mailcheck.js to some of our production pages and I want to describe a bit of the process I went through because I did some things a bit differently and had some fun along the way.

Lets start with a PSA – do not simply drop Mailcheck onto your website as is! In my opinion / findings the default algorithm is way too greedy – aka it will mostly suggest all emails should be It is worth taking the time to tweak mailcheck for your particular userbase, on one wants to see a correction for their proper email address!

The first thing I did was dumped a ton of emails from our database to create a dataset to work with. I could have used Node to write some scripts to test out the Mailcheck behaviour but Python is just so much more convient for doing numerical analysis. Plus its what our data team uses so I could leverage some of their knowledge and code. So now for the fun part – I ended up using PyV8 (a python wrapper for calling out to Google’s V8 javascript engine). With this setup I was able to slice and dice through our production emails using python and pandas calling the exact javascript mailcheck algorithm and collecting my results. After tweaking the algorithm I could take the settings and new js code and put it in production.

Check out this wacky franken script that got the job done (pandas not included):

import PyV8

def init_mailcheck():
  global ctxt
  ctxt = PyV8.JSContext()

def run_sift3Distance(s1,s2):
  script = "Mailcheck.mailcheck.sift3Distance('%s','%s')" %(s1,s2)
  return ctxt.eval(script)

def run_splitEmail(email):
  script = "Mailcheck.mailcheck.splitEmail('%s')" %(email)
  return ctxt.eval(script)

def run_mailcheck(email):
  script = """{
                 email: "%s",
           """ % (email)
  result =  ctxt.eval(script)
  if result:
      result = result.address + '@' + result.domain

  return result

if __name__=="__main__":
  print run_mailcheck("")
  # >>>

by kevinhughes27 at January 28, 2015 10:02 PM

January 25, 2015 09:56 PM

pyIPCA a python library for Incremental PCA

I extracted some of the useful code and nifty examples from the background of my Thesis as a python library for your enjoyment. PCA or Principal Component Analysis is a pretty common data analysis technique, incremental PCA lets you perform the same type of analysis but uses the input data one sample at a time rather than all at once.

The code fully conforms to the scikit-learn api and you should be able to easily use it anywhere you are currently using one of the sklearn.decomposition classes. In fact this library is sort of on the waiting list for sklearn:

IPCA on 2D point cloud shaped like an ellipse

IPCA on 2D point cloud shaped like an ellipse

Check it out if you’re interested and holla at sklearn if you want this feature!

by kevinhughes27 at January 25, 2015 09:56 PM

January 25, 2015 09:55 PM

Do you use your inbox as a running ToDo list? Do you send yourself emails so you remember to do certain things? Do you spend most of your day in a terminal? If you answered yes to all 3 of those questions then you might be interested in this nifty little ruby gem  I just released. Its called gmail_todo and its made for quickly emailing yourself a ToDo note from the command line, think of the precious seconds you’ll save! Now when you remember something you need to do and you are in a terminal rather than alt-tabbing (or god forbid reaching for the mouse) you can quickly type ‘todo “get milk”‘ and voila an email will appear in your inbox. As an added bonus my gem prepends “[ToDo]” to your subject for easy filtering!

Check it out on Github and RubyGems

by kevinhughes27 at January 25, 2015 09:55 PM

January 25, 2015 09:55 PM

A few days ago I launched a free app for Shopify that automates the process of sending your customers tax receipts for their donations to your non-profit Shopify site. Its a pretty cool little app – I think it is a really good example of how to build a simple piece of automation using webhooks. I fully open sourced the code here: so have a look!

by kevinhughes27 at January 25, 2015 09:55 PM

December 06, 2014 06:47 PM

Apipie example recording with Minitest

I’ve been using the excellent apipie gem on a new project I’m working on and I wanted to share my solution for example recording with Minitest. The gem documentation has an example of how to specify which functional tests should be shown as an Api example if you happen to be using Rspec but nothing for Minitest – what to do?

If you read the source of the example recorder it’s actually pretty smart – it loads the apipie_examples.json file of the previously recorded examples and only adds new ones or updates existing examples. With this knowledge I came up with an approach – I added a special string #apidoc to the test name of the tests I want included in my Api docs. Then I made this Rake task to run only my Api controller tests and filter which tests using the string:

task :api_examples do
  Dir["#{Rails.root}/test/controllers/api/*controller_test.rb"].each do |controller|
    pipe = IO.popen("APIPIE_RECORD=examples bundle exec ruby -Itest #{controller} -n /#apidoc/")
    while (line = pipe.gets); print line; end

Also make sure you have config.show_all_examples = 1 in your apipie initializer file.

That’s how I did and its been working nicely for me, let me know in the comments if you have a better solution!

by kevinhughes27 at December 06, 2014 06:47 PM

November 20, 2014 02:37 AM

Some fun with D3.js

I’ve been intrigued by D3.js and the impressive visualizations I’ve seen people make for quite some time and I finally got to check it out for myself. I wanted to write this blog post sharing my experiences and revelations from the perspective of someone who at one point spent a fair amount of time with Matlab/Octave and my personal fave matplotlib.

The first major point I want to make is that D3.js is not a plotting library – it should not be compared directly to Matlab/Octave or matplotlib because they serve different purposes. D3.js is a visualization library, sure this visualization may be a graph but you shouldn’t be playing with D3.js until you already understand your data and simply want to show it off. While exploring and figuring out your data matplotlib is going to be way faster and more effective (note there are some wrappers building standard plotting functions ontop of D3.js like NVD3 if javascript is really your thing).

Once you’ve figured out your data and its time to show-off D3.js really shines and here is the main reason why in my opinion. Its due to a fundamental shift in a way of thinking that makes D3.js so powerful: rather than calling plot with an array of x and y values I am binding an array of javascript objects to my plot and then telling it which attributes to use for different things. Essentially you are pulling more data into the plot and you can write code at each level – let me show you the snippet that made me realize how powerful this shift in thinking is:

var chart =".chart");

var tip = d3.tip()
.attr('class', 'd3-tip')
.offset([-10, 0])
.html(function(d) { return "<span>" + d.text + "</span>"; });;

var bar = chart.selectAll("g")
.attr("transform", function(d, i) { return "translate(" + i * barWidth + ",0)"; });

.attr("y", function(d) { return y(d.value); })
.attr("height", function(d) { return height - y(d.value); })
.attr("width", barWidth - 1)
.on('mouseout', tip.hide);

The important parts to note are this: I grab the value for my y axis of the rect from `d.value` where `d` is one of the objects in my `data` array that I am plotting. So far this is pretty standard but look at the tip function that gets called on mouseover – it shows a span which contains `d.text`! I’ve attached another piece of data here called `text` that can be used to display more information! This was the wow moment for me because this kind of thing isn’t possible in matplotlib and showing a simple tooltip is really on the beginning of what you can do with all this added context.

Also something about plots with a :hover effect is kind of awesome!

If you haven’t played with D3 yet and have some cool data I definetly recommend it, I used it to make some pretty sweet visualizations/tools for our Parity Ultimate Frisbee League here in Ottawa, check’em out:

by kevinhughes27 at November 20, 2014 02:37 AM

October 30, 2014 11:07 AM

A third of the top 100 papers are about software

How many of the papers that are in the top 100 most cited about software?

21, with an additional 12 papers which are not specifically about software itself, but about methods or statistics that were implemented later in software. When you take a step back and think about the myriad areas of research and the stratospheric numbers of citations the top 100 get, it is quite remarkable that one fifth of the papers are actually about software. I mean really about software, not software as an afterthought. Some examples:

To put in perspective how rarified the air is in the top 100 citations, the if we combined all citations received by all JMLR papers in the last five years (according to SCImago), this one gigantic paper would not even make it into the top 100.

Yes, yes, citations do not directly measure the quality of the paper, and there are size of community effects and all that. To be frank, being highly cited seems to be mostly luck.

In the spirit of open science, here is a bar plot showing these numbers, and here is my annotated table which I updated from the original table. For a more mainstream view of the data, look at the Nature article.

by Cheng Soon Ong at October 30, 2014 11:07 AM

September 07, 2014 05:05 PM

Shogun in NYC

In late August, I was invited to NYC to present Shogun at an open-source Machine Learning software workshop (link), organised by John Langford. Seeing Shogun being recognised as a major player along with big ML/stats libraries like TheanoStanTorchLibLinearVowpalWabbit, etc really got my excited.

I talked to most of the other project's developers and a few very interesting possible collaborations came up. For example, Shogun's unique way to automagically generate interfaces to most computing languages via swig. It has been in our pipeline of ideas for a long time to pull this functionality out of Shogun, and offer it to other projects in a modular way. Sergey has put together (link) a simple prototype here and will continue to work on this soon.

Another thing is that we would like to integrate some (fixed) models from Stan into Shogun, for example to complement our collection of variational inference methods for Gaussian Processes with a full blown MCMC based approach.

While talking to Gunnar Raetsch, we had the idea to host a hackfest where we bring together all Shogun core-developers for a week, working on more sophisticated projects that are not suitable for Google Summer of Code. A generic framework for parallelising/distributing algorithms in Shogun would be a first idea, extending ideas from lambday's GSoC 2013 project. This would again also be useful for other ML libraries and I in fact talked to John Langford about using ideas from VowpalWabbit here. We made a list of ideas (link) for such a hack and are currently trying to get funding for it.

The meeting was video-taped and I will put links for this soon.

by (Heiko Strathmann) at September 07, 2014 05:05 PM

August 19, 2014 12:36 AM

Shogun workshop 2014

Another super nice event that happened in Berlin in July was the second Shogun workshop that me and the Shogun team organised. We had a hands-on session and a main workshop day full of talks, cool people, and lots of Machine Learning. It is a great pleasure working with the Shogun team, and I am very happy that we were able to push this year's workshop through.

I talked a bit about what Shogun is and what it tries to be, and gave some directions for future work. If you missed the event, we got full video coverage!

Workshop program


by (Heiko Strathmann) at August 19, 2014 12:36 AM

August 19, 2014 12:24 AM

It's already a while ago, in July, I presented Shogun at the EuroPython in Berlin. This was a super nice conference full of interesting people and projects. My talk was recorded and can be found [here], slides are [here].

Another super interesting project was presented by Thomas Wiecki: Probabilistic programming with PyMC3 [video]. This is a very interesting project, a bit similar to Stan, which allows to plug together probabilistic models and then do HMC for inference, powered by automatic differentiation (which I am currently super excited about). Unlike Stan, this happens all in Python which makes things super comfortable (example notebook). As I am working on kernel-based samplers (Kameleon MCMC & soon more), this might be a good vehicle to make them public, exploiting the nice auto-diff tool (Theano) that PyMC uses.

by (Heiko Strathmann) at August 19, 2014 12:24 AM

July 28, 2014 12:39 PM

Open Machine Learning Workshop

Just in case there are people who follow this blog but not John Langford's, there is going to be an open machine learning workshop on 22 August 2014 in MSR, New York, organised by Alekh Agarwal, Alina Beygelzimer, and John Langford.

As it says on John's blog: If you are interested, please email msrnycrsvp at and say “I want to come” so we can get a count of attendees for refreshments.

by Cheng Soon Ong at July 28, 2014 12:39 PM

July 22, 2014 12:00 AM

Machine Learning Distro

What would you include a linux distribution to customise it for machine learning researchers and developers? Which are the tools that would cover the needs of 90% of PhD students who aim to do a PhD related to machine learning? How would you customise a mainstream linux distribution to (by default) include packages that would allow the user to quickly be able to do machine learning on their laptop?

There are several communities which have their own custom distribution:

  • Scientific Linux which is based on Red Hat Enterprise Linux is focused making it easy for system administrators of larger organisations. The two big users are FermiLab and CERN who each have their own custom "spin". Because of its experimental physics roots, it does not have a large collection of pre-installed scientific software, but makes it easy for users to install their own.
  • Bio-Linux is at the other end of the spectrum. Based on Ubuntu, it aims to provide a easy to use bioinformatics workstation by including more than 500 bioinformatics programs, including graphical menus to them and sample data for testing them. It is targeted at the end user, with simple instructions for running it Live from DVD or USB, to install it, and to dual boot it.
  • Fedora Scientific is the latest entrant, providing a nice list of numerical tools, visualisation packages and also LaTeX packages. Its documentation lists packages for C, C++, Octave, Python, R and Java. Version control is also not forgotten. A recent summary of Fedora Scientific was written as part of Open Source Week.

It would seem that Fedora Scientific would satisfy the majority of machine learning researchers, since it provides packages for most things already. Some additional tools that may be useful include:

  • tools for managing experiments and collecting results, to make our papers replicable
  • GPU packages for CUDA and OpenCL
  • Something for managing papers for reading, similar to Mendeley
  • Something for keeping track of ideas and to do lists, similar to Evernote

There's definitely tons of stuff that I've forgotten!

Perhaps a good way to start is to have the list of package names useful for the machine learning researcher in some popular package managers such as yum, apt-get, dpkg. Please post your favourite packages in the comments.

by Cheng Soon Ong at July 22, 2014 12:00 AM

June 25, 2014 09:07 PM

ICML paper: Kernel Adaptive Metropolis Hastings

$\DeclareMathOperator*{\argmin}{arg\,min}$ $\def\Mz{M_{\mathbf{z},y}}$

In this year's ICML in Bejing, Arthur Gretton, presented our paper on Kernel Adaptive Metropolis Hastings [link, poster]. His talk slides are based on Dino Sejdinovic ones [link]. Pitty that I did not have travel funding. The Kameleon is furious!

The idea of the paper is quite neat: It is basically doing Kernel-PCA to adapt a random walk Metropolis-Hastings sampler's proposal based on the Markov chain's history. Or, more fancy: Performing a random walk on an empirically learned non-linear manifold induced by an empirical Gaussian measure in a Reproducing Kernel Hilbert Space (RKHS). Sounds scary, but it's not: the proposal distribution ends up being a Gaussian aligned to the history of the Markov chain.

We consider the problem of sampling from an intractable highly-nonlinear posterior distribution using MCMC. In such cases, the thing to do (tm) usually is Hamiltonian Monte Carlo (HMC) (with all its super fancy extensions on Riemannian manifolds etc). The latter really is the only way to sample from complicated densities by using the geometry of the underlying space. However, in order to do so, one needs access to target likelihood and gradient.

We argue that there exists a class of posterior distributions for that HMC is not available, while the distribution itself is still highly non-linear. You get them as easy as attempting binary classification with a Gaussian Process. Machine Learning doesn't get more applied. Well, a bit: we sample the posterior distribution over the hyper-parameters using Pseudo-Marginal MCMC. The latter makes both likelihood and gradient intractable and the former in many cases has an interesting shape. See this plot which is the posterior of parameters of an Automatic Relevance Determination Kernel on the UCI Glass dataset. This is not an exotic problem at all, but it has all the properties we are after: non-linearity and higher order information unavailable. We cannot do HMC, so we are left with a random walk, which doesn't work nicely here due to the non-linear shape.

The idea of our algorithm is to do a random walk in an infinite dimensional space. At least almost.

  • We run some method to get a rough sketch of the distribution we are after.
  • We then embed these points into a RKHS; some infinite dimensional mean and covariance operators are going on here, but I will spare you the details here.
  • In the RKHS, the chain samples are Gaussian, which corresponds to a manifold in the input space, which in some sense aligns with what we have seen of the target density yet.
  • We draw a sample $f$ from that Gaussian - random walk in the RKHS
  • This sample is in an infinite dimensional space, we need to map it back to the original space, just like in Kernel PCA
  • It turns out that is hard (nice word for impossible). So let's just take a gradient step along some cost function that minimises distances in the way we want: \[\argmin_{x\in\mathcal{X}}\left\Vert k\left(\cdot,x\right)-f\right\Vert _{\mathcal{H}}^{2}\]
  • Cool thing: This happens in input space and gives us a point whose embedding is in some sense close to the RKHS sample.
  • Wait, everything is Gaussian! Let's be mathy and integrate the RKHS sample (and the gradient step) out.

Et voilà, we get a Gaussian proposal distribution in input space\[q_{\mathbf{z}}(\cdot|x_{t})=\mathcal{N}(x_{t},\gamma^{2}I+\nu^{2}M_{\mathbf{z},x_{t}}HM_{\mathbf{z},x_{t}}^{\top}),\]where $\Mz=2\eta\left[\nabla_{x}k(x,z_{1})|_{x=y},\ldots,\nabla_{x}k(x,z_{n})|_{x=y}\right]$ is based on kernel gradients (which are all easy to get). This proposal aligns with the target density. It is clear (to me) that using this for sampling is better than just an isotropic one. Here are some pictures of bananas and flowers.

The paper puts this idea in the form of an adaptive MCMC algorithm, which learns the manifold structure on the fly. One needs to be careful about certain details, but that's all in the paper. Compared to existing linear adaptive samplers, which adapt to the global covariance structure of the target, our version adapts to the local covariance structure. This can all also be mathematically formalised. For example, for the Gaussian kernel, the above proposal covariance becomes \[\left[\text{cov}[q_{\mathbf{z}(\cdot|y)}]\right]_{ij} = \gamma^{2}\delta_{ij} + \frac{4\nu^{2}}{\sigma^{4}}\sum_{a=1}^{n}\left[k(y,z_{a})\right]^{2}(z_{a,i}-y_{i})(z_{a,j}-y_{j}) +\mathcal{O}(n^{-1}),\] where the previous points $z_a$ influence the covariance, weighted by their similarity $k(y,z_a)$ to current point $y$.

Pretty cool! We also have a few "we beat all competing methods" plots, but I find those retarded and will spare them here - though they are needed for publication ;)

Dino and me have also written a pretty Python implementation (link) of the above, where the GP Pseudo Marginal sampling is done with Shogun's ability to importance sample marginal likelihoods of non-conjugate GP models (link). Pretty cool!

by (Heiko Strathmann) at June 25, 2014 09:07 PM

June 03, 2014 12:01 AM

Google Summer of Code 2014

GSoC 2014 is between 19 May and 18 August this year. The students should now be just sinking their teeth into the code, and hopefully having a lot of fun while gaining invaluable experience. This amazing program is in its 10th year now, and it is worth repeating how it benefits everyone:

  • students - You learn how to write code in a team, and work on projects that are long term. Suddenly, all the software engineering lectures make sense! Having GSoC in your CV really differentiates you from all the other job candidates out there. Best of all, you actually have something to show your future employer that cannot be made up.

  • mentors - You get help for your favourite feature in a project that you care about. For many, it is a good introduction to project management and supervision.

  • organisation - You recruit new users and, if you are lucky, new core contributors. GSoC experience also tends to push projects to be more beginner friendly, and to make it easier for new developers to get involved.

I was curious about how many machine learning projects were in GSoC this year and wrote a small ipython notebook to try to find out.

Looking at the organisations with the most students, I noticed that the Technical University Vienna has come together and joined as a mentoring organisation. This is an interesting development, as it allows different smaller projects (the titles seem disparate) to come together and benefit from a more sustainable open source project.

On to machine learning... Using a bunch of heuristics, I tried to identify machine learning projects from the organisation name and project titles. I found more than 20 projects with variations of "learn" in them. This obviously misses out projects from R some of which are clearly machine learning related, but I could not find a rule to capture them. I am pretty sure I am missing others too. I played around with some topic modelling, but this is hampered by the fact that I could not figure out a way to scrape the project descriptions from the dynamically generated list of project titles on the GSoC page.

Please update the source with your suggestions!

by Cheng Soon Ong at June 03, 2014 12:01 AM

April 28, 2014 11:32 AM

Amazing blog by Sergey

Hi there. Our fellow Shoguneer and C++ guru Sergey Lisitsyn has started a new blog. Be sure to check this out – contains classic stuffs.

Statutory Warning: You may feel inferior regarding your C++ coding abilities :P

by heavensdevil6909 at April 28, 2014 11:32 AM

April 28, 2014 11:30 AM

I used this code just to see whether shogun works or not. I configured shogun with --interfaces=java_modular option and did a make install in /usr/local. It has to have jblas installed in /usr/share/java/jblas.jar

1. First I checked if jblas works fine. I tried this example -

import org.jblas.*;

public class jblas_test {
  public static void main(String[] args) {
    double[][] data = new double[][]
                      {{ 1, 2, 3, 4, 5},
                      { 6, 7, 8, 9, 10},
                      {11, 12, 13, 14, 15}};
    DoubleMatrix matrix = new DoubleMatrix(data);

    DoubleMatrix vector = new DoubleMatrix(new double[]{3, 3, 3, 3,3});
    DoubleMatrix result = matrix.mmul(vector);
    System.out.println(result.rows+"x"+result.columns+": "+result);
    System.out.println("Jblas working fine");

2. Then I compiled and ran it with -

[rahul@cfdvs4-2 jblas]$ javac -cp ".:/usr/share/java/jblas.jar"

[rahul@cfdvs4-2 jblas]$ java -cp ".:/usr/share/java/jblas.jar" jblas_test

3x1: [45.000000; 120.000000; 195.000000]
Jblas working fine

3. Next step was to get a minimal shogun example run. I wrote a simple code -

import org.shogun.*;

public class helloworld {
  static {
      System.out.println("Loaded modshogun API");
    } catch(UnsatisfiedLinkError e) {
      System.out.println("Couldn't load modshogun");

  public static void main(String[] args) {
    System.out.println("shogun works");

4. Then I compiled and ran it with -

[rahul@cfdvs4-2 test]$ javac -cp ".:/usr/local/share/java/shogun.jar"
[rahul@cfdvs4-2 test]$ java -cp ".:/usr/local/share/java/shogun.jar" -Djava.library.path=".:/usr/local/lib/jni/" helloworld
Loaded modshogun API
shogun works

5. Now was the time to run some actual shogun code. I need shogun for string kernel classification, so I tried out . Just changed System.loadLibrary("modshogun"); to System.load("/usr/local/lib/jni/"); at line #9. And also added this line after line 51.

System.out.println(out.rows+"x"+out.columns+": "+out);

Then compiled and ran -

[rahul@cfdvs4-2 test]$ javac -cp ".:/usr/share/java/jblas.jar:/usr/local/share/java/shogun.jar"
[rahul@cfdvs4-2 test]$ java -cp ".:/usr/share/java/jblas.jar:/usr/local/share/java/shogun.jar" -Djava.library.path=".:/usr/local/lib/jni/" classifier_domainadaptationsvm_modular
1x10: [-1.000000, 1.000000, -1.000000, -1.000000, -1.000000, -1.000000, 1.000000, 1.000000, 1.000000, -1.000000]

Feels terrific :D

by heavensdevil6909 at April 28, 2014 11:30 AM

April 28, 2014 11:30 AM

I have submitted the proposal (it can be found here). I’m having a much clearer view of the tasks that has to be done. I’ll be working on two other entrance issues before the official coding day begins, namely the graph coloring issue and the elliptic curve functions issue.

In my understanding, the whole work can be summarized as the following -

  1. use greedy graph coloring of the precision matrix (power of precision matrix?) for finding a set of probing vectors, \{v_{j}\} (task for the project includes integrating an existing library. I checked out a few libraries and seems like Joe Culberson’s Graph Coloring code is a really good candidate. Its written in C and provides two greedy methods for vertex coloring. More tests for our specific need has to be done)
  2. for each probing vector, we need to compute v^{T}log(Q)v, Q being the precision matrix, log(Q) is the matrix-logarithm
  3. for computing log(Q) v in the above expression, Cauchy’s integral formula (coming from complex analysis) for matrix function is used, which can be discretized, giving a rational approximation of log(Q) v, which involves solving N different systems of linear equation. Solving these systems involves invoking a preconditioned CG solver (in the project, we’ll  be integrating this from an existing library)
  4. the systems have complex coefficients (coming from conformal mapping needed for the quadrature rule of the above integration) which is given by an existing algorithm in Hale et. al. The precision of this approximation is defined by a theorem, which depends on the extremal eigenvalues. We can calculate the number N for desired accuracy using this theorem. The algorithm (for finding complex integration weights and shifts) needs Jacobi elliptic curve functions (in Driscoll’s SC-toolbox ellipkkp and ellipjc) and extremal eigenvalues. (task for the project is to integrate Krylstat’s implementation of this)
  5. combining everything for the expression of log-determinant involves writing a class which, which will combine all the subtasks. I had a discussion with Heiko on this. While initially we’ll use this to compute this on one computer, later plans include replacing this so that a OpenMPI program can run the subtasks on a cluster with low communication cost and speeded up execution. Will discuss about more details later on

The main paper describes several other techniques for handling special cases (like, when the matrix is ill-conditioned, etc). I’ll read up about these in more detail and update this page later.

by heavensdevil6909 at April 28, 2014 11:30 AM

April 28, 2014 11:28 AM

Let me start this with a happy note. I am really glad for my proposal for “Estimators of large-scale sparse Gaussian” has been accepted in Google summer of code, 2013. I’m really looking forward to an awesome summer. I  promised myself to write more often regarding this, but I’ve been really busy with a task that was needed for this project. I hope I can include some of that experience in this post too.

Some theoretical background about the project -

The aim of this project is to estimate log-determinant (up to an arbitrary precision) of a very large sparse precision matrix (inverse of covariance matrix) that arises in the log-likelihood expression of a multivariate Gaussian distribution. A direct method (like the one that I already added there in Shogun, CStatistics::log_det()) relies on Cholesky factorization of the matrix, which, in practice, may not be that sparse and often cannot be even stored in the memory and hence are not feasible in most of the practical scenarios  . The idea of this project borrows a concept from complex analysis (Cauchy’s integral formula for matrix functions) which represents a matrix function using a contour integral in the complex plane. Rational approximation of this integral for matrix function (logarithm of a matrix in this case) times a vector leads to a shifted family of linear systems with complex shifts, weights and constant, which can be estimated up to an arbitrary precision.

The task for estimating log-determinant here then becomes to sample the trace of the log of matrix using a set of vectors (called probing vectors, generated using greedy graph coloring), and in that expression, fit the log-matrix times vector using the rational approximation formula,


There has been quite a few changes in the design and structure of the framework than I have initially had in mind. Heiko suggested some really cool ideas regarding the way it should work.

by heavensdevil6909 at April 28, 2014 11:28 AM

April 28, 2014 11:27 AM

This is my first post after being selected for GSoC this year, been really really occupied with stuffs so far. So I’ll just quickly list down the things that I’ve done so far and things I am planning to do next.

We have designed the framework for implementing the log-determinant project. The background of this project can be found here (I’m really happy to see my name listed there :) thanks to Heiko! :) ).  For this project, we need to work with complex numbers, which was not supported by Shogun yet. The first task was to incorporate std::complex<T> with shogun and that required

  • adding the datatype as a primitive type
  • checking switch over all ptypes
  • adding support for the template classes that will use this
  • adding support for the parameter framework
  • and finally check serialization with this.

I added a new type for std::complex<double> as complex64_t and all required support.

We also needed Jacobi elliptic functions that we will be needing for the rational appromixation of the matrix logarithm. This part is also completed.

The next step was to come up with the basic structure of the framework. So far, the class diagram looks like the following -


Not all the classes are shown, and a few mistakes are there (please pardon me). I’m going to implement the base classes one by one, and for the sake of simplicity, we’ll start with a direct computation of matrix log (thanks to Eigen3) instead of approximation. So my main focus for this week would be to code up basic stuffs, focus on the framework development rather than numerical issues, fixing errors+unit-test+documentation+leak-check all the newly added classes thoroughly. Once the basic framework seems promising, the rest of the stuffs can be added iteratively one by one.

See you next weeek! Adios!

by heavensdevil6909 at April 28, 2014 11:27 AM

April 28, 2014 11:26 AM

As we planned, to keep the focus more on a working framework initially, I started with a direct logarithm of dense matrices that Eigen3 unsupported module provides (available for Eigen3.1.0 and later) instead of the rational approximation. This week, I have implemented the framework for dense matrix linear operators using Gaussian trace samples (\mathcal{N}(0,I)) and that shows that for a small 2\times 2 matrix, say,

C=\begin{bmatrix} 2&1 \\ 1&3 \end{bmatrix}

we can approximate the log(\left|C\right|)  by estimating E(s^{T}log(C)s) up to a certain precision with a large number of estimates. For example, log(\left|C\right|)=1.609438, and for 1,000,000 log-det estimates with Gaussian samples, we get an expected estimate E(s^{T}log(C)s)=1.609546, s \sim\mathcal{N}(0,I).

The framework has been developed in a modular way, with all independent base classes and implementations first and then the dependent ones. I added a unit-test for (almost) every component that has been added to ensure that they behave as we might expect them to do. Currently the framework has -

  • CLinearOperator<T> base class, CDenseMatrixOperator<T> implementation of this, a unit-test.
  • CJobResult base class, CScalarResult<T> and CVectorResult<T> implementation of this, a unit-test for scalar.
  • CJobResultAggregator base class, CStoreScalarAggregator<T> implementation of this and a unit-test.
  • CIndependentJob base class, CDenseExactLogJob implementation of this and a unit-test.
  • CIndependentComputationEngine base class, CSerialComputationEngine implementation, and a unit-test
  • COperatorFunction<T> base class, CDenseMatrixExactLog implementation of this, a unit-test
  • CTraceSampler base, CNormalSampler implementation, a unit-test
  • CLogDetEstimator class and a unit-test

I have a small program ready (very much similar to the CLogDetEstimator unit-test) with this which I’ll add to the libshogun examples. This shows how the framework works -

  • The sample method of CLogDetEstimator computes a number of log-det estimates of a linear operator function
  • In sample, it first computes the stuff that are required for the rest of the computation job (call init on COperatorFunction and CTraceSampler). For example, CDenseMatrixExactLog implementation computes the log of the dense matrix and sets that as the linear operator in the operator function in its init. CLogRationalApproximation will compute complex shifts, weights etc using Jacobi elliptic functions in init, etc. CNormalSampler initializes the number of samples it should generate per log-det estimate (which is 1 in this case). CProbingSampler implementation will use graph coloring of the sparse linear operator and set the number of colors as the number of estimates, all in its init.
  • Then for the required number of log-det estimates, sample uses the submit_job method of COperatorFunction to create a number of jobs per sample, and keeps the JobResultAggregators with itself. submit_job internally creates a number of jobs (based on the implementation) with one aggregator, submits the jobs to the computation engine, which may or may not start computing those job immediately (based on implementation) and passes the aggregator to sample.
  • In serial implementation of the computation engine, it starts computing the jobs immediately as soon as they are submitted (call compute method on the job) and blocks until the computation is done. For CDenseExactLogJob, the computation task is to compute log(C)s first (apply linear operator on vector, result is a vector), and then s^{T}log(C)s (vector-vector dot product). The vector is then safely discarded.
  • sample then waits for all computation jobs to be completed with the engine’s wait_for_all method. Serial implementation returns immediately in this case since all jobs are already computed.
  • sample calls finalize on all the job result aggregators which shapes the final aggregation of the job results into CScalarResults. It then computes an average of the estimates and returns.

Plan for next week

To implement the CLogRationalApproximation class for dense matrix that computes weights, shifts in init (requires the eigenvalues, currently thought of using Eigen3 for that), and then CLogRationalApproximationIndividual, which creates num_shifts jobs for each trace sample and moves the shifts into the operator. A CLogRationalApproximationIndividualJob will use Eigen3’s complex solver for direct solving. I planned to keep CLinearSolver as a base class for all the solvers, which I’ll try to implement next week, along with its DirectLinearSolver implementation. CStoreVectorAggregator has to be implemented as well.

Ah, its been too long for a report! I just realized it! :)

That’s all folks! See you next week!

by heavensdevil6909 at April 28, 2014 11:26 AM

April 28, 2014 11:26 AM

[Reposted from Shogun's mailing list]

default is libshogun.

run, within a build directory inside shogun/, cmake ..

if testing is required, run cmake -DENABLE_TESTING=ON .. and then make and run ctest –output-on-failure, this will run the tests as well as unit-tests.

If valgrind check is needed, run cmake -DENABLE_TESTING=ON -DBUILD_DASHBOARD_REPORTS=ON .. and run ctest –output-on-failure. this will run valgrind on all the tests. If only unit-testing is required, then use ctest -R unit. If verbose is required, then run it with ctest -V (or -VV) -R unit.

alternatively,  just use -DENABLE_TESTING=ON and run valgrind on any suspected test

valgrind “-q” “–tool=memcheck” “–leak-check=yes” “–show-reachable=yes” “–workaround-gcc296-bugs=yes” “–num-callers=50″ ./tests/unit/shogun-unit-test “–gtest_filter=SparseMatrixOperator.*”

To build it for another interface, refer to Viktor’s mail which is -

this will build you libshogun, without interfaces. if you want modular interfaces then you have to turn it on, either with cmake cmd argument or with cmake gui. e.g. if you want python modular interface:
cmake -DPythonModular=ON ..

the defines for the various interfaces:
. .. python modular: -DPythonModular=ON
. .. ruby modular: -DRubyModular=ON
. .. octave modular: -DOctaveModular=ON
. .. csharp modular: -DCSharpModular=ON
. .. python modular: -DJavaModular=ON
. .. lua modular: -DLuaModular=ON

there are some other funky defines like:
. .. -DENABLE_TESTING=ON  turns on unit testing and all the example (compilation) and run for the interfaces you’ve chosen to compile.
NOTE: for running the tests, after a successful ‘make’ instead of the old ‘make tests’ use ctest:
ctest –output-on-failure

. .. -DBUNDLE_EIGEN=ON your system does not ship the right eigen version (3.1.2 or later)? just use this define, and cmake will automagically download/compile and bundle eigen 3.1.4 into shogun

. .. -DBUNDLE_JSON=ON same as the previous case just with libjson-c

and now something more…FANCY!
cmake comes with a nice cpack utility that basically can generate all kinds of packages, not only .tar.gz and .tar.bz2 but like debian’s .deb or redhat’s rpm, osx’ mpkg etc. so hopefully with this we soon be able to ship various nightly built binary packages for you.
of course this work is still under progress…. if you want it sooner, come and help us, so we can get it for you sooner.

anyways, enjoy!
and of course if you run into some problems while trying to cmake shogun, just write here on the mailing list or come to the irc channel and ask somebody!


Hi Klye

your best bet is to use cmake gui, i.e. ccmake on UNIX systems.
there you can browse through the possible options, and i’ve written a small description for all the possible options.
for the 2 things you were looking for:

–disable-svm-light  in cmake is -DUSE_SVMLIGHT=OFF
–enable-svm-light  in cmake is -DUSE_SVMLIGHT=ON

–enable-swig-directors is -DUSE_SWIG_DIRECTORS=ON

we are in progress with updating the ./src/INSTALL file where we’d describe things in more detail than my little email.


by heavensdevil6909 at April 28, 2014 11:26 AM

April 28, 2014 11:25 AM

Recently I bumped into a great introductory video series on concurrent programming using C++11 by Bo Qian on youtube. In this I am just trying to note down important parts and playing with stuffs. Some of the examples may be directly used as it is in the video series.

Basic threading environment in C++11

The following example introduces with the basic constructs.

 * filename: test1.cpp
 * a simple example of using threads in c++11
 * to compile, use
 *   g++ -lpthread -std=c++0x test1.cpp
 * on linux environment

#include <iostream>
#include <thread>
using namespace std;

// a stupid thread function
void thread_function(int n)
  for (int i=0; i<n; ++i)
  // each thread has a unique id
    cout << "thread with id " << this_thread::get_id() << " says hello " << i << endl;
  cout << "address of n in t1 thread is " << &n << endl;

// a stupid thread function that takes a reference
void thread_function_ref(int& n)
  for (int i=0; i<n; ++i)
    cout << "thread with id " << this_thread::get_id() << " says hello " << i << endl;
  cout << "address of n in t2 (reference) thread is " << &n << endl;

// a stupid functor
class functor
  int state;
  void operator()(int n) {
  for (int i=0; i<n; ++i)
    cout << "thread with id " << this_thread::get_id() << " says hello "<< i << endl;

// a stupid main function
int main()
  int n = 3;

// thread creation -
// ----------------
  // thread using normal function - takes a callable object as the first
  // argument and a number of other arguments which are to be passed as
  // arguments to the callable object (function, lambda expressions, functor)
  // note that the arguments are always passed by value! a way to pass the
  // arguments by reference is shown in the next example
  thread t1(thread_function, n);

  // using std::ref we can pass the argument (n in this case) as a reference
  // just passing n won't do the trick!
  thread t2(thread_function, std::ref(n));
  // to verify, we print the address of n in the main thread and in the child
  // threads of the thread functions. let's see!
  cout << "address of n in main thread is " << &n << endl;

  // this is cool! using this we can share memory between threads. we can
  // also completely hand over a memory to another thread using std::move
  // but more on that later

// joining and stuffs -
// --------------------
  // we may decide to join the thread, in which the creator thread will wait
  // until the execution of the child thread is finished. here, however we
  // decide not to do that right away

// more funky ways of thread creation -
// -----------------------------------
  // thread using lambda function - lambda functions are cool! check more
  // about these on
  thread t3([n]() {
    // inherits variable n from current scope and copies it to the thread
    // stack at different address
    for (int i=0; i<n; ++i)
      cout << "thread with id " << this_thread::get_id() << " says hello " << i << endl;
    cout << "address of n in t3 thread is " << &n << endl;

  thread t4([&n]() {
    // inherits variable n from current scope and uses the same address
    for (int i=0; i<n; ++i)
      cout << "thread with id " << this_thread::get_id() << " says hello " << i << endl;
    cout << "address of n in t4 thread is " << &n << endl;

  thread t5([](int n) {
    // doesn't inherit anything - we pass it by value - same as t1
    for (int i=0; i<n; ++i)
      cout << "thread with id " << this_thread::get_id() << " says hello " << i << endl;
    cout << "address of n in t5 thread is " << &n << endl;
  }, n);

  // similarly we can pass it by reference as t2 - but its getting boring

// thread using functor -
// ----------------------
  // creating an object first
  functor f1;
  thread t6(f1, n);

  // using anonymous object (note the extra paranthesis)
  thread t7((functor()), n);

// some more boring stuffs -
// -------------------------
  // something to do for the main thread
  for (int i=0; i<n; ++i)
    cout << "thread with id " << this_thread::get_id() << " says hello " << i << endl;

  // join/detatch all threads
  if (t1.joinable())

  return 0;

This code works. When I compiled and ran on my system (Fedora-19-x86_64) it ran, except it printed out something that’s pretty tough to read!

[ thread_test]$ ./a.out
address of n in main thread is thread with id thread with id 0x7fff867e5aec0x7f30625da7000x7f3061dd9700 says hello  says hello
thread with id 0139845777069824 says hello
thread with id 139845768677120 says hello 1
thread with id 139845777069824 says hello 2
address of n in t1 thread is thread with id 0x7f3061dd8e0c0x7f3060dd7700
thread with id  says hello 0
thread with id 139845760284416 says hello 1
thread with id 139845760284416 says hello 2
thread with id 139845751891712address of n in t4 thread is  says hello 0x7fff867e5aec0

thread with id 139845751891712 says hello 1
thread with id 139845751891712 says hello 2
address of n in t5 thread is 0x7f30605d5e04
thread with id 139845768677120 says hello 1
thread with id 139845768677120 says hello 2
address of n in t3 thread is 0x2138340
139845785462528 says hello 1
thread with id thread with id 139845785462528 says hello 2
address of n in t1 thread is 0x7f30625d9e0c
139845743499008 says hello 0
thread with id 139845743499008 says hello 1
thread with id 139845743499008 says hello 2
thread with id 139845785470848 says hello 0
thread with id 139845785470848 says hello 1
thread with id 139845785470848 says hello 2
thread with id 139845735106304 says hello 0
thread with id 139845735106304 says hello 1
thread with id 139845735106304 says hello 2

This is because all the threads are running for the same resource stdout and causing data race condition. This must be handled using mutex which provides lock/unlock mechanisms for shared resources. The following example shows how.

Handling data race condition using mutex

 * filename: test2.cpp
 * a simple example of using a shared resource, such as cout,
 * among multiple threads securely using mutex

#include <iostream>
#include <thread>
#include <mutex>

using namespace std;

class printer
  ostream& os;
  mutex& mu;
  // making default constructor, copy constructor and
  // assignment operator implicitly disabled for apparently 
  // no reason
  printer(const printer&)=delete;
  const printer& operator=(const printer&)=delete;

  // the constructor that we will be using
  printer(mutex& m):os(cout), mu(m){}

  // the function that we will be using
  // I tried using string& s as a parameter but didn't work!
  void shared_print(string s, int i)
    // before using the resource that is os, we first need to
    // lock the mutex, which could have normally be done using
    // mutex::lock and mutex::unlock methods. however, if in
    // between lock and unlock some exception happens, then
    // the mutex is never going to be unlocked again until the
    // main thread dies, therefore all other threads will be
    // kept on waiting till death
    // lock_guard provides a Resource Acquisition Is Initialization
    // (RAII) sort of thing, which calls the unlock in its destructor
    // therefore, whenever the control goes outside of this block and
    // guard gets killed, the mutex will be unlocked!
    // sweet!
    lock_guard<mutex> guard(mu);
    os << s << i << endl;

void thread_function(int& n, printer& pr)
  for (int i=0; i<n; ++i)
    // tried using std::ref which gave weird errors
    // that it has been deleted!! any clue??

int main()
  // mutex is the main tool for employing mutual exclusion from
  // shared resources
  mutex mu;

  printer pr(mu);
  int n = 4;

  // main thread and child thread use same printer object
  // to print stuffs. in fact everyone wishes to print stuffs
  // using cout should use this! otherwise cout will be exposed
  // to vulnerabilities and somebody can do nasty stuffs.
  // its really importantn that we never ever give the access
  // of the ostream from printer to anybody!
  thread t1(thread_function, std::ref(n), std::ref(pr));

  // its a good practice to surround the following within
  // a try-catch block, so that if the following part of the
  // code throws an exception, the child thread still gets joined
  try {
    for (int i=0; i<n; ++i)
  } catch(...) {


  return 0;

This gives a pretty output

[ thread_test]$ ./a.out

Number of threads to be created is often a good question. It makes sense to create as many threads as there are cpus available. Otherwise a lot of time would be wasted in context switching which is a bad thing for program performance.


function provides a way to do that.

In the next example, a different kind of lock guard is shown at action, which provides a deferred locking mechanism.

 * filename: test3.cpp
 * this shows usage of unique_lock

#include <iostream>
#include <thread>
#include <mutex>

using namespace std;

// similar to what we have seen before, locks the data
// and then writes to it
void thread_function(int& data, mutex& mu, void (*f)(int&))
  // we shouldn't create a lock_guard here because there is
  // no point in locking the data between the iterations of
  // the loop - the other thread would then have to wait until
  // this function returns from one thread
  unique_lock<mutex> guard(mu, defer_lock);
  for (int i=0; i<10; ++i)
    // here we can create the lock as lock_guard and before the
    // next iteration the mutex will be unlocked. but to avoid
    // the overhead of creating and destroying a lock_guard obj
    // each time, we can use unique lock's lock/unlock mechanism
    // under defered lock
    //lock_guard<mutex> guard(mu);
    cout << this_thread::get_id() << ": says data is " << data << endl;

int main()
  mutex mu;
  // data that the threads try to write into
  // must be synchronized to avoid data race condition
  // using mutex
  int data = 0;

  // creating a thread that tries to increase the data
  thread t1(thread_function, std::ref(data), std::ref(mu), [](int& n){n++;});
  // another thread which tries to decrease the data
  thread t2(thread_function, std::ref(data), std::ref(mu), [](int& n){n--;});


  return 0;

Output is shown below. after the execution is over, the value of data is same as before, as one might expect.

[ thread_test]$ ./a.out
140657387300608: says data is 1
140657387300608: says data is 2
140657387300608: says data is 3
140657387300608: says data is 4
140657387300608: says data is 5
140657387300608: says data is 6
140657378907904: says data is 5
140657378907904: says data is 4
140657378907904: says data is 3
140657378907904: says data is 2
140657378907904: says data is 1
140657378907904: says data is 0
140657378907904: says data is -1
140657378907904: says data is -2
140657378907904: says data is -3
140657378907904: says data is -4
140657387300608: says data is -3
140657387300608: says data is -2
140657387300608: says data is -1
140657387300608: says data is 0

Some more pointers are coming up next! Wrapping up this one cause its getting too big.

by heavensdevil6909 at April 28, 2014 11:25 AM

April 28, 2014 10:47 AM

Tests to be added – one for the mean/variance for trace sampler and another with a huge matrix but not diagonal but having subdiagonal entries.

Add a test for distance two coloring on the sparse matrix operator itself and then trace samples compared with distance one coloring on power of the sparse matrix.

very important to have things in a very polished/documented/tested way.  missing features are not that important. rather have the existing things working well. there can be many examples given, eigen solvers, linear solvers, log det esitmators, computation framework examples. Use matplotlib for plots, iypthon notebook –pylab inline.

by heavensdevil6909 at April 28, 2014 10:47 AM

April 28, 2014 10:44 AM

Related files – under shogun/lib/

  • common.h
  • DataType.h
  • DataType.cpp

first thing – add typedef std::complex<float64_t> complex64_t, then the new SG primitive type.

Todo list:

  • serialization
  • modular typemaps
  • check all switch over ptype
We have this shogun type object, TSGDataType, contains primitive type, container type(vector,matrix), struct type (string,sparse).  TSGDataType is used by shogun parameter framework to represent things, for example for serialization, model-selection, equals method. one has to register parameters with the SG_ADD macro which calls m_parameters->add of the current object. In there, we need new methods for complex, should be a few, in the same style as the existing ones. Then we have to check serialization, which writes types to files. Then modular maps which map the type the the type of the language (eg python complex).

by heavensdevil6909 at April 28, 2014 10:44 AM

April 24, 2014 11:14 PM

Machine Learning Summer School in Reykjavik

I am visiting the Machine Learning Summer School in Reykjavik, Iceland. Having our poster on Kameleon MCMC, a Kernel Adaptive Metropolis-Hastings sampler, with me (arXiv link: to appear in ICML 2014, poster link). Already met loads of interesting new and known people. Iceland is a pretty cool place, looks a bit like this.

by (Heiko Strathmann) at April 24, 2014 11:14 PM

April 18, 2014 12:10 AM

Opaque Pointers Revisited

Opaque pointer (aka d-pointer or pimpl) is a great C++ design pattern useful for prolongated binary interface compatibility, properly hidden implementation and faster compilation. However, it has inherent performance drawback, which could get pretty critical if you care about efficiency. In this post I propose an approach that makes d-pointers less binary compatible but swipes away its inefficiency.

April 18, 2014 12:10 AM

April 08, 2014 06:53 PM

Actions With RAII

You may know RAII as a cool idiom that makes it pretty easy to handle resources finalization automatically. When used properly, it reduces LoC, helps to avoid bugs and gives more safety for free. This makes RAII an important part of modern C++.

April 08, 2014 06:53 PM

March 30, 2014 11:16 AM

Reproducibility is not simple

There has been a flurry of articles recently outlining 10 simple rules for X, where X has something to do with data science, computational research and reproducibility. Some examples are:

Best practices

These articles provide a great resource to get started on the long road to doing "proper science". Some common suggestions which are relevant to practical machine learning include:

Use version control

Start now. No, not after your next paper, do it right away! Learn one of the modern distributed version control systems, git or mercurial currently being the most popular, and get an account on github or bitbucket to start sharing. Even if you don't share your code, it is a convenient offsite backup. Github is the most popular for open source projects, but bitbucket has the advantage of free private accounts. If you have an email address from an educational institution, you get the premium features for free too.

Distributed version control systems can be conceptually daunting, but it is well worth the trouble to understand the concepts instead of just robotically type in commands. There are numerous tutorials out there, and here are some which I personally found entertaining, git foundations and hginit. For those who don't like the command line, have a look at GUIs such as sourcetree, tortoisegit, tortoisehg, and gitk. If you work with other people, it is worth learning the fork and pull request model, and use the gitflow convention.

Please add your favourite tips and tricks in the comments below!

Open source your code and scripts

Publish everything. Even the two lines of Matlab that you used to plot your results. The readers of your NIPS and ICML papers are technical people, and it is often much simpler for them to look at your Matlab plot command than to parse the paragraph that describes the x and y axes, the meaning of the colours and line types, and the specifics of the displayed error bars. Tools such as ipython notebooks and knitr are examples of easy to implement literate programming frameworks that allow you to make your supplement a live document.

It is often useful to try to conceptually split your computational code into "programs" and "scripts". There is no hard and fast rule for where to draw the line, but one useful way to think about it is to contrast code that can be reused (something to be installed), and code that runs an experiment (something that describes your protocol). An example of the former is your fancy new low memory logistic regression training and testing code. An example of the latter is code to generate your plots. Make both types of code open, document and test them well.

Make your data a resource

Your result is also data. When open data is mentioned, most people immediately conjure images of the inputs to prediction machines. But intermediate stages of your workflow are often left out of making things available. For example, if in addition to providing the two lines of code for plotting, you also provided your multidimensional array containing your results, your paper now becomes a resource for future benchmarking efforts. If you made your precomputed kernel matrices available, other people can easily try out new kernel methods without having to go through the effort of computing the kernel.

Efforts such as and provide useful resources to host machine learning oriented datasets. If you do create a dataset, it is useful to get an identifier for it so that people can give you credit.

Challenges to open science

While the articles call these rules "simple", they are by no means easy to implement. While easy to state, there are many practical hurdles to making every step of your research reproducible .

Social coding

Unlike publishing a paper, where you do all your work before publication, publishing a piece of software often means that you have to support it in future. It is remarkably difficult to keep software available in the long term, since most junior researchers move around a lot and often leave academia altogether. It is also challenging to find contributors that can help out in stressful periods, and to keep software up to date and useful. Open source software suffers from the tragedy of the commons, and it quickly becomes difficult to maintain.

While it is generally good for science that everything is open and mistakes are found and corrected, the current incentive structure in academia does not reward support for ongoing projects. Funding is focused on novel ideas, publications are used as metrics for promotion and tenure, and software gets left out.

The secret branch

When developing a new idea, it is often tempting to do so without making it open to public scrutiny. This is similar to the idea of a development branch, but you may wish to keep it secret until publication. The same argument applies for data and results, where there may be a moratorium. I am currently unaware of any tools that allow easy conversion between public and private branches. Github allows forks of repositories, which you may be able to make private.

Once a researcher gets fully involved in an application area, it is inevitable that he starts working on the latest data generated by his collaborators. This could be the real time stream from Twitter or the latest double blind drug study. Such datasets are often embargoed from being made publicly available due to concerns about privacy. In the area of biomedical research there are efforts to allow bona fide researchers access to data, such as dbGaP. It seamlessly provides a resource for public and private data. Instead of a hurdle, a convenient mechanism to facilitate the transition from private to open science would encourage many new participants.

What is the right access control model for open science?

Data is valuable

It is a natural human tendency to protect a scarce resource which gives them a competitive advantage. For researchers, these resources include source code and data. While it is understandable that authors of software or architects of datasets would like to be the first to benefit from their investment, it often happens that these resources are not made publicly available even after publication.

by Cheng Soon Ong at March 30, 2014 11:16 AM

March 28, 2014 06:59 AM

Any Struggles

In C++ you can’t just forget about types. Each variable has its own type that is not going to be changed by any means. What if you really need something heterogeneous? There is a known idiom called Any that enables you to erase the type and recall it later.

March 28, 2014 06:59 AM

March 25, 2014 11:33 PM

I Like Intractable Likelihoods

Last week, I went to the i-like workshop at Oxford university. Pretty cool! All of Britain's statisticians were there and I met many of them for the first time. Check out my two posters (Russian Roulette, Kernel Adaptive Metropolis Hastings). Talks were amazing - as in last NIPS, the main trend is on estimating likelihoods (well, that's the name of the program), either using some other random process such as importance sampling a latent model's marginal likelihood (aka Pseudo-Marginal MCMC), or directly sub-sampling likelihoods or gradients.

These things are important in Machine Learning too, and it is very nice to see the field growing together (even-though there was a talk by a Statistician spending lots of time on re-inventing belief propagation and Junction tree ideas - always such a pitty if this happens simply because communities do not talk to each other enough). Three talks that I really found interesting:

Remi Bardenet talked about sub-sampling approaches to speed up MCMC. This is quite related to the Austerity in MCMC land paper by Welling & Co, with the difference that his tests do not suffer from small number of points in the hypothesis test to decide accept/reject.

Chris Sherlock talked about optimal rates and scaling for Pseudo-Marginal MCMC. There finally are some nice heuristics how to scale PM estimates in a way that the number of iid samples per computation time is optimal. Interestingly, the acceptance rate and the variance of the likelihood estimate can be tweaked separately.

Jim Griffin gave a very interesting talk on adaptive MCMC on discrete, in particular binary, state-spaces - he used them for feature selection (in ML language). His algorithm automatically learns global mutations rates for each of the positions. However, it doesn't take any correlations between the features into account. This might be a very interesting application for our fancy Kameleon sampler (arxiv, code), thinking about this!

Finally, I presented two posters, the one on Playing Russian Roulette with Intractable Likelihoods that I already presented in Reykjavik, and (with Dino) a new poster (link) on the Kernel Adaptive Metropolis Hastings Kameleon that I mentioned above. The corresponding paper is hopefully published very soon. Talking to other scientists about my own work is just great!


by (Heiko Strathmann) at March 25, 2014 11:33 PM

March 02, 2014 11:46 PM

Weekend Project – Install SteamOS

This weekend I championed my way through installing SteamOS (the Debian distro by Valve that will be the installed on the upcoming Steam boxes). I had to do some pretty crazy stuff to get it working including dropping out of the automated install to manually inject grub-pc and then compiling the drivers for my wireless card. All in all it was a triumph!

2013-12-14 19.04.37

and then finally:

2013-12-15 21.16.23

This was an early beta release but they made some weird choices – like handicapping the basic Debian installer by fully automating it and only supporting efi. I was actually a bit disappointed when I finally finished because the end result is not really different from simply installing Ubuntu and setting Steam big picture mode to auto start, I am not sure what exactly I was expecting though. SteamOS is much more for OEMs than the DIY crowd at the moment but I can see that Valve is super invested in Linux at this point with a ton of additions to their own repositories. Good things are going to come of this I can feel it!

* Edit *

Almost all the hacking I had to do has been wrapped in Ye Olde SteamOSe 

* Edit 2 *

Wow Valve released an updated version of the beta addressing a lot of the problems Ye Olde SteamOSe addressed and they allegedly collaborated to get this done! This is why Valve is going to win the next generation – working with the community. Full story here

by kevinhughes27 at March 02, 2014 11:46 PM

March 02, 2014 11:19 PM

I haven’t had time for much machine learning on the side as my new job and learning more about web development has been keeping me pretty busy so I decided to help out the Shogun team by doing some clean-up on the website. The whole thing actually came about because I was trying to find the current status of the shogun build to send to someone on the mailing list and I couldn’t find it on the old site, needless to say it was tough to navigate. My main goal with the re-design was to “flatten” the navigation into a navbar rather then the current system of nested navigation links (as an aside the way this worked was pretty poor, we actually queried a large portion of the database on every page load just to make this nested navigation auto-generated). The project ended up being mostly front-end which isn’t really my thing but its cool cause I got to learn some new stuff, I also got to write some Django “migrations” and maintenance tasks so there was a bit of backend too. I think the re-design was a success and the site is definitely easier to navigate around (as a bonus now we don’t have to query the whole db on each page wooo! – solution put the navbar content in the db too!)

you can see the new site live where its always been:

by kevinhughes27 at March 02, 2014 11:19 PM

March 01, 2014 08:29 PM

GSoC Interview with Sergey and me

Sergey and me gave an interview on Shogun and Google Summer of Code. Here it is:

The internet. More specifically #shogun on Wasn’t IRC that thing that our big brothers used as a socialising substitute when they were teenagers back in the 90s? Anyways. We are talking to two of the hottest upcoming figures in machine learning open-source software, the Russian software entrepreneur Sergey Lisitsyn, and the big German machine Heiko Strathmann.

1.jpg2013-07-10 11.55.11.jpg

Hi guys, glad to meet you. Would you mind introducing yourself?

Sergey (S): Hey, I am Sergey. If you ask me what do I do apart from Shogun - I am currently working as a software engineer and finishing my Master’s studies at Samara State Aerospace University. I joined Shogun in 2011 as a student and now I am doing my best to help guys from the Shogun team to keep up with GSoC 2014.

Heiko (H): Hej, my name is Heiko. I do a Phd in Neuroscience & Machine Learning at the Gatsby Institute in London and joined Shogun three years ago during GSoC. I love open-source since my days in school.


Your project, Shogun, is about Machine Learning. That sounds scary and sexy, but what is it really?

H: My grandmother recently sent me an email asking about this ‘maschinelles Lernen’. I replied it is the art of finding structure in data in an automated way. She replied: Since when are you an artist? And what is this “data”? I showed her the movie PI by Darren Aronofsky where the main character at some point is able to predict stock prices after realising “the pattern”, and said that’s what we want to do with a computer. Since then, she is worried about me because the guy puts a drill into his head in the end….. Another cool application is for example to model brain patterns to allow people to learn how to use a prosthesis faster.

S: Or have you seen your iPhone detects faces? That’s just a Support Vector Machine (SVM). It employs kernels which are inner products of non-linear mappings of Haar features into a reproducing kernel Hilbert Space so that it minimizes ….


Yeah, okok... What is the history of Shogun in the GSoC?

S: The project got started by Sören in his student days around 15 years ago. It was a research only tool for a couple of years before being made public. Over the years, more and more people joined, but the biggest boost came from GSoC...

H: We just got accepted into our 4th year in that program. We had 5+8+8 students so far who all successfully did the program with us. Wow I guess that’s a few million dollars. (EDITOR: actually 105,000$.) GSoC students forced Shogun to grow up in many ways: github, a farm of buildbots, proper unit-testing, a cloud-service, web-demos, etc were all set up by students. Also, the diversity of algorithms from latest research increased a lot. From the GSoC money, we were able to fund our first Shogun workshop in Berlin last summer.


How did you two got into Shogun and GSoC? Did the money play a role?

H: I was doing my undergraduate project back in 2010, which actually involved kernel SVMs, and used Shogun. I thought it would be a nice idea of putting my ideas into it -- also I was lonely coding just on my own. 2010, they were rejected from GSoC, but I eventually implemented my ideas in 2011. The money to me was very useful as I was planning to move to London soon. Being totally broke in that city one year later, I actually paid my rent from my second participation’s stipend - which I got for implementing ideas from my Master’s project at uni. Since 2013, I mentor other students and help organising the project. I think I would have stayed around without the money, but it would have been a bit tougher.

S: We were having a really hard winter in Russia. While I was walking my bear and clearing the roof of the snow, I realised I forgot to turn off my nuclear missile system…..

H: Tales!

S: Okay, so on another cold night I noticed a message on GSoC somewhere and then I just glanced over the list of accepted organizations and Shogun’s description was quite interesting so I joined a chat and started talking to people - the whole thing was breathtaking for me. As for the money - well, I was a student and was about to start my first part-time job as a developer - it was like a present for me but it didn’t play the main role!

H: To make it short: Sergey suddenly appeared and rocked the house coding in lightspeed, drinking Vodka.


But now you are not paid anymore, while still spending a lot of time on the project. What motivates you to do this?

S: This just involves you and you feels like you participate in something useful. Such kind of appreciation is important!

H: Mentoring students is very rewarding indeed! Some of those guys are insanely motivated and talented. It is very nice to interact with the community with people from all over the world sharing the same interest. Trying to be a scientist, GSoC is also very useful in producing tools that myself or my colleagues need, but that nobody has the time to build properly. You see, there are all sorts of synergic effects in GSoC and my day-job at university, such as meeting new people or getting a job since you know how to code in a team.


How does this work? Did you ever publish papers based on GSoC work?

S: Yeah, I actually published a paper based on my GSoC 2011 work. It is called ‘Tapkee: An Efficient Dimension Reduction Library’ and was recently published in the Journal of Machine Learning Research. We started writing it up with my mentor Christian (Widmer) and later Fernando (Iglesias) joined our efforts. It took enormous amount of time but we did it! Tapkee by the way is a Russian word for slippers.

H: I worked on a project on statistical simulation of global ozone data last year. The code is mainly based on one of my last year’s student’s project - a very clever and productive guy from Mumbai who I would never have met without the program, see


So you came all the way from being a student with GSoC up to being an organisation admin. How does the perspective change during this path?

H: I first had too much time so I coded open-source, then too little money so I coded open-source, then too much work so I mentor people coding it open-source. At some point I realised I like this stuff so much that I would like to help organising Shogun and bring together the students and scientists involved. It is great to give back to the community which played a major role for me in my studies. It is also sometimes quite amusing to get those emails by students applying, being worried about the same unimportant things that I worried about back then.

S: It seems to be quite natural actually. You could even miss the point when things change and you became a mentor. Once you are into the game things are going pretty fast. Especially if you have full-time job and studies!


Are there any (forbidden) substances that you exploit to keep up with the workload?

S: It would sound strange but I am not addicted to vodka. Although I bet Heiko is addicted to beer and sausages.

H: Coffeecoffeecoffeee…… Well, to be honest GSoC definitely reduces your sleep no matter whether you are either student, mentor, or admin. By the way, our 3.0 release was labelled: Powered by Vodka, Mate, and beer.


Do you crazy Nerds actually ever go away from your computers?

H: No.

S: Once we all met at our workshop in Berlin - but we weren’t really away from our computers. Why on earth to do that?


Any tips for upcoming members of the open-source community? For students? Mentors? Admins?

H: Students: Do GSoC! You will learn a lot. Mentors: Do GSoC! You will get a lot. Admins/Mentors: Don’t do GSoC, it ruins your health. Rather collect stamps!

S: He is kidding. (whispers: “we need this … come on … just be nice to them”)

H: Okay to be honest: just have fun of what you are doing!


Due to the missing interest in the community, Sergey and Heiko interviewed themselves on their own.



GSoC 2013 blog:

GSoC 2014 ideas:




by (Heiko Strathmann) at March 01, 2014 08:29 PM

March 01, 2014 01:42 AM

Yeah! Shogun this week got accepted to be an organisation participating in the 10th Google Summer of Code. This year, besides mentoring a few projects, I am one of the three project administrators. I am curious how this will be. One first thing to do was to write the application for Shogun - I'm glad it worked! I also will spend a little more time organising things. Apart from trying to find mentors (which requires a lot of talking people into it), I also want to make Shogun (and the students) having more from the program. Last year, I pushed the team to ask all students

  • to write a project report in the form of IPython notebooks (link). These are absolutely great for talking about the GSoC work, impressing people, and having a final piece of work to show for the students.
  • To fully unit-test every module of their algorithm/framework. This is absolutely essential in order to not loose the student's work a few years later when a re-factoring change breaks their code and nobody knows how to fix it. Those tests already saved lots of life since last year.
  • To peer-review each other in pairs of students. This improved documentation here and there and solved some bugs. I want to emphasise this more this year as I think it is a great way of enabling synergistic effects between students.

In addition, we will again screen all the applicants via a set of entrance tasks on our github page (link). I just wrote a large number of such smaller or larger tasks that get students started on a particular project, fix bugs in Shogun, or prepare some larger change. In order to get the students started a bit more easily (contributing to Shogun these days is a non-trivial task), I wrote a little how-to (link) that is supposed to point out our expectations, and what are the first steps towards participating in GSoC. 

Finally, I wrote descriptions for quite a few possible projects, some of them with a number of interesting co-mentors. The full list is here (link). If you are a talented student interested in any of those topics, consider working with us during the summer. It's usually very fun!

  • Variational Learning for Recommendation with Big Data. With Emtiyaz Khan, who I met at last year's workshop for latent Gaussian models. Matrix factorisation and Gaussian Processes, ultra-cool project.
  • Generic Framework for Markov Chain Monte Carlo Algorithms and Stan Interface. With Theo Papamarkou, who I know from my time at UCL Statistics. It's about a modular representation of MCMC within Shogun and a possible interface to STAN for the actual sampling. This would be a major step of Shogun towards probabilistic models.
  • Testing and Measuring Variable Interactions With Kernels. With Dino, who is post-doc at Gatsby and co-author of our optimal kernel for MMD paper. This project is to implement all kernel based interaction measures in Shogun in a unified way. We'll probably use this for research later.
  • A Meta-Language for Shogun examples. With Sören. Write example once, press button to generate in any modular language binding. This would be so useful to have in Shogun!
  • Lobbying Shogun in MLPACK’s automatic benchmarking system. Joint project with Ryan from MLPACK. He already can compare speed of different toolboxes. Now let's compare results.
  • Shogun Missionary & Shogun in Education. With Sören. Write high quality notebooks and eye-candy examples. Very different project as this is about creative technical writing and illustrating methods on cool data rather than hacking new algorithms. I would be very excited if this happened!

Some of the other projects involve cool buzzwords such as Deep Learning, Structured Output, Kernel, Dual solvers, Cluster backends, etc. Join us! :)

by (Heiko Strathmann) at March 01, 2014 01:42 AM

February 25, 2014 02:43 AM

An Excellent Computer Vision Book:

Mastering OpenCV with Practical Computer Vision Projects

I’ve talked about this book before – one of the chapters inspired my vision system for last years FRC competition (you can read it here). I think it’s an excellent book and Packt just sent my an ebook version to do a proper review. Of all the book reviews I’ve done this is the most genuine because I actually found this book useful for work I was doing. They found the right level of being technically interesting, robust and substantial all the while without being too daunting. The source code that accompanies this book is great and I still check back to it when starting new projects ( Its well written and won’t take long to read through – I think it is a worthwhile read for anyone doing computer vision work. Program design isn’t something a lot of computer vision researchers / developers think or talk about a lot (at least in my experience) so to see how others lay out the problem can really help with your own work.

You can grab a copy here:

by kevinhughes27 at February 25, 2014 02:43 AM

January 28, 2014 10:49 PM

Dear Resident Evil 6 Developers

This is the first time I’ve felt compelled to write a real video game review/critique, why is this? because I’ve been a fan of RE since I was a kid and its one of my favourite franchises. While I will say I liked RE6 the game left me very frustrated and I need to talk about it.

RE is increasingly suffering from an identity crisis, is it a survival game? is it an action game? it’s getting more cinematic but at the same time someone is pushing to add more gamification. These goals are kind of at odds, you can’t just smash them all together and expect it to work.

Action vs. Survival
I think going for a bit more action than the early games is fine and in my opinion you had the right balance in RE4, since then things have gotten weird. Melee seems to have gotten way more important to the extent that I’m using it when it doesn’t even make sense (this is detracting from the cinematic feel too). The mutations have also got a bit out of hand and I feel like I’m being penalized for headshots – shooting zombies in the face is one of the reasons I play RE. Most importantly the ammo conservation system is now broken, in RE4 I felt in control of my ammo supply and I could conserve in easy situations, in RE6 I felt like I was always scrambling but not in a good way. Finally when you gave enemies real guns it kind of tossed the ammo conservation aspect out the window – some guy is shooting at me but I have no ammo so I’m gonna run and punch him in the face what? (not very cinematic either..)

Cinematic vs. Gamification 
The gamification is at odds with the cinematic grandeur of the game – I should have my gun raised looking intently at the next door or checking my back but instead I’m looking down at the guy whose brain I just stomped hoping to god that he drops some ammo so I’m not totally screwed in the next room – and then guess what he drops experience, fucking experience. My memories of RE6 should include the epic boss battles or being startled by a surprise monster but instead I remember running around after killing everything picking up swag and kicking boxes in looking for crap. Tell whoever is pushing the gamification to shove it and make the game epic!


  • Reduce/Remove the item drops, it makes the game less cinematic. Same goes for skill points just calculate it somehow
  • Melee is sick don’t get me wrong but it should be used when it makes sense (think cinematic)
  • The mutations and enemy variants made sense in RE4, no need to one up it
  • I like shooting zombies in the face!
  • If enemies have guns we need more ammo

I just bought RE4 HD for PC and I am pumped!

by kevinhughes27 at January 28, 2014 10:49 PM

December 13, 2013 04:45 PM

MLOSS workshop at NIPS 2013

Last week, I went to the Advances in Neural Information Processing Systems (NIPS) for the first time. That was a very nice experience due to the incredibly density of people whose names I know from research papers. In fact, it was too much to take so I had to pick things that sounded interesting - still loads.

The main three buzzwords of the conference for me were: Deep Learning (even Mark Zuckerberg is interested in that these days), Mini-batch, and stochastic gradient descent (aka on-line whatever).

One very interesting workshop I attended on Tuesday was on Machine Learning Open-Source Software (MLOSS), organised by Cheng Soon Ong (who could not be there unfortunately) and Antti Honkela. I presented a short spotlight for Shogun (slide) and had a one hour demo, showing off with our cool IPython notebooks (link) and the cloud Shogun server (link). I got some very encouraging feedback for this, including from Fernando Perez.
I also met a few nice fellow open-source ML coders from scikit-learn.

During the workshop, there was a quite lively discussion about licensing issues, in particular whether to choose GPL or BSD. The python universe for example seems to gain a lot from being BSD-style licensed.

Finally, NIPS is was held close to Lake Tahoe, which is surrounded by incredibly beautiful mountains to hike in. One evening, I met the guy who left those traces ... very exciting, slightly scary...

by (Heiko Strathmann) at December 13, 2013 04:45 PM

November 14, 2013 04:18 PM

Keynotes at ACML 2013

We were very lucky this year to have an amazing set of keynote speakers at ACML 2013 who have made key contributions to getting machine learning into the real world. Here are some links to the open source software projects that they mentioned during their talks. The videos of the talks should be available at some point on the ACML website

We started off with Geoff Holmes, who spoke at MLOSS 06. He told us about how WEKA has been used in industry (satisfying Kiri Wagstaff's Challenge #2), and the new project for streaming data MOA. Later in the day, Chih-Jen Lin told us how important it was to understand both machine learning and optimisation, such that you can exploit the special structure for fast training of SVMs. This is how he obtained amazing speedups in LIBLINEAR. On the second day, Ralf Herbrich (who also gave a tutorial) gave us a behind the scenes tour of TrueSkill, the player matching algorithm used on XBox Live. Source code in F# is available here and the version generalised to track skill over time is available here.

Thanks to Geoff, Chih-Jen and Ralf for sharing their enthusiasm!

by Cheng Soon Ong at November 14, 2013 04:18 PM

October 29, 2013 11:38 PM

GSoC 2013 brings Shogun 3.0

Shogun’s third Google Summer of Code just ended with our participation in the mentor summit at Google’s headquarter in Mountain View and the release of Shogun 3.0 (link) What a great summer! But let’s start at the beginning…

Shogun is a toolbox that offers a unified framework for data-analysis, or in buzz words: machine learning, for a broad range of data types and analysis problems. Those not only include standard tools such as regression, classification, clustering, etc, but also cutting edge techniques from recent developments in research. One of Shogun’s most unique features is its interfaces to a wide range of mainstream computing languages.

In our third GSoC, we continued most of the directions taken in previous years such as asking students to contribute code in the application process for them to be considered. For that, we created a list of smaller introductory tasks for each of the GSoC projects that would become useful later in the project. While allowing students to get used to our development process, and increasing the quality of the applications, this also pushed the projects forward a bit before GSoC even started. The number of applications did not suffer through that (57 proposals from 52 students) but even increased compared to the previous year (48 proposals from 38 students) -- this seems to be a trend.

This summer, we also had former GSoC students mentoring for the first time: Sergey Lisitsyn and me (mentoring two projects). Both of us joined in 2011. In addition, the former student Fernando Iglesias participated again and former student Viktor Gal stayed around to work on Shogun during GSoC (and did some massive infrastructure improvements). These are very nice long term effects of continuous GSoC participation. Thanks to GSoC, Shogun is growing constantly both in terms of code and developers.

As in 2012, we eventually could give away 8 slots to some very talented students. All of them did an awesome job on some highly involved projects covering a large number of topics. Two projects were extensions of previous ones: 


Roman Votjakov extended last year’s project on the popular Gaussian Processes for handling classification problems and Shell Hu implemented a collection of algorithms within last year’s structured output framework (for example for OCR)

Fernando Iglesias implemented a new algorithm called metric learning, which plays well together with existing methods in Shogun. 

Another new algorithm came from Soumyajit De, who has implemented an estimation method for log-determinants of large sparse matrices (needed for example for large-scale Gaussian distributions), and implemented a framework for linear operators and solvers, and fundamentals of an upcoming framework for distributed computing (which is used by his algorithm) on the fly. 

Evangelos Anagnostopoulos worked on feature hashing and random kitchen sinks, two very cool tricks to speed up linear and kernel-based learning methods in Shogun. Kevin Hughes implemented methods for independent component analysis, which can be used to separate mixtures of signals (for example audio, heart-beats, or images) and are well known in the community.

Last but not least, Liu Zhengyang created a pretty web-framework for running Shogun demos from the web browser and did add support for directly loading data from the mldata website. Evgeniy Andreev improved Shogun’s usability via integrating native support for various popular file formats such as CSV and protobuf.



You might have noticed the links in the above text (and images). Most of them are the final reports of the students in the form of IPython notebooks, an awesome new open-source tool that we started using for documentation. We are very proud of these.  See for a list of all notebooks. Also check out the web-demo framework at if you haven't yet.

IPython also features Shogun in the cloud: Former student Viktor Gal did setup which is an IPython notebook server ran by us. It allows you to play with Shogun-python from any web-browser without having to install it. You can try the existing notebooks or write your own. Give it a shot and let us know what you think!

This year’s GSoC also was the most productive one for us ever. We got  more than 2000 commits changing almost 400000 lines in more than 7000 files since our last release before GSoC.

Students! You all did a great job and we are more than amazed what you all have achieved. Thank you very much and we hope some of you will stick around.

Besides all the above individual projects, we encouraged students to work together a bit more to enable synergistic effects. One way we tried to implement this was through a peer review where we paired students to check each others interface documentation and final notebooks. We held the usual meetings with both mentors and students every few weeks to monitor progress and happiness, as well as asking students to write weekly reports. Keeping our IRC channel active every day also helped a lot in keeping things going.

My personal experience with mentoring was very positive. It is very nice to give back to the community. I tried to give them the same useful guidance that I received back then, and probably learned as much as my students did on the way. Having participated in GSoC 2011 and 2012, the change of perspective as a mentor was interesting, in particular regarding the selection process. Time wise, I think Google’s official statement of 5 hours per student per week is underestimating things quite a bit (if you want to get things done), and of course there is no upper bound on time you can spend.

Our plan of pairing external mentors with internal developers worked smoothly. As most of our mentors are scientists who tend to be very busy, it is sometimes hard for them to review all code on their own. Combining  big-picture guidance with the in-depth framework knowledge of the paired core developers allowed for more flexibility when allocating mentors for projects. Keep in mind that Shogun is still being organised by only five people (4 former students) plus a hand full of occasional developers, which makes it challenging to supervise 8 projects.

Another change this year was that writing unit-tests were mandatory to get code merged, which made the number of unit tests grew from 50 to more than 600. In the past years, we had seen how difficult it is to write tests at the end of projects, or maintain untested code. Making students do this on-the-fly drastically increased the stability of their code. A challenging side-effect of this was that many bugs within Shogun were discovered (and eventually fixed) which kept students and developers busy.

 As for Shogun itself, GSoC also boosts our community of users, which became so active this year that decided to organise a the first Shogun workshop in Berlin this summer. We had something over 30 participants from all over the world. The Shogun core team also met for the first time in real life, which was nice! We had a collection of talks, discussions, and hands-on sessions. Click here and here for videos and slides.

October brought the mentor summit, which I attended for the first time. This was such a cool event! There was a hotel with hot-tub, lots of goodies on the google campus as for example an on-site barista (!), a GSoC mentor with a robot-dog, and loads of loads of interesting people from interesting open-source projects. Some of these were new to me, some of them are projects that I have been checking out for more than 10 years now.I attended a few fruitful sessions, for example on open-source software for science. Sören hang out with the people he knew from previous years and the cool Debian guys (for which he is a developer too).

After the summit, the Shogun mentor team went hiking in the south Californian desert - I even climbed a rock.

What a great summer!











by (Heiko Strathmann) at October 29, 2013 11:38 PM

October 29, 2013 09:15 PM

Shogun Toolbox Version 3.0 released!

Dear all,

we are proud to announce the 3.0 release of the Shogun Machine-Learning Toolbox. This release features the incredible projects of our 8 hard-working Google Summer of Code students. In addition, you get other cool new features as well as lots of internal improvements, bugfixes, and documentation improvements. To speak in numbers, we got more than 2000 commits changing almost 400000 lines in more than 7000 files and increased the number of unit tests from 50 to 600. This is the largest release that Shogun ever had! Please visit to obtain Shogun.


Here is a brief description of what is new, starting with the GSoC projects, which deserve most fame:

  • Gaussian Process classification by Roman Votjakov
  • Structured Output Learning of graph models by Shell Hu
  • Estimators for log-determinants of large sparse matrices by Soumyajit De
  • Feature Hashing and random kitchen sinks by Evangelos Anagnostopoulos
  • Independent Component Analysis by Kevin Hughes
  • A web-based demo framework by Liu Zhengyang
  • Metric learning with large margin nearest neighbours by Fernando Iglesias
  • Native support for various popular file formats by Evgeniy Andreev


Everyone likes screenshots. Well, we have got something better! All of the above projects (and more) are now documented in the form of IPython notebooks, combining machine learning fundamentals, code, and plots. Those are a great looking way that we chose to document our framework from now on. Have a look at them and feel free to submit your use case as a notebook!

FGM.html GMM.html HashedDocDotFeatures.html LMNN.html SupportVectorMachines.html Tapkee.html bss_audio.html bss_image.html ecg_sep.html gaussian_processes.html logdet.html mmd_two_sample_testing.html

The web-demo framework has been integrated into our website, go check them out.

Other changes

We finally moved to the Shogun build process to CMake. Through GSoC, added a general clone and equals methods to all Shogun objects, and added automagic unit-testing for serialisation and clone/equals for all classes. Other new features include multiclass LDA, and probability outputs for multiclass SVMs. For the full list, see the NEWS.

Workshop Videos and slides

In case you missed the first Shogun workshop that we organised in Berlin last July, all of the talks have been put online.

Shogun in the Cloud

As setting up the right environment for shogun and installing it was always one of the biggest problems for the users (hence the switching to CMake), we have created a sandbox where you can try out shogun on your own without installing shogun on your system! Basically it's a web-service which give you access to your own ipython notebook server with all the shogun notebooks. Of course you are more than welcome to create and share your own notebooks using this service! *NOTE*: This is a courtesy service created by Shogun Toolbox developers, hence if you like it please consider some form of donation to the project so that we can keep up this service running for you. Try shogun in the cloud.


The release has been made possible by the hard work of all of our GSoC students, see list above. Thanks also to Thoralf Klein and Björn Esser for the load of great contributions. Last but not least, thanks to all the people who use Shogun and provide feedback.

Sören Sonnenburg on behalf of the Shogun team (+ Viktor Gal, Sergey Lisitsyn, Heiko Strathmann and Fernando Iglesias)

by Soeren Sonnenburg at October 29, 2013 09:15 PM

October 29, 2013 08:50 PM

Shogun goes cloud. Try out to interactively play with machine learning algorithms or try any of the interactive demos.

The SHOGUN Machine Learning Toolbox is a collection of algorithms designed for unified learning for a broad range of feature types and learning settings, like classification, regression, or explorative data analysis. For more information visit

by Soeren Sonnenburg at October 29, 2013 08:50 PM

September 14, 2013 12:28 AM

Latent Gaussian Models in Reykjavik

This weekend, I visited the beautiful city Reykjavik in Iceland for the first time. I participated in this year's workshop on Latent Gaussian models (link) (in fact mostly spatial statistics) and also presented a poster (link), which is about our recent work on Russian Roulette for intractable likelihoods (arXiv, blog). Met a lot of nice people doing interesting things there.

by (Heiko Strathmann) at September 14, 2013 12:28 AM

September 03, 2013 06:38 PM

Google Summer of Code 2013

Google Summer of Code (GSoC) 2013 has been an absolute blast! The majority of my heavy coding is over so I wanted to post a bit about the experience. It has been fantastic I’ve learned so much and done so many different and unexpected things. And before I go any further I want to give a big thanks to all the Shogun devs who helped me out and made this program so great and also thanks to Google for running such a kick ass program.

My project was to code several Independent Component Analysis algorithms specifically those based on Approximate Joint Diagonalization of matrices. It was pretty cool and similar enough to my thesis work that I was able to jump right in fairly quickly. It was an interesting change of pace for me, as I like to put it: its called Google Summer of Code not Google Summer of Research and having the code be the number one priority was a welcome change of pace for me. As the focus was code I spent a lot of time translating research papers and author’s source code into production code. I like to think I am quite the wizz with porting between numerical libraries now (matlab -> python,  python -> c etc.) Also I’m now so familiar with NumPy, Octave and Eigen3 (and almost R) that I can pretty much work fluently in each and change between them quickly almost without noticing. Have a look at my recent post One Example Three Languages!

One of the other things I got into this summer was playing with the Shogun Modular interfaces which are created using SWIG. I once tried to play with SWIG for one of my own projects but unfortunately never got far. This summer though I updated a few of the typemaps to add support for NDArray which some of my classes needed. I also playing with updating the ruby modular interface to use the newer and more active NMatrix numerical library (not included in Shogun as of yet though). Anyways playing with typemaps was an interesting experience and I definitely learned more than a few things.

One of the other things I learned was the softer side of class and framework design. I realized that even though I’ve been doing OOP for years one thing I still need more experience with is laying out a new class from scratch. The first time I had to do this I actually had to stop for a second and think, I actually wrote a basic foo bar style class example to double check that what I wanted to do would work. In the end I am quite happy with the class structure I came up with and I look forward to being involved in this type of design more often in the future.

Thats all I can think of now! If you’re a student I highly recommend doing GSoC!

Also here is a link to my final project:

by kevinhughes27 at September 03, 2013 06:38 PM

September 01, 2013 10:33 AM

What does the “OSS” in MLOSS mean?

I was recently asked to become an Action Editor for the Machine Learning and Open Source Software (MLOSS) track of Journal of Machine Learning Research. Of course, I gladly accepted since the aim of the JMLR MLOSS track (as well as the broader MLOSS project) -- to encourage the creation and use of open source software within machine learning -- is well aligned with my own interests and attitude towards scientific software.

Shortly after I joined, one of the other editors raised a question about how we are to interpret an item in the review criteria that states that reviewers should consider the "freedom of the code (lack of dependence on proprietary software)" when assessing submissions. What followed was an engaging email discussion amongst the Action Editors about the how to clarify our position.

After some discussion (summarised below), we settled on the following guideline which tries to ensure MLOSS projects are as open as possible while recognising the fact that MATLAB, although "closed", is nonetheless widely used within the machine learning community and has an open "work-alike" in the form of GNU Octave:

Dependency on Closed Source Software

We strongly encourage submissions that do not depend on closed source and proprietary software. Exceptions can be made for software that is widely used in a relevant part of the machine learning community and accessible to most active researchers; this should be clearly justified in the submission.

The most common case here is the question whether we will accept software written for Matlab. Given its wide use in the community, there is no strict reject policy for MATLAB submissions, but we strongly encourage submissions to strive for compatibility with Octave unless absolutely impossible.

The Discussion

There were a number of interesting arguments raised during the discussion, so I offered to write them up in this post for posterity and to solicit feedback from the machine learning community at large.

Reviewing and decision making

A couple of arguments were put forward in favour of a strict "no proprietary dependencies" policy.

Firstly, allowing proprietary dependencies may limit our ability to find reviewers for submissions -- an already difficult job. Secondly, stricter policies have the benefit of being unambiguous, which would avoid future discussions about the acceptability of future submission.

Promoting open ports

An argument made in favour of accepting projects with proprietary dependencies was that doing so may actually increase the chances of its code being forked to produce a version with no such dependencies.

Mikio Braun explored this idea further along with some broader concerns in a blog post about the role of curation and how it potentially limits collaboration.

Where do we draw the line?

Some of us had concerns about what exactly constitutes a proprietary dependency and came up with a number of examples that possibly fall into a grey area.

For example, how do operating systems fit into the picture? What if the software in question only compiles on Windows or OS X? These are both widely used but proprietary. Should we ensure MLOSS projects also work on Linux?

Taking a step up the development chain, what if the code base is most easily built using proprietary development tools such as Visual Studio or XCode? What if libraries such as MATLAB's Statistics Toolbox or Intel's MKL library are needed for performance reasons?

Things get even more subtle when we note that certain data formats (e.g., for medical imaging) are proprietary. Should such software be excluded even though the algorithms might work on other data?

These sorts of considerations suggested that a very strict policy may be difficult to enforce in practice.

What is our focus?

It is pretty clear what position Richard Stallman or other fierce free software advocates would take on the above questions: reject all of them! It is not clear that such an extreme position would necessarily suit the goals of the MLOSS track of JMLR.

Put another way, is the focus of MLOSS the "ML" or the "OSS"? The consensus seemed to be that we want to promote open source software to benefit machine learning, not the other way around.

Looking At The Data

Towards the end of the discussion, I made the argument that if we cannot be coherent we should at least be consistent and presented some data on all the accepted MLOSS submissions. The list below shows the breakdown of languages used by the 50 projects that have been accepted to the JMLR track to date. I'll note that some projects use and/or target multiple languages and that, because I only spent half an hour surveying the projects, I may have inadvertently misrepresented some (if I've done so, let me know).

C++: 15; Java: 13; MATLAB:11; Octave: 10; Python:9; C: 5; R: 4.

From this we can see that MATLAB is fairly well-represented amongst the accepted MLOSS projects. I took a closer look and found that of the 11 projects that are written in (or provide bindings for) MATLAB, all but one of them provide support for GNU Octave compatibility as well.

Closing Thoughts

I think the position we've adopted is realistic, consistent, and suitably aspirational. We want to encourage and promote projects that strive for openness and the positive effects it enables (e.g., reproducibility and reuse) but do not want to strictly rule out submissions that require a widely used, proprietary platform such as MATLAB.

Of course, a project like MLOSS is only as strong as the community it serves so we are keen to get feedback about this decision from people who use and create machine learning software so feel free to leave a comment or contact one of us by email.

Note: This is a cross-post from Mark's blog at Inductio ex Machina.

by Mark Reid at September 01, 2013 10:33 AM

August 17, 2013 06:47 PM

Becoming a Web Developer

Now that Grad School is over I’m moving on to the next exciting chapter of my career – I’m joining a great company called Shopify in the fall and I am going to be working as a Web Developer! I’m quite excited, in fact so excited I took a course from Udacity to get up to speed on some Web Dev basics.

CS 253 Web Development – Building a Blog with Steve Huffman (for those who don’t know this is the guy who started Reddit so he might know a thing or two about building websites) was a really great course and I would definitely recommend it to anyone who wants to learn a thing or two about the web. I also really liked Steve’s teaching style – while he did a great job explaining things simply he also wasn’t afraid to show how he really works i.e. in the terminal and using Linux/Unix commands etc. The course could have easily hidden all of this away but I think it was important to show – using windows and gui’s just isn’t how people work in this industry so why should the course be taught like so? Good job Steve for keeping it real!

Taking the class was really worthwhile as it helped tie together a bunch of knowledge I had accumulated randomly over the years and it helped make some sense of some of the Django hacking I did once upon a time(I say hacking because I got stuff to work but didn’t totally understand everything).

Here is my hard-earned certificate! I completed the course with High Distinction meaning I did all the homework, the final and the bonus question!

blog - cs253

I also pushed all my code to my GitHub account:

by kevinhughes27 at August 17, 2013 06:47 PM

August 14, 2013 12:00 AM

Code review for science

How good is the software associated with scientific papers? There seems to be a general impression that the quality of scientific software is not that great. How do we check for software quality? Well, by doing code review.

In an interesting experiment between the Mozilla Science Lab and PLoS Computational Biology, a selected number of papers with snippets of code from the latter will be reviewed by engineers from the former.

For more details see the blog post by Kaitlin Thaney.

by Cheng Soon Ong at August 14, 2013 12:00 AM

August 09, 2013 06:58 PM

August 05, 2013 06:10 PM

One Example Three Languages

I wanted to post this example of my Google Summer of Code work because I think its neat. One of the cool things about Shogun is our great SWIG wrapper and our static interface which lets us use Shogun natively in a bunch of different languages. So here is an example program of doing Blind Source Separation using the Jade algorithm from Python, Octave and R:

Blind Source Separation using the Jade Algorithm with Shogun
Based on the example from scikit-learn

Kevin Hughes 2013

import numpy as np
import pylab as pl

from shogun.Features  import RealFeatures
from shogun.Converter import Jade

# Generate sample data
n_samples = 2000
time = np.linspace(0, 10, n_samples)

# Source Signals
s1 = np.sin(2 * time)  # sin wave
s2 = np.sign(np.sin(3 * time))  # square wave
S = np.c_[s1, s2]
S += 0.2 * np.random.normal(size=S.shape)  # add noise

# Standardize data
S /= S.std(axis=0)  
S = S.T

# Mixing Matrix
A = np.array([[1, 0.5], [0.5, 1]])

# Mix Signals
X =,S)
mixed_signals = RealFeatures(X)

# Separating
jade = Jade()
signals = jade.apply(mixed_signals)
S_ = signals.get_feature_matrix()
A_ = jade.get_mixing_matrix();

# Plot results
pl.subplot(3, 1, 1)
pl.title('True Sources')
pl.subplot(3, 1, 2)
pl.title('Mixed Sources')
pl.subplot(3, 1, 3)
pl.title('Estimated Sources')
pl.subplots_adjust(0.09, 0.04, 0.94, 0.94, 0.26, 0.36)
% Blind Source Separation using the Jade Algorithm with Shogun
% Based on the example from scikit-learn
% Kevin Hughes 2013

% Generate sample data
n_samples = 2000;
time = linspace(0,10,n_samples);

% Source Signals
S = zeros(2, length(time));
S(1,:) = sin(2*time);
S(2,:) = sign(sin(3*time));
S += 0.2*rand(size(S));

% Standardize data
S = S ./ std(S,0,2);

% Mixing Matrix
A = [1 0.5; 0.5 1]

% Mix Signals
X = A*S;
mixed_signals = X;

% Separating
sg('set_converter', 'jade');
sg('set_features', 'TRAIN', mixed_signals);
S_ = sg('apply_converter');

% Plot
plot(time, S(1,:), 'b');
hold on;
plot(time, S(2,:), 'g');
set(gca, 'xtick', []) 
title("True Sources");

plot(time, X(1,:), 'b');
hold on;
plot(time, X(2,:), 'g');
set(gca, 'xtick', []) 
title("Mixed Sources");

plot(time, S_(1,:), 'b');
hold on;
plot(time, S_(2,:), 'g');
title("Estimated Sources");
# Blind Source Separation using the Jade Algorithm with Shogun
# Based on the example from scikit-learn
# Kevin Hughes 2013


# Generate sample data
n_samples <- 2000
time <- seq(0,10,length=n_samples)

# Source Signals
S <- matrix(0,2,n_samples)
S[1,] <- sin(2*time)
S[2,] <- sign(sin(3*time))
S <- S + 0.2*matrix(runif(2*n_samples),2,n_samples)

# Standardize data
S <- S * (1/apply(S,1,sd))

# Mixing Matrix
A <- rbind(c(1,0.5),c(0.5,1))

# Mix Signals
X <- A %*% S
mixed_signals <- matrix(X,2,n_samples)

# Separating
sg('set_converter', 'jade')
sg('set_features', 'TRAIN', mixed_signals)
S_ <- sg('apply_converter')

# Plot

plot(time, S[1,], type="l", col='blue', main="True Sources", ylab="", xlab="")
lines(time, S[2,], type="l", col='green')

plot(time, X[1,], type="l", col='blue', main="Mixed Sources", ylab="", xlab="")
lines(time, X[2,], type="l", col='green')

plot(time, S_[1,], type="l", col='blue', main="Estimated Sources", ylab="", xlab="")
lines(time, S_[2,], type="l", col='green')

by kevinhughes27 at August 05, 2013 06:10 PM

August 02, 2013 10:02 PM

Successfully Defended my Thesis!

Yesterday I defended my thesis titled “Subspace Bootstrapping and Learning for Background Subtraction”. Grad School has been a blast but I’m definitely looking forward to employed life!

I’ll post a link to my thesis under publications as soon as Queen’s uploads it to their system.

by kevinhughes27 at August 02, 2013 10:02 PM

July 17, 2013 11:05 AM

Shogun Workshop 2013

Last weekend, our Shogun workshop finally took place in Berlin. It was really cool to meet all those guys in person. We have been working together for quite some time now. The core-team an Shogun's supporters are absolutely awesome. It is great to be part of that.

We had a nice afternoon at c-base (who were so friendly to host us) with some talks by all of our developers, followed by two days of hands-on workshop at the TU-Berlin.

I gave a little talk on two random things you can do with kernels (that are completely unrelated): Gaussian Processes and the kernel MMD. Slides are (download). I also wrote some IPython notebooks for GP-regression (link), GP-probit-classification (link), and two-sample testing with the kernel MMD (link).
One of the results of our discussions was that we will start using those notebook for Shogun's documentation as they allow to combined code, plots, and maths in a web-based viewer.

Finally, here are some picture of us, (pretty nerdy)


by (Heiko Strathmann) at July 17, 2013 11:05 AM

July 15, 2013 03:02 PM

8 Reasons Why Better Nutrition Makes You a Better Developer

Reblogged from The Code Barbarian:

Software developers are not known for having the best nutrition. When it comes to development work, the stereotypical late night Red Bull-fueled coding binge is often not too far from the truth. It's hard to imagine a hackathon without a stack of pizza boxes and a mountain of empty soda bottles. In addition, no good tech firm lets their kitchen run out of chips or Vitamin Water.

Read more… 1,435 more words

As a fellow primal/paleo software developer my experiences have been similar!

by kevinhughes27 at July 15, 2013 03:02 PM

July 09, 2013 11:05 PM

Shogun Toolbox Days 2013 Program and Updates

Dear all,

we are excited that the first Shogun workshop, July 12-14 in Berlin, is getting closer. Thanks to all the people that signed up -- we are sure it will be a packed and inspiring weekend!

We have finalized the schedule for Friday, July 12, taking place at the C-Base (see description below [1]). After an intro where everyone gets to know each other and where we introduce ourselves, Shogun, and Machine Learning in general, there will be some tutorials by Shogun developers. In addition, we will have discussions about various topics and various coffee breaks and lunch. Finally, we will enjoy a summer's evening in Berlin.

On Saturday and Sunday, July 13-14 there will be hands on sessions at the Technical University Berlin [2], where developers are around for more close up discussions and practical guidance. Bring your Laptop if you want to try things.

See the final schedule for more details [1]. We plan to do video recordings of all lectures and will have a live stream [3].

See you there! The Shogun-Team

by Soeren Sonnenburg at July 09, 2013 11:05 PM

July 07, 2013 04:00 PM

GSoC Weekly Report 3

This week I wrote a hdf5 data importer to all demos in shogun-demo. For the sake of that, I made the coordinate scalable, and changed all the demo’s input domain to fit the toy_data_set. Now it can only loads features and labels in australian.libsvm.h5 in shogun-data, I’ll make it available to accept more in next week.

Next week, I’ll finish the web-based ocr demo as in, and a dimensional reduction demo, and add more function on toy_data generator/importer.

July 07, 2013 04:00 PM

June 30, 2013 04:00 PM

GSoC Weekly Report 2

This week, I did a lot of refractoring on shogun-demo. Now the demo framework can auto-generate code(js, css, html) for coordinate-system, json interact, mouse click input, argument handle , and heatmap drawing. demo creator only need to specify what argument demo needed, what shape will be plot, some details for the coordinate system, and a python backend to tell server how data will be processed, then a new demo created! I also rewrite the toy data generator, now it’s fully modular, when demo creator want to add a toy data generator, he can add it with only one line code. toy-data importer are half finished, it’ll will finish in tomorrow. I merged dvalcarce’s binary classification and binary perceptron demo into this repo. They’re now available:, .

next week, I’ll prepare some data set from, for toy data importer use. and I’ll make demos as more as possible. and add exceptional input check on existing framework(defend ddos attack).

June 30, 2013 04:00 PM

June 28, 2013 05:19 PM

Book Review: Instant OpenCV Starter

I was recently contacted by Packt Publishing to do a quick book review of their new OpenCV book Instant OpenCV Starter. I was quite flattered that they contacted me, my internet presence must be improving, and they were going to provide me a free copy to review so I agreed!

I just finished reading the book and I do think it is quite good! Its on the shorter side but its also not that expensive and I would definitely recommend it to people who are just starting with OpenCV (and maybe who aren’t the most rockstar programmers) and also to professors who are going to be teaching a Computer Vision course.

Perhaps the most helpful chapter to newcomers to OpenCV is the introduction and the installation tutorial. The book does a very nice job at describing how to install OpenCV for both Linux and Windows, the latter being quite similar to my own tutorial. I also liked how they added some details about the various dependencies required for building on Linux.

The book wraps up with some fairly simple OpenCV examples and one really cool example of Image Steganography. And as always with Packt the source code is available from their site and is of good quality.

I’d definitely recommend this book to newcomers especially for the install tutorial which is an area that the greater OpenCV community should address better. Also if you are a more advanced OpenCV user or you finish this book and are looking for more I would recommend the OpenCV 2 Cookbook and definitely the Mastering OpenCV with Practical Computer Vision Projects.

by kevinhughes27 at June 28, 2013 05:19 PM

June 26, 2013 04:10 PM

June 26, 2013 01:56 PM

Russian Roulette for intractable Likelihoods

\( \def\mm#1{\boldsymbol{#1}} \DeclareMathOperator{\tr}{tr} \)

While I was working at UCL's Statistics Department in winter, I got involved into a very exciting project in the group of Mark Girolami. It is based around the Pseudo-Marginal Metropolis-Hastings algorithm. In 2003, a Genetics paper [1] described an approach to sample a distribution using the standard Metropolis-Hastings algorithm when the density function is not available by simply replacing it with an unbiased estimate.

For a standard Bayesian inference problem with likelihood \(\pi(y|\theta) \), prior \(\pi(\theta)\), and a proposal \(Q\), rather than using the standard M-H ratio \[ \frac{\pi(y|\theta^{\text{new}})}{\pi(y|\theta)}\times\frac{\pi(\theta^{\text{new}})}{\pi(\theta)}\times \frac{Q(\theta|\theta^{\text{new}})}{Q(\theta^{\text{new}}|\theta)}, \] the likelihood is replaced by an unbiased estimator as

\[ \frac{\hat{\pi}(y|\theta^{\text{new}})}{\hat{\pi}(y|\theta)}\times\frac{\pi(\theta^{\text{new}})}{\pi(\theta)}\times \frac{Q(\theta|\theta^{\text{new}})}{Q(\theta^{\text{new}}|\theta)}.\] Remarkably  the resulting Markov chain converges to the same posterior distribution as the exact algorithm. The approach was later formalised and popularised in [2].

In our project, we exploited this idea to perform inference over models whose likelihood functions are intractable. Example of such intractable likelihoods are for example Ising models or, even simpler, very large Gaussian models. Both of those models' normalising constants are very hard to compute. We came up with a way of producing unbiased estimators for the likelihoods, which are based on writing likelihoods as an infinite sum, and then truncating it stochastically.

Producing unbiased estimators for the Pseudo-Marginal approach is a very challenging task. Estimates have to be strictly positive. This can be achieved via pulling out the sign of the estimates in the final Monte-Carlo integral estimate and add a correction term (which increases the variance of the estimator). This problem is studied under the term Sign problem. The next step is to write the likelihood function as an infinite sum. In our paper, we do this for a geometrically titled correction of a biased estimator obtained by an approximation such as importance sampling estates, upper bounds, or deterministic approximations, and for likelihoods based on the exponential function.

I in particular worked on the exponential function estimate. We took a very nice example from spatial statistics: a worldwide grid of ozone measurements from a satellite that consists of a about 173,405 measurements. We fitted a simple Gaussian model whose covariance matrices are massive (and sparse). In such models of the form \[ \log (\mathcal{N}_x(\mu,\Sigma))=-\log(\det(\Sigma)) - (\mu-x)^T \Sigma^{-1}(\mu-x) + C, \] the normalising constant involves a log-determinant of such a large matrix. This is impossible using classical methods such as Cholesky factorisation \[ \Sigma=LL^T \Rightarrow \log(\det(\Sigma))=2\sum_i\log(L_{ii}), \] due to memory shortcomings: It is not possible to store the Cholesky factor \(L\) since it is not in general sparse. We therefore constructed an unbiased estimator using a very neat method based on graph colourings and Krylov methods from [3].

This unbiased estimator of the log-likelihood is then turned into a (positive) unbiased estimator of the likelihood itself via writing the exponential function as an infinite series \[ \exp(\log(\det(\Sigma)))=1+\sum_{i=1}^\infty \frac{\log(\det(\Sigma))^i}{i!}. \]

We then construct an unbiased estimator of this series by playing Russian Roulette: We evaluate the terms in the series and plug in a different estimator for \(\log(\det(\Sigma))\) for every \(i\); once those values are small, we start flipping a coin every whether we continue the series or not. If we do continue, we add some weights that ensure unbiasedness. We also ensure that it is less likely to continue in every iteration so that the procedure eventually stops. This basic idea (borrowed from Physics papers from some 20 years ago) and some technical details and computational tricks then give an unbiased estimator of the likelihood of the log-determinant of our Gaussian model and can therefore be plugged into Pseudo-Marginal M-H. This allows to perform Bayesian inference over models of sizes where it has been impossible before.

More details can be found on our project page (link, see ozone link), and in our paper draft on arXiv (link). One of my this year's Google summer of Code projects for the Shogun Machine-Learning toolbox is about producing a sophisticated implementation of log-determinant estimators (link). Pretty exciting!

[1]: Beaumont, M. A. (2003). Estimation of population growth or decline in genetically monitored populations. Genetics 164 1139–1160.
[2]: Andrieu, C., & Roberts, G. O. (2009). The pseudo-marginal approach for efficient Monte Carlo computations. The Annals of Statistics, 37(2), 697–725.
[3]: Aune, E., Simpson, D., & Eidsvik, J. (2012). Parameter Estimation in High Dimensional Gaussian Distributions.

by (Heiko Strathmann) at June 26, 2013 01:56 PM

June 23, 2013 04:00 PM

GSoC Weekly Report 1

Sorry that I’ve been busying for my finals last week, I’ll catch up the progress this week. Last week I’ve done some code refractoring on the existing demos, and made the django site looks like a little framework. I planed to do the following things this week:

  1. integrate the data to the exisiting demos. (regression, clustering)
  2. merge the exisiting binary/multiclass classification demo into the framework.
  3. refractor the javascripts.
  4. find a better way to draw heatmap with contour.

June 23, 2013 04:00 PM

June 06, 2013 08:44 AM

10th Anniversary of DIMVA!

I am happy to announce the 10th anniversary of the security conference DIMVA! The conference takes place 17-19 July 2013 in Berlin, Germany. For now 10 years, DIMVA has brought together experts from academia, industry, and government to discuss research on detection of intrusions and malware. In celebration of the 10th anniversary the conference program is packed with highlights:
The conference venue is the Mövenpick Hotel Berlin that is located in the centre of Berlin. The registration for the conference is now open. Do not miss the early bird deadline: June 12! You can find more details at the DIMVA 2013 website.

See you in Berlin!

by (Konrad Rieck) at June 06, 2013 08:44 AM

May 31, 2013 09:33 PM

Google Summer of Code 2013: Acceptance

Yahoo! I have been accepted to Google Summer of Code 2013! This summer I’ll be working on Gaussian Processes for Classification project for the SHOGUN Machine Learning Toolbox. More information about ideas of the project can be found at GSoC 2013 ideas page at the website of the SHOGUN Machine Learning Toolbox. My proposal for the project can be found at here.

I’m very excited to be a part of the team of SHOGUN developers and to have two great mentors: Heiko Strathmann and Oliver Stegle, who will be mentoring me during GSoC 2013. I think this summer should be quite fun :)

Weekly reports and other important and interesting information about my progress on the project will be posted here.

May 31, 2013 09:33 PM

May 29, 2013 04:00 PM

Google Summer of Code 2013 Accepted

Google Summer of Code 2013

After about a month’s preparation, I got accepted into GSoC 2013 by shogun-toolbox. Thanks to all the members of shogun, especially Soeren Sonnenburg, thanks to Google, thanks to all the people who supported me.

In this summer, I’ll devote myself to implementing graphical interactive demos for the ML algorithms provided by shogun-toolbox. Progress will be reported weekly in this blog.

Next I’m going to do some code refractoring on the previous demos, for the maximization of code-reuseing. During the time before the official GSoC startup date(June, 17), I’ll spend about (3-4 hours)/day on the project.

It’s my first time participate in developing open source software, so amazing.

May 29, 2013 04:00 PM

May 27, 2013 08:42 PM

Google Summer of Code 2013

I’ve been accepted to Google Summer of Code 2013! I’ll be working with the great group behind the Shogun Machine Learning Toolbox I am going to be working on Approximate Joint Diagonalization (ADJ) of matrices for Independent  Component Analysis (ICA) and Blind Source Separation (BSS) – think the cocktail party problem.

I’ll be posting about my progress here regularly so stay tuned!

by kevinhughes27 at May 27, 2013 08:42 PM

May 16, 2013 07:28 PM

Open Sourced a bunch of Robot Code today!

I cleaned up and released all our code for the 2013 FRC season. It’s not crazy documented or anything but this was the code that was on our robot so maybe it will help somebody.

Also check out our scouting database using python, tkinter, sqlite3, numpy and matplotlib. It worked great for our scouting needs this year and more importantly the students who worked on it became pretty proficient with some important tools!

by kevinhughes27 at May 16, 2013 07:28 PM

May 13, 2013 06:27 PM

libBGS released!

Today I release some of the code I’ve been using for my masters as libBGS. I created libBGS from modifying the original libBGS by Donovan Parks. The major changes include updating to use OpenCV 2.xx and making better use of the stl when appropriate. I also added several algorithms including a Gaussian Mixture Model (GMM) variant, simple frame differencing and some upgrades to the Eigenbackground implementation.

The code is use-able but still a WIP, I’d like to try and add serialization so that the background models and be saved and loaded nicely.

check out the project page:

by kevinhughes27 at May 13, 2013 06:27 PM

May 08, 2013 09:04 PM


Finally got a chance to blog about the most recent architectural improvements to ARPool – Singletons! Singletons make so much sense for ARPool its insane, I can’t believe we didn’t have it like this before.

Backing up for those who aren’t familiar with what I’m talking about, a singleton is an object for which we enforce the constraint that there will only ever be a single instance of that object. How do we do this? easy make the constructor private, err okay but then how do we instantiate the singleton in the first place? This one isn’t hard but I wouldn’t say easy, we make a public method called getInstance which returns a pointer to the single instance of the class if it has already been initialized or creates it if it has not. The class also has 2 static variables to hold the single instance and a flag indicating whether it has been instantiated yet. The code looks like this:

// header
class Singleton

	static bool instanceFlag;
	static Singleton* single;

	Singleton* getInstance();

	// the rest of the methods...


// cpp file
bool Singleton::instanceFlag = false;
Singleton* Singleton::single = NULL;

Singleton* Singleton::getInstance()
		single = new Singleton();
		instanceFlag = true;
		return single;
		return single;

So why would we want to do this? when is this useful? The best example is whenever you are dealing with hardware or an actual physical element. For example in ARPool there is only one camera so it makes sense to make the camera class a singleton.

The major advantage of using singletons is that you can include the class where ever you like and simply ask for the instance – this can greatly simplify your code because you no longer have to pass all of these objects to each other to access them. A really basic way to think about Singletons is to treat them as safe global objects.

and thats really all there is to it! I’m just going to sit here and smile thinking about how much cleaner ARPool’s code is now mmmmmmmmmmmmm!

by kevinhughes27 at May 08, 2013 09:04 PM

May 06, 2013 12:05 AM

New ARPool Website is Live!

In prep for our big show of ARPool at the Augmented World Expo 2013 (AWE) in Santa Clara California I got the okay to re-do the website a bit. Now I’m a far-cry from a front-end engineer (just have never had enough time to learn!) but I am pretty stoked with what I put together in a couple of hours.

Thanks to these great weekly events called Queen’s Hacks where a bunch of us get together and work on cool side projects I’d been exposed to Twitter Bootstrap, which is essentially the quintessential css and js library for building a modern website / web app. Big shout out to Twitter for this awesome tool and an even bigger shout out for making it open source! Go Twitter!

Anyways I am pretty hack and slash when it comes to css and js so I don’t have anything neat to report other than to say hey check out this flashy new site I made! 

by kevinhughes27 at May 06, 2013 12:05 AM

May 03, 2013 10:57 PM

Shogun Student Applications Statistics for Google Summer of Code 2013

Almost a month has passed since SHOGUN has been accepted for Google Summer of Code 2013. Student application deadline was today (May 6) and shogun received 57 proposals from 52 students. This is quite some increase compared to 2012 (48 applications from 38 students). What is interesting though is that it didn't look that good in the very beginning (see the figure below):

Comparing this to 2012, this curve is much more flat in the beginning but exponentially increasing towards the end. Why is that? We didn't change the way we engaged with students (even though we tried to improve the instructions and added lots of entrance tagged tasks to github issues). We still require patches to be submitted to even be considered. So it is similarly tough to get into gsoc 2013 with us as it was in the previous year.

What is interesting though is that various organizations complained about a slow uptake in the beginning. And it turns out that google did limit the number of student applications from 20 (last year) to 5 (in 2013). This might explain the shape of the curve: Students are more cautious to apply but once the deadline is near the apply to the maximum of 5 to improve their chances. This goes hand-in-hand with the observation that the quality of newly submitted student applications tends to decrease towards the deadline.

So did this new limit hurt? To the contrary! In the end the quality of proposals increased a lot and we were able to even way before the student application deadline start to score/rank students. We are happy to have many very strong candidates this year again. Lets hope we get enough slots to accommodate all of the excellent students and then lets start the fun :)

by Soeren Sonnenburg at May 03, 2013 10:57 PM

April 17, 2013 08:27 PM

CfP: Shogun Machine Learning Workshop, July 12-14, Berlin, Germany

CALL FOR PARTICIPATION: Shogun Machine Learning Workshop, Berlin, Germany, July 12-14, 2013

Data Science, Big-Data are omnipresent terms documenting the need for automated tools to analyze the ever growing wealth of data. To this end we invite practitioners, researchers and students to participate in the first Shogun machine learning workshop. While the workshop is centered around the development and use of the shogun machine learning toolbox, it will also feature general machine learning subjects.

General Information

The workshop will include:
  • A general introduction to machine learning held by Gunnar Raetsch.
  • Introductory talks about e.g. Dimension reduction techniques, Kernel-statistical testing, Gaussian Processes, Structured Output learning.
  • Contributed talks and a poster session, and a poster-spotlight.
  • A discussion panel
  • A hands on session on July 13-14

Do not miss the chance to familiarize yourself with the shogun machine learning toolbox for solving various data analysis tasks and to talk to their authors and contributors. The program of the workshop will cover from basic to advanced topics in machine learning and how to approach them using Shogun, which makes it suitable for anyone, no matter if you are a senior researcher or practitioner with many year's of experience, or a junior student willing to discover much more. Interested?

A tentative schedule is available at

Call for contributions

The organizing committee is seeking workshop contributions. The committee will select several submitted contributions for 15-minute talks and poster presentations. The accepted contributions will also be published on the workshop web site.

Amongst other topics, we encourage submission that

  • are applications / publications utilizing Shogun
  • are highly relevant to practitioners in the field
  • are of broad general interest
  • are extensions to Shogun

Submission Guidelines

Send an abstract of your talk/contribution to before June 1. Notifications will be given on June 7.


Workshop registration is free of charge. However, only a limited number of seats is available. First-come, first-served! Register by filling out the registration form.

Location and Timeline

The main workshop will take place at c-base Berlin (, on July 12. It is followed by additional 2-day hands-on sessions held at TU Berlin on July 13-14.

About the Shogun machine learning toolbox

Shogun is designed for unified large-scale learning for a broad range of feature types and learning settings, like classification, regression, or explorative data analysis. Further information is available at

by Soeren Sonnenburg at April 17, 2013 08:27 PM

What's New

Jan. 26, 2015 -> SHOGUN 4.0.0
Feb. 17, 2014 -> SHOGUN 3.2.0
Jan. 6, 2014 -> SHOGUN 3.1.1
Jan. 5, 2014 -> SHOGUN 3.1.0
Oct. 28, 2013 -> SHOGUN 3.0.0
March 17, 2013 -> SHOGUN 2.1.0
Sept. 1, 2012 -> SHOGUN 2.0.0