| --- Log opened Wed Jan 29 00:00:08 2014 | ||
| --- Day changed Wed Jan 29 2014 | ||
| @wiking | thoralf: i hope that the other unit tests (libmrm etc) are convex problems | 00:00 |
|---|---|---|
| @wiking | as they cannot solve non-convex problems | 00:00 |
| thoralf | lol | 00:00 |
| thoralf | At least they converge. ;) | 00:00 |
| @wiking | well that shouldn't mean anything | 00:01 |
| @wiking | if u test it on a non-convex example | 00:01 |
| @wiking | that's up to luck that it does anything meaningful | 00:01 |
| thoralf | In the linear case it shouldn't be a big deal. | 00:04 |
| @wiking | ? | 00:04 |
| thoralf | Forget what I said. It's simply wrong. | 00:07 |
| thoralf | No excuse. ;) | 00:07 |
| @wiking | okey :D | 00:08 |
| thoralf | Let me know if you can reproduce my observation... | 00:13 |
| thoralf | Have a good night... | 00:13 |
| thoralf | Zzz | 00:13 |
| -!- thoralf [~thoralf@91-65-136-138-dynip.superkabel.de] has quit [Quit: Konversation terminated!] | 00:15 | |
| -!- FSCV [~FSCV@50.7.50.60] has quit [Quit: Leaving] | 01:11 | |
| -!- zxtx [~zv@129-79-241-148.dhcp-bl.indiana.edu] has quit [Ping timeout: 252 seconds] | 03:40 | |
| -!- Skg_ [82c2a6a3@gateway/web/freenode/ip.130.194.166.163] has joined #shogun | 03:53 | |
| Skg_ | Hi, I was wondering if anybody would be able to help me out with an issue I've encountered. I've just built v3.1.1, and have encountered an issue while running through the modular tutorial (http://www.shogun-toolbox.org/doc/en/current/modular_tutorial.html). When I try and call the function Labels, I receive the error "Cannot create new instances of the type 'Labels'" - is there anything that I might have missed that would cause such | 03:55 |
| Skg_ | Ah, as it turns out it's an issue with the tutorial - labels no longer exists, but there are binary/multiclass/regression labels available. | 03:59 |
| Skg_ | The tutorial also appears to have an issue with calling PerformanceMeasures - it looks like my shogun.evaluation does not include PerformanceMeasures. | 04:05 |
| -!- zxtx [~zv@c-98-223-196-32.hsd1.in.comcast.net] has joined #shogun | 04:25 | |
| -!- Skg_ [82c2a6a3@gateway/web/freenode/ip.130.194.166.163] has quit [Ping timeout: 245 seconds] | 04:48 | |
| -!- Skg_ [cba6ea41@gateway/web/freenode/ip.203.166.234.65] has joined #shogun | 07:23 | |
| -!- besser82 [quassel@fedora/besser82] has quit [Ping timeout: 276 seconds] | 07:35 | |
| lisitsyn | argh -30C here I don't wanna get out | 07:40 |
| -!- Skg_ [cba6ea41@gateway/web/freenode/ip.203.166.234.65] has quit [Quit: Page closed] | 07:52 | |
| -!- besser82 [quassel@fedora/besser82] has joined #shogun | 08:01 | |
| -!- lisitsyn [~lisitsyn@80.252.20.67] has quit [Ping timeout: 252 seconds] | 08:37 | |
| -!- benibadman [~benibadma@94.135.236.129] has joined #shogun | 08:56 | |
| -!- zxtx [~zv@c-98-223-196-32.hsd1.in.comcast.net] has quit [Ping timeout: 252 seconds] | 09:00 | |
| -!- zxtx [~zv@c-98-223-196-32.hsd1.in.comcast.net] has joined #shogun | 09:09 | |
| -!- lisitsyn [~lisitsin@mxs.kg.ru] has joined #shogun | 09:21 | |
| -!- benibadm_ [~benibadma@94.135.236.129] has joined #shogun | 10:06 | |
| -!- benibadman [~benibadma@94.135.236.129] has quit [Ping timeout: 272 seconds] | 10:09 | |
| -!- benibadman [~benibadma@94.135.236.129] has joined #shogun | 10:38 | |
| -!- benibadm_ [~benibadma@94.135.236.129] has quit [Ping timeout: 245 seconds] | 10:40 | |
| -!- benibadm_ [~benibadma@94.135.236.129] has joined #shogun | 11:10 | |
| -!- benibadman [~benibadma@94.135.236.129] has quit [Ping timeout: 260 seconds] | 11:13 | |
| @wiking | lisitsyn: :D | 11:16 |
| @wiking | lisitsyn: get a jacket :) | 11:16 |
| lisitsyn | wiking: every cloth is double now :D | 11:19 |
| @wiking | lisitsyn: so change or not to change that is the question *now* | 11:22 |
| -!- votjakovr [~votjakovr@188.134.46.30] has joined #shogun | 11:40 | |
| @wiking | hehe | 12:23 |
| @wiking | we need a spark interface :) | 12:23 |
| @wiking | lisitsyn: here? | 13:07 |
| -!- HeikoS1 [~heiko@pat-191-250.internal.eduroam.ucl.ac.uk] has joined #shogun | 13:14 | |
| @wiking | HeikoS1: yo | 13:24 |
| lisitsyn | wiking: ya | 13:50 |
| @wiking | lisitsyn: have u checked mesos or spark? | 13:52 |
| lisitsyn | wiking: hmm no | 13:54 |
| @wiking | k | 14:21 |
| @wiking | :D | 14:21 |
| HeikoS1 | wiking: yo! | 15:37 |
| -!- lisitsyn [~lisitsin@mxs.kg.ru] has quit [Quit: Leaving.] | 16:02 | |
| @wiking | HeikoS1: ok so the plan for the new backend is actualy this: spark or mesos | 16:12 |
| @wiking | HeikoS1: http://spark.incubator.apache.org/docs/latest/python-programming-guide.html | 16:13 |
| @wiking | check the Interactive Use part | 16:13 |
| @wiking | basically depending on a env variable | 16:14 |
| @wiking | it'll run your task either on a spark cluster (or mesos) or on 4 local cores :P | 16:14 |
| HeikoS1 | wiking: looks and sounds good, I will check this out | 16:32 |
| -!- lisitsyn [~lisitsyn@80.252.20.67] has joined #shogun | 17:13 | |
| -!- benibadm_ [~benibadma@94.135.236.129] has quit [] | 17:43 | |
| -!- iglesiasg [~iglesiasg@524AE0A7.cm-4-3d.dynamic.ziggo.nl] has joined #shogun | 18:32 | |
| -!- mode/#shogun [+o iglesiasg] by ChanServ | 18:32 | |
| -!- shogun-notifier- [~irker@7nn.de] has joined #shogun | 18:33 | |
| shogun-notifier- | shogun: Parijat Mazumdar :develop * ccd01c4 / / (12 files): https://github.com/shogun-toolbox/shogun/commit/ccd01c4ed7d60742af893115b879905090dfd97f | 18:33 |
| shogun-notifier- | shogun: minor changes in KMeans | 18:33 |
| shogun-notifier- | shogun: Fernando Iglesias :develop * 1bc8681 / / (12 files): https://github.com/shogun-toolbox/shogun/commit/1bc868162dad0f4d61552b66fef4c8fc8c2e49e6 | 18:33 |
| shogun-notifier- | shogun: Merge pull request #1831 from mazumdarparijat/fastkmeans | 18:33 |
| shogun-notifier- | shogun: | 18:33 |
| shogun-notifier- | shogun: minor changes in KMeans | 18:33 |
| -!- lambday [67157c4d@gateway/web/freenode/ip.103.21.124.77] has joined #shogun | 18:44 | |
| lambday | HeikoS1: hi... there? | 18:44 |
| HeikoS1 | lambday: hi! | 18:44 |
| HeikoS1 | how are things? | 18:44 |
| lambday | HeikoS1: hi.. things are good... | 18:44 |
| lambday | HeikoS1: I was studying about multithread things.. and bumped into things like thread pooling... as lisitsyn was saying... | 18:45 |
| lambday | HeikoS1: I was working on a prototype.. | 18:45 |
| lambday | which maintains a job queue | 18:45 |
| lambday | and there are a bunch of working threads.. which wait for jobs to be available on that queue | 18:46 |
| lambday | and as soon as they get jobs... they wake up and start executing | 18:46 |
| lambday | this is the idea we initially had | 18:46 |
| lambday | right? | 18:46 |
| lambday | oh but before that.... your exams are over? | 18:46 |
| HeikoS1 | lambday: yeah that was the idea | 18:46 |
| HeikoS1 | lambday: exams are over | 18:47 |
| HeikoS1 | finally :) | 18:47 |
| lambday | HeikoS1: how was it? | 18:47 |
| lambday | hehhe :D | 18:47 |
| HeikoS1 | lambday: mixed | 18:47 |
| HeikoS1 | lambday: so yeah this could be a nice system of implementing the multithread stuff | 18:47 |
| HeikoS1 | maybe something easier would be good to start with as a prototype for the interfaces | 18:47 |
| lambday | HeikoS1: yeah... and from what I checked out, c++11 has a really cool library for multithread things | 18:47 |
| lambday | HeikoS1: and IMO its better if we could use that... cause pthread is too system specific... | 18:48 |
| HeikoS1 | lambday: I know, that might be useful | 18:48 |
| HeikoS1 | lambday: I dont know whether we are allowed to do that | 18:48 |
| HeikoS1 | lisitsyn: might have an answer to it :) | 18:48 |
| -!- HeikoS1 [~heiko@pat-191-250.internal.eduroam.ucl.ac.uk] has left #shogun [] | 18:48 | |
| -!- HeikoS1 [~heiko@pat-191-250.internal.eduroam.ucl.ac.uk] has joined #shogun | 18:48 | |
| lisitsyn | allowed to do what? | 18:48 |
| lambday | HeikoS1: if the computation engine part is drawn out of shogun and put into an external library, then probably it can be... | 18:48 |
| lambday | and we don't extend SGObject for these classes | 18:49 |
| lambday | may be then?? | 18:49 |
| HeikoS1 | lambday: we will still need an internal wrapper for things | 18:49 |
| HeikoS1 | so the first step is to write a very very easy multithread implementation | 18:49 |
| HeikoS1 | to finilise the interfaces | 18:49 |
| HeikoS1 | and then once that works think about the backend | 18:49 |
| lambday | HeikoS1: currently what I wrote is an extension of CIndependentComputationEngine... (not fully working yet) | 18:50 |
| lambday | but it pretty much does what we earlier discuss about | 18:50 |
| lambday | calls the compute on job... | 18:50 |
| HeikoS1 | lambday: ok that sounds good | 18:50 |
| HeikoS1 | lambday: so, why dont we start with an openmp backend | 18:50 |
| HeikoS1 | that seems very simpole | 18:50 |
| lambday | but for the "wait_for_all" method, it would have been really nice if we could have used c++11's future/promise things | 18:50 |
| HeikoS1 | with very simple queuing | 18:50 |
| HeikoS1 | lambday: ah yeah | 18:50 |
| HeikoS1 | lambday: maybe its a good idea to change that then | 18:51 |
| lambday | HeikoS1: but it all depends on whether we can support c++11's std::thread | 18:51 |
| lambday | :-/ | 18:51 |
| lisitsyn | why not? | 18:51 |
| HeikoS1 | lisitsyn: is shogun fully c++11? | 18:52 |
| HeikoS1 | then lets use that | 18:52 |
| HeikoS1 | as a first backend | 18:52 |
| lisitsyn | HeikoS1: what's fully c++11? | 18:52 |
| lambday | HeikoS1: we can check for its support | 18:52 |
| lambday | like currently it does for a bunch of c++11 features | 18:52 |
| HeikoS1 | but still lets keep the interface in a way that things can be done from the modular interfaces | 18:52 |
| lisitsyn | afaik it is supported on major compilers | 18:53 |
| lisitsyn | so I'd not be afraid of it | 18:53 |
| lambday | anyone with g++4.7+ should have it | 18:53 |
| lambday | vc++ also supports std::thread now... | 18:53 |
| lisitsyn | you probably know boost has threadpool | 18:53 |
| lambday | yeah... | 18:53 |
| lisitsyn | I bet it is worth to take some ideas from it | 18:54 |
| lambday | haven't used it though | 18:54 |
| lambday | HeikoS1: lisitsyn: so do you think I should send a PR with the current implementation of my thread pool things... and then may be we can discuss? | 18:54 |
| lisitsyn | yeah sure | 18:54 |
| lambday | I'll finish it soon then | 18:55 |
| lisitsyn | just let me know when to take a look | 18:55 |
| HeikoS1 | lambday: yeah maybe thats a good idea | 18:55 |
| lisitsyn | cause I am a bit | 18:55 |
| lisitsyn | lost | 18:55 |
| lisitsyn | :D | 18:55 |
| HeikoS1 | lambday: a working example (like kernel matrix) would be good also | 18:55 |
| lambday | lisitsyn: hehe :D | 18:55 |
| lambday | HeikoS1: yeah I thought of that... | 18:55 |
| HeikoS1 | since again interfaces are important, how to specify the parts | 18:55 |
| HeikoS1 | lambday, lisitsyn also talk to viking about the spark stuff he mentioned | 18:55 |
| lisitsyn | damn it is fcking cod | 18:55 |
| lambday | HeikoS1: another thing... the job.... I think we should use a callable object as job instead of making it a class... | 18:55 |
| lisitsyn | cold* | 18:55 |
| lambday | spark stuff?... sorry I am not updated :( | 18:56 |
| lambday | lisitsyn: -10 degrees? :P | 18:56 |
| lisitsyn | lambday: -30C | 18:56 |
| lambday | fuck!! :-o | 18:56 |
| lambday | HeikoS1: if we use job as a callable object, we can use std::packaged_task, functors, functions etc... | 18:57 |
| lambday | in many cases making it a subclass of some generic job class would cause a lot of overhead | 18:58 |
| lisitsyn | one great drawback of std threading stuff | 18:58 |
| lisitsyn | in java you just extend it | 18:58 |
| lambday | lisitsyn: what is it? | 18:58 |
| lisitsyn | no way here | 18:58 |
| lisitsyn | you just either use it or fck you | 18:58 |
| lisitsyn | :D | 18:58 |
| HeikoS1 | lambday: yeah an object might be better | 18:58 |
| HeikoS1 | so its easier to use things | 18:58 |
| HeikoS1 | I agree | 18:58 |
| lambday | lisitsyn: we'll have to think of some trade off then :-/ | 18:59 |
| lambday | HeikoS1: I was thinking of separating the computation part and do the experiments... and then post it somewhere, so that we can discuss... | 18:59 |
| lambday | HeikoS1: to do the changes in the existing framework | 18:59 |
| lisitsyn | also | 19:00 |
| lambday | later we'll integrate | 19:00 |
| lisitsyn | it would be cool to have it more implementation agnostic | 19:00 |
| lisitsyn | I mean | 19:00 |
| lisitsyn | std::thread won't work with mpi | 19:00 |
| lisitsyn | but we can imagine thread is just mpi kind of thread | 19:00 |
| lambday | ummm.. std::thread on linux uses pthread, right? | 19:01 |
| lisitsyn | lambday: should be | 19:04 |
| lambday | for a thread pool kind of scenario, we gotta spend some time on the job scheduling part... may be job queue won't be just a queue... but a priority queue or some other fancy DS | 19:04 |
| lambday | cause not all jobs we'll be using would be independent | 19:04 |
| lambday | but initially a simple queue would do.. | 19:04 |
| lambday | right? | 19:04 |
| lambday | lisitsyn: yeah I think it does.. | 19:05 |
| lisitsyn | lambday: well it should have some queue but shouldn't know what queue | 19:05 |
| lambday | ya... but there has to be some mechanism to specify our required computation order.... may be a DAG? | 19:06 |
| lambday | thread pools would be computing different types of jobs | 19:07 |
| lambday | I mean the threads in the pool | 19:07 |
| lisitsyn | lambday: heh well yeah | 19:07 |
| lisitsyn | it is the questiong | 19:07 |
| lisitsyn | question* | 19:07 |
| lambday | ya... somehow we gotta make it easy for the application programmer to just specify their required order... and submit different types of jobs to the engine... then just wait for all... | 19:09 |
| lambday | ideally it should be that simple... :-/ | 19:09 |
| shogun-buildbot | build #48 of deb4 - python3 is complete: Failure [failed test python modular] Build details are at http://buildbot.shogun-toolbox.org/builders/deb4%20-%20python3/builds/48 blamelist: Fernando Iglesias <fernando.iglesiasg@gmail.com>, Parijat Mazumdar <mazumdarparijat@gmail.com> | 19:09 |
| lambday | but anyway I will complete the current implementation with a simple queue (using std::queue) and then discuss may be? | 19:10 |
| -!- thoralf [~thoralf@91-65-138-183-dynip.superkabel.de] has joined #shogun | 19:13 | |
| thoralf | Heyhey. | 19:13 |
| shogun-buildbot | build #411 of precise - libshogun is complete: Success [build successful] Build details are at http://buildbot.shogun-toolbox.org/builders/precise%20-%20libshogun/builds/411 | 19:16 |
| HeikoS1 | lambday, lisitsyn keep in mind the thing is called *independent jobs* | 19:18 |
| HeikoS1 | so thats just embarassingly parallel stuff | 19:18 |
| lambday | HeikoS1: well, that would make it pretty simple then... I was wondering if we can use it in some other cases.. when a bunch of independent jobs has to be followed by another bunch of independent jobs.. | 19:19 |
| lambday | but initially that's too much to worry about :-/ | 19:20 |
| lambday | HeikoS1: things will work from modular interface as well.. we'll specify jobs from the modular interface as well.. engine will be there.. shouldn't matter what they use in the backend.. | 19:22 |
| HeikoS1 | lambday: yeah ordering would be cool | 19:23 |
| HeikoS1 | lambday: but I think its kind of hard to specify that within the framwork | 19:23 |
| HeikoS1 | why not write the code this way | 19:23 |
| HeikoS1 | that uses the independent-for framework | 19:23 |
| HeikoS1 | sorry for my long delays here btw ;) | 19:23 |
| lambday | HeikoS1: no worries man!.. | 19:23 |
| lambday | HeikoS1: well, currently what we had in mind can only compute one bunch of independent jobs.. and then wait for all... | 19:24 |
| HeikoS1 | lambday: yes | 19:24 |
| lambday | if we need another bunch... we'll have to create another engine.. | 19:24 |
| HeikoS1 | yep thats true | 19:24 |
| HeikoS1 | ah you mean you want to combine things? | 19:24 |
| lambday | HeikoS1: I was wondering if we can design it someway that we can use the same engine... | 19:24 |
| HeikoS1 | lambday: but thats just queuing isnt it? | 19:25 |
| HeikoS1 | since the engine has a certain capacity | 19:25 |
| lambday | HeikoS1: but can we guarantee that a job from job grup J1 {j11,j12...j1n} has computed *before* J2 {j21,j22,....j2n} in that case? | 19:26 |
| lambday | I mean, say, if the result from job group J1 is used by the job group j2 | 19:27 |
| HeikoS1 | lambday: I see | 19:27 |
| HeikoS1 | currently this is only possible via wait_for_all | 19:27 |
| lambday | yeah | 19:27 |
| lambday | but then also, we cannot check before computing some job of the second batch if the required result has already been computed | 19:27 |
| -!- parijat [671b082a@gateway/web/freenode/ip.103.27.8.42] has joined #shogun | 19:27 | |
| lambday | so if we want that flexibility... | 19:28 |
| lambday | ummm.. | 19:28 |
| HeikoS1 | lambday: I see | 19:28 |
| HeikoS1 | mmmh | 19:28 |
| HeikoS1 | I feel we should start with a replacement for the current parallel stuff | 19:28 |
| HeikoS1 | which is all just parfor basically | 19:28 |
| HeikoS1 | no dependencies | 19:29 |
| HeikoS1 | we can always add that | 19:29 |
| lambday | parfor as in openmp style? | 19:29 |
| HeikoS1 | yeah just tasks that have independent parts | 19:29 |
| HeikoS1 | to unify this within shogun would already be a great improvement | 19:30 |
| HeikoS1 | lisitsyn: how would you represent categorical features in shogun? | 19:30 |
| HeikoS1 | lisitsyn: as strings? | 19:30 |
| lisitsyn | HeikoS1: I think map<int,string> + int | 19:31 |
| -!- parijat [671b082a@gateway/web/freenode/ip.103.27.8.42] has quit [Client Quit] | 19:32 | |
| lambday | HeikoS1: I'll check this then.. currently keeping in mind regarding the kernel matrix computation part... if I can make this example working.. then voila! | 19:33 |
| lambday | HeikoS1: instead of putting things in thread_param and get_kernel_matrix_helper, we;ll specify it as job.. and inside the get_kernel_matrix, create jobs and submit to the engine.. | 19:34 |
| lambday | this is what I had planned | 19:35 |
| lambday | the main code would be easy to read.. | 19:35 |
| lambday | plus memory things can be handled using std::ref and std::move ... | 19:36 |
| lambday | HeikoS1: lisitsyn: will brb after dinner | 19:37 |
| HeikoS1 | lambday: good plan | 19:39 |
| HeikoS1 | yeah if this works, we can translate a few of shoguns parallel things to this much cleaner approach | 19:40 |
| HeikoS1 | and if thats fine, then we can start thinking about x-validation on a batch cluster backend like thing | 19:40 |
| -!- travis-ci [~travis-ci@ec2-107-20-100-146.compute-1.amazonaws.com] has joined #shogun | 19:49 | |
| travis-ci | [travis-ci] it's Fernando Iglesias's turn to pay the next round of drinks for the massacre he caused in shogun-toolbox/shogun: http://travis-ci.org/shogun-toolbox/shogun/builds/17851887 | 19:49 |
| -!- travis-ci [~travis-ci@ec2-107-20-100-146.compute-1.amazonaws.com] has left #shogun [] | 19:49 | |
| -!- iglesiasg [~iglesiasg@524AE0A7.cm-4-3d.dynamic.ziggo.nl] has quit [Quit: Leaving] | 20:01 | |
| -!- thoralf [~thoralf@91-65-138-183-dynip.superkabel.de] has quit [Ping timeout: 272 seconds] | 20:13 | |
| @wiking | lambday: i've just suggested that maybe we should start moving on with the computing framework using spark... | 20:17 |
| @wiking | lambday: or at least do a mesos implementation of it | 20:17 |
| lambday | re | 20:25 |
| lambday | wiking: spark.. okay | 20:26 |
| lambday | cool!!! | 20:27 |
| lambday | wiking: so do you think it would be possible to come up with unified computation framework for multithread and cluster? like.. from an application programmer's point of view.. they will just have to use different engines... that's all | 20:28 |
| @wiking | lambday: well according to spark | 20:30 |
| @wiking | that's doable | 20:30 |
| @wiking | see http://spark.incubator.apache.org/docs/latest/python-programming-guide.html | 20:31 |
| @wiking | check the "Interactive Use" part | 20:31 |
| @wiking | you can either use spark like: MASTER=spark://IP:PORT ./pyspark | 20:31 |
| @wiking | or | 20:31 |
| lambday | wiking: checking | 20:31 |
| @wiking | MASTER=local[4] ./pyspark | 20:31 |
| @wiking | for 'use four cores on the local machine' | 20:31 |
| lambday | oh that's classic! | 20:32 |
| @wiking | so depending on the MASTER env variable | 20:32 |
| @wiking | u use various 'cluster' solutions | 20:32 |
| @wiking | which is what we want | 20:33 |
| @wiking | of course we'll have several drawbacks for this | 20:33 |
| @wiking | like creating a parallel for loop with openmp is very easy | 20:33 |
| @wiking | #pragma etc... | 20:33 |
| @wiking | but this way we'll have to implement our own way to do this | 20:34 |
| @wiking | to be able to handle both cases | 20:34 |
| @wiking | 1) different cores but on one machine | 20:34 |
| @wiking | 2) totally separated computing nodes | 20:34 |
| @wiking | so we'll have to have something like a function | 20:35 |
| @wiking | to do a parallel for loop | 20:35 |
| @wiking | and of course the biggest problem is that spark doesn't have c++ api : | 20:37 |
| @wiking | :D | 20:37 |
| lambday | wiking: yeha.... | 20:37 |
| lambday | wiking: yeah I was just about to say that | 20:38 |
| @wiking | mesos does have | 20:38 |
| lambday | everything has to be on py or java.. :-/ | 20:38 |
| @wiking | lambday: mesos is c++ implemented so we can use that | 20:38 |
| @wiking | but then we loose all the good stuff of spark | 20:38 |
| lambday | wiking: no way we can use it backwards? :P | 20:39 |
| @wiking | lambday: "If you want to pass data through an external C++ program, there's already an operation on Spark's distributed datasets (RDDs) called pipe(). You give it a command, and it will launch that process, pass each element of the dataset to its stdin, and return an RDD of strings representing the lines printed to the process's stdout." | 20:39 |
| lambday | provide some wrappers and use swig? | 20:39 |
| @wiking | lambday: well java has JNI | 20:39 |
| @wiking | so it's possible of course | 20:39 |
| @wiking | but that's really a pain in the ass | 20:39 |
| lambday | yep | 20:39 |
| @wiking | i've been working with JNI and it's really hell | 20:39 |
| lambday | wiking: hmm... but the way you said it can both be used for multicore and cluster.. is really cool!.. | 20:41 |
| lambday | so for that much flexibility I wonder if we can take the pain | 20:42 |
| lambday | I don't have much exp with JNI and not spark for that matter but I can check | 20:42 |
| @wiking | well | 20:42 |
| @wiking | i'll have to check this pipedrdd | 20:42 |
| @wiking | https://spark.incubator.apache.org/docs/0.6.2/api/core/spark/rdd/PipedRDD.html | 20:43 |
| @wiking | :) | 20:43 |
| @wiking | we could write the serializer for this :)))) | 20:43 |
| lambday | wiking: I'll check it out... play with it a bit | 20:44 |
| lambday | lol python guide asks to read the scala part first :D | 20:45 |
| @wiking | mesos seems to be easy of course | 20:46 |
| @wiking | http://mesos.apache.org/documentation/latest/app-framework-development-guide/ | 20:46 |
| lambday | wiking: HeikoS1: I will come back in a few mins... | 20:57 |
| @wiking | https://github.com/zhgwen/spark_cpp_api | 21:06 |
| @wiking | hehehe | 21:06 |
| @wiking | kind of hard with the documentation | 21:07 |
| lambday | HeikoS1: wiking back | 21:16 |
| lambday | wiking: lol.. chinese? | 21:16 |
| lambday | lol we shouldn't worry about that.. there are people among us who can help us with it :D | 21:16 |
| lambday | wiking: would this help? http://spark.incubator.apache.org/docs/latest/running-on-mesos.html | 21:18 |
| @wiking | mmm well that's just how to run spark on mesos | 21:21 |
| @wiking | but that doesn't help us with having spark | 21:21 |
| -!- thoralf [~thoralf@91-65-138-183-dynip.superkabel.de] has joined #shogun | 21:23 | |
| thoralf | Re. | 21:24 |
| -!- votjakovr [~votjakovr@188.134.46.30] has quit [Quit: WeeChat 0.4.0] | 21:27 | |
| -!- shogun-notifier- [~irker@7nn.de] has quit [Quit: transmission timeout] | 21:33 | |
| -!- iglesiasg [~iglesiasg@524AE0A7.cm-4-3d.dynamic.ziggo.nl] has joined #shogun | 22:18 | |
| -!- mode/#shogun [+o iglesiasg] by ChanServ | 22:18 | |
| -!- lambday [67157c4d@gateway/web/freenode/ip.103.21.124.77] has quit [Ping timeout: 245 seconds] | 22:33 | |
| -!- chetna [0e8b5206@gateway/web/freenode/ip.14.139.82.6] has joined #shogun | 22:46 | |
| chetna | Hello :) I am really enthusiastic to contribute in the field of machine learning and projects on similar lines. Going through the idea page for GSoC-2013 I found many ideas which are related to the coursework I have persuaded back in my college. I would like to know how can I start working and hacking into the codebase for Shogun. | 22:57 |
| @iglesiasg | chetna, Hey, I suggest you to have a look to the initiation issues in GitHub | 22:58 |
| chetna | iglesiasg: Could you please direct me to the link for the same. | 22:59 |
| @iglesiasg | chetna, https://github.com/shogun-toolbox/shogun/issues?milestone=none&state=open | 23:00 |
| @iglesiasg | click on the "entrance" label (initiation as I said before is not the right word) | 23:00 |
| @iglesiasg | you will find the label on the left hand side of the page | 23:00 |
| chetna | iglesiasg: Thanks :) I'll look through the issues with the above mentioned tag :) | 23:02 |
| @iglesiasg | chetna, make a comment in the issue if you decide to start working on one, just so we avoid wasting manpower replicating work | 23:04 |
| chetna | iglesiasg: For Sure :) | 23:04 |
| -!- chetna_ [0e8b5206@gateway/web/freenode/ip.14.139.82.6] has joined #shogun | 23:15 | |
| -!- chetna [0e8b5206@gateway/web/freenode/ip.14.139.82.6] has quit [Ping timeout: 245 seconds] | 23:17 | |
| -!- lisitsyn [~lisitsyn@80.252.20.67] has quit [Ping timeout: 245 seconds] | 23:28 | |
| -!- chetna_ [0e8b5206@gateway/web/freenode/ip.14.139.82.6] has quit [Ping timeout: 245 seconds] | 23:55 | |
| --- Log closed Thu Jan 30 00:00:16 2014 | ||
Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!