IRC logs of #shogun for Thursday, 2017-08-03

« Wednesday, 2017-08-02 Index Friday, 2017-08-04 »
--- Log opened Thu Aug 03 00:00:22 2017
-!- olinguyen [81615ad9@gateway/web/freenode/ip.129.97.90.217] has quit [Quit: Page closed]05:36
-!- mikeling [uid89706@gateway/web/irccloud.com/x-mkoruuyhyohpgnhz] has joined #shogun08:08
@wikingmikeling, hey09:24
@wikinghere?09:24
mikelingwiking: hey09:24
mikelingyes09:24
@wikingok so about the biginput09:25
@wikingYes most of the time the idea is that you need to change the source as well to index_t09:25
@wikingBut this is not like a must09:25
@wikingIt depends09:25
@wikingNow the way we could do this09:25
@wikingThat you do the changes commit by commit09:25
@wikingmeaning you change in one place and all the connected places09:25
@wikingAnd send in the commit09:25
@wikingAnd then I can review09:25
@wikingas there's no real rule of thumb09:25
@wikingAnd yes09:25
@wikingI can rebase the barnch09:26
@wikingTo the latest09:26
@wikingthing09:26
mikelingwiking: yeah, that's great. Thank you09:27
@wikingOk there's a lot of conflict09:27
@wikingBut I'll do it as soon as possible09:27
mikelingbut since the index_t no longer pointer to int32_t https://github.com/shogun-toolbox/shogun/commit/e8c66f0d849696f890af5645622e2feb2eef29b3#diff-1a988b6d5203d889a25d01f2fb6ffb72R6209:27
mikelingactually, I had try to do it commit by commit09:27
mikelingbut the true is many places get involved into this, like I said in the email09:28
mikelingsome places we use index_t09:28
mikelingbut we actually pass a int32_t to it09:28
mikelingit works before since they are equal but it's not after we point index_t to int64_t09:29
mikelinglike add_vector(), if we try to build the  feature/BigInput branch directly, we will get no matching member function fo call to 'add_vector' in many places since we pass a int32_t* to index_t*.09:52
@wikingOk just a sec10:25
@wikingHad to do some meeting10:25
@wikingmikeling, ok I've pushed the rebased one10:34
mikelingwiking: ok, thank you. Let me update it10:39
-!- travis-ci [~travis-ci@ec2-54-221-76-249.compute-1.amazonaws.com] has joined #shogun10:46
travis-ciit's Viktor Gal's turn to pay the next round of drinks for the massacre he caused in shogun-toolbox/shogun: https://travis-ci.org/shogun-toolbox/shogun/builds/26051396610:46
-!- travis-ci [~travis-ci@ec2-54-221-76-249.compute-1.amazonaws.com] has left #shogun []10:46
Trixiswiking: so somehow i seem to have managed to load shogun on the cluster using qrsh. Only today tho12:18
Trixisyesterday it didnt work because openblas wasnt there, today it mysteriously appeared12:18
@wikingqrsh?12:19
Trixiswiking: basically instead of submitting the job via the fire and forget command, i open a rsh session on one of the compute nodes12:21
Trixisbecause for some reason that way my path doesnt get brutalised12:21
@wikingjesus12:22
Trixisidk12:22
@wikingthis thing is not good12:22
Trixislike its a total clsuterfuck12:22
@wikingIf you need to do things like this :(12:22
Trixisye12:22
Trixisoh and all the nodes im using should be identical in specs / config, so idk why there was no openblas yesterday12:23
Trixis(the group im part of has its own nodes)12:24
@wikingheheh12:25
@wikingOk that seems like a setup problem12:25
@wikingi'm a bit really confused12:25
@wikingOr like undeceive whether the jnilib should contain libshogun and any other library as a static lib ....12:26
@wikingAs that way none of these problems would exists12:26
Trixiswiking: it seems to contain libshogun12:28
Trixisjust not openblas12:28
@wikingyeye12:28
@wikingI mean I know what is the status quo12:28
@wikingI'm just wondering what should be the actual optimal solution12:28
Trixisah, right12:29
@wikingi mean honestly12:30
@wikingWhat you would like to hat with an uber jar is12:30
@wikingHaving all these stuff12:30
@wikingthere12:30
@wikingNo?12:30
@wiking:)12:30
Trixisyeah, lol12:32
Trixisshogun is actually the only library that has to be supplied locally (but thats because the chemoinformatics library used seems to require being included in an uberjar)12:33
-!- iglesiasg [~iglesiasg@217.119.234.214] has joined #shogun12:50
-!- mode/#shogun [+o iglesiasg] by ChanServ12:50
-!- iglesiasg [~iglesiasg@217.119.234.214] has quit [Quit: leaving]12:59
-!- iglesiasg [~iglesiasg@217.119.234.214] has joined #shogun12:59
-!- mode/#shogun [+o iglesiasg] by ChanServ12:59
-!- phyj [d24d1af4@gateway/web/freenode/ip.210.77.26.244] has joined #shogun13:07
-!- phyj [d24d1af4@gateway/web/freenode/ip.210.77.26.244] has quit [Quit: Page closed]13:13
-!- iglesiasg [~iglesiasg@217.119.234.214] has quit [Quit: leaving]15:49
-!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has joined #shogun15:50
-!- mode/#shogun [+o HeikoS] by ChanServ15:50
@HeikoSmicmn: hi16:13
micmnHeikoS: hey16:14
@HeikoSjust reading your email16:14
@HeikoSso I like the idea of making features a) state-less and b) read-only in general16:14
@HeikoSthen a lot of problems go away16:14
@HeikoSand the underlying SGReferenced data can just be shared16:14
@HeikoSwhen operations on features are called that would change them, generate a copy16:15
@HeikoSmake all iterators const16:15
@HeikoSmicmn: wiking ^16:15
@HeikoSno need for rw imho16:15
@wikingJust a sec16:16
@wikingIn a meeting16:16
@HeikoSmicmn: so all algorithms that currently change the data (I dont think those are many, e.g. x-validation blows up), would create copies first16:17
micmnso which methods would modify the features?16:17
@HeikoSpreprocessors16:18
@HeikoSBUT16:18
@HeikoSsome of them can also be implemeted to work on the fly16:18
@HeikoSbut generally, algorithms shouldt change the features16:18
@HeikoSso the answer is: None :D16:18
@HeikoSmicmn: does that make things simpler?16:19
micmnyeah :P16:19
micmnbut many algorithms with dense features16:20
@HeikoSsorry what do you mean?16:20
micmnwork directly on the underlying matrix16:20
@HeikoSoh yes16:20
@HeikoSstill ok16:20
@HeikoSjust const16:20
@HeikoS(as a first step)16:20
@HeikoSthen next step is iterators16:20
@HeikoSand final step is using your new "Matrix" class16:20
@HeikoSbut first step would be to make the matrix getter const I guess16:21
@HeikoSah16:21
@HeikoSsubsets16:21
micmnyep16:21
@HeikoSwell as of now16:21
@HeikoSI think this causes a copy anyways16:21
@HeikoSright?16:21
micmnyeah16:21
@HeikoSget_feature_matrix() returns a copy if a subset is active16:21
@HeikoSwhich sucks big time16:21
@HeikoSbut this doesnt hinder making features const16:22
@HeikoSso step 1 can be done16:22
@HeikoSnext step is iterators, or operations to replace the working with the matrix16:22
@HeikoSbut first step is OK16:22
@HeikoSmicmn: thoughts?16:22
micmnmmm16:23
micmnso you'll keep the current subset mechanism?16:23
@HeikoSmaybe for now yes16:23
@HeikoSas making const and state-less is first step16:23
@HeikoSI mean this is a small step16:23
@HeikoSbut with subsets16:24
@HeikoSsince feature object will be stateless16:24
@HeikoSadding a subset will have to return a new instance16:24
@HeikoS(with the same matrix, but with one subset added to the stack=16:24
@HeikoS)16:24
@HeikoSsee what I mean?16:24
micmnyeah more or less like in the gist16:24
@HeikoSlink?16:24
micmnhttps://gist.github.com/micmn/ea9e149f7c11a7b8d7ddb5f2c6682c6316:25
@HeikoSah have16:25
@HeikoSnot quite16:25
@HeikoSno modification methods on features16:25
@HeikoSthey are const16:25
micmnyeah I refer to the view() method16:25
@HeikoSsubsetted_feats = feats.add_subset()16:26
@HeikoSor more like16:26
@HeikoSsubsetted = feats.subset_view(inds)16:26
@HeikoSand the no more add_* call16:26
micmnHeikoS, ah the point that I still don't get is: iterate on what? depends on the features type i.e. densefeatures -> sgvector16:31
@HeikoSiterate on dot product for example16:32
@HeikoSlet me find a gist16:32
@HeikoShttps://gist.github.com/lisitsyn/a6d8ff6e8690431f967c5318c3750919#file-gistfile1-txt-L12916:32
@HeikoShttps://gist.github.com/lisitsyn/a6d8ff6e8690431f967c5318c3750919#file-gistfile1-txt-L10916:33
@HeikoSmicmn:  have to go to a talk now16:33
micmnok thx16:33
@HeikoSbut will still read/write here16:33
@HeikoSso lets discuss more16:33
@HeikoSsee what I mena with the iterators?16:33
micmnso is the iterator that performs the operation16:34
micmnand we'll have different type of iterators16:34
-!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has quit [Ping timeout: 276 seconds]16:38
-!- olinguyen [81615ad9@gateway/web/freenode/ip.129.97.90.217] has joined #shogun17:40
mikelingwiking: ping17:51
@wikingpong17:53
@wikingmicmn, sorry had a fucking nightmare meeting over here17:53
micmn:(17:53
@wikingOk back17:54
@wikingI read the log17:54
@wikingBtw we never do has_more in c++17:55
@wikingIt's java17:55
@wikingIt's end()17:55
mikelingwiking: Hi, what's this w for https://github.com/shogun-toolbox/shogun/commit/952906e79a60e69bfa8fa6f6303f967671c79d57#diff-b684d0a89dbd67566975b83e09135053R22418:00
mikelingis that a SGVector?18:00
mikelingbecause I got error: use of undeclared identifier 'w' w = new_w; when building it18:02
@wikingmmm18:03
@wikingI think w is a float64_t vector18:03
@wikingWonder where it was originally declared18:03
mikelingcould we define it as SGVector<float64_t> m_w?18:04
@wikingyeah18:04
@wikingI just wonder when did it disappear :)))18:05
-!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has joined #shogun18:23
-!- mode/#shogun [+o HeikoS] by ChanServ18:23
@HeikoSmicmn: see what I mean with the iterators?18:24
@HeikoSolinguyen: hi!18:24
olinguyenhey!18:24
@HeikoSolinguyen: fine with me with the lagged features18:24
@HeikoSbut let's finish the RF stuff first18:25
olinguyenagreed18:25
@HeikoSso then we also have RF in the plots18:25
micmnHeikoS: so it iterates over the features and i.e. feats->dot(idx, w) becames iterator->dot(w)18:26
mikelingwiking: I got this out-of-line definition of 'CWDSVMOcas' does not match any declaration in 'shogun::CWDSVMOcas' for https://github.com/shogun-toolbox/shogun/commit/952906e79a60e69bfa8fa6f6303f967671c79d57#diff-2a3c04af057ddaaf9cfa245feef22d22R8118:26
mikelingmmm, but I think it's same with its declaration18:27
@HeikoSmicmn: yes, so no need to touch the vectors18:28
micmnok cool18:28
@HeikoSolinguyen: tada18:30
@HeikoSfirst patch merged18:30
@wikingmmm18:30
@wikingmikeling, w8 a second18:30
@wikingI have to check this18:30
mikelingok18:30
@wikingAs develop actually should work on gpl repo already18:30
@wikingbut now there's something weir18:31
@wikingd18:31
@wiking:S18:31
olinguyengreat :) (and finally lol)!18:31
@HeikoSolinguyen: it takes a few iterations usually18:31
@HeikoSask the others ;)18:31
mikelingwiking: and I also don't know why the CMath::min() has error https://pastebin.mozilla.org/902875818:31
mikelingsince as I both of the parameter are int32_t which shouldn18:32
mikeling* shouldn't have problem18:32
@wikingmmm18:32
@wikingchecking18:32
-shogun-buildbot:#shogun- Build deb3 - interfaces #77 is complete: Failure [java (failure)] - http://buildbot.shogun-toolbox.org:8080/#builders/37/builds/7718:57
olinguyenHeikoS: did I cause those errors: MulticlassLabels.java:53: error: constructor MulticlassLabels(DoubleMatrix) is already defined in class MulticlassLabels19:00
olinguyenWhy is it only complaining for java?19:00
@HeikoSolinguyen: ah crap19:01
@HeikoSI should have waited for the last travis build ;)19:01
@HeikoSolinguyen: lets fix this before doing anything else19:01
olinguyensure19:01
@HeikoSmy fault19:01
olinguyenno worries, i think i saw it earlier but ignored19:01
@HeikoSolinguyen: ok the problem is this19:03
@HeikoSyou know we use SWIG to auto generate the interfaces19:03
olinguyenright19:03
@HeikoSso for every method/constructor, swig generates a method in the target language19:03
@HeikoSlike CMulticlassLabels::CMulticlassLabels(const SGMatrix<float64_t> confidences)19:03
@HeikoSand it inserts the language-specific type into the SG* types19:03
@HeikoSwhich is DoubleMatrix in java19:04
@HeikoS(we use a package called jblas which defines it)19:04
@HeikoSnow the thing is19:04
@HeikoSthat jblas treats both vectors and matrices as DoubleMatrix19:04
@HeikoSand since there already was a method: CMulticlassLabels::CMulticlassLabels(const SGVector<float64_t> src)19:04
@HeikoSswig generates the same method signature twice19:04
@HeikoSand therefore kaboom19:05
olinguyeni get it19:05
@HeikoSone fix could be19:05
@HeikoSrather than adding this constructor, you could add a method that sets the confidences19:05
@HeikoSolinguyen: understand what I mean?19:06
@HeikoSwiking: we need to get rid of jblas, this is stupid that it is not typesafe between vector and matrix19:06
olinguyenyea got it19:06
@HeikoSolinguyen: can you send a PR for that asap?19:06
olinguyenyep, i'll get on it now19:06
@HeikoSwe now broke the develop build (it is red)19:06
@HeikoSso that has priority19:06
@HeikoSas we are blocking the other guys work (they cannot check build success, it is always red)19:07
@HeikoSI have to run now, but will check back in a few hrs19:07
olinguyenshould be done by then!19:07
@HeikoSok cool thx19:07
@HeikoSolinguyen: as for the "I didnt write these tests", thanks for the reply, and all good then19:07
-!- HeikoS1 [~heiko@eduroam-int-pat-8-18.ucl.ac.uk] has joined #shogun19:10
-!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has quit [Ping timeout: 258 seconds]19:13
-!- travis-ci [~travis-ci@ec2-54-145-26-220.compute-1.amazonaws.com] has joined #shogun19:19
travis-ciit's Olivier's turn to pay the next round of drinks for the massacre he caused in shogun-toolbox/shogun: https://travis-ci.org/shogun-toolbox/shogun/builds/26068606019:19
-!- travis-ci [~travis-ci@ec2-54-145-26-220.compute-1.amazonaws.com] has left #shogun []19:19
-!- HeikoS1 [~heiko@eduroam-int-pat-8-18.ucl.ac.uk] has quit [Ping timeout: 240 seconds]19:23
Trixiswiking: any idea if libsvm does any multithreading?19:49
-!- HeikoS [~heiko@host-92-0-169-11.as43234.net] has joined #shogun23:12
-!- mode/#shogun [+o HeikoS] by ChanServ23:12
olinguyenHeikoS: "The job exceeded the maximum time limit for jobs, and has been terminated." but I think all the other tests passed23:18
--- Log closed Fri Aug 04 00:00:24 2017
« Wednesday, 2017-08-02 Index Friday, 2017-08-04 »

Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!