IRC logs of #shogun for Tuesday, 2016-08-30

« Monday, 2016-08-29 Index Wednesday, 2016-08-31 »
--- Log opened Tue Aug 30 00:00:27 2016
CaBai'm having some difficulties running evaluations...01:05
CaBahttps://github.com/shogun-toolbox/shogun/blob/develop/src/shogun/evaluation/ROCEvaluation.cpp#L7901:05
CaBathis check in ROCEvaluation.cpp:79 fails for me in some cases, rather randomly01:05
CaBait uses CLabel::get_value() to collect the binary labels01:06
CaBahowever, this junk for me01:06
CaBaas in this mwe: https://gitlab.unique-internet.de/snippets/501:06
CaBacan somebody give me a hint here?01:06
CaBa(the values are mostly positive in most my test cases, seems random memory though...)01:09
shogun-buildbotbuild #63 of memleak - valgrind is complete: Failure [failed memory check]  Build details are at http://buildbot.shogun-toolbox.org/builders/memleak%20-%20valgrind/builds/63  blamelist: Viktor Gal <vigsterkr@gmail.com>05:52
shogun-buildbotbuild #1061 of nightly_none is complete: Failure [failed compile]  Build details are at http://buildbot.shogun-toolbox.org/builders/nightly_none/builds/1061  blamelist: Viktor Gal <vigsterkr@gmail.com>06:00
shogun-buildbotbuild #1190 of nightly_default is complete: Failure [failed test notebooks]  Build details are at http://buildbot.shogun-toolbox.org/builders/nightly_default/builds/1190  blamelist: Viktor Gal <vigsterkr@gmail.com>06:39
-!- sanuj [~sanuj@59.97.245.250] has joined #shogun07:19
sanuji just passed gsoc final evaluation \o/07:20
sanujlisitsyn, there?07:27
sanujlisitsyn, i have a question related to plugins09:48
lisitsynpartially09:56
sanujwe do: MetaClass<base_class> clazz = manifest.class_by_name<base_class>("class");09:57
sanujwhen loading plugins09:57
sanujwhat if there is a method in "class" which is not present in "base_class"09:57
sanujsay: method()09:57
sanujSome<base_class> clazz_obj = clazz.instance();09:58
sanujclazz_obj->method()09:58
sanujwill give error09:58
sanujbut why are we treating it as an object of "base_class"?09:58
sanujlisitsyn, ^09:59
lisitsynobviously you can't call what is absent in base_class09:59
lisitsyn:)09:59
sanujlisitsyn, so everything should be present in base class?10:00
sanuji was pluginizing MCLDA10:00
sanujgot this error10:00
lisitsynwell then it chould be Some<Classifier>10:00
sanujit was Some<CNativeMulticlassMachine>10:01
sanuji changed it to10:01
sanujSome<CMCLDA>10:01
sanujand everything worked as expected10:01
sanujso do we need to do this base class complication?10:01
lisitsynehmm10:01
lisitsynwell you've got to have all the methods10:02
lisitsynin the class you're referring through10:02
sanujlisitsyn, but it will create many constraints for someone writing a plugin for shogun10:02
lisitsynyeah somehow10:03
sanujso the problem here is10:03
sanujCMCLDA has set_features()10:03
sanujand CNativeMulticlassMachine doesn't10:04
sanujand if someone makes objects by the plugin way, no parameters can be passed in the constructor as well, for now.10:04
lisitsynyes that's all clear10:06
lisitsynwe'd have to limit something to be generic10:06
sanujokay10:06
sanujlisitsyn, so i'll move set_features to CNativeMulticlassMachine10:07
lisitsynI think set_features is present in some of its base classes10:07
sanujyeah, let me see10:07
sanujlisitsyn, i have created a dir "plugins" in shogun's root10:08
sanujthe inheritance is like this10:11
sanujCMCLDA > CNativeMulticlassMachine > CMulticlassMachine > CBaseMulticlassMachine > CMachine > CSGObject10:12
-!- sanuj [~sanuj@59.97.245.250] has quit [Ping timeout: 252 seconds]10:16
-!- sanuj [~sanuj@117.203.14.172] has joined #shogun10:29
@wikingCaBa: hey11:13
@wikingCaBa: ping me when you are around11:14
CaBawiking: ping11:26
@wikingok can you repeat the problem you are facing?11:26
CaBawiking: often in roc eval this [1] check fails for me11:27
CaBa[1] https://github.com/shogun-toolbox/shogun/blob/develop/src/shogun/evaluation/ROCEvaluation.cpp#L8811:27
@wikingyou dont have negatives?11:27
CaBai have almost balanced classes and use stratified x-val with 10 folds11:28
CaBai don't believe that11:28
CaBaand i don't understand how this check can work with get_value()11:28
CaBabecause when i get_value() on my labels, i don't see my labels and also the sign of what i get doesn't represent my labels11:28
@wikingeh11:29
@wikingthen something is wrong there )11:29
CaBais CBinaryLabel::get_value() [inherited from CLabel] supposed to get me the same as CBinaryLabel::get_label() [not defined in CLabel]?11:30
@wikingnoup11:31
@wikingthey are different11:31
CaBawiking: what's get_value() for?11:33
@wikingyeah this is a common misconception11:34
CaBawiking: somewhere in the docs i came across the term "confidence"?11:36
@wikingyep it's the confidence value (get_value)11:37
CaBawiking: as in "how sure" you are about a label?11:38
@wikingy11:38
CaBawiking: why does ROCEvaluation check that in order to determine the number of pos / neg labels then?11:39
@wikingbut this is being used afaik for regression label as well11:40
@wikingsee11:41
@wikingundocumented/libshogun/labels_binary_fit_sigmoid.cpp11:41
@wikingundocumented/libshogun/classifier_libsvm_probabilities.cpp11:41
@wikingrather the latter one11:42
CaBawiking: ok. confidence. but why is get_value() used in the evaluation classes?11:44
-!- sanuj [~sanuj@117.203.14.172] has quit [Ping timeout: 255 seconds]11:45
CaBawiking: maybe i shouldn't be trying to understand shogun, i'm sure the bug is somewhere on my side, i mean all the label and eval code wasn't touched in years, i probably works :P11:59
CaBawiking: i'm just curious to understand how all this is meant to work12:00
@wikingsorry man i've got down with some tasks i have to finish12:31
@wikingwill try to get back to you asap12:31
@wiking:)12:31
@wikingthese are super valid questions12:31
@wikingthat i should all add to FAQ12:31
@wiking:))))12:32
CaBahehe12:32
CaBaok, cool, let me know when you have time12:32
@wikingbtw lisitsyn Saurabh7 CaBa  and all the others here12:32
@wikingi guess the best for the FAQ would be on the wiki page of the project12:32
@wikingnot like now at the doxygen page12:33
@wikingor?12:33
CaBayou mean github wiki?12:33
@wikinghttp://www.shogun-toolbox.org/doc/en/latest/faq.html12:33
@wikingyep12:33
CaBai'd opt for github, yes12:34
CaBawiking: interesting. CContingencyTableEvaluation casts the labels to CBinaryLabels and uses get_label() - ROCEvaluation asserts LT_BINARY but doesn't cast and doesn't use get_label()12:51
@wikingCaBa, ok i've started this http://github.com/shogun-toolbox/shogun/wiki/FAQ13:13
Saurabh7CaBa: uhm , if theres an assert for binary, get_labels() should be used i guess :)13:27
Saurabh7ah the other one was changed to get_label here https://github.com/shogun-toolbox/shogun/commit/3ed97f5610477da1da39103f86e0a767d0628b1313:28
CaBaSaurabh7: hm. unfortunately with no reason in the commit message13:43
CaBaSaurabh7: get_confidence() is the predecessor of get_value()?13:44
Saurabh7yep should be13:44
CaBaSaurabh7: so what do you think about roceval? is that a bug?13:44
Saurabh7yeah, looks like it, couldbe easily verified with a test case13:47
CaBaSaurabh7: well existing testcases don't seem to capture that?14:03
-!- sanuj [~sanuj@59.97.247.24] has joined #shogun15:19
-!- sanuj [~sanuj@59.97.247.24] has quit [Ping timeout: 265 seconds]15:41
-!- sanuj [~sanuj@117.220.48.180] has joined #shogun15:53
CaBaOK, i added the cast to CBinaryLabel and used get_label() instead of get_value()16:00
CaBahttps://github.com/shogun-toolbox/shogun/compare/develop...lkuchenb:fix/rocEvalBinaryLabels16:00
CaBaNow I also get results that look more normal16:00
CaBawiking / Saurabh7 maybe somebody could take a look at this16:01
-!- sanuj [~sanuj@117.220.48.180] has quit [Ping timeout: 265 seconds]16:07
CaBabtw16:40
CaBa==3446== Conditional jump or move depends on uninitialised value(s)16:40
CaBa==3446==    at 0x73575CE: shogun::CROCEvaluation::evaluate_roc(shogun::CLabels*, shogun::CLabels*) (ROCEvaluation.cpp:79)16:40
CaBa==3446==    by 0x73570BC: shogun::CROCEvaluation::evaluate(shogun::CLabels*, shogun::CLabels*) (ROCEvaluation.cpp:22)16:41
CaBavalgrind also reported access to uninitialized values before the patch16:41
@wikingCaBa, why did you move it into evaluate_roc_binary16:56
@wiking?16:56
@wikingi mean why just not patch the original function? :)16:56
-!- sanuj [~sanuj@117.220.48.180] has joined #shogun17:05
CaBawiking: was just a quick way to integrate the cast without renaming any variables17:08
@wiking:)17:08
CaBawiking: any clue whether this is the right way to go?17:12
@wikingmmm you could have done an explicit cast17:13
@wikingas17:13
@wiking(or static_cast)17:13
@wikingas17:13
@wikingASSERT(predicted->get_label_type()==LT_BINARY)17:14
@wikingso it's for sure a BinaryLabels class17:14
CaBawiking: hm? the assert is still there17:15
CaBawiking: https://github.com/shogun-toolbox/shogun/compare/develop...lkuchenb:fix/rocEvalBinaryLabels#diff-2c552fe2477b2349ad25afef8024b7f1R12717:15
CaBawiking: the assert was there also before. it was just never casted.17:15
@wikingthat's wha ti mean17:16
@wikingsince you had that assert17:16
@wikingyou could have just done17:16
@wiking(BinaryLabels*)17:16
CaBawiking: that's what happens right now. first ASSERT, then the cast. i'm lost ;)17:17
@wikingmmm17:17
@wikingok so i literally17:18
@wikingdo not understand why17:18
@wikingevaluate_roc_binary17:18
CaBawiking: i just oriented myself at the other (working) evaluation:17:19
CaBahttps://github.com/shogun-toolbox/shogun/blob/develop/src/shogun/evaluation/ContingencyTableEvaluation.cpp#L2617:19
CaBawiking: there is also a helper function call for the more specific label type17:19
CaBacompute_scores()17:19
CaBaso i kinda took that approach to ROCEvaluation. but i don't mind one function / two functions. i was more wondering if you think this really was the issue here (CLabel::get_value() vs. CBinarlyLabel::get_label()), because i find it hard to believe that ROC evaluation didn't work for years...17:21
@wikinghehhhaheheh17:21
@wikingyou would be surprised17:21
@wikingaobut many things17:21
CaBa😱17:21
@wiking:>17:37
@wikingctcp17:37
lisitsynBAZD17:37
@wikingyes bazdmeg17:37
@wiking:)17:37
CaBawiking: you prefer this in one function? is there a way to confim that this is actually correct?17:42
@wikingCaBa, currently looking into the code :)17:42
@wikingtrying to figure out wtf17:42
@wiking:)17:42
CaBawiking: ah. thanks.17:42
@wikingCaBa, what's the model you are using actually?18:18
CaBawiking: hm?18:59
@wikingwell i guess you are doing this with a given model18:59
CaBawiking: RealFeatures + GaussianKernel + LibSVM19:00
@wikingk19:00
@wikingCaBa, will u send a pr? :)19:07
CaBawiking: sure. figured anything out?19:10
CaBawiking: i'll clean that up first19:19
CaBasanuj: ping19:30
CaBawhat's the deal with MultitaskROCEvaluation?19:31
CaBawiking: you got a PR. i made a new fix, now i understand your confusion - i hadn't seen that evaluate() was already a wrapper for evaluate_roc() ;-) i was acting under the assumption that evaluate_roc() was the public interface.19:44
CaBawiking: i also changed MultitaskROCEvaluation.cpp which failed to build after the fix.19:45
CaBa(doesn't fail to build now)19:45
-!- sanuj [~sanuj@117.220.48.180] has quit [Remote host closed the connection]22:01
CaBaSaurabh7: *ping*23:42
-!- OXPHOS [92bd15c7@gateway/web/freenode/ip.146.189.21.199] has joined #shogun23:44
CaBahi OXPHOS23:57
--- Log closed Wed Aug 31 00:00:28 2016
« Monday, 2016-08-29 Index Wednesday, 2016-08-31 »

Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!