IRC logs of #shogun for Tuesday, 2017-07-25

« Monday, 2017-07-24 Index Wednesday, 2017-07-26 »
--- Log opened Tue Jul 25 00:00:10 2017
-!- olinguyen [81615ad9@gateway/web/freenode/ip.129.97.90.217] has quit [Quit: Page closed]00:54
-!- geektoni [~geektoni@93-34-234-212.ip52.fastwebnet.it] has joined #shogun09:09
-!- iglesiasg [~iglesiasg@217.119.234.214] has joined #shogun10:30
-!- mode/#shogun [+o iglesiasg] by ChanServ10:30
-!- zxtx [~zxtx@sunnythinking.org] has quit [Ping timeout: 260 seconds]10:40
-!- zxtx [~zxtx@sunnythinking.org] has joined #shogun10:43
-!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has joined #shogun10:49
-!- mode/#shogun [+o HeikoS] by ChanServ10:49
@sukey[https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/3951 opened by geektoni11:24
@HeikoSgeektoni: still around?11:44
@HeikoSgeektoni: cool to catch you11:46
@HeikoSso how is the error going?11:46
geektoniHeikoS: not good11:46
geektoniI don't understand why it happens11:46
@HeikoSgeektoni: can you explain me very briefly the mechanics of what is happening?11:46
geektonido you mean the new CrossValidation?11:47
@HeikoSyes11:47
@HeikoSlike how you store things11:47
@HeikoShow they are emmited11:47
geektonibasically, each time you do a cross validation run11:47
geektonia CrossValidationStorage object is created11:47
@HeikoSis that something that was there before?11:48
geektonithis object has a vector which will contains the results of each fold evaluation.11:48
@HeikoSor did you create it?11:48
geektoniI created it11:48
@HeikoSok cool11:48
@HeikoSand the error comes from a double free?11:49
geektoninope, it came from a code region where it should not throw any error11:50
@HeikoSwhat exactly happens?11:50
@HeikoSwhat is the error?11:50
geektonihttps://github.com/geektoni/shogun/blob/parameter_observers_cross_validation/src/shogun/evaluation/CrossValidation.cpp#L19311:51
geektonithis line cause the memory error.11:51
@HeikoSah11:51
@HeikoSand what is it?11:52
@HeikoSthe error?11:52
geektonihttps://gist.github.com/geektoni/f376b5c411dca9be884734dae7ee6354#file-backtrace11:52
@HeikoSthx11:52
geektonifrom the backtrace, I can see there is something with TParameters11:53
geektonibut yeah, I don't understand why :P11:53
@HeikoSwhat does valgrind say to that?11:53
@HeikoSif you don't store the indices, then it doesnt happen?11:53
@HeikoSmaybe the indices are freed somewhere when they shouldnt11:54
geektoniyeah, if I remove these lines https://github.com/geektoni/shogun/blob/parameter_observers_cross_validation/src/shogun/evaluation/CrossValidationStorage.cpp#L3411:55
geektonithe error disappear11:55
geektonibut then valgrind says that there are many memory leaks11:55
@HeikoSbtw11:55
@HeikoShttps://github.com/geektoni/shogun/blob/parameter_observers_cross_validation/src/shogun/evaluation/CrossValidationStorage.cpp#L10111:55
@HeikoSyou need to REF here as well no?11:56
@HeikoSalso no need to NULL check11:56
@HeikoSSG_UNREF does that11:56
@HeikoSalso11:56
@HeikoSalways REF before UNREF to avoid memory error when new/old objects are the same11:56
@HeikoSin fact, I think it might be forgotten REF the memory error11:56
geektonikk, I didn't add the SG_REF because I read that the clone() method does it for me :P11:56
@HeikoSclone returns object with ref counter set to 1?11:57
@HeikoSwell ok, but in a method where you receive a pointer, you dont know that riht?11:57
@HeikoSso you need to REF it definitely11:57
geektonimmh, but it's ref counter will be still set to 1 right?11:58
@HeikoSit might be that after having called the method, you will have to UNREF (if clone returns object with counter set to 1)11:58
@HeikoSbut that is style11:58
@HeikoSbut there might be a problem11:58
@HeikoSimagine m_test_true_result == results11:58
@HeikoSand the ref counter is 111:58
@HeikoSund then you unref11:58
@HeikoSthen it gets deleted11:59
@HeikoSand then you assign it to itself11:59
@HeikoSand boom11:59
geektonimmh11:59
geektoniok, I'll try to add SG_REF to the setters12:00
@HeikoSI mean check if that might cause problems12:01
@HeikoSbut add it anyways, to avoid problems in the future12:01
geektonikk12:01
@HeikoSgeektoni: and can you use valgrind to find out where the memory was freed?12:01
geektoniHeikoS: mmh, actually I've never tried to use it that way12:02
@HeikoSI think it should tell you on a bad read12:02
@HeikoS"memory freed by"12:02
geektoniohh, that would be nice12:03
geektoniso, I'll let you know what I'll find out12:04
geektonimaybe is just a matter of SG_REF12:04
geektoniI still don't understand why it fails on a train_locked() method, though :/12:05
@HeikoSbecause that is the first place where the labels are used12:05
@HeikoS(my hypothesis)12:06
geektonimmh12:07
geektoniwell, I'll try and we'll ses how it goes.12:07
@HeikoSlet me know12:07
@HeikoSwill grab a coffee and then check back12:08
geektoniHeikoS: I'll be back in a few hours with some news12:08
-!- geektoni [~geektoni@93-34-234-212.ip52.fastwebnet.it] has quit [Remote host closed the connection]12:09
@wikingelias!12:25
@wikingHeikoS, ping12:26
@HeikoSwiking: pong12:34
@wikingsup?12:34
@wikinggot a second?12:34
@HeikoSwiking: sure!12:34
@HeikoSall good here, and you?12:34
@wikingso-so12:34
@wikingbusy12:34
@wikingalthough i shoulnd't be :)12:35
@wikingbut yeah getting better12:35
@wikinganyhow so i was wondering12:35
@wikingwhat we should do with those cases12:35
@wikingwhere we test fixtures12:35
@wikingfor a specific seed12:35
@HeikoSyeah!12:35
@HeikoSlike meta integration12:35
@wikingwhichi will cause a lot of headache now with12:35
@HeikoSor also fixtures12:35
@wikingc++11 random12:35
@wikingyep indeed12:35
@HeikoSyes12:35
@wikingmeta is currently a major problem12:35
@wikingwith GMM12:36
@wikingetc12:36
@HeikoSmeta is a problem indeed12:36
@HeikoSfor the unit tests with fixtures12:36
@HeikoSI think we can put in a deterministic drop in replacement for the random stream12:36
@HeikoSthe numbers it outputs might not be "correct"12:36
@HeikoSbut this is not the point in those tests anyways12:36
@wikingah you mean that instead of using some blob generator12:37
@wikingthat actually uses prng12:37
@wikingjust have a stream of precreated array of doubleS?12:37
@HeikoSyeah nonsense numbers12:37
@HeikoSI would still use the blobs12:37
@HeikoSbut the blobs random input is fixed12:37
@HeikoSso all the transformations are still at place12:37
@wiking'random input'?12:37
@HeikoSof blobs12:37
@HeikoSi.e. randn12:37
@wikingaaah yeah yeah12:38
@wikingi see what you mean12:38
@wikingmmm12:38
@wikingit's gonna be tricky12:38
@HeikoSthis would only work for unit tests12:38
@wikingsomehow mock it :P12:38
@wikingthat's really not easy to do12:38
@wikingthat12:38
@wiking actually12:38
@HeikoScant we just replace all the CRandom instances with CFakeRandom automatically?12:38
@wikingbecause you would actually need to mock12:38
@wikingstd::uniform_int_distribution()12:38
@HeikoSlike make that a feature of the random class12:38
@wikingnono we do not have CRandom anymore12:38
@wikingthat's the point12:38
@HeikoSrandom->enable_deterministic_test_mode()12:38
@HeikoSyeah or the individual rngs12:39
@wikingthose are all std::12:39
@wikingwe do not wrap them anymore12:39
@HeikoSI see12:39
@wikingthat is the point12:39
@HeikoSokok12:39
@HeikoSthen mock yes12:39
@wikingbecause wrapping all the possible distrib12:39
@HeikoSbut thats not easy indeed12:39
@wikingwould be just cumbersome12:39
@HeikoSI dont know then :(12:39
@HeikoSit isnt possible to set the state of the std:: rng?12:40
@HeikoSit is12:40
@HeikoSbut the transformations are not portable12:40
@HeikoSthats the problem right?12:40
@wikingyeah the prng is actually good12:40
@wikingit's portable12:40
@wikingthe transforms aren't12:40
@HeikoSwhat is the community suggestion to solve this?12:40
@wikingnone :)12:40
@HeikoSand what platforms are outliers?12:40
@wikingthis is really depending on the actual12:41
@wikinglibstdc++ implementation12:41
@HeikoSsigh12:41
@wikingso gnu would have one12:41
@wikingclang has it's own12:41
@wikingmsvc obviously12:41
@HeikoSokok12:41
@wikingbut yeah as many libstdc++12:41
@HeikoSlets think more12:41
@HeikoSthis is just for testing12:41
@wikingyes12:41
@HeikoSunit testing of fixed seeds12:41
@wikingwell i mean in this case12:41
@HeikoSso I mean we are not loosing any functionality or introduce bugs12:41
@wikingit doesn't matter12:41
@wikingif you fix the seed or not :P12:41
@wikingnono12:42
@HeikoSso why dont we disable the tests on non-gnu?12:42
@wikingmmm because who knows :D12:42
@wikingi mean we cannot be 100% certain12:42
@wikingthat this is the only difference12:42
@wikingor gonna be ever12:42
@HeikoStrue12:42
@HeikoSwe still have other unit tests12:42
@HeikoSfor things that are not random12:42
@HeikoS(like most things )12:42
@wikingyes12:42
@wikingmost of them are good12:42
@wikingsee the pr12:42
@wikingonly about 5 tests fail12:43
@wikingor 612:43
@HeikoSI think then we have to compromise12:43
@wikingout of which most of them are actual meta tests12:43
@HeikoSyeah the integration testing12:43
@HeikoSso then we have to accept that that is not portable and rely on unit tests on those cases12:43
@HeikoStravis still catches errors when someone fucks up an algorithm12:43
@HeikoSbut people with the "disabled test" os will have to rely on travis or buildbot, or something12:44
@HeikoSnot nice12:44
@HeikoSbut what can we do12:44
@HeikoSlisitsyn: you there?12:44
@HeikoSwiking: btw we can release bsd pretty soon I guess12:45
@wikingHeikoS, that is awesomeo12:45
@wikingquestion when12:45
@HeikoSjust needs lisitsyn to change the headers12:45
@wikingdo we wait end of aug12:45
@wikingor before?12:45
@HeikoSdoesnt matter12:45
@wikingk12:45
@HeikoSnext release12:45
@HeikoSand develop as soon as its done12:45
@HeikoSso with the headers12:45
@wikingyeah ok... that is fine12:46
@HeikoSneed some sed magic that kills all the GPL ones in their different formats and stuff12:46
@HeikoSand replaces with BSD with the authors from git history12:46
@HeikoSmy preference would be a "This is BSD3, see LICENSE.md" but I think that is borderline12:46
@HeikoShttps://github.com/SheffieldML/GPy/blob/devel/GPy/core/gp.py#L112:47
@wikingmmm idk12:47
@HeikoSbut whatever12:47
@wikinglets do the same thing as facebook12:47
@wikingbsd + patent :D12:47
@wikingthat way anybody who uses shogun12:48
@wikingwe have their licenses :)))12:48
@wikingand pattents12:48
@HeikoShahaha :D12:48
@wiking*patents12:48
@wikingjoking12:48
@HeikoSyes ok12:48
@HeikoSso yeah all that is missing is those headers, and all the corner cases that go with that12:48
@HeikoSand then a final grep GPL in the repository to check12:48
@HeikoSand gogo12:48
@wikingcool12:49
@wikingbtw how does sklearn handle this story of different randoms?12:49
@wikingor they have some common random in py12:49
@wiking?12:49
@HeikoSI think you can pass rng instances in there12:51
@HeikoSby defaultl they use np.random12:51
@wikinghttp://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html12:51
@wiking    If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.12:51
@wikingic12:51
@HeikoSwiking: oh that reminds me12:52
@HeikoSoli needs probability outputs for RF12:52
@HeikoSand that is a nice c++ task for him12:52
@HeikoSbut I am not too sure how that works in the bagging machine setup12:52
@HeikoS"The predicted class probabilities of an input sample are computed as the mean predicted class probabilities of the trees in the forest"12:52
@HeikoSthis is what skl does12:52
@HeikoSwiking: is that, I just pass the MeanRule and all good?12:53
@wikingi mean there are all those12:53
@wikingCCombinationRule stuff12:54
@wikingyou can pass MeanRule instead of the MajorityVote12:54
@wikingHeikoS, we use MeanRule for regression12:58
@wikingbut of course you can use that for classification as well12:58
@HeikoSok, I sent oli an email and explained this to hium12:59
@HeikoStelling that he needs to dive into the code a bit and understand the dynamics in there12:59
@HeikoSthx12:59
@wiking        rand_forest=RandomForest(feats_train,train_labels,20,1)13:05
@wiking        rand_forest.set_feature_types(ft)13:05
@wiking        rand_forest.set_machine_problem_type(PT_REGRESSION)13:05
@wiking        rand_forest.set_combination_rule(MeanRule())13:05
@wikingso you can still have rand_forest.set_combination_rule(MeanRule())13:05
@wiking for classification imo13:05
@HeikoSok cool13:06
@HeikoSwhat would be even nicer13:06
@HeikoSif you build a RF13:06
@HeikoSthat it automatically does that for you, i.e. the labels contain both the class labels as well as the averaged ones13:06
@HeikoSlike in SVM containing the distance and the label13:06
@HeikoSIll ask him to do so13:06
@wikingah13:07
@wikingyeah that could be done imo easily13:07
Trixis"SerializableXmlFile.java:34: error: cannot find symbol modshogunJNI.delete_SerializableXmlFile(swigCPtr)" build error :|14:31
-!- geektoni [~geektoni@93-34-128-40.ip49.fastwebnet.it] has joined #shogun15:06
-!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has quit [Ping timeout: 248 seconds]15:07
@wikingTrixis, moooo15:10
Trixiswiking: hey15:10
@wikingTrixis, did u compile it with xml support?15:11
Trixisturned xml off by accdient15:11
Trixisrecompiling now15:11
-!- wiking_ [~wiking@huwico/staff/wiking] has joined #shogun15:36
-!- mode/#shogun [+o wiking_] by ChanServ15:36
-!- wiking [~wiking@huwico/staff/wiking] has quit [Read error: Connection reset by peer]15:37
-!- wiking_ is now known as wiking15:37
-!- iglesiasg [~iglesiasg@217.119.234.214] has quit [Quit: leaving]15:38
Trixiswiking: "java.lang.UnsatisfiedLinkError: /homes/dhelekal/shogun/build/src/interfaces/java_modular/libmodshogun.so: /usr/lib64/libgomp.so.1: version `GOMP_4.0' not found (required by /homes/dhelekal/shogun/build/src/interfaces/java_modular/libmodshogun.so)"16:17
Trixisstill16:17
Trixiseven after i disabled all the libraries16:17
Trixiswiking: whats weird is that the library you gave me, and the library i just compiled are linked to the same libgomp as per ldd "libgomp.so.1 => /sw/opt/gcc-4.9.3/lib64/libgomp.so.1 (0x00007f4943be2000)"16:24
@sukey[https://github.com/shogun-toolbox/shogun] Issue https://github.com/shogun-toolbox/shogun/issues/3952 opened by lacava16:29
@sukey[https://github.com/shogun-toolbox/shogun] Issue https://github.com/shogun-toolbox/shogun/issues/395216:29
@wikingTrixis, ok so i read about it a bit16:49
@wikingunfortunately there needs to be some additional stuff16:49
@wikingthat'll get you the libgomp.a into the shared lib16:50
@wiking(libmodshogun.so)16:50
Trixiswiking: also after running make install, i now have both libmodshogun.so and libshogun.so in ~/lib/shogun/cli16:51
@wikingTrixis, https://stackoverflow.com/a/2768938716:51
Trixishm16:53
Trixisshit16:53
-!- olinguyen [81615ad9@gateway/web/freenode/ip.129.97.90.217] has joined #shogun16:58
@wikinghhehe yeah17:02
Trixiswiking: also why did the one you gave me run without that?17:05
@wiking:o17:06
@wikingdunno really :D17:06
@wikingwhat makes me really mad17:10
@wikingthat cmake does not do it for us17:10
Trixisye17:13
Trixisi personally hate cmake (one of the reasons why i avoid c++ despite being able to write in it at an OK level)17:13
@wikinguse babel17:14
@wikingno17:14
@wikingwhat is that17:14
@wikingbSOMETHING17:14
@wikingfrom google17:14
@wiking:D17:14
Trixisbabel is a cheminformatics toolkit, lol17:15
Trixiswiking: one of the reasons why i prefer java, because maven makes builds easy... (and its not as slow as python)17:15
@wikingbuscar17:15
@wikingbazel17:16
@wiking:D17:16
@wikingmmm17:16
@wikinghttps://github.com/shogun-toolbox/shogun/issues/395217:16
@wikingpuzzled by this error17:16
@wiking:D17:16
Trixiswiking: the issue im getting seems to be something weird with the way the cluster is configured...17:51
Trixisi tried running my program on the login node17:51
Trixisworks fine17:51
Trixis(obviously i had to kill it right after the library got loaded)17:51
Trixis(+ the login node has no computational power at all)17:51
-!- HeikoS [~heiko@host-92-0-169-11.as43234.net] has joined #shogun18:24
-!- mode/#shogun [+o HeikoS] by ChanServ18:24
Trixiswiking: so yeah, i'm quite mad at our IT :\18:46
-!- HeikoS [~heiko@host-92-0-169-11.as43234.net] has quit [Ping timeout: 240 seconds]19:00
-!- olinguyen [81615ad9@gateway/web/freenode/ip.129.97.90.217] has quit [Quit: Page closed]19:20
-!- geektoni [~geektoni@93-34-128-40.ip49.fastwebnet.it] has quit [Quit: Leaving.]19:34
-!- geektoni [~geektoni@93-34-128-40.ip49.fastwebnet.it] has joined #shogun19:34
-!- HeikoS [~heiko@host-92-0-169-11.as43234.net] has joined #shogun19:54
-!- mode/#shogun [+o HeikoS] by ChanServ19:55
-!- geektoni [~geektoni@93-34-128-40.ip49.fastwebnet.it] has quit [Ping timeout: 260 seconds]19:55
-!- HeikoS [~heiko@host-92-0-169-11.as43234.net] has quit [Ping timeout: 255 seconds]20:04
-!- olinguyen [81615ad9@gateway/web/freenode/ip.129.97.90.217] has joined #shogun20:10
-!- HeikoS [~heiko@host-92-0-169-11.as43234.net] has joined #shogun21:45
-!- mode/#shogun [+o HeikoS] by ChanServ21:45
-!- HeikoS [~heiko@host-92-0-169-11.as43234.net] has quit [Ping timeout: 268 seconds]22:15
-!- micmn [~micmn@ec2-54-67-24-92.us-west-1.compute.amazonaws.com] has quit [Ping timeout: 260 seconds]23:14
-!- micmn [~micmn@ec2-54-67-24-92.us-west-1.compute.amazonaws.com] has joined #shogun23:15
--- Log closed Wed Jul 26 00:00:11 2017
« Monday, 2017-07-24 Index Wednesday, 2017-07-26 »

Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!