Open in new window / Try shogun cloud
--- Log opened Tue Jul 25 00:00:10 2017
-!- olinguyen [81615ad9@gateway/web/freenode/ip.] has quit [Quit: Page closed]00:54
-!- geektoni [] has joined #shogun09:09
-!- iglesiasg [~iglesiasg@] has joined #shogun10:30
-!- mode/#shogun [+o iglesiasg] by ChanServ10:30
-!- zxtx [] has quit [Ping timeout: 260 seconds]10:40
-!- zxtx [] has joined #shogun10:43
-!- HeikoS [] has joined #shogun10:49
-!- mode/#shogun [+o HeikoS] by ChanServ10:49
@sukey[] Pull Request opened by geektoni11:24
@HeikoSgeektoni: still around?11:44
@HeikoSgeektoni: cool to catch you11:46
@HeikoSso how is the error going?11:46
geektoniHeikoS: not good11:46
geektoniI don't understand why it happens11:46
@HeikoSgeektoni: can you explain me very briefly the mechanics of what is happening?11:46
geektonido you mean the new CrossValidation?11:47
@HeikoSlike how you store things11:47
@HeikoShow they are emmited11:47
geektonibasically, each time you do a cross validation run11:47
geektonia CrossValidationStorage object is created11:47
@HeikoSis that something that was there before?11:48
geektonithis object has a vector which will contains the results of each fold evaluation.11:48
@HeikoSor did you create it?11:48
geektoniI created it11:48
@HeikoSok cool11:48
@HeikoSand the error comes from a double free?11:49
geektoninope, it came from a code region where it should not throw any error11:50
@HeikoSwhat exactly happens?11:50
@HeikoSwhat is the error?11:50
geektonithis line cause the memory error.11:51
@HeikoSand what is it?11:52
@HeikoSthe error?11:52
geektonifrom the backtrace, I can see there is something with TParameters11:53
geektonibut yeah, I don't understand why :P11:53
@HeikoSwhat does valgrind say to that?11:53
@HeikoSif you don't store the indices, then it doesnt happen?11:53
@HeikoSmaybe the indices are freed somewhere when they shouldnt11:54
geektoniyeah, if I remove these lines
geektonithe error disappear11:55
geektonibut then valgrind says that there are many memory leaks11:55
@HeikoSyou need to REF here as well no?11:56
@HeikoSalso no need to NULL check11:56
@HeikoSSG_UNREF does that11:56
@HeikoSalways REF before UNREF to avoid memory error when new/old objects are the same11:56
@HeikoSin fact, I think it might be forgotten REF the memory error11:56
geektonikk, I didn't add the SG_REF because I read that the clone() method does it for me :P11:56
@HeikoSclone returns object with ref counter set to 1?11:57
@HeikoSwell ok, but in a method where you receive a pointer, you dont know that riht?11:57
@HeikoSso you need to REF it definitely11:57
geektonimmh, but it's ref counter will be still set to 1 right?11:58
@HeikoSit might be that after having called the method, you will have to UNREF (if clone returns object with counter set to 1)11:58
@HeikoSbut that is style11:58
@HeikoSbut there might be a problem11:58
@HeikoSimagine m_test_true_result == results11:58
@HeikoSand the ref counter is 111:58
@HeikoSund then you unref11:58
@HeikoSthen it gets deleted11:59
@HeikoSand then you assign it to itself11:59
@HeikoSand boom11:59
geektoniok, I'll try to add SG_REF to the setters12:00
@HeikoSI mean check if that might cause problems12:01
@HeikoSbut add it anyways, to avoid problems in the future12:01
@HeikoSgeektoni: and can you use valgrind to find out where the memory was freed?12:01
geektoniHeikoS: mmh, actually I've never tried to use it that way12:02
@HeikoSI think it should tell you on a bad read12:02
@HeikoS"memory freed by"12:02
geektoniohh, that would be nice12:03
geektoniso, I'll let you know what I'll find out12:04
geektonimaybe is just a matter of SG_REF12:04
geektoniI still don't understand why it fails on a train_locked() method, though :/12:05
@HeikoSbecause that is the first place where the labels are used12:05
@HeikoS(my hypothesis)12:06
geektoniwell, I'll try and we'll ses how it goes.12:07
@HeikoSlet me know12:07
@HeikoSwill grab a coffee and then check back12:08
geektoniHeikoS: I'll be back in a few hours with some news12:08
-!- geektoni [] has quit [Remote host closed the connection]12:09
@wikingHeikoS, ping12:26
@HeikoSwiking: pong12:34
@wikinggot a second?12:34
@HeikoSwiking: sure!12:34
@HeikoSall good here, and you?12:34
@wikingalthough i shoulnd't be :)12:35
@wikingbut yeah getting better12:35
@wikinganyhow so i was wondering12:35
@wikingwhat we should do with those cases12:35
@wikingwhere we test fixtures12:35
@wikingfor a specific seed12:35
@HeikoSlike meta integration12:35
@wikingwhichi will cause a lot of headache now with12:35
@HeikoSor also fixtures12:35
@wikingc++11 random12:35
@wikingyep indeed12:35
@wikingmeta is currently a major problem12:35
@wikingwith GMM12:36
@HeikoSmeta is a problem indeed12:36
@HeikoSfor the unit tests with fixtures12:36
@HeikoSI think we can put in a deterministic drop in replacement for the random stream12:36
@HeikoSthe numbers it outputs might not be "correct"12:36
@HeikoSbut this is not the point in those tests anyways12:36
@wikingah you mean that instead of using some blob generator12:37
@wikingthat actually uses prng12:37
@wikingjust have a stream of precreated array of doubleS?12:37
@HeikoSyeah nonsense numbers12:37
@HeikoSI would still use the blobs12:37
@HeikoSbut the blobs random input is fixed12:37
@HeikoSso all the transformations are still at place12:37
@wiking'random input'?12:37
@HeikoSof blobs12:37
@HeikoSi.e. randn12:37
@wikingaaah yeah yeah12:38
@wikingi see what you mean12:38
@wikingit's gonna be tricky12:38
@HeikoSthis would only work for unit tests12:38
@wikingsomehow mock it :P12:38
@wikingthat's really not easy to do12:38
@wiking actually12:38
@HeikoScant we just replace all the CRandom instances with CFakeRandom automatically?12:38
@wikingbecause you would actually need to mock12:38
@HeikoSlike make that a feature of the random class12:38
@wikingnono we do not have CRandom anymore12:38
@wikingthat's the point12:38
@HeikoSyeah or the individual rngs12:39
@wikingthose are all std::12:39
@wikingwe do not wrap them anymore12:39
@HeikoSI see12:39
@wikingthat is the point12:39
@HeikoSthen mock yes12:39
@wikingbecause wrapping all the possible distrib12:39
@HeikoSbut thats not easy indeed12:39
@wikingwould be just cumbersome12:39
@HeikoSI dont know then :(12:39
@HeikoSit isnt possible to set the state of the std:: rng?12:40
@HeikoSit is12:40
@HeikoSbut the transformations are not portable12:40
@HeikoSthats the problem right?12:40
@wikingyeah the prng is actually good12:40
@wikingit's portable12:40
@wikingthe transforms aren't12:40
@HeikoSwhat is the community suggestion to solve this?12:40
@wikingnone :)12:40
@HeikoSand what platforms are outliers?12:40
@wikingthis is really depending on the actual12:41
@wikinglibstdc++ implementation12:41
@wikingso gnu would have one12:41
@wikingclang has it's own12:41
@wikingmsvc obviously12:41
@wikingbut yeah as many libstdc++12:41
@HeikoSlets think more12:41
@HeikoSthis is just for testing12:41
@HeikoSunit testing of fixed seeds12:41
@wikingwell i mean in this case12:41
@HeikoSso I mean we are not loosing any functionality or introduce bugs12:41
@wikingit doesn't matter12:41
@wikingif you fix the seed or not :P12:41
@HeikoSso why dont we disable the tests on non-gnu?12:42
@wikingmmm because who knows :D12:42
@wikingi mean we cannot be 100% certain12:42
@wikingthat this is the only difference12:42
@wikingor gonna be ever12:42
@HeikoSwe still have other unit tests12:42
@HeikoSfor things that are not random12:42
@HeikoS(like most things )12:42
@wikingmost of them are good12:42
@wikingsee the pr12:42
@wikingonly about 5 tests fail12:43
@wikingor 612:43
@HeikoSI think then we have to compromise12:43
@wikingout of which most of them are actual meta tests12:43
@HeikoSyeah the integration testing12:43
@HeikoSso then we have to accept that that is not portable and rely on unit tests on those cases12:43
@HeikoStravis still catches errors when someone fucks up an algorithm12:43
@HeikoSbut people with the "disabled test" os will have to rely on travis or buildbot, or something12:44
@HeikoSnot nice12:44
@HeikoSbut what can we do12:44
@HeikoSlisitsyn: you there?12:44
@HeikoSwiking: btw we can release bsd pretty soon I guess12:45
@wikingHeikoS, that is awesomeo12:45
@wikingquestion when12:45
@HeikoSjust needs lisitsyn to change the headers12:45
@wikingdo we wait end of aug12:45
@wikingor before?12:45
@HeikoSdoesnt matter12:45
@HeikoSnext release12:45
@HeikoSand develop as soon as its done12:45
@HeikoSso with the headers12:45
@wikingyeah ok... that is fine12:46
@HeikoSneed some sed magic that kills all the GPL ones in their different formats and stuff12:46
@HeikoSand replaces with BSD with the authors from git history12:46
@HeikoSmy preference would be a "This is BSD3, see" but I think that is borderline12:46
@wikingmmm idk12:47
@HeikoSbut whatever12:47
@wikinglets do the same thing as facebook12:47
@wikingbsd + patent :D12:47
@wikingthat way anybody who uses shogun12:48
@wikingwe have their licenses :)))12:48
@wikingand pattents12:48
@HeikoShahaha :D12:48
@HeikoSyes ok12:48
@HeikoSso yeah all that is missing is those headers, and all the corner cases that go with that12:48
@HeikoSand then a final grep GPL in the repository to check12:48
@HeikoSand gogo12:48
@wikingbtw how does sklearn handle this story of different randoms?12:49
@wikingor they have some common random in py12:49
@HeikoSI think you can pass rng instances in there12:51
@HeikoSby defaultl they use np.random12:51
@wiking    If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.12:51
@HeikoSwiking: oh that reminds me12:52
@HeikoSoli needs probability outputs for RF12:52
@HeikoSand that is a nice c++ task for him12:52
@HeikoSbut I am not too sure how that works in the bagging machine setup12:52
@HeikoS"The predicted class probabilities of an input sample are computed as the mean predicted class probabilities of the trees in the forest"12:52
@HeikoSthis is what skl does12:52
@HeikoSwiking: is that, I just pass the MeanRule and all good?12:53
@wikingi mean there are all those12:53
@wikingCCombinationRule stuff12:54
@wikingyou can pass MeanRule instead of the MajorityVote12:54
@wikingHeikoS, we use MeanRule for regression12:58
@wikingbut of course you can use that for classification as well12:58
@HeikoSok, I sent oli an email and explained this to hium12:59
@HeikoStelling that he needs to dive into the code a bit and understand the dynamics in there12:59
@wiking        rand_forest=RandomForest(feats_train,train_labels,20,1)13:05
@wiking        rand_forest.set_feature_types(ft)13:05
@wiking        rand_forest.set_machine_problem_type(PT_REGRESSION)13:05
@wiking        rand_forest.set_combination_rule(MeanRule())13:05
@wikingso you can still have rand_forest.set_combination_rule(MeanRule())13:05
@wiking for classification imo13:05
@HeikoSok cool13:06
@HeikoSwhat would be even nicer13:06
@HeikoSif you build a RF13:06
@HeikoSthat it automatically does that for you, i.e. the labels contain both the class labels as well as the averaged ones13:06
@HeikoSlike in SVM containing the distance and the label13:06
@HeikoSIll ask him to do so13:06
@wikingyeah that could be done imo easily13:07
Trixis" error: cannot find symbol modshogunJNI.delete_SerializableXmlFile(swigCPtr)" build error :|14:31
-!- geektoni [] has joined #shogun15:06
-!- HeikoS [] has quit [Ping timeout: 248 seconds]15:07
@wikingTrixis, moooo15:10
Trixiswiking: hey15:10
@wikingTrixis, did u compile it with xml support?15:11
Trixisturned xml off by accdient15:11
Trixisrecompiling now15:11
-!- wiking_ [~wiking@huwico/staff/wiking] has joined #shogun15:36
-!- mode/#shogun [+o wiking_] by ChanServ15:36
-!- wiking [~wiking@huwico/staff/wiking] has quit [Read error: Connection reset by peer]15:37
-!- wiking_ is now known as wiking15:37
-!- iglesiasg [~iglesiasg@] has quit [Quit: leaving]15:38
Trixiswiking: "java.lang.UnsatisfiedLinkError: /homes/dhelekal/shogun/build/src/interfaces/java_modular/ /usr/lib64/ version `GOMP_4.0' not found (required by /homes/dhelekal/shogun/build/src/interfaces/java_modular/"16:17
Trixiseven after i disabled all the libraries16:17
Trixiswiking: whats weird is that the library you gave me, and the library i just compiled are linked to the same libgomp as per ldd " => /sw/opt/gcc-4.9.3/lib64/ (0x00007f4943be2000)"16:24
@sukey[] Issue opened by lacava16:29
@sukey[] Issue
@wikingTrixis, ok so i read about it a bit16:49
@wikingunfortunately there needs to be some additional stuff16:49
@wikingthat'll get you the libgomp.a into the shared lib16:50
Trixiswiking: also after running make install, i now have both and in ~/lib/shogun/cli16:51
-!- olinguyen [81615ad9@gateway/web/freenode/ip.] has joined #shogun16:58
@wikinghhehe yeah17:02
Trixiswiking: also why did the one you gave me run without that?17:05
@wikingdunno really :D17:06
@wikingwhat makes me really mad17:10
@wikingthat cmake does not do it for us17:10
Trixisi personally hate cmake (one of the reasons why i avoid c++ despite being able to write in it at an OK level)17:13
@wikinguse babel17:14
@wikingwhat is that17:14
@wikingfrom google17:14
Trixisbabel is a cheminformatics toolkit, lol17:15
Trixiswiking: one of the reasons why i prefer java, because maven makes builds easy... (and its not as slow as python)17:15
@wikingpuzzled by this error17:16
Trixiswiking: the issue im getting seems to be something weird with the way the cluster is configured...17:51
Trixisi tried running my program on the login node17:51
Trixisworks fine17:51
Trixis(obviously i had to kill it right after the library got loaded)17:51
Trixis(+ the login node has no computational power at all)17:51
-!- HeikoS [] has joined #shogun18:24
-!- mode/#shogun [+o HeikoS] by ChanServ18:24
Trixiswiking: so yeah, i'm quite mad at our IT :\18:46
-!- HeikoS [] has quit [Ping timeout: 240 seconds]19:00
-!- olinguyen [81615ad9@gateway/web/freenode/ip.] has quit [Quit: Page closed]19:20
-!- geektoni [] has quit [Quit: Leaving.]19:34
-!- geektoni [] has joined #shogun19:34
-!- HeikoS [] has joined #shogun19:54
-!- mode/#shogun [+o HeikoS] by ChanServ19:55
-!- geektoni [] has quit [Ping timeout: 260 seconds]19:55
-!- HeikoS [] has quit [Ping timeout: 255 seconds]20:04
-!- olinguyen [81615ad9@gateway/web/freenode/ip.] has joined #shogun20:10
-!- HeikoS [] has joined #shogun21:45
-!- mode/#shogun [+o HeikoS] by ChanServ21:45
-!- HeikoS [] has quit [Ping timeout: 268 seconds]22:15
-!- micmn [] has quit [Ping timeout: 260 seconds]23:14
-!- micmn [] has joined #shogun23:15
--- Log closed Wed Jul 26 00:00:11 2017