| --- Log opened Mon Dec 05 00:00:59 2011 | ||
| -!- blackburn [~blackburn@31.28.51.215] has quit [Quit: Leaving.] | 00:28 | |
| -!- shogun-buildbot [~shogun-bu@7nn.de] has quit [Quit: buildmaster reconfigured: bot disconnecting] | 02:39 | |
| --- Log opened Mon Dec 05 03:23:14 2011 | ||
| -!- shogun-toolbox [~shogun@7nn.de] has joined #shogun | 03:23 | |
| -!- Irssi: #shogun: Total of 5 nicks [0 ops, 0 halfops, 0 voices, 5 normal] | 03:23 | |
| -!- Irssi: Join to #shogun was synced in 7 secs | 03:23 | |
| -!- in3xes_ [~in3xes@180.149.49.230] has joined #shogun | 04:14 | |
| -!- in3xes [~in3xes@180.149.49.230] has quit [Ping timeout: 244 seconds] | 04:18 | |
| -!- in3xes_ is now known as in3xes | 05:13 | |
| -!- in3xes [~in3xes@180.149.49.230] has quit [Ping timeout: 255 seconds] | 05:53 | |
| -!- blackburn [~blackburn@31.28.51.215] has joined #shogun | 07:48 | |
| -!- blackburn [~blackburn@31.28.51.215] has quit [Quit: Leaving.] | 08:13 | |
| -!- sonne|work [~sonnenbu@194.78.35.195] has joined #shogun | 11:40 | |
| -!- blackburn [5bdfb203@gateway/web/freenode/ip.91.223.178.3] has joined #shogun | 12:37 | |
| blackburn | sonne|work: I have reproduced custom kernel error.. | 13:06 |
|---|---|---|
| blackburn | setting triangle from full fails, will try to fix today | 13:06 |
| blackburn | then if you will manage to detect HMM or SVMs errors we can release | 13:06 |
| sonne|work | blackburn: hi | 14:05 |
| blackburn | hi | 14:05 |
| sonne|work | I noticed that 10 tests or so are failing on the train ride to work | 14:05 |
| sonne|work | not the old tests but the newer ones (python tester.py ) | 14:05 |
| sonne|work | I've debugged a few - but it still needs work also from you ... | 14:05 |
| blackburn | sonne|work: do you remember any? | 14:05 |
| sonne|work | all the converters have no test | 14:05 |
| blackburn | sonne|work: converters tests is difficult task | 14:06 |
| sonne|work | basically for non-existent tests one needs to call python generator.py <filename_of_the_python_foo> | 14:06 |
| blackburn | sonne|work: I did tell you already, there is problems.. | 14:06 |
| sonne|work | yeah but we need something reasonable... maybe you can come up with something | 14:06 |
| blackburn | damn, there are | 14:07 |
| blackburn | sonne|work: I can but I need time, I think it would be ok to do it later | 14:07 |
| blackburn | sonne|work: almost all of them requires to add some self-testing methods I can use in python | 14:07 |
| sonne|work | I think this is really important - otherwise we won't know if we broke anything | 14:07 |
| blackburn | sonne|work: I know status of converters 24h a week ;) | 14:08 |
| sonne|work | I don't have time now to discuss but lets do this maybe tonight | 14:08 |
| blackburn | sonne|work: but others - yes, we should fix it | 14:08 |
| blackburn | okay | 14:08 |
| sonne|work | (I thought so too about several other components btw) | 14:08 |
| sonne|work | cu | 14:08 |
| blackburn | I will be online from 23-30 UTC+4 | 14:09 |
| blackburn | cu | 14:09 |
| -!- blackburn [5bdfb203@gateway/web/freenode/ip.91.223.178.3] has quit [Quit: Page closed] | 15:19 | |
| -!- blackburn [~blackburn@188.168.4.192] has joined #shogun | 19:06 | |
| -!- blackburn [~blackburn@188.168.4.192] has quit [Ping timeout: 252 seconds] | 19:13 | |
| -!- blackburn [~blackburn@188.168.4.192] has joined #shogun | 19:14 | |
| blackburn | sonney2k: here | 19:36 |
| -!- shogun-buildbot [~shogun-bu@7nn.de] has quit [Quit: buildmaster reconfigured: bot disconnecting] | 20:12 | |
| -!- shogun-buildbot [~shogun-bu@7nn.de] has joined #shogun | 20:13 | |
| -!- mode/#shogun [+o sonney2k] by ChanServ | 20:17 | |
| blackburn | sonney2k: I wonder why almost all tests are 'ERROR' :D | 20:17 |
| blackburn | bugs.. I haz it | 20:25 |
| 15SAAI18M | shogun: Soeren Sonnenburg master * r77c55d7 / (2 files): fix library and evaluation test - http://git.io/iCkWjA | 20:29 |
| 15SAAI18M | shogun: Soeren Sonnenburg master * r0a36522 / data : require new data for regression tests - http://git.io/Iy6JdA | 20:29 |
| @sonney2k | blackburn, here maybe 3-4 tests fail + all the converter_* ones | 20:29 |
| blackburn | sonney2k: 50% here | 20:29 |
| blackburn | just like united russia election result | 20:29 |
| @sonney2k | united we stand | 20:30 |
| @sonney2k | wait | 20:30 |
| @sonney2k | this happens when you do python tester.py? | 20:30 |
| blackburn | sonney2k: ERROR OK ERROR OK .. | 20:30 |
| @sonney2k | ? | 20:32 |
| @sonney2k | yes or no? | 20:32 |
| blackburn | sonney2k: ah sorry yes | 20:32 |
| blackburn | wait | 20:32 |
| blackburn | python modular? | 20:32 |
| @sonney2k | yes the others don't support that | 20:33 |
| blackburn | then yes | 20:33 |
| @sonney2k | I have an error only in preprocessor_dimensionreductionpreprocessor_modular.py + converter_* | 20:33 |
| blackburn | lol | 20:34 |
| @sonney2k | so which is the first test failing for you? | 20:34 |
| blackburn | regression_krr_modular.py setting 1/2 ERROR | 20:34 |
| blackburn | regression_krr_modular.py setting 2/2 ERROR | 20:34 |
| blackburn | regression_libsvr_modular.py setting 1/2 ERROR | 20:34 |
| blackburn | regression_libsvr_modular.py setting 2/2 ERROR | 20:34 |
| blackburn | regression_svrlight_modular.py setting 1/2 ERROR | 20:34 |
| blackburn | regression_svrlight_modular.py setting 2/2 ERROR | 20:34 |
| blackburn | these ones for example | 20:34 |
| @sonney2k | you don't have lapack/atlas? | 20:35 |
| blackburn | sonney2k: I thought I have | 20:35 |
| @sonney2k | or let me guess different cpu / optimization flags different results? | 20:35 |
| @sonney2k | I made it easy to investigate | 20:36 |
| @sonney2k | just run python tester.py -d regression_krr_modular.py | 20:36 |
| @sonney2k | you will get detailled debug output and be in pdb to investigate | 20:36 |
| @sonney2k | there are two variables then a and b | 20:36 |
| shogun-buildbot | build #361 of r_static is complete: Failure [failed test_1] Build details are at http://www.shogun-toolbox.org/buildbot/builders/r_static/builds/361 blamelist: sonne@debian.org | 20:36 |
| @sonney2k | both should be lists | 20:36 |
| @sonney2k | with the things returned by this function | 20:36 |
| @sonney2k | blackburn, btw did you do git submodule update? | 20:37 |
| @sonney2k | not that we have different versions of data... | 20:37 |
| blackburn | sonney2k: minute | 20:37 |
| @sonney2k | I now forcefully enabled all tests checks in buildbot | 20:38 |
| blackburn | [ -6.67244038e-14 9.76996262e-15 8.28848101e-12 -2.22044605e-14 | 20:39 |
| blackburn | -2.08366657e-12 -7.94919686e-14 4.06741307e-12 -5.29576383e-14 | 20:39 |
| blackburn | -1.99840144e-15 -4.55191440e-15 5.21804822e-15 1.78868031e-12 | 20:39 |
| blackburn | 1.99840144e-15 -5.07416331e-12 -6.25055563e-14 -3.21964677e-15 | 20:39 |
| blackburn | 2.42028619e-14 5.55111512e-16 -7.99360578e-15 4.71844785e-16 | 20:39 |
| blackburn | 4.62185845e-13 6.66133815e-15 5.10702591e-15 -2.10942375e-15 | 20:39 |
| blackburn | -6.69064804e-12 -3.01980663e-14 0.00000000e+00 | 20:39 |
| blackburn | mismatch | 20:39 |
| blackburn | hm not really much | 20:39 |
| @sonney2k | that explains it though | 20:40 |
| blackburn | sonney2k: how? | 20:41 |
| @sonney2k | different atlas etc lib / cpu etc and so different results | 20:41 |
| @sonney2k | these things are strict 100% exact match tests to determine if anything changes... | 20:41 |
| blackburn | sonney2k: oh that's pretty bad way to test machine learning algos | 20:43 |
| @sonney2k | no | 20:44 |
| blackburn | sonney2k: I have no atlas btw | 20:44 |
| @sonney2k | it is very tough to figure out if your eps change is due to a bug or sth else | 20:44 |
| @sonney2k | so running things twice should give 100% same result | 20:44 |
| blackburn | twice sure, but different machines.. | 20:45 |
| @sonney2k | of course this assumes you can init the random number generator etc | 20:45 |
| @sonney2k | sure | 20:45 |
| @sonney2k | that's true but for that one could easily introduce a numerical comparison and allow for some slack | 20:45 |
| blackburn | sonney2k: I do not really know but I have x86 ubuntu here, while you have 64 | 20:46 |
| @sonney2k | tester.py is not that difficult to read and whenever we have floats/numpy matrices we could do a lossy test (if enabled at cmdline) | 20:46 |
| blackburn | it does matter sometimes | 20:46 |
| @sonney2k | problem is that it is not clear how much slack to allow | 20:47 |
| @sonney2k | floating point crap is awful | 20:47 |
| @sonney2k | e.g. a 1e-16 difference might lead to a 1e-2 difference in e.g. svm training | 20:48 |
| @sonney2k | and this could easily be a newly introduced bug (like forgotten variable to add or so) | 20:48 |
| blackburn | sonney2k: I have added a little slack | 20:48 |
| blackburn | distances are ok now | 20:48 |
| blackburn | but svms are not | 20:48 |
| @sonney2k | optional please | 20:49 |
| @sonney2k | (like the -d option with -s --slack) | 20:49 |
| @sonney2k | yeah | 20:49 |
| blackburn | kernels are halfly ok | 20:49 |
| blackburn | sonney2k: something is really wrong with subgradient, sgd and ocas | 20:50 |
| @sonney2k | sgd and ocas work just fine here | 20:50 |
| @sonney2k | subgradient yes broken | 20:50 |
| @sonney2k | well very random | 20:50 |
| blackburn | sonney2k: then everything should look ok | 20:52 |
| blackburn | sonney2k: have you ran ./test_all? | 20:52 |
| @sonney2k | yes - I'd like to get rid of these tests though | 20:52 |
| @sonney2k | that is the stuff we wanted to get rid of for 1.0.0 | 20:53 |
| @sonney2k | but failed | 20:53 |
| blackburn | sonney2k: how much errors did you have? | 20:53 |
| @sonney2k | subgradientsvm, sgd (but this can be ignored we changed default args and that is tested), hmm, localalignment, oligo string, FK, TOP | 20:55 |
| @sonney2k | that's it | 20:55 |
| @sonney2k | btw we should have these new tests for python-static too | 20:55 |
| @sonney2k | but hell no we dont' have them for matlab/octave/r/... | 20:56 |
| blackburn | sonney2k: I'm worried only about HMM :) | 20:57 |
| blackburn | I have never used to use it but | 20:57 |
| @sonney2k | blackburn, so can you come up with some way that we can test your dim red stuff? | 21:12 |
| blackburn | sonney2k: yes but not now | 21:12 |
| @sonney2k | how long do you think will you need? I would rather not want to release with new functions that dont' have tests | 21:12 |
| blackburn | sonney2k: I have to write complex tests | 21:12 |
| @sonney2k | why? | 21:12 |
| shogun-buildbot | build #63 of nightly_default is complete: Failure [failed test_1] Build details are at http://www.shogun-toolbox.org/buildbot/builders/nightly_default/builds/63 | 21:13 |
| blackburn | sonney2k: e.g. to test isomap I have to check if geodesic distances are ~ | 21:13 |
| @sonney2k | can't you just apply $DIMREDMETHOD and compute $SCORE and then compare? | 21:13 |
| blackburn | sonney2k: but for LLE I have to check something much more complex | 21:13 |
| shogun-buildbot | build #77 of nightly_all is complete: Failure [failed compile] Build details are at http://www.shogun-toolbox.org/buildbot/builders/nightly_all/builds/77 | 21:13 |
| blackburn | sonney2k: I'm really sure with LLE, etc | 21:14 |
| @sonney2k | so you want to compute sth like objective value? | 21:14 |
| blackburn | three algos I'm not sure with are NPE, LPP and LLTSA | 21:14 |
| blackburn | sonney2k: yes | 21:14 |
| blackburn | sonney2k: my point is we should fix HMM and release | 21:16 |
| blackburn | sonney2k: do not much worry about dimreduction things, I do a lot of development and testing still | 21:16 |
| blackburn | and now everything looks ok | 21:17 |
| blackburn | btw a little before it wasn't | 21:17 |
| @sonney2k | I am more worried about the dimred stuff than the hmm stuff :) | 21:22 |
| blackburn | sonney2k: ehmm why? | 21:24 |
| blackburn | sonney2k: I did tests on LLE, KLLE and both LTSA and KLTSA | 21:27 |
| blackburn | on different data | 21:27 |
| blackburn | and checked with scikits even | 21:27 |
| blackburn | you underestimate too much | 21:27 |
| @sonney2k | no this is just experience - at some point any algorithm we have in shogun was correct and well tested but the bigger things get the harder it is to ensure that there are no side effects | 21:36 |
| blackburn | it is much faster, flexible and accurate that it was in shogun 1.0 | 21:38 |
| @sonney2k | so please do the test when you find the time - we can now but I am not too happy about it | 21:38 |
| blackburn | sonney2k: test what? | 21:38 |
| blackburn | sonney2k: do you want to hold release without tests? | 21:40 |
| @sonney2k | please do the tests for converter_* / dimred* | 21:41 |
| @sonney2k | no, as I said we can release now | 21:41 |
| blackburn | sonney2k: but hmm?? | 21:43 |
| @sonney2k | I will attempt to fix all tests except converter_* / dimred* first | 21:44 |
| @sonney2k | then | 21:44 |
| blackburn | sonney2k: it can took more than 2 weeks | 21:44 |
| @sonney2k | could you please commit the modifyied tester.py? | 21:46 |
| blackburn | sonney2k: ok | 21:46 |
| blackburn | sonney2k: should it support different tolerances? | 21:46 |
| @sonney2k | yes, just as arg | 21:46 |
| blackburn | sonney2k: btw numpy.all is bad pattern | 21:51 |
| blackburn | there was near or so | 21:51 |
| @sonney2k | I only know any/all | 21:57 |
| 15SAAI18M | shogun: Sergey Lisitsyn master * r380f6e2 / testsuite/python_modular/tester.py : Added tolerance option for tester - http://git.io/UcRuRw | 22:06 |
| @sonney2k | blackburn, I would rather use the relative deviation e.g. (a-b)/(a+b) < eps instead of norm | 22:09 |
| @sonney2k | anyway time to sleep | 22:10 |
| blackburn | norm of (a-b)/(a+b)? | 22:10 |
| 15SAAI18M | shogun: Sergey Lisitsyn master * r21324b9 / (3 files in 2 dirs): Faces application to dimreduction, added word embedding example - http://git.io/Hlu8CQ | 22:13 |
| 15SAAI18M | shogun: Sergey Lisitsyn master * r51970aa / testsuite/python_modular/tester.py : Changed tolerance to max - http://git.io/M_3ezQ | 22:20 |
| 15SAAI18M | shogun: Sergey Lisitsyn master * r8108a74 / examples/undocumented/python_modular/converter_multidimensionalscaling_modular.py : Added testing info for MDS into python example - http://git.io/F_8Fyw | 22:41 |
| -!- blackburn [~blackburn@188.168.4.192] has quit [Quit: Leaving.] | 23:24 | |
| --- Log closed Tue Dec 06 00:00:10 2011 | ||
Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!