| --- Log opened Mon Jan 20 00:00:00 2014 | ||
| -!- shogun-notifier- [~irker@7nn.de] has quit [Quit: transmission timeout] | 00:59 | |
| -!- sonne|osx_ [~sonne@f052020014.adsl.alicedsl.de] has joined #shogun | 03:29 | |
| -!- sonne|osx [~sonne@f052018132.adsl.alicedsl.de] has quit [Ping timeout: 272 seconds] | 03:31 | |
| -!- sonne|osx_ is now known as sonne|osx | 03:31 | |
| shogun-buildbot | build #686 of nightly_default is complete: Failure [failed test] Build details are at http://buildbot.shogun-toolbox.org/builders/nightly_default/builds/686 | 04:12 |
|---|---|---|
| -!- shallhome [~shallhome@193.28.144.96] has joined #shogun | 04:40 | |
| -!- shallhome [~shallhome@193.28.144.96] has quit [Remote host closed the connection] | 05:14 | |
| -!- sonne|osx [~sonne@f052020014.adsl.alicedsl.de] has quit [Quit: sonne|osx] | 07:21 | |
| -!- sonne|osx [~sonne@89.204.130.57] has joined #shogun | 07:49 | |
| sonne|osx | besser82: we have the same pickle problem again | 07:54 |
| sonne|osx | besser82: please enlighten me how you fixed it! | 07:54 |
| -!- lisitsyn [~lisitsin@mxs.kg.ru] has joined #shogun | 07:56 | |
| -!- sonne|osx [~sonne@89.204.130.57] has quit [Ping timeout: 252 seconds] | 08:00 | |
| -!- sonne|osx [~sonne@89.204.137.33] has joined #shogun | 08:04 | |
| -!- sonne|osx [~sonne@89.204.137.33] has quit [Client Quit] | 08:05 | |
| -!- sonne|osx [~sonne@24-134-74-216-dynip.superkabel.de] has joined #shogun | 08:19 | |
| -!- sonne|osx [~sonne@24-134-74-216-dynip.superkabel.de] has quit [Client Quit] | 08:21 | |
| besser82 | sonne|work: which pickle-problem @ which test / build? I can haz linxx, plz? | 09:43 |
| sonne|work | besser82: well all python3 tests fail due to me adding the pickle string based comparison | 09:46 |
| sonne|work | besser82: so IIRC you figured there is some encoding problem.... | 09:47 |
| besser82 | sonne|work: encoding ain't the only one ;) | 09:47 |
| besser82 | sonne|work: but in fact the major one | 09:47 |
| besser82 | sonne|work: lemme cycle through all recent commits to see what changed and I'll give you some advice ;) | 09:48 |
| sonne|work | besser82: well it is just this pickle.dumps(objs) == comparison in tester.py | 09:50 |
| besser82 | sonne|work: then there is no general solution (tm); the most problematic thing is the incompat between numpy and py3-numpy, which can't be solved without external workaround like storing the numpy-datatypes with numpy's internal serialization methods, unfortunately. :( | 10:41 |
| sonne|work | besser82: well all I want is that this doesn't crash! | 10:41 |
| besser82 | sonne|work: you mean the segfaults, don't you? | 10:42 |
| sonne|work | yes | 10:42 |
| besser82 | sonne|work: those were never fixed IIRC | 10:42 |
| -!- lambday [67157f36@gateway/web/freenode/ip.103.21.127.54] has joined #shogun | 10:43 | |
| sonne|work | besser82: I thought you did sth with the encoding? | 10:43 |
| besser82 | sonne|work: because of we cannot restore py2_mod stuff with py3_mod... | 10:43 |
| besser82 | sonne|work: I did some experiments with that, but that actually wasn't successful :( | 10:44 |
| besser82 | sonne|work: It's not only the encoding / datatypes used in pickle; it's basically more :( | 10:44 |
| besser82 | sonne|work: I tried sth. with manually hex-editing the pickles to have str-data stored as py3-bytes, but then there is some other serious trouble, when opening them with py3_modular :( | 10:46 |
| besser82 | sonne|work: the only way I see to get that fixed is to have different pickles for py2 and py3 | 10:48 |
| besser82 | sonne|work: but hoding the same data... | 10:48 |
| besser82 | sonne|work: which brings us to the next question: How to get that done? | 10:49 |
| besser82 | sonne|work: I mean, those pickled must have been created somewhen; but how? | 10:49 |
| sonne|work | besser82: well it is just some ascii file that we create | 11:07 |
| sonne|work | besser82: how would embed some ascii string python2/3 compatible | 11:07 |
| adrin | sonne|work, besser82: hey, what's up? Good morning. | 11:24 |
| sonne|work | adrin: good morning | 11:24 |
| besser82 | sonne|work: basically I would try to use some py3::bytes-type which is perfectly binary-compat with char* | 11:24 |
| sonne|work | adrin: we are trying to get python3 to work ... | 11:24 |
| besser82 | sonne|work: but the problem here is py2 doesn't know "bytes" and has str only, which is the same as char*, too | 11:25 |
| adrin | if you put a "b" before the string, it will be interpreted as a byte-char in py3, and will be ignored in py2.7, why doesn't that work here? | 11:25 |
| sonne|work | adrin: who says it wouldn't :) | 11:26 |
| besser82 | adrin: Hey! G'Mornin! Yes, as sonne|work says all about Python3 && shogun | 11:26 |
| sonne|work | adrin: we are just not python3 experts | 11:26 |
| sonne|work | that is something we can fix at tonights stammtisch :) | 11:26 |
| adrin | besser82, sonne|work: there are two prefixes for strings in python, "u" and "b" | 11:26 |
| sonne|work | adrin: in python 2.6 etc too? | 11:26 |
| adrin | "u" is ignored by py3, and "b" is ignored by 2.7 | 11:27 |
| adrin | no py2.6 doesn't know about them | 11:27 |
| adrin | sonne|work: these two were introduced in py2.7, to make people able to have strings compatible in both 2 and 3 pythons, but you will need to have at least py2.7 | 11:27 |
| besser82 | sonne|work, adrin: for Py2.6 one needs to `import __future__` within some exception | 11:28 |
| besser82 | sonne|work, adrin: like try: import __future__ except: #noop | 11:28 |
| sonne|work | adrin: how do we get these if we are reading a file? | 11:28 |
| besser82 | sonne|work, adrin: but that won't fix the problem with pickles :-P | 11:29 |
| adrin | sonne|work: so the problem is that you want to read a file and you need to make sure you're interpreting it as ASCII bytes? | 11:29 |
| besser82 | adrin: no, we are importing a pickle made with proto 2 | 11:30 |
| besser82 | adrin: and for some reason py3_modular segfaults when restoring the py2_mod data from that pickle ;) | 11:30 |
| sonne|work | sry no time for that jsut now | 11:30 |
| sonne|work | adrin: will you be around in the evening? | 11:30 |
| adrin | sonne|work: yeah sure. I'll try to see what I can do until then. | 11:31 |
| adrin | besser82: In the mean time, it would be nice if you could send me a py2_mod data so that I can try to load it. | 11:38 |
| besser82 | adrin: you can fetch them from here: https://github.com/shogun-toolbox/shogun-data/tree/master/testsuite/tests | 11:39 |
| besser82 | adrin: here is a list of those failing / segfaulting tests: http://buildbot.shogun-toolbox.org/builders/deb4%20-%20python3/builds/36 | 11:39 |
| adrin | besser82: thanks | 11:40 |
| besser82 | adrin: the BIG problems is the binary-incompat of numpy-arrays && those segfaults, when restoring data from python_modular of shogun ;) np. You're welcome :D | 11:40 |
| -!- lambday [67157f36@gateway/web/freenode/ip.103.21.127.54] has quit [Ping timeout: 272 seconds] | 11:41 | |
| adrin | besser82: I see | 11:41 |
| besser82 | adrin: btw. manually converting str to bytes in pickle didn't work. :( | 11:42 |
| besser82 | adrin: like sed -e 's!^S\'!B\'!g' pickle.file | 11:43 |
| -!- bb_ [458fc8de@gateway/web/freenode/ip.69.143.200.222] has joined #shogun | 15:19 | |
| -!- bb_ [458fc8de@gateway/web/freenode/ip.69.143.200.222] has quit [Ping timeout: 272 seconds] | 15:39 | |
| -!- sonne|osx [~sonne@89.204.138.241] has joined #shogun | 15:40 | |
| sonne|osx | adrin, besser82 btw we can of course generate the data again with python2.7 or python3 if that helps | 15:46 |
| besser82 | sonne|osx: that would be the easiest solution ;) | 15:46 |
| besser82 | sonne|osx: we need separate pickles for py2 && py3 and all would be fine | 15:47 |
| sonne|osx | besser82: would this ensure that it can be read with python2.7 then? | 15:47 |
| sonne|osx | besser82: nah | 15:47 |
| sonne|osx | one for all | 15:47 |
| besser82 | sonne|osx: one for all isn't possible ---> numpy | 15:47 |
| sonne|osx | besser82: so there is no way to load numpy data from python3? | 15:47 |
| besser82 | sonne|osx: it is when pickle is from py3 | 15:48 |
| besser82 | sonne|osx: they are not interchangeable | 15:48 |
| besser82 | sonne|osx: that's why numpy has own im/export methods for data ;) | 15:48 |
| sonne|osx | besser82: and stuff like pickle.load(file_obj, encoding='latin1') ? | 15:48 |
| besser82 | sonne|osx: will be readble, but numpy data is a mess then | 15:49 |
| besser82 | sonne|osx: as to be seen in tesstsuite on py3, which fail but don't segfault ;) | 15:49 |
| sonne|osx | besser82: hmmhh btw will you be around for the stammtisch tonight? | 15:49 |
| besser82 | sonne|osx: sure! | 15:49 |
| besser82 | sonne|osx: not till the end, but at least ~2 h | 15:50 |
| sonne|osx | yeah sure | 15:51 |
| sonne|osx | besser82: no idea why the numpy people did it this way | 15:52 |
| sonne|osx | sucks badly that you cannot load your old data | 15:52 |
| besser82 | sonne|osx: you can but only with some extra steps | 15:52 |
| sonne|osx | besser82: which are? | 15:52 |
| besser82 | sonne|osx: you need to load from the py-ver the pickle was made with and then export the numpy-data with the export-method of numpy | 15:53 |
| besser82 | sonne|osx: that export is gurateed to be readable from every recent python | 15:53 |
| sonne|osx | besser82: which would be savenpy? or what | 15:53 |
| sonne|osx | c'mon | 15:54 |
| sonne|osx | :( | 15:54 |
| besser82 | sonne|osx: load pickle | 15:55 |
| besser82 | sonne|osx: $my_numpy_array.save($file_handle) | 15:55 |
| besser82 | sonne|osx: open other python-ver | 15:56 |
| besser82 | sonne|osx: create empty numpy-array | 15:56 |
| besser82 | sonne|osx: $new_array.load($my_export_from_old_pyver) | 15:56 |
| sonne|osx | besser82: not good - maybe we can work around this with python's stringIO | 15:57 |
| sonne|osx | adrin: what do you think? Maybe we store numpy stuff with stringIO - then we could keep one single format? | 15:58 |
| besser82 | sonne|osx: why using stringIO? | 15:58 |
| sonne|osx | besser82: string that behaves like a file handle | 15:58 |
| besser82 | sonne|osx: i mean numpy has own methods for that... ??? | 15:59 |
| sonne|osx | besser82: so we could write that string out instead of pickle | 15:59 |
| sonne|osx | and it would be compatible | 15:59 |
| besser82 | sonne|osx: that can be done with those numpy.methods, too | 15:59 |
| besser82 | sonne|osx: those are gurateed to be compat within different python-interp | 16:00 |
| sonne|osx | besser82: lets talk about that in the evening | 16:00 |
| besser82 | sonne|osx: kk | 16:00 |
| * sonne|osx off train | 16:00 | |
| -!- sonne|osx [~sonne@89.204.138.241] has quit [Quit: sonne|osx] | 16:00 | |
| -!- lambday [67157e37@gateway/web/freenode/ip.103.21.126.55] has joined #shogun | 16:04 | |
| -!- parijat [671b082a@gateway/web/freenode/ip.103.27.8.42] has joined #shogun | 16:10 | |
| adrin | besser82: me and my office mate were looking at the errors/data | 16:11 |
| besser82 | adrin: Many thx! and what did you find out? | 16:12 |
| adrin | besser82: he's never had such a problem with pickle, but he never used different python versions | 16:12 |
| besser82 | adrin: same for us, too ;) | 16:12 |
| adrin | besser82: nothing yet, but we're trying to see how to tell pickle how to interpret strings | 16:13 |
| besser82 | adrin: thanks for your work ;) | 16:13 |
| adrin | besser82: I personally prefer to handle it that way, or generating the data using utf strings, and tell pickle in py2 to interpret them as utf, I guess then there is no need for multiple data files, right? | 16:14 |
| besser82 | adrin: would that work with numpy-arrays? | 16:14 |
| adrin | besser82: don't know yet, but we have py2.6, 2.7, and 3.3 available. Personally I don't recommend to worry/support 2.6. 2.7 and 3.3 should be enough | 16:16 |
| besser82 | adrin: I would apprechiate support for py2.6 because of RHEL6 ;) | 16:16 |
| besser82 | adrin: But those details i might add somewhen later then ;) | 16:16 |
| adrin | besser82: I see. Then we'll test 2.6 too. Shall I ask for a peice of code which generates the pickle data + a sample saved data + the code which is supposed to read it? We found many files which can't read a data, but it would be nice if we had just an example, if available. If not, we'll try with a simple string/np.ndarray/set/list file if even with such a simple example the problem perists. | 16:19 |
| besser82 | adrin: I think sonney2k has that code or some examples about generation.... | 16:21 |
| besser82 | adrin: but trying with some toy data would at least point into the right direction, I think. | 16:21 |
| adrin | besser82: ok, we'll do it. BTW, is the stammtisch a real physical one or you guys were talking about a virtual stammtisch? :D | 16:23 |
| -!- gsomix [~gsomix@46.20.65.64] has quit [Ping timeout: 264 seconds] | 16:28 | |
| besser82 | adrin: virtual one on irc ;) | 16:33 |
| besser82 | adrin: distances are a bit problematic ;) | 16:33 |
| besser82 | adrin: you, toralf and sonney2k possibly could meet physically Berlin ;) | 16:34 |
| besser82 | adrin: me is from Hildesheim (30 km south of HAnnover) | 16:34 |
| -!- snik [~snik@lap-13-03.ag3.mpi-sb.mpg.de] has joined #shogun | 16:37 | |
| besser82 | snik: hi! You must be one of adrin's office mates, right?!? | 16:37 |
| snik | Hi! besser82, yes, i am! | 16:38 |
| besser82 | snik: nice to meet you! :D | 16:39 |
| snik | Nice to meet you too! :) | 16:39 |
| besser82 | snik: and thank for helping with that py3-thing! :D | 16:39 |
| adrin | besser82: haha, we're based in Saarbruecken, Berlin is a bit far :D | 16:39 |
| snik | oh, we have just started.. | 16:39 |
| besser82 | adrin: kk, that's a bit far to Berlin, too ;) | 16:40 |
| adrin | besser82: why the 'latin_1' decoding option for numpy arrays is not an option here? I didn't understand why it's not desireable. | 17:53 |
| besser82 | adrin: the problem is: numpy-array-pickles are not binary compatible between py2 and py3 :( | 17:54 |
| besser82 | adrin: one can encode / decode the pickled-data using 'latin-1', but the outcome is just garbage | 17:55 |
| adrin | besser82: ok, let me see, it didn't seem garbage, let me check | 17:56 |
| adrin | besser82: my mistake, it IS garbage. will be back soon :D | 17:57 |
| snik | ok, right, it really is! | 17:57 |
| besser82 | adrin, snik: numpy.save/load() works between pyvers, but pickle seems not. :( | 17:58 |
| -!- genrut [~genrut@193.28.144.131] has joined #shogun | 17:59 | |
| snik | besser82: pickle is really an issue, visibly even for the larger community, especially with the protocol thingy | 18:00 |
| besser82 | snik: unfortunately, yes :( | 18:00 |
| -!- genrut [~genrut@193.28.144.131] has quit [Ping timeout: 246 seconds] | 18:09 | |
| -!- parijat [671b082a@gateway/web/freenode/ip.103.27.8.42] has quit [Ping timeout: 272 seconds] | 18:40 | |
| adrin | besser82: I guess we failed :D, the problem is the ndarray representation is in unicode format, and can not be read in python 2 | 18:56 |
| adrin | besser82: we tried to save using py3, and load in py2, but for this reason it doesn't work | 18:57 |
| besser82 | adrin: save like pickle? | 18:57 |
| besser82 | adrin: or numpy.save? | 18:57 |
| adrin | besser82: yeah, in theory it should work if the protocol is 2, but it doesn't | 18:57 |
| adrin | besser82: did you guys try the numpy.save/savez? I would vote for them if they work between python versions | 18:58 |
| adrin | besser82: we thought if we could save the data in py3 in a way that py2 could read it, then somehow the problem is solved, but even this solution didn't work | 18:59 |
| besser82 | adrin: let me discuss that with sonney2k on tonight's stammtisch... | 18:59 |
| besser82 | adrin: the other ugly thing is those segfaults of python3_modular, when trying to recreate data from py2 pickle... :( | 19:00 |
| besser82 | adrin: any thoughts on that? | 19:00 |
| snik | besser82, adrin: yes, i too vote for numpy.save and savez options, but may be we should give a second round of check to see pickling in py3 and reading/loading in py2 works/does not work. | 19:01 |
| adrin | besser82: we believe they are not segfaults and are the exceptions of python which are being interpreted by cmake as segfault. Can this be the case? | 19:01 |
| snik | besser82: i am looking into the segfaults of all those Python_modulars, i will get back if can find some thing on it | 19:01 |
| besser82 | adrin: nope, those segfaults are segfaults :( | 19:02 |
| besser82 | adrin: but not in python directly | 19:02 |
| besser82 | adrin: the python_modular just interfaces to libshogun | 19:02 |
| besser82 | adrin: and the segfault come from there... | 19:02 |
| adrin | besser82: couldn't it be because the data is no loaded properly? | 19:02 |
| besser82 | adrin: I suspect the incompat between py3-ustr and char* | 19:03 |
| besser82 | adrin: py2::str == char* | 19:03 |
| besser82 | adrin: py3::ustr == somewhat homebrewed by python-devs | 19:03 |
| besser82 | adrin: but py3::bytes == char* ;) | 19:04 |
| adrin | besser82: oh yeah you're right. It perfectly makes sense if we're passing some str data to the C code, but if the data is just a numpy array and a bunch of numbers, why would it give a segfault? | 19:04 |
| besser82 | adrin: nope that just isn't numpy-only | 19:05 |
| besser82 | adrin: there is some json-style data | 19:05 |
| besser82 | adrin: which is stored as py2::str and get converted to py3::ustr on import ---> crash | 19:05 |
| besser82 | adrin: we need to find a way to store that json-string somehow both py-ver will read correctly | 19:06 |
| besser82 | adrin: but howss | 19:06 |
| besser82 | adrin: but how? ;) | 19:06 |
| adrin | besser82: yeah it makes sense, I hate to have to change the C code to support the unicode | 19:06 |
| adrin | besser82: the quick solution would be to convert the encoding of all the strings into bytearray before giving it to the C code, which is straightforward | 19:07 |
| adrin | besser82: but I'm not sure if that's a desireable solution | 19:07 |
| besser82 | adrin: mhhh | 19:07 |
| besser82 | adrin: that would need some custom unpickler, right?!? | 19:07 |
| adrin | besser82: it might need to feed the encoding parameter while unpickling only in py3, and maybe also converging the ustr to bytearray right after loading the data. | 19:09 |
| adrin | bessesr82: about the np.savez, I just checked and it seems the data saved in py3 can be perfectly loaded in py2 | 19:11 |
| adrin | besser82:brb | 19:12 |
| besser82 | adrin: as I said :D kk | 19:20 |
| snik | besser82, adrin: leaving office now, see you guys tomorrow! Will sleep over the problem, hopefully we will shoot it down soon! :) | 19:25 |
| besser82 | snik: allrighty! Gute Nacht, sch?nen Feierabend :D Many thx for your help! | 19:25 |
| snik | besser82: my pleasure, if i can be of any help to the community! Danke, gleichfalls :) | 19:27 |
| -!- thoralf [~thoralf@91-65-136-3-dynip.superkabel.de] has joined #shogun | 19:28 | |
| thoralf | Heyhey :) | 19:28 |
| besser82 | thoralf: Hey! | 19:29 |
| thoralf | S?ren told you're missing me. ;) | 19:29 |
| -!- iglesiasg [~iglesiasg@524AE0A7.cm-4-3d.dynamic.ziggo.nl] has joined #shogun | 19:31 | |
| -!- mode/#shogun [+o iglesiasg] by ChanServ | 19:31 | |
| @iglesiasg | sup people | 19:32 |
| besser82 | thoralf: yes, the stammtisch feels so empty w/o you ;) | 19:32 |
| besser82 | iglesiasg: nuthang new ;) And yourself? | 19:32 |
| @iglesiasg | besser82, in process of git bisect-ing :) | 19:33 |
| besser82 | iglesiasg: Nice! Sound promising to get that testsuite running ;) | 19:33 |
| -!- pickle27 [d8bfe7da@gateway/web/freenode/ip.216.191.231.218] has joined #shogun | 19:45 | |
| @sonney2k | iglesiasg, how is it going? | 19:56 |
| @sonney2k | And heya good evening gents :) | 19:57 |
| @sonney2k | pickle27, any news from across the world? | 19:57 |
| pickle27 | not really! | 19:58 |
| @iglesiasg | sonney2k, compiling another revision :D | 19:58 |
| pickle27 | had a question about gsoc - when is the app deadline? | 20:00 |
| -!- votjakovr [~votjakovr@188.134.46.30] has joined #shogun | 20:00 | |
| @iglesiasg | let's check | 20:00 |
| @iglesiasg | pickle27, http://www.google-melange.com/gsoc/events/google/gsoc2014 | 20:01 |
| @iglesiasg | there you go | 20:01 |
| @iglesiasg | valentine's day :DD | 20:01 |
| besser82 | sonney2k: heyya! | 20:01 |
| besser82 | adrin: ping? | 20:01 |
| @sonney2k | pickle27, verrry soon... | 20:01 |
| pickle27 | cool! I was just curious - are we submitting all of those to the idea list or are we going to clean it up a bit | 20:01 |
| adrin | besser82: ping-reply | 20:02 |
| @sonney2k | hey votjakovr! | 20:02 |
| besser82 | adrin: wb :D | 20:02 |
| @sonney2k | pickle27, we need to bring it into gsoc'13 shape | 20:02 |
| adrin | besser82: had to rush for dinner, mensa-last-minute-food :D | 20:02 |
| @iglesiasg | I think there will be updates to the ideas list | 20:02 |
| besser82 | lisitsyn, lisitsyn1: AUSWEIS BITTE! :-P | 20:02 |
| votjakovr | sonney2k: hi :) | 20:03 |
| @sonney2k | adrin, besser82 so regarding numpy and pickle - why not save things with numpy to a file - then read it in and pickle it as string / byte array? | 20:05 |
| @sonney2k | or snik ^^ | 20:05 |
| @iglesiasg | pickle27, btw, I saw last week the video you posted on fb about shopify | 20:06 |
| @iglesiasg | pickle27, the company looks really great man! | 20:06 |
| lisitsyn1 | besser82: mein ausweis ist weg | 20:06 |
| pickle27 | yeah its a pretty good time here | 20:06 |
| @iglesiasg | awesome | 20:06 |
| pickle27 | btw if anyone is looking for a job .. just let me know | 20:06 |
| besser82 | lisitsyn1: :-P | 20:06 |
| lisitsyn1 | pickle27: remote? ;) | 20:07 |
| @iglesiasg | pickle27, will ask in three years :D | 20:07 |
| -!- parijat [671b082a@gateway/web/freenode/ip.103.27.8.42] has joined #shogun | 20:07 | |
| pickle27 | we have some remote people, we mostly try and get people to relocate atm | 20:07 |
| adrin | sonny2k: I'm confused, if you save numpy data with numpy.save/savez, you can load it in another python version. About strings, we need to set the encoding while loading in py3, and maybe then change the encoding before passing to the C code | 20:07 |
| besser82 | sonney2k: as adrin said... | 20:08 |
| pickle27 | lisitsyn1: I also joined the sublime text bandwagon | 20:08 |
| besser82 | sonney2k: numpy-arrays in pickles are not compat between py2 / py3 and never will | 20:08 |
| lisitsyn1 | pickle27: cool, I think I am going to do that too | 20:08 |
| lisitsyn1 | oh sh I just had 13 hrs workday after fulltime working weekend | 20:09 |
| @sonney2k | adrin, besser82 the point is we want a *single* .pickle file to write to | 20:09 |
| lisitsyn1 | this makes no sense :D | 20:09 |
| besser82 | sonney2k: as it looks like atm: not a chance | 20:09 |
| besser82 | sonney2k: or we will definetively need an own impl. of Unpickler | 20:10 |
| @sonney2k | adrin, besser82 so we could do a trick to store numpy arrays - we save them to a file with save, read them in as ascii string - followed by some decoding | 20:10 |
| besser82 | sonney2k: that is not needed | 20:10 |
| besser82 | sonney2k: one can save that data with numpy's own methods and read it in as needed | 20:10 |
| adrin | sonny2k: I guess you're mixing two seperate problems, strings, and numpy arrays. | 20:10 |
| besser82 | sonney2k: the string-problem is mostly related to the segfaults, when restoring modshogun-data from pickles | 20:11 |
| @sonney2k | besser82, but how do you then save shogun's objects and numpy arrays when they are in the same list? | 20:11 |
| @sonney2k | adrin, no no | 20:11 |
| besser82 | sonney2k: in different pickles :-P | 20:12 |
| besser82 | sonney2k: one simply cannot have a pickle with that data to be compatible between py2 / py3 | 20:12 |
| besser82 | sonney2k: the other approach would be json or hdf5 | 20:13 |
| besser82 | sonney2k: hdf5 using pytables | 20:13 |
| -!- thoralf [~thoralf@91-65-136-3-dynip.superkabel.de] has quit [Ping timeout: 265 seconds] | 20:14 | |
| besser82 | sonney2k: the numpy-devs discussed that for ~3 monthes | 20:14 |
| besser82 | sonney2k: how to keep pickles compat | 20:14 |
| besser82 | sonney2k: but one cannot accomplish | 20:15 |
| besser82 | sonney2k: if there would have been a way they would have done it | 20:15 |
| -!- thoralf [~thoralf@91-65-136-3-dynip.superkabel.de] has joined #shogun | 20:17 | |
| pickle27 | lisitsyn1: I thought you already did? we talked about it a while ago | 20:18 |
| lisitsyn1 | pickle27: nah no | 20:18 |
| lisitsyn1 | wanted to | 20:18 |
| pickle27 | its pretty sweet, plus there is vintage mode if there is some vim command you want | 20:18 |
| pickle27 | btw do we still use tabs in shogun? | 20:19 |
| @sonney2k | besser82, ohh well - we could of course save numpy data as sth else but in an extra file - nah we have too many already and we could certainly save each element in the list we are returning into the same file with some minor logic | 20:24 |
| @sonney2k | besser82, we could drop returning numpy objects altogether but instead work with scalars / the shogun objects directly | 20:25 |
| @sonney2k | pickle27, how is it going with the website? | 20:25 |
| besser82 | sonney2k: which still will not be pickable... | 20:25 |
| pickle27 | I've been poking around | 20:25 |
| pickle27 | its complicated because we want all that information just also to be simpler | 20:26 |
| adrin | besser82: if the json dump for ndarray could work, why wouldn't numpy-devs use it? | 20:26 |
| besser82 | adrin: they do sth. similar in their load/save methods ;) | 20:26 |
| adrin | besser82: but anyway I'm not much in favor of dumping floating points as serialized-text data | 20:27 |
| besser82 | adrin: yes, that might be problematic | 20:27 |
| @sonney2k | besser82, sure it would - look we don't need the all the magic | 20:33 |
| @sonney2k | besser82, just in the examples we return instead of the numpy array | 20:33 |
| @sonney2k | some string | 20:33 |
| @sonney2k | and that /can/ be pickled and compared | 20:34 |
| -!- parijat [671b082a@gateway/web/freenode/ip.103.27.8.42] has quit [Ping timeout: 272 seconds] | 20:34 | |
| @sonney2k | or even we return the numpy array but iterate over objects if they are a list/tuple and convert them to some string representation - aka binary npy format | 20:35 |
| -!- lambday [67157e37@gateway/web/freenode/ip.103.21.126.55] has quit [Ping timeout: 272 seconds] | 20:35 | |
| @sonney2k | hey thoralf - I am still confused what to do with the bmrm fix /patch of yours | 20:36 |
| thoralf | sonney2k: Me too. ;) | 20:36 |
| adrin | sonney2k: so you're suggesting to make everything numpy-dump-able? | 20:37 |
| thoralf | sonney2k: I'm was pretty sure that I did the right thing, but Michal did the whole thing, so I need to check even more careful. ;) | 20:37 |
| @sonney2k | adrin, well we have the following setup: we return some list / tuple of results in our examples | 20:38 |
| @sonney2k | adrin, some of them are scalars, some SGObjects and some is numpy stuff | 20:39 |
| @sonney2k | adrin, in our regression tests we simply check if the objects are the same / similar | 20:39 |
| @sonney2k | so why not for numpy stuff add some extra treatment? | 20:39 |
| @sonney2k | thoralf, yeah to bad he didn't respond to your last Q | 20:40 |
| adrin | sonney2k: yeah sure, if it solves the problem it makes sense. | 20:42 |
| thoralf | sonney2k: Any suggestion how to proceed? | 20:42 |
| thoralf | I could reason (in more detail) about how I think it is, but maybe someone else could double-check my arguments... | 20:43 |
| @sonney2k | thoralf, well it is michael who knows - the only idea I have is take some big example train it with bmrm and then apply the patch and see if results are *exactly* the same | 20:44 |
| @sonney2k | iglesiasg, done with bisect? | 20:45 |
| @iglesiasg | sonney2k, Bisecting: 3 revisions left to test after this (roughly 2 steps) | 20:45 |
| @iglesiasg | I am going to see those revisions as well... | 20:45 |
| @iglesiasg | but I am suspicious | 20:45 |
| @iglesiasg | all the revisions I have found so far during the process were good... | 20:46 |
| adrin | sonney2k, besser82: I need to catch the bus, see you tomorrow. | 20:47 |
| @sonney2k | adrin, cu! | 20:47 |
| besser82 | adrin: kk, sch?nen Feierabend && Gute Nacht! | 20:47 |
| thoralf | sonney2k: But since his implementation is definitely wrong (producing memory errors when setting beta), I don't expect the same results. | 20:48 |
| @sonney2k | iglesiasg, if you figure that out the only remaining test is evaluation_clustering_simple | 20:48 |
| @iglesiasg | arrrgh | 20:49 |
| @sonney2k | today was a tough day in berlin - walking on ice | 20:49 |
| @iglesiasg | I am starting to get concerned that there is something wrong going on here | 20:49 |
| thoralf | sonney2k: I think the only question is if my patch is correct (beside the memory integrity). | 20:49 |
| @sonney2k | I've never seen more people fall... | 20:49 |
| thoralf | sonney2k: But you're idea is good: We could try to solve the same problem and then compare duality gap or something like that. | 20:52 |
| @sonney2k | thoralf, actually we have some test even | 20:53 |
| @sonney2k | just since yesterday :) | 20:53 |
| thoralf | Or use the weight vector and use the (training) data to compute sth. like average square mean between predicted and trained value | 20:54 |
| @sonney2k | thoralf, no no - it should be 100% identical | 20:54 |
| thoralf | sonney2k: I think the implementation is wrong! | 20:55 |
| thoralf | The beta is initialized wrong! | 20:55 |
| thoralf | So why should it return the *same* results? | 20:55 |
| @sonney2k | thoralf, muahh then you should email vojtech / CC michael with your explanation. they are $DEITY | 20:56 |
| lisitsyn1 | don't fix it | 20:56 |
| lisitsyn1 | :D | 20:57 |
| lisitsyn1 | it is going to be a hydra | 20:57 |
| -!- charry [~charry@91.210.101.37] has joined #shogun | 20:57 | |
| @sonney2k | lisitsyn, yes marvin :) | 20:58 |
| thoralf | lisitsyn1: I already fixed it. ;) | 20:58 |
| thoralf | sonney2k: vojtech-who? | 20:58 |
| @sonney2k | thoralf, vojtech franc | 20:59 |
| thoralf | Hmm, okay. | 21:07 |
| @iglesiasg | sonney2k, I screwed up | 21:10 |
| @iglesiasg | a revision that yesterday failed, now it doesn't fail | 21:11 |
| @iglesiasg | only reason I found, because I am compiling using ccache and something is messing around with this | 21:11 |
| @sonney2k | iglesiasg, :`-( | 21:12 |
| @iglesiasg | yep, exactly | 21:12 |
| @sonney2k | iglesiasg, did you uninstall previous releases? | 21:12 |
| @iglesiasg | sonney2k, no | 21:12 |
| @sonney2k | perhaps you have a include conflict? | 21:13 |
| @sonney2k | iglesiasg, ohh you have to! | 21:13 |
| @iglesiasg | :( | 21:13 |
| @iglesiasg | how do I do that? | 21:13 |
| @iglesiasg | I was basically | 21:13 |
| @iglesiasg | git co some_revision | 21:13 |
| @iglesiasg | ccache make -j2 | 21:13 |
| @iglesiasg | sudo make install | 21:13 |
| @iglesiasg | in tests/integration/python_modular $ python tester.py linear_time_mmd.py | 21:13 |
| -!- negativ [~negativ@91.210.101.170] has joined #shogun | 21:14 | |
| @iglesiasg | sonney2k, what should I have done? | 21:14 |
| @iglesiasg | sonney2k, rm -rf build each time? uninstall how? | 21:14 |
| @sonney2k | iglesiasg, always remove the installed shogun dir incl headers/libshogun/python stuff | 21:14 |
| @sonney2k | yes | 21:14 |
| @sonney2k | and rm-rf build too all the time | 21:14 |
| @wiking | include conflict should have been fixed already by one of besser82's commit | 21:15 |
| -!- charry [~charry@91.210.101.37] has quit [Ping timeout: 252 seconds] | 21:15 | |
| @sonney2k | wiking, doesn't help if we git bisect old releases | 21:15 |
| @wiking | adding make uninstall target to cmake is a fairly easy process, there are howtos for it so if somebody is up for that should do it | 21:15 |
| @wiking | sonney2k: indeed | 21:16 |
| @iglesiasg | aargh fuck me man, this is so annoying | 21:16 |
| @sonney2k | and that is what we are doing | 21:16 |
| @wiking | sonney2k: bisect within a docker image | 21:16 |
| @sonney2k | wiking, how does that help? | 21:16 |
| @sonney2k | I mean rm'ing is far easier | 21:16 |
| @wiking | well you can just delete the container... | 21:16 |
| @wiking | and start from the next commit | 21:16 |
| @iglesiasg | sonney2k, btw I am not sure what dirs yo umean with "remove the installed shogun dir incl headers/libshogun/python stuff" | 21:17 |
| @wiking | iglesiasg: e.g. /usr/local/include/shogun /usr/local/lib/libshogun* /usr/local/lib/python2.7/sites-package/shogun | 21:18 |
| @wiking | something like this | 21:18 |
| @wiking | if u have installed it | 21:18 |
| @wiking | of cours | 21:18 |
| @iglesiasg | wiking, got it, thanks! | 21:18 |
| @iglesiasg | wiking, I was not sure if there were those ones or something else inside shogun tree dir | 21:18 |
| @wiking | iglesiasg: but you dont have to make install to be able to run ctests | 21:18 |
| @iglesiasg | wiking, the problem is that in this case I cannot try it with ctest | 21:19 |
| @iglesiasg | wiking, I have to run by hand the tester | 21:19 |
| @iglesiasg | wiking, let me show you why | 21:19 |
| @wiking | iglesiasg: then just set PYTHONPATH | 21:19 |
| @wiking | and LD_LIBRARY_PATH | 21:19 |
| @iglesiasg | why so? | 21:19 |
| @iglesiasg | to what? | 21:19 |
| @wiking | so that u dont need the make install | 21:19 |
| @wiking | well why else would you kill yourself with make install? :) | 21:19 |
| @sonney2k | true - you don't need make install then | 21:19 |
| @wiking | and then do rm -rf all the time after a check | 21:20 |
| @iglesiasg | don't get it | 21:20 |
| @iglesiasg | I thought swig interfaces required the sudo make install | 21:20 |
| @wiking | iglesiasg: well u dont need to do make install in order to be able to do a simple check | 21:20 |
| @wiking | iglesiasg: no they dont | 21:20 |
| @iglesiasg | I guess the pythonpath ld_library_path thing fix that | 21:20 |
| @wiking | iglesiasg: what's the script u r trying to run? | 21:20 |
| @iglesiasg | wiking, I am running the integration test on linear_time_mmd.py | 21:21 |
| @wiking | mmm | 21:21 |
| @iglesiasg | so tests/integration/python_modular/tester.py linear_time_mmd.py | 21:21 |
| @wiking | well you might have a smal problem | 21:21 |
| @wiking | with the version | 21:21 |
| @wiking | where imports are not from modshogun | 21:21 |
| @wiking | but from shogun | 21:21 |
| @iglesiasg | ok... | 21:21 |
| @iglesiasg | I think I am cool with doing install and removing the directories actually | 21:22 |
| @iglesiasg | just would like to avoid repeating the process all over again :) | 21:22 |
| @iglesiasg | sonney2k, but I think it is better if I try to do this connecting remotely to fatbot | 21:24 |
| @iglesiasg | each installation takes far too long locally | 21:24 |
| @iglesiasg | hmm but will I have problems there with the installation issue?? | 21:25 |
| @sonney2k | iglesiasg, don't install | 21:25 |
| @sonney2k | like wiking said | 21:25 |
| @sonney2k | it really is better | 21:25 |
| @iglesiasg | sonney2k, sure ok | 21:25 |
| @iglesiasg | but tell me what to do instead | 21:25 |
| @iglesiasg | set PYTHONPATH, LD_LIBRARY_PATH | 21:25 |
| @iglesiasg | to? | 21:25 |
| @iglesiasg | I don't really follow why that fixes it | 21:26 |
| @iglesiasg | I would understand that setting them tells make where to install the stuff | 21:26 |
| @sonney2k | iglesiasg, PYTHONPATH=path/to/modshogun ( the directory) | 21:26 |
| @sonney2k | iglesiasg, LD_LIBRARY_PATH=/path/to/libshogun (the dir again) | 21:26 |
| @sonney2k | and then python example.py | 21:26 |
| @sonney2k | iglesiasg, well if there is no default install of shogun -> no clash | 21:27 |
| @iglesiasg | sonney2k, the directory means from where I want to execute the python example.py? | 21:27 |
| -!- gsomix [~gsomix@46.20.65.64] has joined #shogun | 21:28 | |
| @sonney2k | iglesiasg, no the dir where the modshogun.so / libshogun*.so* are in | 21:28 |
| @iglesiasg | sonney2k, i am not sure what to do really | 21:30 |
| @sonney2k | iglesiasg, 1) remove shogun install'd files | 21:32 |
| @sonney2k | 2) build again | 21:33 |
| @sonney2k | 3) set python/ld_lib path | 21:33 |
| @sonney2k | 4) run the example | 21:33 |
| @sonney2k | if any of this fails ask here | 21:33 |
| gsomix | sonney2k, hey. sry, I cannot to participate in today's hacking night. :( my laptop got wet by milk, need to dry. | 21:33 |
| @iglesiasg | aaaah I think I got it now | 21:33 |
| @iglesiasg | sonney2k, the key was telling me that 3) goes after 2) :) | 21:34 |
| gsomix | nothing serious, but keyboard does not work properly | 21:34 |
| @sonney2k | gsomix, you can type pretty well still ;) | 21:35 |
| gsomix | sonney2k, nope | 21:38 |
| gsomix | shift, ctrl, alt keys are crazy | 21:38 |
| @iglesiasg | sonney2k, was it ssh to bug.7nn.de for fatbot? | 21:41 |
| @sonney2k | sonney2k, fatbot.shogun-* | 21:42 |
| @iglesiasg | ssh to fatbot.shogun.toolbox.org is not working at all | 21:42 |
| -!- infomix [~infomix@91.210.100.173] has joined #shogun | 21:42 | |
| @sonney2k | iglesiasg, works totally fine here | 21:43 |
| @iglesiasg | it sort of hangs up here | 21:43 |
| @sonney2k | iglesiasg, well then do it locally... | 21:43 |
| @iglesiasg | sonney2k, a little bit weird that I ping without a trouble though | 21:44 |
| @sonney2k | lisitsyn, wiking can you still ssh fatbot? | 21:47 |
| -!- gsomix [~gsomix@46.20.65.64] has quit [Ping timeout: 272 seconds] | 21:47 | |
| @wiking | sonney2k: i could about 4 hours ago | 21:48 |
| @wiking | and still i can | 21:49 |
| @wiking | y? | 21:49 |
| @wiking | 15388 sonne 20 0 8135m 36m 392 R 99.6 0.1 7685:48 python | 21:49 |
| @wiking | 15391 sonne 20 0 12.9g 36m 392 R 99.6 0.1 7726:18 python | 21:49 |
| @wiking | they are running now for a while | 21:49 |
| @iglesiasg | wiking, would you mind adding a moment my public key? | 21:51 |
| @iglesiasg | wiking, the one that is there is an old one | 21:51 |
| @wiking | iglesiasg: in your home? | 21:51 |
| @wiking | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDq1H9qVAPm7JZkhy+Mk2cwwExAJJgbnI0S9bT4ZO03caNWuBaP92c/mHNQi7x1+IyCCwd4jqInMrQ6MDTqz1YGZqsWknRzYxT0hs8Z1SSD06gs5APBfkaVpMgyuwLFyKTjqmib9H9vyZnTMj/d+ZYXxlb8DNT78aJqEnDRe8rNps4gCGOq5bJNkcaIV/aL5IJmwLmofjvhtD2O4I6U8pUgat6ulsJFlDTDCha3lEfMcP4KeqH7uSK/pCuT8Z1Xp0LaGgQjOlrHtTRJ73b3U5bXRKPMn+UmifdqXjIB5Vhbj0KDxnqcP3k8cJK27gUDXa8ApTj9oWOMFK6/qQxE/qLB fernando.iglesiasg@gmail.com | 21:52 |
| @wiking | this is what you have in authorized_keys atm | 21:52 |
| @wiking | if u want something else | 21:52 |
| @wiking | gimme | 21:52 |
| @wiking | and tell me if i should append or overwrite | 21:52 |
| @iglesiasg | wiking, let me see one second | 21:53 |
| @wiking | iglesiasg: u can actually give out public key in public.... that's the idea :P | 21:57 |
| @wiking | but ok adding | 21:57 |
| @wiking | iglesiasg: tyr now | 21:57 |
| @wiking | should be working | 21:57 |
| @iglesiasg | wiking, nope actually :S | 22:00 |
| @wiking | mmm interesting | 22:01 |
| @wiking | oh maaan | 22:02 |
| @wiking | your problem is the username. u r trying with iglesiasg but your username on fatbot is iglesias | 22:02 |
| @wiking | without the last g on the end | 22:02 |
| @wiking | iglesias | 22:03 |
| @iglesiasg | wiking, I tried iglesias too ;) | 22:03 |
| @iglesiasg | ssh iglesias@fatbot.shogun-toolbox.7nn.de | 22:03 |
| @iglesiasg | Permission denied (publickey). | 22:03 |
| @wiking | plz try now again | 22:03 |
| @wiking | and and u r trying the wrong host as well | 22:03 |
| @wiking | :DDD | 22:03 |
| @wiking | fatbot.shogun-toolbox.org | 22:03 |
| @wiking | so | 22:04 |
| @wiking | ssh iglesias@fatbot.shogun-toolbox.org | 22:04 |
| @wiking | u have 2 more seconds to try as after that i'm gone | 22:06 |
| @wiking | iglesiasg: if u want access to 7nn.de | 22:06 |
| @wiking | i can add there your public key as well | 22:06 |
| @wiking | no actually there i cannot | 22:06 |
| @wiking | as there i dont have sudo | 22:06 |
| -!- thoralf [~thoralf@91-65-136-3-dynip.superkabel.de] has quit [Quit: Konversation terminated!] | 22:07 | |
| @iglesiasg | wiking, .org worked, thanks! | 22:07 |
| -!- votjakovr [~votjakovr@188.134.46.30] has quit [Quit: WeeChat 0.4.0] | 22:07 | |
| @sonney2k | ohh man | 22:07 |
| @sonney2k | iglesiasg, now finally figure it out :) | 22:08 |
| -!- infomix [~infomix@91.210.100.173] has quit [K-Lined] | 22:18 | |
| -!- negativ [~negativ@91.210.101.170] has quit [K-Lined] | 22:18 | |
| -!- rymrym [~rymrym@193.104.110.37] has joined #shogun | 22:21 | |
| @sonney2k | argh | 22:26 |
| -!- pickle27 [d8bfe7da@gateway/web/freenode/ip.216.191.231.218] has quit [Ping timeout: 272 seconds] | 22:35 | |
| -!- shogun-notifier- [~irker@7nn.de] has joined #shogun | 22:37 | |
| shogun-notifier- | shogun-data: Soeren Sonnenburg :master * 6b289bd / testsuite/tests/ (2 files): https://github.com/shogun-toolbox/shogun-data/commit/6b289bd5e22d88ec6ad5918063906267a7b65d81 | 22:37 |
| shogun-notifier- | shogun-data: update data for fixed clustering evalution | 22:37 |
| shogun-notifier- | shogun: Soeren Sonnenburg :develop * 2483b5a / / (3 files): https://github.com/shogun-toolbox/shogun/commit/2483b5ab18f9195b104e682cfbbb4d49baacd39d | 22:37 |
| shogun-notifier- | shogun: fix ClusteringMutualInformation - init matrix as zero fixing valgrind errors and tests | 22:37 |
| @sonney2k | finally - iglesiasg so now only the mmd stuff remains! | 22:38 |
| -!- pickle27 [d8bfe7da@gateway/web/freenode/ip.216.191.231.218] has joined #shogun | 22:38 | |
| @iglesiasg | sonney2k, I am doing my best! | 22:40 |
| @iglesiasg | sonney2k, the process is much faster now compiling in fatbot | 22:42 |
| @iglesiasg | sonney2k, I hope it will work out this time to detect the revision that broke the test | 22:42 |
| besser82 | sonney2k: How do we want to get that py_mod SGObj thing started? | 22:43 |
| @sonney2k | besser82, no idea | 22:43 |
| @sonney2k | besser82, maybe gsoc task | 22:44 |
| besser82 | sonney2k: and Who to mentor? | 22:44 |
| @sonney2k | too big I am afraid | 22:44 |
| besser82 | sonney2k: what about that cmake-stuff && gsoc? | 22:44 |
| besser82 | sonney2k: shall I put that to iglesia's gdrive? | 22:44 |
| @sonney2k | besser82, sure! | 22:48 |
| besser82 | sonney2k: kk | 22:48 |
| @sonney2k | besser82, I shouldn't be mentoring more than 2 things so not sure if examples is a good thing for me - though I have a plan... the only problem to decide is whether we want some intermediate language to write the examples in or not | 22:49 |
| besser82 | sonney2k: not really, keeping the examples lang by lang would be good | 22:53 |
| besser82 | sonney2k: just sharing the reference data | 22:53 |
| @sonney2k | besser82, no no | 22:56 |
| @sonney2k | it is too much effort to write the examples lang by lang | 22:56 |
| @sonney2k | and they should be very simple just for reference | 22:56 |
| @sonney2k | more complex stuff should be a notebook | 22:56 |
| -!- rymrym [~rymrym@193.104.110.37] has quit [K-Lined] | 22:57 | |
| besser82 | sonney2k: ok, ok | 23:00 |
| besser82 | sonney2k: but what "intermediate" language to chose? | 23:01 |
| @sonney2k | no idea | 23:01 |
| lisitsyn1 | javascript! | 23:01 |
| @sonney2k | maybe wiking has an idea? | 23:02 |
| lisitsyn1 | w00t you are toalking abouuut? | 23:03 |
| @sonney2k | lisitsyn, writing shogun examples in some intermediate language / or jinja templates so that it can easily be converted to all $LANG we support | 23:04 |
| lisitsyn1 | sonney2k: think that's possible? | 23:05 |
| lisitsyn1 | sounds ok but I have no idea | 23:05 |
| lisitsyn1 | sonney2k: then we can follow common naming schemes! | 23:05 |
| lisitsyn1 | e.g. it is just wrong to have get_stuff in java | 23:06 |
| lisitsyn1 | nobody uses that convention in java | 23:06 |
| lisitsyn1 | code looks un-idiomatic | 23:06 |
| lisitsyn1 | wrrrrong | 23:06 |
| @sonney2k | lisitsyn, sure - if you look at the examples they all look the same except for some very minor deviations | 23:06 |
| @sonney2k | lisitsyn, yeah swig supports that even... | 23:06 |
| lisitsyn1 | hmm then we should have it | 23:07 |
| lisitsyn1 | it must be really confusing to use _ in java | 23:07 |
| @sonney2k | lisitsyn, yeah ... doing examples just once would have some huge advantage too | 23:09 |
| pickle27 | I could be okay to mentor that project if besser82 has enough on his plate | 23:09 |
| @sonney2k | lisitsyn, we could return only shogun objects | 23:09 |
| @sonney2k | lisitsyn, and then these can be serialized with SerializableAsciiFile | 23:10 |
| @sonney2k | and read back in and compared with equals | 23:10 |
| @sonney2k | so we have cross language tests | 23:10 |
| @sonney2k | and all examples do the same / it is ensured that they all give the same correct output | 23:10 |
| @sonney2k | pickle27, didn't you have sth else already? | 23:11 |
| shogun-buildbot | build #37 of deb4 - python3 is complete: Failure [failed test python modular] Build details are at http://buildbot.shogun-toolbox.org/builders/deb4%20-%20python3/builds/37 blamelist: Soeren Sonnenburg <sonne@debian.org> | 23:11 |
| pickle27 | I have the opencv integration | 23:11 |
| pickle27 | I only want to mentor one project but if no one picks that one then i'd be happy to help on examples | 23:11 |
| @sonney2k | pickle27, OK | 23:13 |
| @sonney2k | pickle27, any idea what intermediate language one could use? | 23:13 |
| @sonney2k | I would want to keep it as simple as possible | 23:13 |
| @sonney2k | no idea if jinja2 templates would do... | 23:14 |
| pickle27 | honestly most of the examples are already super basic because of the load helper methods | 23:15 |
| @sonney2k | pickle27, yes they are intended to be only reference examples | 23:15 |
| pickle27 | I would probably do a second take and move all of the data generation into a contained class | 23:15 |
| @sonney2k | pickle27, we would want to make things even more simple / reference like | 23:15 |
| @sonney2k | pickle27, and put all the higher level stuff into notebooks etc | 23:15 |
| @sonney2k | pickle27, I think jinja2 is sufficient - I mean the shogun stuff has the very same syntax across languages | 23:17 |
| @sonney2k | and data is supposed to be loaded from disk likely even with shogun. | 23:17 |
| pickle27 | yeah I think that would work | 23:17 |
| @sonney2k | pickle27, me might want to have a few examples still using this loader crap | 23:17 |
| @sonney2k | but that again has the same syntax | 23:17 |
| @sonney2k | pickle27, problem is if we pose this for a student we should try *one* minimal example - like e.g. compute the Gaussian Kernel or so | 23:18 |
| pickle27 | I would make all the examples use the loader | 23:18 |
| pickle27 | then it would be super simple to generate them | 23:19 |
| @sonney2k | pickle27, why not - I think we could have some cmake rule that converts *.py.jinja2 into *.py | 23:20 |
| @sonney2k | and if other *.py's exist it would just not touch things | 23:20 |
| @sonney2k | so we *could* have additional examples if stuff doesn't work out | 23:20 |
| pickle27 | yeah totally | 23:22 |
| pickle27 | btw just to be clear I'm talking about making sure all these are consistent and then use them to gen examples: https://github.com/shogun-toolbox/shogun/blob/develop/examples/undocumented/python_modular/tools/load.py | 23:22 |
| @sonney2k | pickle27, yes | 23:23 |
| @sonney2k | pickle27, we are still talking about hundreds of examples | 23:23 |
| pickle27 | okay just checking we're on the same page | 23:23 |
| @sonney2k | so it will be quite some pile of work | 23:24 |
| pickle27 | yeah i know | 23:24 |
| pickle27 | there is probably lots that we don't really need slash are basically repeated from unit tests | 23:24 |
| @sonney2k | also they need to be changed to just return shogun objects - e.g. the kernel's can only be reasonably tested when they return the kernel matrix | 23:25 |
| @sonney2k | but that would be an SGMatrix - so we need to put them in some CustomKernel or have some other datatype for that that has a .equals | 23:26 |
| pickle27 | actually just thinking now, it might be cool to try and move a lot of those really simple tests to be automatically copied from unit tests | 23:27 |
| pickle27 | cause we're packing a lot of duplication right now aren't we? | 23:27 |
| pickle27 | I know that all of the examples I wrote for gsoc were basically the same as my unit tests | 23:28 |
| pickle27 | we could have a flag or something that says this unit test is also a basic example and then it gets copied in | 23:28 |
| pickle27 | anyways I got to run but I might look into that after Im done on the website see if I can put something together | 23:29 |
| -!- pickle27 [d8bfe7da@gateway/web/freenode/ip.216.191.231.218] has quit [Quit: Page closed] | 23:30 | |
| @sonney2k | pickle27 it is actually even more simple - a unit test would compute e.g. the gaussian kernel and check if values are as expected - in the example you just have the GaussianKernel and done | 23:31 |
| shogun-buildbot | build #2142 of deb3 - modular_interfaces is complete: Failure [failed test python modular] Build details are at http://buildbot.shogun-toolbox.org/builders/deb3%20-%20modular_interfaces/builds/2142 blamelist: Soeren Sonnenburg <sonne@debian.org> | 23:47 |
| --- Log closed Tue Jan 21 00:00:01 2014 | ||
Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!