| --- Log opened Wed Nov 13 00:00:22 2013 | ||
| -!- shogun-notifier- [~irker@7nn.de] has quit [Quit: transmission timeout] | 00:31 | |
| -!- iglesiasg [~iglesiasg@s83-179-44-135.cust.tele2.se] has quit [Quit: Leaving] | 00:53 | |
| -!- hushell [~hushell@8-12.ptpg.oregonstate.edu] has quit [Ping timeout: 264 seconds] | 03:19 | |
| shogun-buildbot_ | build #616 of nightly_default is complete: Failure [failed notebooks] Build details are at http://buildbot.shogun-toolbox.org/builders/nightly_default/builds/616 | 04:15 |
|---|---|---|
| -!- hushell [~hushell@c-24-20-167-124.hsd1.or.comcast.net] has joined #shogun | 04:39 | |
| -!- hushell [~hushell@c-24-20-167-124.hsd1.or.comcast.net] has quit [Ping timeout: 240 seconds] | 07:36 | |
| -!- new_lido [~walid@41.218.173.7] has joined #shogun | 07:56 | |
| -!- new_lido [~walid@41.218.173.7] has quit [Ping timeout: 244 seconds] | 08:00 | |
| -!- hushell [~hushell@c-24-20-167-124.hsd1.or.comcast.net] has joined #shogun | 08:01 | |
| -!- new_lido [~walid@41.218.179.85] has joined #shogun | 08:11 | |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has joined #shogun | 08:19 | |
| -!- benibadman [~benibadma@94.135.236.129] has joined #shogun | 08:27 | |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has quit [Quit: sonne|osx] | 08:42 | |
| -!- benibadm_ [~benibadma@94.135.236.129] has joined #shogun | 08:49 | |
| -!- benibadman [~benibadma@94.135.236.129] has quit [Ping timeout: 246 seconds] | 08:52 | |
| -!- iglesiasg [~iglesiasg@s83-179-44-135.cust.tele2.se] has joined #shogun | 09:27 | |
| -!- mode/#shogun [+o iglesiasg] by ChanServ | 09:27 | |
| -!- iglesiasg [~iglesiasg@s83-179-44-135.cust.tele2.se] has quit [Quit: Leaving] | 09:55 | |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has joined #shogun | 10:03 | |
| -!- lisitsyn1 [~lisitsyn@80.252.20.67] has quit [Ping timeout: 245 seconds] | 10:38 | |
| sonne|osx | guys have you seen? the next europython is in berlin :) | 10:40 |
| sonne|osx | maybe we should just have our shogun workshop before / after that date :) | 10:40 |
| sonne|osx | http://www.ep14.org/ | 10:41 |
| lisitsyn | ha | 10:43 |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has quit [Quit: sonne|osx] | 10:44 | |
| -!- hushell [~hushell@c-24-20-167-124.hsd1.or.comcast.net] has quit [Ping timeout: 246 seconds] | 10:59 | |
| -!- iglesiasg [~iglesias@2001:6b0:1:1da0:9895:4f04:e0e5:ccbc] has joined #shogun | 12:13 | |
| -!- mode/#shogun [+o iglesiasg] by ChanServ | 12:13 | |
| @iglesiasg | BTW, MKL experts | 12:26 |
| @iglesiasg | did you see this mail on the mailing lists regarding MKL regression? | 12:26 |
| @wiking | rer | 13:06 |
| @wiking | sonne|work: ping? | 13:10 |
| @wiking | btw why do we need 2 pass reading for processing a CSV file? | 13:16 |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has joined #shogun | 13:29 | |
| @wiking | sonne|work: ping | 13:31 |
| sonne|osx | wiking: pong | 13:32 |
| @wiking | sonne|osx: 1) why do we need 2 pass reading for processing a CSV file? 2) so gunnar doesn't need any help from our side, apart from signatures? | 13:33 |
| sonne|osx | wiking: well it requires much less memory | 13:34 |
| sonne|osx | 50% actually | 13:35 |
| sonne|osx | wiking: do you know of any python based github markdown format renderer? | 13:39 |
| @wiking | sonne|osx: so the thing is that libarchive has a problem with seeking :P | 13:42 |
| @wiking | sonne|osx: mmm none pops up in my head | 13:42 |
| @wiking | i guess u googled around already | 13:42 |
| sonne|osx | wiking: but couldn't you then open /close and again open/close ? | 13:43 |
| @wiking | sonne|osx: well i hoped that i dont have to | 13:43 |
| @wiking | sonne|osx: but seems that's the only option | 13:44 |
| sonne|osx | wiking: rationale really is - if you load a small .csv it doesn't matter that you read it twice | 13:44 |
| @wiking | sonne|osx: yeah but that's the thing | 13:44 |
| sonne|osx | wiking: if you read a big one - you wouldn't be able to load it in memory when you read it just once | 13:44 |
| @wiking | i guess if u want to load a gz-ed or bzip2 csv | 13:44 |
| @wiking | then u rather have a big feature matrix :P | 13:45 |
| sonne|osx | of course you can use hdf5/protobuf based files that have known number of vectors etc | 13:45 |
| @wiking | sonne|osx: reallocing memory ? | 13:45 |
| sonne|osx | wiking: well yeah but not 100% reliable | 13:46 |
| @wiking | sonne|osx: i mean libarchive is actually designed in mind for big archives | 13:46 |
| @wiking | i.e. tar.bz etc | 13:46 |
| @wiking | which of course is going to be super tricky if we want to support that | 13:46 |
| @wiking | e.g. we dont want to :P | 13:46 |
| @wiking | so just one file with various compression would be ok to support i guess | 13:47 |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has quit [Quit: sonne|osx] | 14:02 | |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has joined #shogun | 14:03 | |
| sonne|osx | wiking: yeah... | 14:03 |
| @wiking | sonne|osx: i want to change from FILE* to some other type of stream | 14:06 |
| sonne|osx | wiking: to what? | 14:06 |
| sonne|osx | I mean from FILE you can get the fd | 14:06 |
| sonne|osx | but hmmh | 14:07 |
| @wiking | the problem is that we are currently passing around FILE* | 14:07 |
| @wiking | and obviously if we want to use libarchive | 14:07 |
| @wiking | we have to use for that something else | 14:08 |
| @wiking | as it doesn't provide FILE* stream | 14:08 |
| sonne|osx | wiking: yeah sure but what do you need to use? | 14:08 |
| @wiking | well some kind of an abstract stream | 14:09 |
| @wiking | where i can both wrap a single file fd (or FILE*) and a more complex libarchive stream | 14:09 |
| sonne|osx | wiking: sure do it then | 14:10 |
| @wiking | as for example now i'm not able to do a close/open operation on a libarchive handle... as i only have a handle | 14:10 |
| @wiking | that doesn't say anything about the filename or fd it was used to open that archive | 14:10 |
| @wiking | hence i do not even know what i need to reopen | 14:10 |
| @wiking | sonne|osx: ok... i mean there's one very straightforward way to do this | 14:10 |
| @wiking | if we can use std:: PP | 14:11 |
| @wiking | :P | 14:11 |
| sonne|osx | wiking: could you use fileno(FILE*) | 14:18 |
| sonne|osx | wiking: and then fdreopen? | 14:18 |
| sonne|osx | ha fdreopen doesn't exist | 14:19 |
| @wiking | sonne|osx: i do not have a way to get a FILE* or fd from libarchive that actually streaming the uncompressed content | 14:20 |
| sonne|osx | I see | 14:21 |
| @wiking | there's only archive_read_data_into_fd | 14:23 |
| @wiking | but that is putting all the content of an archive into the given fd | 14:24 |
| -!- taylan [d5f4a885@gateway/web/freenode/ip.213.244.168.133] has joined #shogun | 14:24 | |
| @wiking | which is basically a shit in sense of that it writes everything into a file | 14:24 |
| taylan | Hi Everyone.. | 14:25 |
| @iglesiasg | sup | 14:25 |
| taylan | I have a question about the C2 parameter.. | 14:25 |
| @wiking | of course this way it's much easier as that FD can be used to generate the FILE* | 14:25 |
| @wiking | and after that we just do the usual business | 14:25 |
| taylan | apparently this parameter controls the weights of the classes, but I don't understand why there is an indirection for it, i.e dividing by C1 before setting teh weights.. | 14:27 |
| taylan | can someone point me any clues? | 14:27 |
| @iglesiasg | taylan, what class? | 14:27 |
| taylan | so ok, I am looking at the libsvm class now, but I think this is the same for other implementations as well | 14:28 |
| taylan | sth like this: float64_t weights[2]={1.0,get_C2()/get_C1()}; | 14:28 |
| taylan | so C2 indirectly sets the weight of classes.. | 14:28 |
| @iglesiasg | taylan, are C1 and C2 weights por positive and negative examples? | 14:29 |
| taylan | if I am working with an imbalanced dataset, I would like to set this seperately..But I feel like I am missing sth as it is the same for all implementations | 14:29 |
| @iglesiasg | taylan, well, I see that set_C accepts two arguments | 14:29 |
| taylan | I thought C is the cost parameter for misclassification.. | 14:31 |
| taylan | and weights represent the imbalance of the dataset | 14:32 |
| taylan | so ideally I would keep weights constant to the ratio of positive and negative samples, and do a grid search on C | 14:32 |
| taylan | but I cannot do a grid search on C without actually changing the weights | 14:33 |
| @iglesiasg | C is the typical regularization | 14:33 |
| @iglesiasg | and in this case you get the option to set different regularization for each of the classes | 14:33 |
| @iglesiasg | because that is good when you have skewed class distributions | 14:34 |
| @iglesiasg | I don't quite see where the problem comes | 14:34 |
| taylan | I think there is no problem, it's probably I misunderstand the concept.. | 14:35 |
| taylan | so setting the weight in svm is actually shifting the regularization parameter internally, is that correct? | 14:36 |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has quit [Quit: sonne|osx] | 14:38 | |
| @iglesiasg | taylan, mmm I am not sure if you are calling weight to the regularization | 14:39 |
| @iglesiasg | I believe the alphas are sometimes called weights | 14:39 |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has joined #shogun | 14:40 | |
| taylan | ok, I see in libsvm code that the weights are actually the regularization for each class.. | 14:41 |
| taylan | This means that I don't need to do any grid search for the c2 parameter, just on C1 will be enough | 14:42 |
| taylan | I have some other questions, I'd appreciate if you can help | 14:42 |
| @iglesiasg | taylan, it can be that libsvm allows you to set regularization (your "weights") for each example | 14:43 |
| @iglesiasg | it is just more general | 14:43 |
| @iglesiasg | but you probably don't want to do model selection allowing every weight to be different | 14:44 |
| taylan | ok , I think it's clear now.. | 14:45 |
| taylan | from libsvm docs: -wi weight : set the parameter C of class i to weight*C, for C-SVC (default 1) | 14:45 |
| lisitsyn | oh well | 14:45 |
| lisitsyn | it happens all the time and we still didn't manage to fix it | 14:45 |
| lisitsyn | taylan: it is known issue but we don't have good solution to search just for C but not C1 C2 | 14:46 |
| lisitsyn | iglesiasg: hey there | 14:46 |
| @iglesiasg | hello hello | 14:47 |
| lisitsyn | taylan: it is just about too much code that uses this C1 C2 :) I am sorry you have to do double work here | 14:47 |
| taylan | so my other question is mainly about training huge datasets. I run out of memory with > 20K samples on 7 features..any tips on this? | 14:47 |
| lisitsyn | with your machine I mean | 14:47 |
| lisitsyn | taylan: just 20K samples and it fails? | 14:47 |
| taylan | lisitsyn, work is no problem..I just want to understand.. | 14:47 |
| lisitsyn | I find it strange | 14:47 |
| taylan | yep, on that range | 14:47 |
| taylan | let me give it a try to be sure.. | 14:47 |
| lisitsyn | taylan: how much memory do you have? | 14:47 |
| taylan | 8GB | 14:48 |
| taylan | I mean I can up the memory of course, but I just have the impression that I am doing sth wrong.. | 14:48 |
| lisitsyn | taylan: no no should be totally enough | 14:48 |
| taylan | SystemError: Out of memory error, tried to allocate 1599840004 bytes using malloc. | 14:49 |
| taylan | This is with 20K samples | 14:49 |
| taylan | It actually fails in the first iteration | 14:49 |
| lisitsyn | 1.6 gb | 14:50 |
| lisitsyn | strange | 14:50 |
| sonne|osx | yeah should work | 14:50 |
| taylan | before running it I have 2.7GB free.. | 14:50 |
| lisitsyn | taylan: can you share the code doing that? | 14:50 |
| taylan | sorry, 2.7GB full | 14:50 |
| taylan | ah, ok, I think it tries to allocate this 4 times, since I set the parallel to 4.. | 14:52 |
| taylan | sth like this: | 14:53 |
| taylan | self.gridSearch = GridSearchModelSelection(self.crossValidation, self.paramTree) p = Parallel() p.set_num_threads(4) self.gridSearch.set_global_parallel(p) best_combination = self.gridSearch.select_model(True) | 14:53 |
| taylan | sorry, couldn't paste well enough -)) | 14:53 |
| taylan | ok, let's say that my dataset is actually a lot bigger..is there any way to use memory mapped files for feature storage? | 14:54 |
| dsockwell | i had a weird segfault with gridsearch a few months ago | 14:54 |
| lisitsyn | well your dataset is pretty small right now | 14:54 |
| dsockwell | haven't tried the new release or bothered debugging too much because it was a toy project | 14:54 |
| sonne|osx | we had an issue with a memory leak w/ modular interfaces | 14:54 |
| lisitsyn | taylan: are you doing binary classification? | 14:54 |
| taylan | I see, so this might point to a memory leak somewhere.. | 14:54 |
| sonne|osx | but no idea why it would alloc 1.5GB | 14:55 |
| dsockwell | but you may benefit from running valgrind on your project | 14:55 |
| taylan | yes, binary classification.. | 14:55 |
| lisitsyn | taylan: I see absolutely no reason to allocate that much memory | 14:55 |
| taylan | I will give it a try, everything points me to using C++ now, so I wll port the code to C++. I have been using python, it might have an effect as well | 14:55 |
| dsockwell | i had to recompile libshogun targeting i486 to get valgrind to run | 14:55 |
| lisitsyn | taylan: no there is no matter in python | 14:55 |
| lisitsyn | it is about internal things | 14:56 |
| lisitsyn | dsockwell: why? | 14:56 |
| taylan | ok, I will debug this, and come back if I find sth. Just wanted to make sure I'm not doing anything stupid.. | 14:56 |
| lisitsyn | did you get illegal instruction? | 14:56 |
| dsockwell | i had compiled it targeting an i7 and valgrind choked on some of the instructions | 14:56 |
| dsockwell | they're just behind | 14:56 |
| lisitsyn | dsockwell: ah you should have updated valgrind ;) | 14:56 |
| dsockwell | i did | 14:56 |
| -!- krispin [0e8bb973@gateway/web/freenode/ip.14.139.185.115] has joined #shogun | 14:56 | |
| lisitsyn | valgrind had an issue with i5 and i7 | 14:56 |
| lisitsyn | newer one doesn't have it | 14:57 |
| dsockwell | since when is it gone? | 14:57 |
| dsockwell | i might have been doing that before the fix came | 14:57 |
| lisitsyn | dsockwell: ah okay | 14:57 |
| taylan | ok guys, thanks a lot for your feedback..I have one last question though. | 14:57 |
| lisitsyn | dsockwell: then just fyi now it is ok | 14:57 |
| taylan | is there any extension in shogun that can do mpi like parallelization? | 14:57 |
| sonne|osx | with valgrind you should always just use debug mode and no optimization flags | 14:57 |
| taylan | so if I want to run my training on multiple machines, any clues? | 14:57 |
| lisitsyn | taylan: so it would be helpful if you track down where allocation happens | 14:58 |
| dsockwell | anyway taylan valgrind should see through python into libshogun which is C++ | 14:58 |
| lisitsyn | taylan: Heiko has some environment for that iirc | 14:58 |
| taylan | lisitsyn: i will investigate it.. | 14:58 |
| lisitsyn | taylan: other way round you can send me some snippet | 14:58 |
| lisitsyn | I can take a look | 14:58 |
| lisitsyn | but later tonight | 14:58 |
| @iglesiasg | taylan, you may want to check out GraphLab for MPI stuff | 14:58 |
| taylan | lisitsyn: It's ok, I believe I can learn a couple of things debugging..some sort of masochism ;) | 14:59 |
| taylan | iglesiasg: thanks, i will have a look at it now.. | 14:59 |
| dsockwell | yes if it's your first time using valgrind, put a cloth down to catch the blood | 15:00 |
| @iglesiasg | haha | 15:02 |
| lisitsyn | masochist and sadist were put into a jail, masochist cries - hurt me and sadist laughs - I won't I won't | 15:03 |
| lisitsyn | taylan: well if there are leaks - they shouldn't be here, if there are no leaks but it still wants all of your memory I'd suspect a bug here | 15:04 |
| -!- krispin [0e8bb973@gateway/web/freenode/ip.14.139.185.115] has quit [Ping timeout: 250 seconds] | 15:05 | |
| -!- lisitsyn [~lisitsin@mxs.kg.ru] has quit [Quit: Leaving.] | 15:10 | |
| @wiking | asdf | 15:14 |
| sonne|osx | or a user error | 15:15 |
| dsockwell | my personal guess is bad reference counting | 15:21 |
| -!- FSCV [~FSCV@204.45.30.198] has joined #shogun | 15:24 | |
| @wiking | lol | 15:25 |
| @wiking | https://www.destroyallsoftware.com/talks/wat | 15:25 |
| -!- shogun-notifier- [~irker@7nn.de] has joined #shogun | 15:29 | |
| shogun-notifier- | shogun-web: Soeren Sonnenburg :master * 1056a50 / / (8 files): https://github.com/shogun-toolbox/shogun-web/commit/1056a50fa005e604e8ee03959b90140936ed0ce1 | 15:29 |
| shogun-notifier- | shogun-web: create feature matrix under /page/features | 15:29 |
| @sonney2k | iglesiasg, wiking, http://shogun-toolbox.org/page/features | 15:32 |
| @sonney2k | (shift reload the page if it looks weird) | 15:33 |
| @wiking | mmm | 15:33 |
| @sonney2k | wiking, *lol* | 15:35 |
| @sonney2k | wiking, the link you sent it is really hilarious :)) | 15:37 |
| @wiking | yeps | 15:38 |
| @wiking | watMAN | 15:38 |
| @wiking | sonney2k: how do we force regen of the matrix? | 15:39 |
| @wiking | i see i guess feature_matrix.csv update would solve that | 15:40 |
| @wiking | the feature groups should be *strong* | 15:41 |
| @iglesiasg | feature matrix back again then! | 15:41 |
| @wiking | i.e. *General Features* *Supported Operating Systems* etc | 15:41 |
| @sonney2k | wiking, well easy just some css | 15:42 |
| @wiking | sonney2k: go ahead ;) | 15:42 |
| @sonney2k | wiking, the rotated text was not so easy.. | 15:45 |
| @wiking | sonney2k: r u feeling comfortable | 15:45 |
| @wiking | with std::iostream? | 15:45 |
| @wiking | or not so much | 15:45 |
| @wiking | instead of FILE* | 15:46 |
| @sonney2k | http://www.youtube.com/watch?v=9UP1LVNhTis | 15:46 |
| @sonney2k | german though | 15:46 |
| @sonney2k | german radio - moderator freak out | 15:47 |
| @sonney2k | iglesiasg, well it is not really back yet - still hidden. we should get rid of the ? first | 15:53 |
| @sonney2k | iglesiasg, and then proper integration... | 15:54 |
| @sonney2k | wiking, you just run util/matrix.py - that will fetch the latest .csv | 15:54 |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has quit [Quit: sonne|osx] | 15:56 | |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has joined #shogun | 15:57 | |
| @wiking | sonney2k: got it | 15:58 |
| @wiking | sonney2k: iostream? | 15:58 |
| sonne|osx | wiking: we are talking the c library libarchive? | 16:06 |
| sonne|osx | iglesiasg: any news on the markdown rendered stuff? | 16:07 |
| @iglesiasg | sonne|osx, what I told you yesterday, I am stuck with one thing | 16:07 |
| sonne|osx | iglesiasg: didn't see it with what? | 16:07 |
| @iglesiasg | so I pretty much put the javascript code in a new template | 16:08 |
| @iglesiasg | https://github.com/iglesias/shogun-web/blob/readme/templates/readme.html | 16:08 |
| @iglesiasg | but that is not showing anything | 16:08 |
| @iglesiasg | actually, even if remove the line that includes the markdown.js, it is still the same thing going on | 16:08 |
| @iglesiasg | so I was trying to debug that using the chrome dev tools | 16:09 |
| @iglesiasg | but don't really get what is going on | 16:09 |
| @iglesiasg | sonne|osx, do you see anything missing? | 16:09 |
| @wiking | sonne|osx: yeps | 16:10 |
| @wiking | sonne|osx: btw have u tried this: https://pypi.python.org/pypi/Markdown | 16:10 |
| sonne|osx | iglesiasg: I guess CORS issues - what does the console say? | 16:13 |
| @iglesiasg | no idea what CORS is | 16:13 |
| @iglesiasg | wikipedia... | 16:13 |
| sonne|osx | wiking: not sure if this is sufficient but I would actually prefer a server side solution | 16:14 |
| sonne|osx | iglesiasg: http://en.wikipedia.org/wiki/Cross-origin_resource_sharing | 16:14 |
| @iglesiasg | yep, I was there | 16:14 |
| sonne|osx | iglesiasg: any errors on the console? | 16:14 |
| @iglesiasg | sonne|osx, Uncaught SyntaxError: Unexpected token ILLEGAL | 16:15 |
| sonne|osx | hmmhh | 16:17 |
| sonne|osx | iglesiasg: or you render server side - maybe that is sufficient for us | 16:17 |
| @wiking | sonne|osx: https://github.com/shogun-toolbox/shogun/issues/1619#issuecomment-28402597 | 16:18 |
| sonne|osx | iglesiasg: so no .js but just the markdown python -pkg | 16:18 |
| @wiking | sonne|osx: there | 16:18 |
| @iglesiasg | got to run now | 16:18 |
| @iglesiasg | will ask you later again | 16:18 |
| @wiking | sonne|osx: full explanation about what we need | 16:18 |
| sonne|osx | wiking: sounds good - so you could wrap simple files with that (filename based and so could open / reopen) | 16:20 |
| sonne|osx | wiking: so we have some abstract class that defines these operations | 16:20 |
| @wiking | sonne|osx: purely derived from SGObject? | 16:21 |
| sonne|osx | maybe we should rename CFile to something else and then use CFile for that | 16:21 |
| sonne|osx | wiking: yes | 16:21 |
| sonne|osx | maybe CFile -> CDataFile | 16:21 |
| sonne|osx | then define a new CFile with open/read/write/rewind | 16:22 |
| @wiking | sonne|osx: or just have CFileHandle | 16:22 |
| @wiking | or CFileStream | 16:22 |
| sonne|osx | that could be a normal FILE underneath or sth | 16:22 |
| sonne|osx | wiking: yes or that | 16:22 |
| sonne|osx | sounds good to me | 16:22 |
| @wiking | CFileStream like an abstract class | 16:22 |
| @wiking | and then inherit from there the simple posix based file reading | 16:23 |
| sonne|osx | so one could make it work with std::iostream / filename / http urls or anything filish | 16:23 |
| @wiking | and the libarchived based one | 16:23 |
| @wiking | yeps | 16:23 |
| @wiking | what we can do is that simply have like CLibarchiveStream as default | 16:24 |
| @wiking | if libarchive is available | 16:24 |
| @wiking | as it supports raw file reading | 16:24 |
| @wiking | i.e. null-compression files | 16:24 |
| @wiking | as well as compressed files | 16:24 |
| @wiking | and if libarchive is not available we use the posix backend by default | 16:24 |
| sonne|osx | you mean you would add some convenience constructor or what? | 16:25 |
| sonne|osx | if you just work with the abstract class it doesn't really matter... | 16:25 |
| @wiking | sonne|osx: no... i mean that we leave CFile as is (almost.. remove some of the ctors) | 16:25 |
| @wiking | and then if u call | 16:25 |
| sonne|osx | an then? | 16:25 |
| @wiking | CFile(const char* fname, char rw='r', const char* name=NULL); | 16:25 |
| @wiking | then that would use a default backend (depending what is available) | 16:26 |
| @wiking | see above | 16:26 |
| @wiking | of course there would be another ctor | 16:26 |
| @wiking | where u can set your own backend | 16:26 |
| @wiking | or prefered one | 16:26 |
| @wiking | and as well get reference on the CFileStream | 16:26 |
| sonne|osx | wiking: ahh ok | 16:27 |
| @wiking | but before doing this | 16:27 |
| sonne|osx | so CFile gets some set_stream() ? | 16:27 |
| @wiking | yes | 16:27 |
| sonne|osx | sounds good! | 16:27 |
| @wiking | but now i'm just looking into io/stream | 16:27 |
| @wiking | i dont know | 16:27 |
| @wiking | either we try to somehow merge these abstract classes somehow | 16:28 |
| @wiking | or the completely different way to do libarchive support | 16:28 |
| @wiking | is to add only to StreamingFile libarchive | 16:28 |
| @wiking | i.e. StreamingAsciiFile.h | 16:28 |
| @wiking | as actually libarchive as it's arch is more suitable for StreamingFile | 16:30 |
| @wiking | (see the problem with seeking | 16:30 |
| @wiking | ) | 16:30 |
| sonne|osx | wiking: na what you proposed sounds pretty good so I would do that rather | 16:32 |
| @wiking | mmm | 16:33 |
| @wiking | i'm just looking at this | 16:33 |
| @wiking | and actually having this in streaming | 16:34 |
| @wiking | is basically only changing | 16:34 |
| @wiking | IOBuffer | 16:34 |
| @wiking | and that's aal | 16:34 |
| @wiking | that's all | 16:34 |
| @wiking | instead of this whole hacking | 16:36 |
| @wiking | ok i'll have a look in this | 16:37 |
| @wiking | as i really hate actually all these io classes hanging around | 16:37 |
| @wiking | The vote ends Wednesday 13 November 2013 at 17h 27min 00s (local Paris). | 16:39 |
| @wiking | aaah | 16:39 |
| @wiking | in 1 hour we have the release of results for the shogun e.V | 16:39 |
| @wiking | Please wait for 47 minutes 35s. | 16:39 |
| sonne|osx | wiking: finally... | 16:44 |
| shogun-notifier- | shogun-web: Soeren Sonnenburg :master * 64cd4da / static/media/feature_matrix.csv,templates/matrix.html,util/matrix.py: https://github.com/shogun-toolbox/shogun-web/commit/64cd4dabb7cd54c45f946166f999e5f071e66ee9 | 17:30 |
| shogun-notifier- | shogun-web: add overview table for related toolboxes | 17:30 |
| shogun-notifier- | shogun-web: Soeren Sonnenburg :master * a3e207a / templates/matrix.html,util/matrix.py: https://github.com/shogun-toolbox/shogun-web/commit/a3e207a424779a29c3a823148af49bff212e5fb2 | 17:41 |
| shogun-notifier- | shogun-web: make feature bold | 17:41 |
| sonne|osx | wiking: iglesiasg votes are in ... http://adoodle.org/index.php?action=showresults&survey=82ca76510d645ecf85961fcaffd36181 | 17:42 |
| -!- benibadm_ [~benibadma@94.135.236.129] has quit [Ping timeout: 264 seconds] | 17:55 | |
| @wiking | \o/ | 17:56 |
| @wiking | sonne|osx: coffin fw within should = streaming? | 18:00 |
| -!- krispin [0e8bb972@gateway/web/freenode/ip.14.139.185.114] has joined #shogun | 18:14 | |
| -!- Saurabh7 [~Saurabh7@115.248.130.148] has joined #shogun | 18:22 | |
| -!- lisitsyn [~lisitsyn@80.252.20.67] has joined #shogun | 18:31 | |
| -!- lambday [67157f37@gateway/web/freenode/ip.103.21.127.55] has joined #shogun | 18:35 | |
| sonne|osx | wiking: coffin what? | 18:36 |
| -!- benibadman [~benibadma@port-92-206-233-160.dynamic.qsc.de] has joined #shogun | 18:38 | |
| -!- sonne|osx [~sonne@f053042012.adsl.alicedsl.de] has quit [Quit: sonne|osx] | 18:40 | |
| -!- lisitsyn [~lisitsyn@80.252.20.67] has quit [Quit: Leaving.] | 18:41 | |
| -!- krispin [0e8bb972@gateway/web/freenode/ip.14.139.185.114] has quit [Ping timeout: 250 seconds] | 18:43 | |
| @wiking | sonney2k: fw=framework | 18:44 |
| -!- lisitsyn [~lisitsyn@80.252.20.67] has joined #shogun | 18:46 | |
| -!- new_lido [~walid@41.218.179.85] has quit [Quit: Leaving] | 19:05 | |
| @wiking | ok woah | 19:09 |
| @wiking | i've just downloaded 22gigs in 10 seconds :D | 19:09 |
| lisitsyn | wiking: with quantum computer? | 19:11 |
| lisitsyn | or what? | 19:11 |
| lisitsyn | wiking: ahh you put it to /dev/null directly? ;) | 19:11 |
| @wiking | noup | 19:12 |
| @wiking | zfs | 19:12 |
| @wiking | and gigabit | 19:12 |
| lisitsyn | wiking: but hard drive throughput? | 19:13 |
| lisitsyn | how can that be? | 19:13 |
| -!- new_lido [~walid@41.218.179.85] has joined #shogun | 19:15 | |
| @wiking | zfs | 19:16 |
| @wiking | the zfs cache is 14 gigs | 19:16 |
| @wiking | lisitsyn: now i've upgraded myself from working with win7 to working on rdf+sparql | 19:30 |
| @wiking | fucking hell... i'm really suprised how semantic web got boosted in the last 5 years | 19:30 |
| -!- hushell [~hushell@8-12.ptpg.oregonstate.edu] has joined #shogun | 19:41 | |
| lisitsyn | wiking: I have no idea what these rdfs sparqls are :D | 19:42 |
| lisitsyn | potlw uwcs goyus | 19:42 |
| lisitsyn | ;) | 19:42 |
| @wiking | :D | 19:52 |
| @wiking | semantic web | 19:52 |
| @wiking | it's your friend | 19:52 |
| @wiking | see: freebase.org | 19:52 |
| @wiking | it's fucking crazy | 19:52 |
| @wiking | and amazing | 19:52 |
| @wiking | we should try to use shogun on that | 19:52 |
| @wiking | (i see it dying on it) | 19:52 |
| @wiking | the database itself is 250gigs decompressed :) | 19:52 |
| -!- gsomix [~gsomix@188.168.3.111] has joined #shogun | 19:58 | |
| gsomix | good evening | 19:58 |
| gsomix | sorry for my absence | 19:58 |
| @iglesiasg | sonney2k, 100% of votes, amazing :D | 19:59 |
| @iglesiasg | haha this is funny | 19:59 |
| @iglesiasg | we have taken measures to make it anonymous | 19:59 |
| @iglesiasg | and now it isn't at all, because everyone voted, and voted the same :D | 20:00 |
| -!- benibadman [~benibadma@port-92-206-233-160.dynamic.qsc.de] has quit [Remote host closed the connection] | 20:01 | |
| -!- benibadman [~benibadma@port-92-206-233-160.dynamic.qsc.de] has joined #shogun | 20:02 | |
| lisitsyn | iglesiasg: three logicians walk into the bar | 20:06 |
| lisitsyn | you know that joke yes? :) | 20:06 |
| -!- benibadman [~benibadma@port-92-206-233-160.dynamic.qsc.de] has quit [Ping timeout: 272 seconds] | 20:07 | |
| @iglesiasg | mmm I am not sure! | 20:07 |
| @iglesiasg | the one of half a beer? | 20:07 |
| lisitsyn | so the bartender asks | 20:07 |
| lisitsyn | do ALL of you want some bear? | 20:07 |
| lisitsyn | 1: I don't know | 20:07 |
| lisitsyn | 2: I don't know | 20:07 |
| lisitsyn | 3: YES! | 20:07 |
| @iglesiasg | hahaha | 20:07 |
| lisitsyn | it is the same catch as we all know who voted for who | 20:08 |
| @iglesiasg | all right, I will catch you later guys | 20:13 |
| -!- iglesiasg [~iglesias@2001:6b0:1:1da0:9895:4f04:e0e5:ccbc] has quit [Quit: Ex-Chat] | 20:13 | |
| @wiking | lisitsyn: :D | 20:18 |
| @wiking | let's import 250 gigs of rdf | 20:19 |
| @wiking | and then lets index it | 20:21 |
| @wiking | yey what amazing world we live in ;) | 20:21 |
| -!- benibadman [~benibadma@port-92-206-233-160.dynamic.qsc.de] has joined #shogun | 20:28 | |
| @wiking | "that the huge amount of memory assigned to the indexing tool (32GByte) will only | 20:31 |
| @wiking | be needed during the final optimization of the created Solr Index" | 20:31 |
| @wiking | heh | 20:31 |
| @wiking | i remember the time i bought a 640 megz harddrive | 20:31 |
| @wiking | ;P | 20:31 |
| -!- shogun-notifier- [~irker@7nn.de] has quit [Quit: transmission timeout] | 20:41 | |
| -!- hushell [~hushell@8-12.ptpg.oregonstate.edu] has quit [Ping timeout: 272 seconds] | 20:43 | |
| -!- hushell [~hushell@8-12.ptpg.oregonstate.edu] has joined #shogun | 20:54 | |
| gsomix | nite | 21:01 |
| @wiking | gnite | 21:09 |
| -!- lambday [67157f37@gateway/web/freenode/ip.103.21.127.55] has quit [Quit: Page closed] | 21:37 | |
| -!- iglesiasg [~iglesiasg@s83-179-44-135.cust.tele2.se] has joined #shogun | 22:41 | |
| -!- mode/#shogun [+o iglesiasg] by ChanServ | 22:41 | |
| @iglesiasg | sup guys | 22:41 |
| -!- benibadman [~benibadma@port-92-206-233-160.dynamic.qsc.de] has quit [Remote host closed the connection] | 22:55 | |
| -!- benibadman [~benibadma@port-92-206-233-160.dynamic.qsc.de] has joined #shogun | 22:56 | |
| -!- benibadm_ [~benibadma@port-92-206-233-160.dynamic.qsc.de] has joined #shogun | 22:57 | |
| -!- benibadman [~benibadma@port-92-206-233-160.dynamic.qsc.de] has quit [Ping timeout: 245 seconds] | 23:00 | |
| @wiking | notmuch | 23:13 |
| @wiking | crunching data | 23:13 |
| @wiking | :) | 23:13 |
| -!- benibadm_ [~benibadma@port-92-206-233-160.dynamic.qsc.de] has quit [Remote host closed the connection] | 23:28 | |
| -!- benibadman [~benibadma@port-92-206-233-160.dynamic.qsc.de] has joined #shogun | 23:29 | |
| -!- benibadman [~benibadma@port-92-206-233-160.dynamic.qsc.de] has quit [Ping timeout: 248 seconds] | 23:33 | |
| @iglesiasg | hehe as usual then | 23:35 |
| --- Log closed Thu Nov 14 00:00:24 2013 | ||
Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!