| --- Log opened Thu May 24 00:00:19 2018 | ||
| -shogun-buildbot:#shogun- Build nightly trusty deb #165 is complete: Success [build successful] - http://buildbot.shogun-toolbox.org:8080/#builders/26/builds/165 | 03:02 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Issue https://github.com/shogun-toolbox/shogun/issues/4300 | 03:33 |
|---|---|---|
| @sukey | [https://github.com/shogun-toolbox/shogun] Issue https://github.com/shogun-toolbox/shogun/issues/4300 opened by vigsterkr | 03:33 |
| -!- shogun-buildbot [~shogun-bu@7nn.de] has quit [Ping timeout: 245 seconds] | 03:45 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] New commit https://github.com/shogun-toolbox/shogun/commit/ee3da845c7e82a1cb4a2d08cdcaa3ded2a3d23e3 by vigsterkr | 03:46 |
| -!- shogun-buildbot [~shogun-bu@7nn.de] has joined #shogun | 03:46 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4301 opened by vinx13 | 04:33 |
| -shogun-buildbot:#shogun- Build nightly jessie deb #153 is complete: Success [build successful] - http://buildbot.shogun-toolbox.org:8080/#builders/20/builds/153 | 05:04 | |
| -shogun-buildbot:#shogun- Build nightly stretch deb #116 is complete: Success [build successful] - http://buildbot.shogun-toolbox.org:8080/#builders/38/builds/116 | 05:04 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4302 opened by vinx13 | 05:32 |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has joined #shogun | 07:43 | |
| shubham808 | Hello, today i will be implementing the progress bar in most of the algorithm from the wiki page i made | 07:44 |
| wuwei | wiking: i'm still stuck in refcount problem | 07:52 |
| wuwei | we can't unref in transformer::apply | 07:52 |
| wuwei | it doesnt work for swig, for example in python | 07:53 |
| wuwei | processed = transformer.apply(input) | 07:53 |
| wuwei | input is unrefed, and then swig tries to delete input again | 07:54 |
| wuwei | that causes segment fault | 07:54 |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4301 closed by vinx13 | 08:08 |
| @wiking | wuwei, hey | 08:36 |
| wuwei | wiking: hi | 08:36 |
| @wiking | mmmm | 08:36 |
| @wiking | yo umean the cycle when you do transpormer.apply(input) { SG_REF(input)...... SG_UNREF(input)? } | 08:37 |
| @wiking | ? | 08:37 |
| wuwei | apply { SG_UNREF(input); SG_REF(result); } | 08:37 |
| wuwei | that's what we have in apply | 08:38 |
| wuwei | that works in cpp | 08:38 |
| @wiking | mmm | 08:38 |
| wuwei | but the input object in swig becomes invalid, and swig tries to delete again | 08:38 |
| @wiking | it should be | 08:38 |
| @wiking | apply { SG_REF(input) ..... SG_UNREF(input); SG_REF(result); | 08:39 |
| @wiking | still | 08:39 |
| @wiking | it could still drop the stack | 08:39 |
| @wiking | the reason you need to do the SG_REF in the beginning | 08:39 |
| @wiking | that you make sure that nobody tries to delete input while you do stuff with it | 08:39 |
| @wiking | but on the end you should unref it | 08:40 |
| @wiking | since you dont use it anymore (swig might still be using it | 08:40 |
| wuwei | sure | 08:40 |
| wuwei | but then we should always unref input mannually | 08:40 |
| wuwei | e.g. feats = transformer->apply(feats) | 08:40 |
| wuwei | in this case, memory leaks | 08:40 |
| @wiking | you mean if things is in place? | 08:41 |
| @wiking | yeah but you should unref the output | 08:41 |
| @wiking | anyways | 08:41 |
| wuwei | yeah, we need result = apply(feats); unref(feats) | 08:42 |
| wuwei | that's a bit annoying, we always need to use a new variale name | 08:44 |
| @wiking | so lets say in c++ | 08:44 |
| @wiking | you would do this | 08:44 |
| @wiking | input = new DenseFeatures... | 08:44 |
| @wiking | auto result = transform::apply(input); | 08:44 |
| @wiking | sg_unref(input) | 08:45 |
| @wiking | no | 08:45 |
| @wiking | so the last 2 lines should be | 08:45 |
| @wiking | sg_unref(input); | 08:45 |
| @wiking | sg_unref(result); | 08:45 |
| @wiking | right? | 08:45 |
| wuwei | exactly | 08:45 |
| @wiking | ok so the problem could be in this case is | 08:45 |
| @wiking | that if you do a SG_REF(input).... SG_UNREF(input) | 08:46 |
| @wiking | inside apply | 08:46 |
| @wiking | then actually input will be freed | 08:46 |
| @wiking | right? | 08:46 |
| wuwei | yeah | 08:46 |
| @wiking | because before transform::apply(input) | 08:46 |
| @wiking | input.ref_count() = 0 | 08:47 |
| @wiking | ok so in this case you can do internally a trick | 08:47 |
| @wiking | that inside transporm::apply() | 08:47 |
| @wiking | if(input.ref_count() > 1) SG_UNREF(ref_count) | 08:49 |
| @wiking | SG_UNREF(input) | 08:49 |
| wuwei | that works, really tricky :) | 08:49 |
| @wiking | :P | 08:49 |
| @wiking | it's a bit shitty | 08:49 |
| @wiking | but should work | 08:49 |
| @wiking | it shoudl work... | 08:49 |
| @wiking | imo | 08:49 |
| @wiking | but lemme know if you still have problems | 08:50 |
| wuwei | great! tests passed :) | 08:50 |
| wuwei | btw i'm a little confused about alphabet in string features | 08:52 |
| wuwei | on how histogram in alphabet is maintained | 08:52 |
| @wiking | mmm | 08:52 |
| @wiking | good question | 08:52 |
| @wiking | :D | 08:52 |
| wuwei | in StringFeatures::obtain_from_chars, histogram is not updated | 08:53 |
| wuwei | and if you then try to create a copy, StringFeatures(original.get_features(), original.get_alphabet()) | 08:54 |
| wuwei | it fails | 08:54 |
| wuwei | cuz check_alphabet_size won't pass | 08:54 |
| wuwei | that's whats happening on travis | 08:54 |
| @sukey | [https://github.com/shogun-toolbox/shogun] Issue https://github.com/shogun-toolbox/shogun/issues/4303 opened by vigsterkr | 08:54 |
| wuwei | :) | 08:54 |
| @sukey | [https://github.com/shogun-toolbox/shogun] Issue https://github.com/shogun-toolbox/shogun/issues/4303 vigsterkr added label: "development tasks" | 08:54 |
| @wiking | ooogh | 08:54 |
| @wiking | lemme check | 08:55 |
| wuwei | you could also check the example preprocessor_sortulongstring.py | 08:56 |
| wuwei | in fact i think when you calls obtain_from_char, then you will get characters beyond the alphabet you have | 08:57 |
| @sukey | [https://github.com/shogun-toolbox/shogun] Issue https://github.com/shogun-toolbox/shogun/issues/4304 vigsterkr added label: "development tasks" | 09:07 |
| @sukey | [https://github.com/shogun-toolbox/shogun] Issue https://github.com/shogun-toolbox/shogun/issues/4304 opened by vigsterkr | 09:07 |
| @wiking | mmm | 09:10 |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] | 09:33 | |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has joined #shogun | 09:33 | |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] | 09:40 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4305 opened by shubham808 | 09:40 |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has joined #shogun | 09:41 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4301 reopened by vinx13 | 09:56 |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4301 closed by vinx13 | 11:27 |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] | 11:30 | |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has joined #shogun | 11:30 | |
| -!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has joined #shogun | 11:32 | |
| -!- mode/#shogun [+o HeikoS] by ChanServ | 11:32 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4297 merged by karlnapf | 11:54 |
| @sukey | [https://github.com/shogun-toolbox/shogun] New commit https://github.com/shogun-toolbox/shogun/commit/911d35d1bc5667d86f84c1077ab005299324f901 by karlnapf | 11:55 |
| -shogun-buildbot:#shogun- Build trusty - libshogun - viennacl #478 is complete: Failure [failed update shogun (failure)] - http://buildbot.shogun-toolbox.org:8080/#builders/6/builds/478 | 11:55 | |
| -shogun-buildbot:#shogun- Build deb1 - libshogun #449 is complete: Failure [failed update shogun (failure)] - http://buildbot.shogun-toolbox.org:8080/#builders/10/builds/449 | 11:55 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Issue https://github.com/shogun-toolbox/shogun/issues/4299 closed by karlnapf | 11:56 |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] | 11:57 | |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has joined #shogun | 11:59 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4302 merged by karlnapf | 12:11 |
| @sukey | [https://github.com/shogun-toolbox/shogun] New commit https://github.com/shogun-toolbox/shogun/commit/5d65aa6955079a5f37e6804189c58d11ddf74ad2 by karlnapf | 12:11 |
| @HeikoS | wiking: jo! | 12:12 |
| @wiking | ho | 12:12 |
| @HeikoS | shall I dare to remove RealFetaures from SWIG? | 12:12 |
| @HeikoS | or do that after release? | 12:12 |
| @wiking | after | 12:15 |
| @wiking | better | 12:15 |
| @wiking | this one is really not about api changes | 12:15 |
| @wiking | i mean they are there | 12:15 |
| @wiking | but not feature complete :) | 12:15 |
| @wiking | so let's leave that for 7.0 | 12:15 |
| @wiking | or? | 12:15 |
| @HeikoS | yeah I was thinking that | 12:16 |
| @wiking | btw: how do u feel about this https://github.com/shogun-toolbox/shogun/issues/4304 | 12:16 |
| @HeikoS | yep I think that's good | 12:16 |
| @HeikoS | for internal use | 12:16 |
| @wiking | yeah i mean external even :) | 12:16 |
| @wiking | i mean think about somebody wanna do some automl shit | 12:16 |
| @wiking | :D | 12:16 |
| @wiking | using shogun you need to manually do some mappers | 12:16 |
| @wiking | for each model | 12:17 |
| @wiking | this way you can autodiscover | 12:17 |
| @wiking | what u could use | 12:17 |
| @wiking | and yeah i've added the other one | 12:17 |
| @wiking | https://github.com/shogun-toolbox/shogun/issues/4303 | 12:17 |
| @wiking | just to see where and how it blows up | 12:17 |
| @wiking | :D | 12:17 |
| @HeikoS | yep! | 12:18 |
| @HeikoS | good tests those are | 12:18 |
| @HeikoS | framework tests | 12:18 |
| @HeikoS | shogun is slowly becoming a framework ;D | 12:18 |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] | 12:18 | |
| @wiking | btw i'm not so sure what should we do for things like this | 12:21 |
| @wiking | https://github.com/shogun-toolbox/shogun/blob/develop/src/shogun/multiclass/LSHKNNSolver.cpp#L99-L110 | 12:22 |
| @wiking | as this is going to be copy-pasted around | 12:22 |
| @wiking | and that's not good :) | 12:22 |
| @wiking | i mean similar code | 12:22 |
| @wiking | and actually if you have | 12:23 |
| @wiking | a test for the enum | 12:23 |
| @wiking | if ((lhs->get_feature_class() == C_DENSE) && (rhs->get_feature_class() == C_DENSE)) | 12:23 |
| @wiking | instead of doing auto features = lhs->as<CDenseFeatures<float64_t>>(); | 12:23 |
| @wiking | auto features = (CDenseFeatures<float64_t>*)lhs; | 12:23 |
| @wiking | is what you want | 12:23 |
| @wiking | lisitsyn, ^ | 12:23 |
| @wiking | and there was another problem HeikoS with linalg | 12:25 |
| @wiking | 1) viannacl seems to be a bit active again.. yey! | 12:25 |
| @wiking | 2) say we have a simple model implemented PURELY linalg:: | 12:26 |
| @HeikoS | about the first thing | 12:27 |
| @HeikoS | this is a special case somehow as you need to provide falcon with the template | 12:27 |
| @wiking | yeye but i mean | 12:27 |
| @HeikoS | argument | 12:27 |
| @wiking | you could give a lambda | 12:27 |
| @wiking | that describes what to do within the typecheck | 12:27 |
| @HeikoS | but in general, cant this be solved via a clean API | 12:27 |
| @wiking | so that's 'usersupplied' | 12:27 |
| @HeikoS | like the algorithm asks the features for a DotIterator | 12:28 |
| @HeikoS | and then if that is not implemented, there is an error | 12:28 |
| @HeikoS | because dense and sparse should be behind a similar API imo | 12:28 |
| @HeikoS | from an algorithm perspective | 12:28 |
| @HeikoS | for string it is different of course | 12:29 |
| @HeikoS | but there again | 12:29 |
| @wiking | yeah but the check is the same | 12:29 |
| @wiking | something = C_STRING | 12:29 |
| @HeikoS | the only algorithms that can deal with strings AND real are kernel based algorithms | 12:29 |
| @wiking | and then you do some casting | 12:29 |
| @wiking | and la | 12:29 |
| @HeikoS | and there we could hide things behind the kernel API | 12:29 |
| @HeikoS | so the algorithm never knows the feature type | 12:29 |
| @HeikoS | just asks for the kernel between features | 12:29 |
| @HeikoS | and then that is computed if possible | 12:30 |
| @HeikoS | and otherwise error | 12:30 |
| @wiking | yeah but that's only kernelmethods | 12:30 |
| @HeikoS | do we have other thigns that support string and real? | 12:30 |
| @HeikoS | or in general, things that support feature types that cant be joined behind the same API? | 12:30 |
| @wiking | mmm | 12:31 |
| @wiking | i think you are mixing now | 12:31 |
| @wiking | i'm really just solely talking about | 12:31 |
| @wiking | dispatching code | 12:31 |
| @wiking | and how to avoid insane amount of | 12:31 |
| @wiking | copypasting around | 12:31 |
| @wiking | not literally copypasting | 12:31 |
| @wiking | but kind of having the same | 12:31 |
| @HeikoS | for the falcon thing, i cant think of another way | 12:32 |
| @wiking | ok but think about now the other KNN | 12:32 |
| @wiking | say balltree | 12:32 |
| @wiking | and we have to do the same shit | 12:33 |
| @wiking | if.... foo() else bar() | 12:33 |
| @HeikoS | like balltree for various feature types? | 12:33 |
| @wiking | elseif asdf | 12:33 |
| @wiking | that's how every train() | 12:33 |
| @wiking | will look like if you support more than | 12:33 |
| @wiking | 1 type of features | 12:33 |
| @HeikoS | I would argue this is only the case when we use external libraries | 12:34 |
| @HeikoS | as the solver itself should be written against a "distance" or "dot" api | 12:34 |
| @HeikoS | for which no casting should be necessary | 12:34 |
| @HeikoS | or rather no dispatching | 12:34 |
| @HeikoS | or maybe just dispatching under the hood, but not inside the train method | 12:34 |
| @wiking | what does under the hood mean in this context for u? | 12:35 |
| @HeikoS | for example in KNN | 12:35 |
| @HeikoS | i want to access pairwise distances | 12:35 |
| @HeikoS | so I have an API that I give my features and it gives me distances | 12:35 |
| @HeikoS | no matter what the features are | 12:35 |
| @HeikoS | inside train method I mean | 12:36 |
| @wiking | mmm and what if i want to do this ? :) | 12:36 |
| @wiking | what i'm trying to say is that | 12:36 |
| @wiking | say we have eveyrthing ready with plugins | 12:36 |
| @wiking | and somebody walks in with YOLO2000 | 12:36 |
| @wiking | and he wants to glue stuff | 12:36 |
| @HeikoS | (good algorithm btw, YOLO2000) | 12:36 |
| @HeikoS | not sure I understand | 12:37 |
| @wiking | there's a lot of code that needs to be written | 12:37 |
| @wiking | which is just copypaste-like stuff | 12:37 |
| @wiking | i mean for example | 12:38 |
| @wiking | where you just cant do an algo with DotIterator | 12:38 |
| @wiking | right? | 12:38 |
| @wiking | :) | 12:38 |
| @HeikoS | DotIterator, KernelIterator, DistanceIterator | 12:38 |
| @HeikoS | those go a long way | 12:38 |
| @HeikoS | but yeah there will be more | 12:38 |
| @HeikoS | and for those, there is the problem of dispatching of course | 12:39 |
| @HeikoS | I dont know how to solve it | 12:39 |
| @HeikoS | but I dont know any algorithms right now where this would pop up tbh | 12:39 |
| @HeikoS | e.g. GPs need the feature matrix in cases | 12:40 |
| @HeikoS | but it doesnt make sense with anything else that dense real data | 12:40 |
| @HeikoS | some string algorithms need to access the strings | 12:40 |
| @HeikoS | but these also dont make sense with other features than string | 12:40 |
| @HeikoS | so the only thing i can think of that mixes those are kernel algorithms | 12:41 |
| @wiking | say u need llt cholesky | 12:41 |
| @wiking | of your input | 12:41 |
| @HeikoS | yeah GP need that | 12:41 |
| @wiking | that works both for dense and sparse | 12:41 |
| @wiking | so you walk in and say | 12:41 |
| @HeikoS | cholesky for sparse doesnt make too much sense | 12:41 |
| @wiking | lalala | 12:41 |
| @wiking | ? | 12:41 |
| @wiking | ok lets look another decomopsition | 12:42 |
| @HeikoS | cholesky factor is always dense, even if it is of a sparse matrix | 12:42 |
| @wiking | :D | 12:42 |
| @wiking | i mean you input is needed | 12:42 |
| @wiking | as is | 12:42 |
| @wiking | and not via iterable | 12:42 |
| @HeikoS | SGMatrix features.get_factorisation | 12:42 |
| @HeikoS | SGMatrix features::get_sparse_factorisation() | 12:43 |
| @HeikoS | we dont have that | 12:43 |
| @wiking | in this case | 12:43 |
| @wiking | why do you care if its get_sparse_factorisation or get_factorisation | 12:43 |
| @wiking | :) | 12:43 |
| @HeikoS | and I agree if used linalg directly, that doesnt work as linalg wants a typed matrix | 12:43 |
| @wiking | i mean features will tell | 12:43 |
| @HeikoS | some factorizations are sparse | 12:43 |
| @HeikoS | some are dense | 12:43 |
| @HeikoS | cholesky is always dense | 12:43 |
| @HeikoS | so both work | 12:44 |
| @HeikoS | dense_feats.get_cholesky() | 12:44 |
| @HeikoS | sparse_feats.get_cholesky() | 12:44 |
| @wiking | yeah but now you are doing something | 12:44 |
| @wiking | that we kept undoing | 12:44 |
| @wiking | :) | 12:44 |
| @wiking | loading operators over objects | 12:44 |
| @HeikoS | sure | 12:44 |
| @wiking | and till now everyghing was converging | 12:45 |
| @wiking | that you wanna do this via linalg | 12:45 |
| @wiking | now suddenly we'll have a mixture of things | 12:45 |
| @wiking | i mean i dont mind | 12:45 |
| @HeikoS | DotIterator is the same no? | 12:45 |
| @wiking | just currently this is not at all cleared up | 12:45 |
| @wiking | what sense? | 12:45 |
| @HeikoS | there is some external code that you give features and it provides you with a way of accessing them | 12:45 |
| @HeikoS | I mean I agree with you | 12:45 |
| @HeikoS | just saying this is what I meant when I said one can behind things behind an API | 12:46 |
| @HeikoS | linalg is different | 12:46 |
| @HeikoS | it currently wants SGMatrix<float64_t> | 12:46 |
| @wiking | ah so this is 'under the hood' | 12:46 |
| @wiking | yeah sure | 12:46 |
| @wiking | i mean that's just because its way unfinished | 12:46 |
| @HeikoS | yeah | 12:46 |
| @wiking | i mean currently we dont even do | 12:47 |
| @wiking | linalg::dot | 12:47 |
| @wiking | over sparse vectors | 12:47 |
| @wiking | :P | 12:47 |
| @HeikoS | yeah | 12:47 |
| @HeikoS | actually | 12:47 |
| @HeikoS | now this discussion makes me think whether we even want this | 12:47 |
| @HeikoS | since this makes it mandatory inside the train method to know the exact type | 12:47 |
| @HeikoS | which means we need to have the copy paste mess | 12:47 |
| @HeikoS | but what we want in the train method is just "dot product" | 12:48 |
| @HeikoS | or "cholesky factor" | 12:48 |
| @HeikoS | since that is how the algorithm is defined | 12:48 |
| @HeikoS | so linalg design-- | 12:48 |
| @HeikoS | not sure what is the best way out of this | 12:48 |
| @HeikoS | I personally really like the DotIterator | 12:48 |
| @HeikoS | since this solves so many problems | 12:48 |
| @HeikoS | you can do basic type dispatching once before you start iterating (cheap) | 12:49 |
| @HeikoS | you can offer it for sparse/dense under the same API | 12:49 |
| @HeikoS | you know that you will iterate so it can be done fast | 12:49 |
| @wiking | sorry back | 12:49 |
| @wiking | ok so there's another story here | 12:50 |
| @wiking | and this is the story of | 12:50 |
| @wiking | YOLO2001 | 12:50 |
| @HeikoS | lol | 12:50 |
| @wiking | say i use linalg:: all the way | 12:50 |
| @wiking | so my way of thinking was | 12:50 |
| @wiking | lalala i'm nice and good | 12:50 |
| @wiking | now it should be device agnostic | 12:51 |
| @wiking | is it really? :D | 12:51 |
| @wiking | ie i dont see what will trigger something running on GPU or CPU | 12:51 |
| @wiking | in this case | 12:51 |
| @HeikoS | i think CPU/GPU should be define within the train method | 12:51 |
| @wiking | why? | 12:52 |
| @HeikoS | as it is quite hard to infer what is best there | 12:52 |
| @wiking | what i mean | 12:52 |
| @HeikoS | and algorithm design changes quite a bit | 12:52 |
| @wiking | why cant i have either? | 12:52 |
| @wiking | really how? | 12:52 |
| @wiking | kmeans? | 12:52 |
| @HeikoS | well, there is GPU versions of many algorithms and they look quite different | 12:52 |
| @HeikoS | e.g. for SVM solvers | 12:52 |
| @wiking | again YOLO2001 is a simple algo | 12:53 |
| @wiking | no magic shit | 12:53 |
| @HeikoS | i see | 12:53 |
| @wiking | i do a) b) and thats all | 12:53 |
| @wiking | measure distances | 12:53 |
| @wiking | and then whatever | 12:53 |
| @HeikoS | well as it is now, the caller of linalg has to decide | 12:53 |
| @wiking | yeah | 12:53 |
| @HeikoS | i.e. author of YOLO2001 | 12:53 |
| @wiking | i know | 12:53 |
| @HeikoS | not the guy who runs it | 12:53 |
| @wiking | which imo is not the best | 12:53 |
| @wiking | see tf | 12:53 |
| @wiking | those are all agnostic | 12:54 |
| @wiking | right :) | 12:54 |
| @HeikoS | yeah I know | 12:54 |
| @HeikoS | thats better | 12:54 |
| @wiking | i mean you cannot switch runtime | 12:54 |
| @wiking | i give u that | 12:54 |
| @wiking | but on the other hand | 12:54 |
| @wiking | operators are agnostic of device | 12:54 |
| @HeikoS | they are | 12:54 |
| @wiking | i mean from the user (algo implementer) perspective | 12:54 |
| @HeikoS | indeed | 12:54 |
| @HeikoS | and as such, this shouldnt be in the algo | 12:54 |
| @HeikoS | i agree | 12:54 |
| @wiking | so | 12:54 |
| @wiking | fickfuck | 12:55 |
| @wiking | :) | 12:55 |
| @wiking | fyi https://github.com/thrust/thrust | 12:55 |
| @wiking | STL-like stuff | 12:55 |
| @wiking | that is totally device agnostic | 12:55 |
| @wiking | :P | 12:55 |
| @HeikoS | but still | 12:56 |
| @HeikoS | ML algorithms are different | 12:56 |
| @HeikoS | the algorithms DO change when doing CPU / GPU | 12:56 |
| @wiking | mmm | 12:56 |
| @wiking | i would beg to differ | 12:56 |
| @wiking | it's not *always* the case | 12:57 |
| @HeikoS | and sure in tf you can choose | 12:57 |
| @HeikoS | yeah not always of course | 12:57 |
| @wiking | so i mean | 12:57 |
| @HeikoS | just I have seen many instances of it | 12:57 |
| @wiking | if your operators | 12:57 |
| @wiking | are agnostic | 12:57 |
| @wiking | say linalg or tf | 12:57 |
| @wiking | then as the story tells you | 12:57 |
| @wiking | actually things work | 12:57 |
| @wiking | pretty good | 12:57 |
| @wiking | = tf | 12:57 |
| @HeikoS | and also I have seen many instances of really inefficient things being done on the GPU since somebody just took some tf code and made it run on GPu | 12:57 |
| @HeikoS | and in tf, most things run on GPU since most things are matrix operations anyways, so the whole library is built for that | 12:58 |
| @HeikoS | it is more that is ALSO runs things on cpu if one wants | 12:58 |
| @HeikoS | it's complicated :/ | 12:58 |
| @wiking | no it's not | 12:59 |
| @wiking | i dont think its' that complicated | 12:59 |
| @HeikoS | at the end of the day, I guess a framework should give freedom to both algorithm authors and users | 12:59 |
| @wiking | its just maybe things need some changes | 12:59 |
| @wiking | as as of now | 12:59 |
| @wiking | or as things is now | 12:59 |
| @wiking | you cannot drive as a user anything | 12:59 |
| @HeikoS | i agree | 12:59 |
| @wiking | i mean honestly | 13:00 |
| @wiking | if we have SGVector | 13:00 |
| @HeikoS | mmh but it is also hard | 13:00 |
| @wiking | that is strictly user facing interface | 13:00 |
| @HeikoS | how does the user specify that a shogun algo should run on gpu? | 13:00 |
| @wiking | and there we do on an operator[](index_t i) | 13:00 |
| @HeikoS | does he have to specify for every vector where it is? | 13:00 |
| @HeikoS | or just globally? | 13:00 |
| @wiking | a check every time where the memory resides | 13:00 |
| @wiking | then i would think that this is something that is user bsed and not hidden behind | 13:00 |
| @wiking | the algo | 13:00 |
| @wiking | because if you wanna strictly confine within an alog | 13:01 |
| @wiking | *algo | 13:01 |
| @wiking | where is what done | 13:01 |
| @wiking | then the person should know what the heck he is doing :) | 13:01 |
| @HeikoS | true | 13:01 |
| @wiking | and no need for operator[] (index_t i) { assert_on_cpu(); | 13:01 |
| @wiking | for SGVector | 13:01 |
| @wiking | because then you do GPUVector | 13:01 |
| @wiking | and that's all | 13:02 |
| @wiking | :) | 13:02 |
| @wiking | so there's a mixing of things here | 13:02 |
| @wiking | that actually are pointing to different directions | 13:02 |
| @HeikoS | yes there is like multiple bigger design questions | 13:02 |
| @wiking | and currently we try to sit on 2 horses | 13:02 |
| @wiking | but we are actually sitting on the ground | 13:02 |
| @HeikoS | we are stitting in the horse-poo man | 13:03 |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4285 synchronized by vinx13 | 13:46 |
| @wiking | HeikoS, here? | 14:23 |
| @wiking | have u tried this | 14:25 |
| @wiking | http://h2o-release.s3.amazonaws.com/h2o/master/3867/docs-website/h2o-py/docs/modeling.html?highlight=klime#h2oklimeestimator | 14:25 |
| @wiking | anyhow things like this we dont have at all | 14:27 |
| -!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has quit [Ping timeout: 240 seconds] | 15:09 | |
| -!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has joined #shogun | 15:24 | |
| -!- mode/#shogun [+o HeikoS] by ChanServ | 15:24 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4285 synchronized by vinx13 | 15:30 |
| wuwei | wiking: how do u see alphabet things: histogram not updated, alphabet type is wrong, when you call obtain_from_char from string features | 15:58 |
| wuwei | that's the last thing to be fixed now for my pr | 15:58 |
| @wiking | mmm | 15:58 |
| @wiking | i have now a meeeting | 15:58 |
| @wiking | but after that i can check into that | 15:58 |
| @wiking | ok? | 15:58 |
| @wiking | sorry | 15:58 |
| @wiking | today way too many interrupts | 15:59 |
| wuwei | sure | 15:59 |
| wuwei | no worry | 15:59 |
| -!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has quit [Ping timeout: 264 seconds] | 16:03 | |
| wuwei | so here are update for today: 1) updated notebooks 2) got ref count fixed 3) fixed some bug when i refactored preprocessors (most tests passed on travis now); tomorrow i'm going to: 1) fix string preproc 2) get doc updated 3) check & refactor ICA converters for the new api | 16:10 |
| @wiking | thnx! | 16:18 |
| @wiking | ok so now i'll look into the histogram thiingy | 16:18 |
| -!- HeikoS [~heiko@eduroam-int-pat-8-249.ucl.ac.uk] has joined #shogun | 16:19 | |
| -!- mode/#shogun [+o HeikoS] by ChanServ | 16:19 | |
| -!- HeikoS [~heiko@eduroam-int-pat-8-249.ucl.ac.uk] has quit [Quit: Leaving.] | 16:49 | |
| -!- Elfarouk [9cc5ae08@gateway/web/freenode/ip.156.197.174.8] has joined #shogun | 17:08 | |
| -!- Elfarouk [9cc5ae08@gateway/web/freenode/ip.156.197.174.8] has quit [Ping timeout: 260 seconds] | 17:18 | |
| -!- Elfarouk [9cc5ae08@gateway/web/freenode/ip.156.197.174.8] has joined #shogun | 17:21 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4294 synchronized by FaroukY | 17:32 |
| Elfarouk | wikingm Are you here? | 17:36 |
| Elfarouk | wiking | 17:36 |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has joined #shogun | 17:37 | |
| Elfarouk | Is anyone here? | 18:21 |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4293 synchronized by geektoni | 18:24 |
| -!- smatrix [9d276d40@gateway/web/freenode/ip.157.39.109.64] has joined #shogun | 18:41 | |
| smatrix | hi | 18:42 |
| -!- smatrix [9d276d40@gateway/web/freenode/ip.157.39.109.64] has quit [Client Quit] | 18:42 | |
| -!- shubham808 [0e8bf0fb@gateway/web/cgi-irc/kiwiirc.com/ip.14.139.240.251] has quit [Quit: http://www.kiwiirc.com/ - A hand crafted IRC client] | 19:21 | |
| -!- smatrix [9d276d40@gateway/web/freenode/ip.157.39.109.64] has joined #shogun | 19:41 | |
| -!- smatrix [9d276d40@gateway/web/freenode/ip.157.39.109.64] has quit [Client Quit] | 19:42 | |
| @wiking | Elfarouk, yes now | 19:52 |
| Elfarouk | wiking: I am having a problem with serialisation of the new class. I tried changing all the Eigen::Matrix to SGMatrix and all the Eigen::Vector to SGVector and the files compiled nicely. But when I compiled the entire project, I got a serialisation error | 19:58 |
| Elfarouk | that SGMatrix<stan::math::var> can not be serialised | 19:58 |
| Elfarouk | Do you think it'd be better to go back to Eigen::Matrix / Eigen::Vector ? | 20:06 |
| @wiking | mmm | 20:14 |
| @wiking | ok | 20:14 |
| @wiking | so | 20:14 |
| @wiking | for stan::math::var | 20:14 |
| @wiking | use Eigen::Matrix / Eigen::Vector for the time being | 20:15 |
| @wiking | but for Eigen::Matrix / Eigen::Vector <double> | 20:15 |
| @wiking | use the SGstuff | 20:15 |
| @wiking | for stan::math::var we need to fihure out something | 20:15 |
| Elfarouk | Alright then, I'll do that | 20:17 |
| Elfarouk | Sure, I also emailed Heiko and he told me that serialization is only supported now for basic types | 20:18 |
| -!- ChanServ [ChanServ@services.] has quit [shutting down] | 21:07 | |
| -!- ChanServ [ChanServ@services.] has joined #shogun | 21:13 | |
| -!- ServerMode/#shogun [+o ChanServ] by weber.freenode.net | 21:13 | |
| -!- ChanServ [ChanServ@services.] has quit [shutting down] | 21:22 | |
| -!- ChanServ [ChanServ@services.] has joined #shogun | 21:31 | |
| -!- ServerMode/#shogun [+o ChanServ] by weber.freenode.net | 21:31 | |
| -!- Trixis is now known as Guest40548 | 21:34 | |
| -!- Guest40548 is now known as Trixis | 21:53 | |
| -!- Trixis is now known as Guest14447 | 21:54 | |
| -!- Guest14447 [~Trixis@unaffiliated/trixis] has quit [Quit: ZNC 1.6.5 - http://znc.in] | 21:57 | |
| -!- Trixis_ [~Trixis@unaffiliated/trixis] has joined #shogun | 21:57 | |
| -!- Trixis_ is now known as Trixis | 21:57 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4294 synchronized by FaroukY | 22:06 |
| -!- iglesiasg [~iglesias@f119189.upc-f.chello.nl] has joined #shogun | 22:36 | |
| -!- Elfarouk [9cc5ae08@gateway/web/freenode/ip.156.197.174.8] has quit [Quit: Page closed] | 22:38 | |
| -!- Elfarouk [695d4ff2@gateway/web/freenode/ip.105.93.79.242] has joined #shogun | 22:38 | |
| iglesiasg | oh did not notice you were around, Elfarouk. hi! | 22:47 |
| Elfarouk | Hey iglesiasg | 22:47 |
| iglesiasg | How is it going? | 22:48 |
| Elfarouk | Good, finished all comments on the new PR. Now addressing the changes required in the cookbook. Part of it will need to change factory.h so that a class can be instantiated with a factory so i am just having a look at previous prs to see how they did it | 22:49 |
| iglesiasg | Yeah, I just had a look at the new PR | 22:49 |
| iglesiasg | I think there are still more things pending on that one though | 22:49 |
| iglesiasg | Did you have a chat with wiking regarding the antipattern comment? | 22:50 |
| Elfarouk | I did, but we're having a problem with serialisation | 22:50 |
| Elfarouk | right now SGVector<stan::math::var> can't be serialised | 22:50 |
| Elfarouk | cause only basic types can be serialised | 22:50 |
| iglesiasg | Ok | 22:51 |
| Elfarouk | So I tried switching everything to SGVector and SGMatrix, but it broke it because of the serialisation, so he told me to revert them back | 22:51 |
| iglesiasg | What broke? | 22:51 |
| iglesiasg | I mean, what part of the code you are writing/executing relies on serialization? | 22:52 |
| Elfarouk | Just the: SGVector<stan::math::var> and SGMatrix<stan::math::var> | 22:52 |
| iglesiasg | I don?t understand | 22:52 |
| Elfarouk | originally in the PR I was using Eigen::Matrix< stan::math::var, Dynamic, Dynamic> | 22:53 |
| iglesiasg | ok | 22:53 |
| iglesiasg | more specific | 22:53 |
| Elfarouk | wiking told me that's anti shogun pattern | 22:53 |
| iglesiasg | you were using it in a member? | 22:53 |
| iglesiasg | in implementation code? | 22:53 |
| Elfarouk | Ohhh yes, as a member | 22:53 |
| iglesiasg | in the signature of a method? | 22:53 |
| Elfarouk | and in part of the implementation of a code | 22:53 |
| iglesiasg | ok, clear | 22:53 |
| iglesiasg | it is good that you understand these things | 22:53 |
| iglesiasg | saying an anti pattern per se does not explain you much | 22:54 |
| Elfarouk | Yea so we said for now lets see it but we definitely need to address it | 22:54 |
| iglesiasg | do you understand why using SGVector rather than Eigen types? | 22:54 |
| Elfarouk | lets leave it* | 22:54 |
| iglesiasg | for the member | 22:54 |
| Elfarouk | I think now I understand because of the serialization. You only support serializing members which are basic data types | 22:55 |
| Elfarouk | SGVector is a basic datatype | 22:55 |
| Elfarouk | but now Eigen::Matrix | 22:55 |
| Elfarouk | not* | 22:55 |
| iglesiasg | serialization is important if you want to serialize :) | 22:55 |
| iglesiasg | afaik, the whole serialization topic is somewhat WIP | 22:55 |
| iglesiasg | so do not worry about it **unless* you have to use serialization for the stuff you are doing | 22:56 |
| Elfarouk | :D alright | 22:56 |
| iglesiasg | that's why I asked before what part of your code **relies** on serialization | 22:56 |
| iglesiasg | apart from serialization, it is about swig and type maps | 22:56 |
| Elfarouk | Ahaa I see. | 22:56 |
| Elfarouk | I'll go back to the cookbook and then have a look at the new comments on the new PR | 22:57 |
| iglesiasg | the stuff that is part of the "public" interface should in principle be accessible from the swig interfaces | 22:57 |
| iglesiasg | that requires mapping C++ types to whatever interface language is | 22:57 |
| iglesiasg | of course the tool, swig, does this for free for very common types | 22:58 |
| iglesiasg | but it cannot know every possible type in advance! | 22:58 |
| iglesiasg | hope that clarifies a bit | 22:58 |
| Elfarouk | Ohh I see now. So If you want to include sth in the "public" interface, then it needs to be serializable so that SWIG can be able to handle it? | 22:59 |
| iglesiasg | no, it does not need to be serializable | 22:59 |
| iglesiasg | connection to other languages via swig is in-memory | 23:00 |
| iglesiasg | otherwise imagine the overhead! :O | 23:00 |
| iglesiasg | serialization goes beyond in-memory transactions | 23:00 |
| Elfarouk | I see | 23:01 |
| iglesiasg | Anyhow, yes, I agree. Since we are getting a bit stuck and going often back and forth, which I understand can be frustrating for you, let's focus on getting the cookbook pr done | 23:01 |
| iglesiasg | then we switch to the cost functions one | 23:01 |
| iglesiasg | I think the cookbook should be soon finished | 23:02 |
| iglesiasg | for the cost functions we will need to discuss and sync more | 23:02 |
| Elfarouk | Sure. Once I am done with the cookbook and its merged we can do a 15 minute hangout call maybe | 23:02 |
| Elfarouk | just discuss next steps | 23:02 |
| iglesiasg | yep | 23:03 |
| iglesiasg | and it is very good in that sense that you have already opened this second pr | 23:03 |
| iglesiasg | as it is a base to guide the discussion | 23:03 |
| Elfarouk | Cool then. I'll let you know once I am done :D | 23:04 |
| iglesiasg | sure | 23:04 |
| iglesiasg | just keeping it as these last few days is perfect | 23:04 |
| iglesiasg | we check github notifications and all that :) | 23:05 |
| Elfarouk | Alright then :D | 23:05 |
| iglesiasg | Good night! | 23:23 |
| Elfarouk | Good night~ | 23:26 |
| --- Log closed Fri May 25 00:00:21 2018 | ||
Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!