IRC logs of #shogun for Friday, 2017-02-24

« Thursday, 2017-02-23 Index Saturday, 2017-02-25 »
--- Log opened Fri Feb 24 00:00:38 2017
-!- goksinen [~goksinen@rrcs-50-75-193-138.nyc.biz.rr.com] has joined #shogun00:13
-!- goksinen [~goksinen@rrcs-50-75-193-138.nyc.biz.rr.com] has quit [Ping timeout: 240 seconds]00:18
-!- goksinen [~goksinen@rrcs-50-75-193-138.nyc.biz.rr.com] has joined #shogun00:21
-!- goksinen [~goksinen@rrcs-50-75-193-138.nyc.biz.rr.com] has quit [Remote host closed the connection]00:22
-!- goksinen [~goksinen@rrcs-50-75-193-138.nyc.biz.rr.com] has joined #shogun00:22
-!- goksinen [~goksinen@rrcs-50-75-193-138.nyc.biz.rr.com] has quit [Read error: Connection reset by peer]02:27
-!- goksinen [~goksinen@rrcs-50-75-193-138.nyc.biz.rr.com] has joined #shogun02:28
@sukeyNew Commit "replaced std::vector<index_t> with SGVector<index_t>" to shogun-toolbox/shogun by lambday: https://github.com/shogun-toolbox/shogun/commit/25f28f3b116f4c800a4abb85a788b8d9ffade88a06:34
-!- lambday_ [c40f1710@gateway/web/freenode/ip.196.15.23.16] has joined #shogun07:09
-!- mode/#shogun [+o lambday_] by ChanServ07:09
-!- travis-ci [~travis-ci@ec2-54-80-85-96.compute-1.amazonaws.com] has joined #shogun07:27
travis-ciit's lambday's turn to pay the next round of drinks for the massacre he caused in shogun-toolbox/shogun: https://travis-ci.org/shogun-toolbox/shogun/builds/20486154207:27
-!- travis-ci [~travis-ci@ec2-54-80-85-96.compute-1.amazonaws.com] has left #shogun []07:27
-!- lambday [31cf349d@gateway/web/freenode/ip.49.207.52.157] has quit [Ping timeout: 260 seconds]07:30
@sukeyPull Request #3635 "LinalgRefactor - Memory Transfer Mutex"  synchronized by OXPHOS - https://github.com/shogun-toolbox/shogun/pull/363507:35
-!- mhlmz [4e0d7782@gateway/web/freenode/ip.78.13.119.130] has joined #shogun10:13
-!- lambday_ [c40f1710@gateway/web/freenode/ip.196.15.23.16] has quit [Quit: Page closed]13:04
-!- lambday [6a3317ac@gateway/web/freenode/ip.106.51.23.172] has joined #shogun14:15
-!- mode/#shogun [+o lambday] by ChanServ14:15
-!- goksinen is now known as wudayoda15:54
-!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has joined #shogun16:17
-!- mode/#shogun [+o HeikoS] by ChanServ16:17
@HeikoSwiking: jojo16:18
@HeikoSw16:18
@HeikoSlambday: jo16:18
@lambdayHeikoS: hey16:18
@HeikoSlambday: all good?16:18
@lambdayHeikoS: yeah16:18
@lambdayHeikoS: working on a benchmark :)16:19
@HeikoSbenchmarking what?16:19
@lambdayCDynamicObjectArray.get_element() vs std::vector::operator[]..16:20
@lambdayget_element() is quite slower.. takes 1.5x compared to vector []...  perf says it's mostly the SG_REF that takes up most of the time.. so tried with CDynamicObjectArray::get_array()[...], but even that's slower..16:22
@lambdaygotta see whether this makes any significant impact in our code.. cause if the usage is less, the difference won't be much16:23
@lambdaydown to assembly now :D16:23
@HeikoSlambday: no matter what16:24
CaBaHeikoS: heya16:24
@HeikoSwe have to get rid of the std::vector16:24
@HeikoSlambday: btw I wrote some comments on your last commit16:24
@lambdaylet me check16:24
@HeikoSCaBa: hi!16:25
CaBaHeikoS: remind me, back then when we talked about building those ms-param trees for combined kernels - did we conclude that there was no way around creating a separate combined kernel for every subkernel parameter combination or did you still want to show me how that works properly through the ms-param interface? ^^16:26
@HeikoSCaBa: ah I am unreliable16:30
@HeikoSI dont know16:30
@HeikoSbut I do know that I wanted to write some code for you16:30
@HeikoSlambday: hey16:31
@HeikoSlambday: so re benchmarking16:31
@HeikoSlambday: it is not the right time to do that16:31
@HeikoSlambday: because it should be done after we fixed the problems in the code16:31
CaBaHeikoS: http://stackoverflow.com/questions/40359913 <- remember, this one? ;)16:31
@HeikoSlambday: now it takes time to do benchmarking and then it is less likely that the actual problems are fixed16:31
@HeikoSlambday: and in the last commit, you make it worse in fact16:32
@HeikoSmore unsigned/signed comparisons, possible underflows, possible compile errors16:32
@lambdayHeikoS: was just checking before I replace vector<CSGObject> with DynamicObjectArray16:32
@HeikoSit doesnt matter for now16:32
@HeikoSbecause the std::vector has to go16:32
@HeikoSotherwise bugs16:32
@HeikoSonce we have fixed that, we can speed it up16:33
@lambdayHeikoS: where? I replaced std::vectors with SGVector... next task is replacing vector<vector> with SGMatrix16:33
@HeikoSand it is good to do that16:33
@lambdayso got rid of a few size_t's already16:33
@lambdaysending patch step by step, otherwise it's gonna be too big16:33
@HeikoSmy point is, while there is any of the std::vector guys active in loops that do index SG* structures, there should be no benchmarking16:33
@HeikoSbecause benchmarking comes after working code, see what I mean?16:34
@HeikoSlambday: agree on step by step16:34
@lambdayHeikoS: can you point me which line are you refering to?16:34
@lambdayIIRC there is no vector<index_t> anymore.. all SG16:34
@HeikoSyes but there is the nested one16:34
@HeikoSand also the one with CKernel in it16:35
@HeikoShttps://github.com/shogun-toolbox/shogun/blob/25f28f3b116f4c800a4abb85a788b8d9ffade88a/src/shogun/statistical_testing/internals/mmd/CrossValidationMMD.h#L10816:36
@HeikoSlambday: gotta check travis logs16:36
@HeikoSlambday: sorry dont want to be too picky here, but really there are priorities16:37
@HeikoSbrb16:37
-!- HeikoS_mobile [~Mutter@eduroam-int-pat-8-161.ucl.ac.uk] has joined #shogun16:38
HeikoS_mobilelambday: back on mobile16:38
@lambdayHeikoS_mobile: haha which client you're using..16:38
HeikoS_mobileMother16:39
HeikoS_mobilelambday: let's do the benchmarking in another feature branch once this one is merged16:40
HeikoS_mobilelambday: merging has top priority and there are quite a few things missing yet16:40
@lambdayHeikoS_mobile: nah I was benchmarking a separate code that I have written.. just two different types of kernel mgr.. one with dynamic object array and another with vector... benchmark was just the access operator..16:41
@lambdayHeikoS_mobile: cool..16:41
HeikoS_mobilelambday: you see the fact that we never work in mere but always get distracted with some other thing while merge status is 90% is what let to all these problems16:42
HeikoS_mobileMore than happy to speed all the structures up and benchmarking them of course after16:42
@lambdayHeikoS_mobile: yeah I agree16:42
@lambdayHeikoS_mobile: what about vectors that don't contain CSGObjects?16:44
@lambdayHeikoS_mobile: CDynamicArray<T> ??16:44
HeikoS_mobileYes for basic types that16:44
HeikoS_mobileWhat is your T?16:44
@lambdayHeikoS_mobile: custom structs...16:45
HeikoS_mobileAnd other question is: do you need dynamic size16:45
@lambdayHeikoS_mobile: yes16:45
HeikoS_mobileWhy is that? All loop sizes should be known before no?16:45
@lambdayHeikoS_mobile: nah those are run-time variables16:46
HeikoS_mobileSo?16:46
HeikoS_mobileI can't imagine a case in kernel testing where you can't know the size before you run something16:47
@lambdayHeikoS_mobile: so the size is dynamic..16:47
HeikoS_mobileRuntime size is fine16:47
HeikoS_mobileI mean you should know size at allocation time16:47
HeikoS_mobileSince loop sizes are known16:47
@lambdayHeikoS_mobile: yeah..16:47
@lambdayHeikoS_mobile: or maybe not.. cause imagine kernel selection, where the user simply adds kernels16:48
HeikoS_mobileYeah, so?16:48
@lambdayHeikoS_mobile: when he's done, he tests it.. can't know at allocation time16:48
HeikoS_mobileThat's a single reallocate16:49
HeikoS_mobileWith zero recycling of content16:49
* CaBa imagines HeikoS_mobile like this right now... https://files.photomics.org/tmp/moirc.jpg16:49
HeikoS_mobileOr am I missing something?16:49
HeikoS_mobileCaBa: yep ;)16:49
HeikoS_mobileIn Talk16:49
CaBawhatever that is ^^16:50
@lambdayHeikoS_mobile: I am not sure what you mean.. so we allocate a space for 10.. if the number exceeds that, we allocate 10 more.. then do something like shrink_to_fit/leave it as it is?16:50
HeikoS_mobileThough I am using the phone in portrait16:50
HeikoS_mobilelambday: these guys can be put to a dynamic one16:50
HeikoS_mobileBut once the algorithm runs16:50
HeikoS_mobileYou don't need resize as you know how many kernels there are16:50
HeikoS_mobilelambday: btw that's how dynamic array works16:51
@lambdayHeikoS_mobile: yeah resizing is not required in this use-case..16:51
@lambdayHeikoS_mobile: yeah16:51
HeikoS_mobileAllocating 10 more16:51
@lambdayHeikoS_mobile: so that's what we need, right16:51
HeikoS_mobileAnd also std vector iirc16:51
HeikoS_mobileYou can even is list16:52
HeikoS_mobileBecause once you start, you can copy al pointers into a static array16:52
@lambdayHeikoS_mobile: yeah.. before the move semantics, std::vector did realloc/copy-ctor once the space got exhausted.. but since cpp11, std::move saves up a lot of time16:52
HeikoS_mobileI see16:52
@lambdayHeikoS_mobile: static array?16:53
HeikoS_mobileWell still there is no need for resize in your code loops16:53
@lambdayHeikoS_mobile: copying ptrs to another array is not required, we can simply access the underlying contiguous array..16:54
HeikoS_mobileYep sure16:54
HeikoS_mobileSo Our discussion was about the need for resize16:54
@lambdayplus, some of the use-cases may have unique_ptr and not ptrs, so not really copy-able16:55
HeikoS_mobileAnd my point was that you know the size of all structures once you start running the algorithm16:55
HeikoS_mobileHence no need for resize16:55
HeikoS_mobileWithin any loop16:55
@lambdayHeikoS_mobile: okay will create a new vector in that case16:56
@lambdayif size changes in future calls, it will create new memory16:57
HeikoS_mobileThe size doesn't change anyways most of the time iirc, or does it?16:57
HeikoS_mobileResize vector btw pads with zeros16:57
@lambdayit can change in use-case similar to this : mmd.set_num_null_samples(100), mmd.perform_test(), mmd.set_num_null_samples(200), mmd.perform_test()....16:58
@lambdayif you want to plot varying number of null-samples16:58
HeikoS_mobileBut just once right?16:58
HeikoS_mobileOnce per change16:59
@lambdayonce per change...16:59
HeikoS_mobileThat was my point, neglectable delay, big improvement in semantic clarity16:59
@lambdayHeikoS_mobile: okay, realloc it is..17:00
@lambdayHeikoS_mobile: agreed on clarity17:00
HeikoS_mobileI guess the dynamic object array can sts vector thing will have a bigger impact17:00
-!- wudayoda [~goksinen@rrcs-50-75-193-138.nyc.biz.rr.com] has quit [Read error: Connection reset by peer]17:00
HeikoS_mobileSo really worth looking into that17:00
HeikoS_mobileOnce the code size t thing is resolved17:00
HeikoS_mobile*Vs are vector17:01
@lambdayHeikoS_mobile: hehe yeah got that.. yeah will have a look17:01
-!- wudayoda [~goksinen@rrcs-50-75-193-138.nyc.biz.rr.com] has joined #shogun17:02
HeikoS_mobileBtw also please watch out for compiler warnings like the one I posted above17:02
@lambdayHeikoS_mobile: stl vectors have improved a lot (at least on gcc/clang) over time..17:03
@lambdayHeikoS_mobile: yeah will grep for size_t once I am done with this..17:03
HeikoS_mobilelambday: yeah we should investigate these speedups, and then apply it to shoguns structures17:04
HeikoS_mobileBig ++ from my side17:04
HeikoS_mobileDetox and benchmark project relevant17:04
HeikoS_mobileBut let's do that shogun wide, not just in testing code17:05
HeikoS_mobilelambday: Grep doesn't help you as much as watching compiler warnings ;)17:05
HeikoS_mobilelambday: maybe we can internally change the backend of shoguns vectors to stl at some point17:11
@lambdayHeikoS_mobile: gotta be careful in that :) we have SGVectors passed by value at many places.. changing with stl is gonna make deep copy in that case... const& always..17:13
HeikoS_mobilelambday: also talked with wiking to check this numfocus project on data structures and potentially using it17:13
HeikoS_mobilelambday: sure sure I meant more the backend of dynamic array17:14
HeikoS_mobileNot sg*17:14
HeikoS_mobileThat's more this numfocus lib17:14
HeikoS_mobileInteresting stuff definitely17:14
@lambdayHeikoS_mobile: yeah for these common data structures we shouldn't use things written from scratch.. unless we want a finer control over the allocator.. it's better to use some lib.. custom allocators can be done with stl17:15
HeikoS_mobileYep ++17:15
HeikoS_mobileImprove shogun with this stuff17:15
@lambdayonly thing to worry is the swig..17:16
HeikoS_mobileOk, but for now, let's fix the branch ;) I'll brb in a bit17:16
@lambdayHeikoS_mobile: yeah..17:16
@lambdayI like stl cause we don't add an extra dependency on yet another library... it comes with the pkg and it is continuously improving..17:18
-!- HeikoS_mobile [~Mutter@eduroam-int-pat-8-161.ucl.ac.uk] has quit [Quit: Mutter: www.mutterirc.com]17:18
-!- mhlmz [4e0d7782@gateway/web/freenode/ip.78.13.119.130] has quit [Ping timeout: 260 seconds]17:29
@HeikoSlambday: yep!17:52
@HeikoSwiking: you awake?17:54
@HeikoSlambday: still running?17:57
@HeikoSlambday:  I just had this thought we better merge the branch soon since we are currently reviewed for GSoC17:57
@HeikoSlisitsyn: ^17:57
@lambdayHeikoS: no.. was moving to eclipse17:57
@HeikoSthe editor ? ;)17:57
@lambdayHeikoS: only for refactoring though :P17:58
@lambdayHeikoS: for the rest, vim for the win :D17:58
@HeikoShehe17:58
@lambdayHeikoS: when shall we merge back?17:58
@HeikoSgreen CI was the goal17:59
@lambdayapart from that one test being diva, we're good!17:59
@HeikoSand the left column of your project17:59
@lambdayHeikoS: yeah.. I'll do that asap.. but it's not causing failure17:59
@HeikoSyeh I know18:00
@HeikoSit is just the input parser bug18:00
@HeikoSlambday: im hesitant to merge without the consent of us three18:00
@HeikoSthough I think we should do it today, even if there is still this last error18:00
@HeikoSbecause of the review thing18:01
@HeikoSit should at least compile18:01
@lambdaywiking: there?18:01
@HeikoSwell one more day wont harm18:01
@HeikoSsize_t is kind of a blocker actually18:01
@HeikoSso lets see tomorrow18:01
@HeikoSlambday: btw we should have a hangout when things are done with testing, and you share some of the profiling insights you got18:04
@lambdayHeikoS: totally18:05
@lambdayHeikoS: once I am done with this, I was thinking of writing a small plan for the performance project.. about the right tools, about things to look for.. etc etc18:06
@lambdayHeikoS: may also put a link to this video : https://www.youtube.com/watch?v=nXaxk27zwlk18:06
@HeikoSyep good18:06
@HeikoSyeah cool18:07
@HeikoSif you have concrete mini projects we can do in shogun, please write them down18:07
@HeikoSwe can put that into gsoc18:07
-!- OXPHOS [92bd305b@gateway/web/freenode/ip.146.189.48.91] has joined #shogun18:16
OXPHOS@HeikoS: there?18:20
@HeikoSOXPHOS: hi yes18:20
@HeikoSgood to see you here!18:20
@HeikoSOXPHOS: was just thinking about attempting to rebase your branch18:20
OXPHOShi! :)18:20
@HeikoSOXPHOS: I thought we should have a little plan of what is left to merge18:21
@HeikoSOXPHOS: you know these new github project things?18:21
@HeikoShttps://github.com/shogun-toolbox/shogun/projects18:21
@HeikoScould have one for the linalg merge in there18:21
OXPHOS@HeikoS: cool! Never used it before18:23
OXPHOSA few things I want to confirm with you:18:23
@HeikoSOXPHOS: its basically a sticky note todo list18:23
OXPHOSthat's good enough18:24
@HeikoSwould be goo dif you could opulate that then we all have an idea of hwats missing18:24
@HeikoSOXPHOS: shoot18:24
OXPHOS1. I'll remove the bool support of the linalg::sum methods since Eigen no longer supports bool addition18:24
@HeikoSOXPHOS: thats fine!18:25
@HeikoSdoesnt make sense anyways18:26
OXPHOS2. Cholesky works well with all versions of Eigen on my machine. So I guess it's not a Eigen version issue18:28
OXPHOS@HeikoS: copy that18:28
OXPHOSSo I prefer to disable it18:28
@HeikoSOXPHOS: the test you mean?18:28
OXPHOSyes18:28
@HeikoSOXPHOS: you know the eigen version of travis?18:28
OXPHOS3.2.0-818:28
@HeikoSand that works on your machine?18:28
OXPHOSyes18:28
@HeikoSuff18:28
@HeikoSweird!18:28
@HeikoSand you could not reproduce the travis error, even when running the same docker image?18:29
OXPHOSI did see the same error in Docker18:29
OXPHOSon my machine18:29
@HeikoSah ok18:29
-!- mikeling [uid89706@gateway/web/irccloud.com/x-xiyrisaihwgoizbh] has quit [Quit: Connection closed for inactivity]18:29
OXPHOSI traced back to Eigen. So TriangularView.solve() returns inf instead of correct values18:30
@HeikoSbut the same eigen3 version on your own machine doesnt do that18:30
OXPHOSnope18:30
@HeikoSok then. I suggest you disable the unit test, and then put it on your todo-before-merge list18:31
@HeikoSso we can merge the PR I guess?18:31
@HeikoSsorry I didnt get that it wasnt the eigen version18:31
@HeikoSOXPHOS: shall I merge it?18:32
OXPHOS@HeikoS: It's ready to merge on my end18:32
@sukeyPull Request #3534 "LinalgRefactor - Cholesky - CPU only"  merged by karlnapf - https://github.com/shogun-toolbox/shogun/pull/353418:33
@sukeyNew Commit "Merge pull request #3534 from OXPHOS/linalg_cholesky18:33
@sukeyLinalgRefactor - Cholesky - CPU only" to shogun-toolbox/shogun by karlnapf: https://github.com/shogun-toolbox/shogun/commit/c1d17d9e7173683de3b8f9612d5871099be843b218:33
OXPHOSyep it's nice to have an organized to-do list18:33
@HeikoSOXPHOS: I just realised you cannot edit the projects page on github18:33
@HeikoSOXPHOS: so whats missing?18:33
@HeikoSlambday: you still around?18:46
OXPHOSI feel like for the users who keep calling to_gpu() without gpu backends worth tons of warnings18:47
@lambdayHeikoS: yes.. arranging food..18:47
@HeikoSOXPHOS: yes if you give it some love, then these projects look very nice18:47
@HeikoSlambday: I think for projects, the best columns is:18:47
@HeikoS"TODO, TODO optional, in progress, DONE,18:47
@HeikoSjust realised thats mostly aligned with this trello thing where you move things from left to right18:47
@HeikoSOXPHOS: oh I agree18:48
@lambdayHeikoS: can't "milestones" do that?18:48
@HeikoSlambday: they are for issues18:48
@HeikoSbut yes also18:48
@HeikoSmore like the "in progress" thing is good to hjvae18:48
@lambdayHeikoS: sounds exactly like JIRA :D18:49
@HeikoSOXPHOS: I was just thinking of running a GPU implementation in Shogun  in a test or so18:49
@HeikoSOXPHOS: but actually you are right18:49
@lambdayOXPHOS: yo18:49
@HeikoSOXPHOS: they deserve :)18:49
OXPHOS@lambday hi there!18:49
@lambdayOXPHOS: you're a superhero :)18:49
OXPHOS@lambday why is that : )18:50
@lambdayOXPHOS: nice job with linalg :)18:50
OXPHOS@lambday thanks!18:50
@lambdayalthough I haven't had a look lately... but it surely is nice18:50
@HeikoSOXPHOS: btw, we can also add a project for the cereal stuff ... but thats after we merged18:50
OXPHOS@HeikoS: sure18:50
OXPHOS@HeikoS: I was thinking some devs may switch the backend on and off without muting to_gpu() everytime, we can offer them the -DENABLE_GPU_WARNING18:51
OXPHOS@lambday yep haven't heard from you for a while18:51
@HeikoSOXPHOS: yep sounds good18:52
@lambdayOXPHOS: yeah, had some personal commitments :) I'm back to shogun only from this week18:53
OXPHOS@lambday glad to see you back!18:55
OXPHOSI wasn't too engaged either. Slower than I thought18:55
@HeikoSsteady is more important that speedy!!18:56
OXPHOS:D18:57
OXPHOSheading for lunch18:59
OXPHOSsee u around18:59
@HeikoSsee you OXPHOS19:02
@lambdayHeikoS: I am going off for tonight.. will be back tomorrow..19:17
@HeikoSlambday: cool see you tomorrow!19:17
@lambdaysee you :)19:17
-!- lambday [6a3317ac@gateway/web/freenode/ip.106.51.23.172] has quit []19:17
-!- HeikoS_mobile [~Mutter@82-132-219-141.dab.02.net] has joined #shogun19:29
-!- HeikoS_mobile [~Mutter@82-132-219-141.dab.02.net] has quit [Client Quit]19:31
-!- HeikoS [~heiko@untrust-out.swc.ucl.ac.uk] has quit [Ping timeout: 240 seconds]19:39
-!- kmcquisten [d8338942@gateway/web/freenode/session] has joined #shogun21:54
-!- kmcquisten [d8338942@gateway/web/freenode/session] has quit [Changing host]21:54
-!- kmcquisten [d8338942@gateway/web/freenode/ip.216.51.137.66] has joined #shogun21:54
kmcquistenHey all.  Hope you're well today.21:54
kmcquistenI was wondering if someone was around to discuss a bit of difficulty I'm having.21:55
-!- OXPHOS [92bd305b@gateway/web/freenode/ip.146.189.48.91] has quit [Ping timeout: 260 seconds]22:01
kmcquistenAnyone here?22:22
-!- kmcquisten [d8338942@gateway/web/freenode/ip.216.51.137.66] has quit [Quit: Page closed]22:33
-!- suhas2go [uid201652@gateway/web/irccloud.com/x-qtzwrunazcwxdoow] has joined #shogun23:37
--- Log closed Sat Feb 25 00:00:39 2017
« Thursday, 2017-02-23 Index Saturday, 2017-02-25 »

Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!