IRC logs of #shogun for Friday, 2017-05-26

« Thursday, 2017-05-25 Index Saturday, 2017-05-27 »
--- Log opened Fri May 26 00:00:46 2017
-!- mikeling [uid89706@gateway/web/irccloud.com/x-ddqbkongnepvwlue] has joined #shogun01:31
@sukeyPull Request #3810 "Port kernelMachine to openmp"  synchronized by MikeLing - https://github.com/shogun-toolbox/shogun/pull/381004:11
mikelingwiking: Hi, how could I specify the tests can run with mutithread? Or I should add some codes to test the mutithread situation07:12
mikeling* specify the tests run with mutithread07:15
mikelingsorry :/07:15
-!- iglesiasg [~iglesiasg@217.119.234.214] has joined #shogun09:13
-!- mode/#shogun [+o iglesiasg] by ChanServ09:13
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has joined #shogun09:30
-!- mode/#shogun [+o HeikoS] by ChanServ09:30
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has quit [Ping timeout: 240 seconds]09:51
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has joined #shogun09:53
-!- mode/#shogun [+o HeikoS] by ChanServ09:53
@sukeyWiki page: Heiko-Strathmann edited on shogun-toolbox/shogun by karlnapf10:33
@sukeyWiki page: Heiko-Strathmann edited on shogun-toolbox/shogun by karlnapf10:38
mikelingping HeikoS10:57
mikelingHi, How could I validate the openmp really works?  omp_get_num_threads() always return 1 even I export OMP_NUM_THREADS as 4. Could I force things run with mutithreads?10:59
-!- geektoni [~geektoni@93-34-128-38.ip49.fastwebnet.it] has joined #shogun11:25
@wikingmikeling, hey11:43
@wikingomp_get_num_threads == 1 ?11:43
@wikingeven if OMP_NUM_THREADS=4?11:43
@wikingit's strange11:43
@wikingwhat machine you are testing on?11:44
mikelingwiking: Ubuntu 17.0411:57
mikelingi7 with 4 cores11:58
@wikingok and if you run anythign with openmp11:58
@wikingit returns 1?11:58
@wikingwhat's your compiler?11:58
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has quit [Ping timeout: 268 seconds]11:58
mikelingGcc12:00
lisitsyneh?12:00
lisitsyn:)12:00
mikelingSo?12:00
mikelingDoesn't gcc support openmp?12:01
lisitsynit does 146% sure :P12:02
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has joined #shogun12:18
-!- mode/#shogun [+o HeikoS] by ChanServ12:18
@wikingmikeling, mmm it's weird though12:26
@wikingit should be multithreaded12:26
@wikingcan you try to gdb or debug somehow12:26
mikeling sure, please wait a second12:27
@wikingwhat is the parallel->get_num_threads() value by default12:27
@wikingi cannot believe that we are back to square one12:27
@wikingwith the parallel execution12:27
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has quit [Remote host closed the connection]12:31
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has joined #shogun12:32
-!- mode/#shogun [+o HeikoS] by ChanServ12:32
mikelingwiking: here is the output https://pastebin.mozilla.org/902276112:39
@wikingmikeling, amazing12:40
@wikinglemme debug12:40
mikelingwiking: and parallel->get_num_threads also https://pastebin.mozilla.org/902276012:41
mikelingand here is my compiler information gcc (Ubuntu 6.3.0-12ubuntu2) 6.3.0 2017040612:42
@wikingcrazy12:42
@wikingagaaaaaaaaaaaaaain12:42
@wikingcraaaaaaaaaazy12:42
@wikingthis is develop branch right?12:43
* wiking we need a fucking unit test for this! :)12:43
mikelingno, it's on the kernel_machine_clean_up(with that pr) branch12:44
@wikingyeah but12:44
mikelinglet me try it on develop12:44
@wikingif you try it in a different place12:44
@wikingwhere you get parallel->...12:44
@wikingit should be the same12:44
mikelinggive me a few minutes to try it on develop, just for sure ;)12:45
@wikingmmm12:45
@wikingwait wait12:45
@wikingmmmmmmmmmmmmmmm12:46
@wikingok wait a second :)))))12:46
@wikingnum_threads = omp_get_num_threads();12:46
@wikingis in a12:46
@wiking+#ifdef HAVE_OPENMP12:46
@wiking+#pragma omp single12:46
@wikingsection12:46
@wiking:)))12:46
@wikingmmm12:47
@wiking        #pragma omp single12:47
@wiking        {12:47
@wiking            num_threads=omp_get_num_threads();12:47
@wiking            num_vec=idx_a2-idx_a1+1;12:47
@wiking            step=num_vec/num_threads;12:47
@wiking        }12:47
@wikingis this supposed to return the available threads?12:47
@wikingit should no?12:47
mikeling so, it actually only have one thread been detected?12:48
mikelingyep12:48
mikelingI believe so12:48
@wikingjust compiling12:48
@wikingbut yeah this is actually problematic12:48
@wiking:(((((12:48
mikelingI guess the nature of omp_get_num_threads() is about to get all available threads12:48
@wikingit should work12:48
@wikingbut then again12:48
@wikingit's a shit12:48
@wikingbecause this is not controlled by12:49
@wikingSHOGUN_NUM_THREADS12:49
@wiking:(12:49
@wikingso that needs patching12:49
mikelingwiking: it's on the develop branch https://pastebin.mozilla.org/902276512:50
mikeling:)12:50
@wikinghuh12:50
@wikingok12:50
@wikinggimme some time to debug12:50
@wikingi'll get back to you12:50
@wikingwith an urgent ticket as well12:50
@wikingto get this fixed all over the place12:50
@wiking:)12:50
mikelingok, thank u12:50
@wikingjust have to come up with a soltuion12:51
mikelingplease tell me there are anything I can help12:51
@wikingkk12:52
@wikingi'll do as soon as12:52
@wiking In a sequential section of the program omp_get_num_threads returns 112:58
@wiking:(12:58
@wikingok12:58
@wikingtesting some shit12:58
@wikingmikeling, (gdb) p parallel->get_num_threads()12:58
@wiking$6 = 812:59
@wiking(gdb) p omp_get_num_threads()12:59
@wiking$8 = 112:59
@wiking:)12:59
@wikingok12:59
@wikingwe r fucked :)))012:59
@wikingok i'll come up with a fix12:59
@wikingand then i'll ping u in that commit12:59
CaBahola wiking13:04
@wikingCaBa, servus, wie gehst?13:26
@wiking*gehtst13:26
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has quit [Quit: Leaving.]13:41
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has joined #shogun13:41
-!- mode/#shogun [+o HeikoS] by ChanServ13:41
CaBawiking: jó13:42
CaBawiking: i live in the wrong half of germany for 'servus' though ;)13:44
CaBawiking: how are you?13:44
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has quit [Ping timeout: 245 seconds]13:46
@wikinggood good13:46
@wikingbusy13:46
@wikingas usual13:47
@wikinghow about you13:47
@wiking?13:47
CaBawiking: thesis crisis as usual, but otherwise good as well ;)13:48
@wiking:)13:49
CaBasummer has finally come to berlin \o/13:49
-!- geektoni [~geektoni@93-34-128-38.ip49.fastwebnet.it] has left #shogun []13:57
lisitsynwiking: we don't setup the number of threads in parallel?14:04
@wikinglisitsyn, we do14:04
lisitsynwhat's the bug then?14:05
@wikingmmm14:05
lisitsynwe don't setup OMP number of threads I mean14:05
lisitsyn?14:05
@wikingactually nothing i've realised14:05
@wikingbut we should be using there14:05
lisitsynno?14:05
@wikingparallel14:05
@wikingmikeling, so i've did some debugging ... and in distance machine the setup works14:05
@wikingso i'm wondering why you get =1 for threads in KernelMachine.14:06
@wikingbut we should definitely use another construct14:06
@wikingthan14:06
@wikingnum_threads = omp_get_num_threads();14:06
lisitsynah fck14:07
lisitsynI see14:07
lisitsynyes we don't ever use omp_get_num_threads I gues14:07
lisitsynwe just don't setup it right?14:07
@wikingmmm no14:09
@wikingit should actually work14:09
@wikingParallel::set_num_threads sets14:09
@wikingomp_set_num_threads14:09
lisitsynok then I am missing everything14:09
lisitsynis it happening in tests?14:09
@wikingso it should be consistent14:09
lisitsynfork?14:09
@wikingi'm just not understanding14:09
@wikinghow does it =1 in case of mikeling14:09
lisitsyncan it be that gtest forks and reset it this way?14:10
@wikingnono14:10
@wikingi've just tested it14:10
lisitsynok no idea then14:10
lisitsyn:D14:10
@wikingok it works good14:12
@wikingwith DistanceMachine14:12
@wikingit was my fault in the beginning14:12
@wikingi'll try patching now the develop :)14:12
@wikingand use KernelMachine14:12
@wikinghow do you fetch the patch of a PR?14:13
@wikingmikeling,14:20
@wikingBreakpoint 1, shogun::CKernelMachine::apply_get_outputs () at ../src/shogun/machine/KernelMachine.cpp:34814:20
@wiking348num_threads = omp_get_num_threads();14:20
@wiking(gdb) n14:20
@wiking349step = num_vectors / num_threads;14:20
@wiking(gdb) p num_threads14:20
@wiking$1 = 314:20
mikelingwiking: pong14:20
@wiking(gdb) r --gtest_filter=KRRNystrom.*14:20
@wikingand i started gdb with14:20
@wikingSHOGUN_NUM_THREADS=3 gdb /home/wiking/shogun/build/bin/shogun-unit-test14:20
@wikingif i run without SHOGUN_NUM_THREADS=314:21
mikelingwiking: ok, let me have a try14:21
mikelingcould Iuse SHOGUN_NUM_THREADS when compile it? e.g cmake SHOGUN_NUM_THREADS=3 ../14:22
@wiking(gdb) p num_threads14:22
@wiking$1 = 814:22
mikelingalright14:22
@wikingin case i dont use SHOGUN_NUM_THREADS14:22
@wikingso14:22
@wikingi mean SHOGUN_NUM_THREADS is a runtime var14:22
@wikingyou dont need to compile anything14:22
@wikingjust make sure that cmake detects OpenMP14:22
@wikingi.e. check in your build14:22
@wikingsrc/shogun/lib/config.h14:23
@wiking#define HAVE_OPENMP 114:23
@wikingso it should work14:25
@wikingbtw make sure that you fix the typo in #pragma opm single14:25
mikelinghmmm but the  HAVE_OPENMP do equal to 114:25
@wikingthat should be automatically generated in your14:26
@wikingsrc/shogun/lib/config.h14:26
@wikingi.e. build/src/shogun/lib/config.h14:26
mikelingyeah, I mean I found it in config.h and it equal to one14:26
@wikingcool14:27
@wikingso you hsould have openmp14:27
@wikingso in that case num_threads should equal to your num cores14:27
-!- iglesiasg [~iglesiasg@217.119.234.214] has quit [Ping timeout: 240 seconds]15:00
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has joined #shogun15:01
-!- mode/#shogun [+o HeikoS] by ChanServ15:02
-!- iglesiasg [~iglesiasg@217.119.234.214] has joined #shogun15:03
-!- mode/#shogun [+o iglesiasg] by ChanServ15:03
-!- leagoetz [~leagoetz@x4db43ef6.dyn.telefonica.de] has joined #shogun15:03
-!- iglesiasg [~iglesiasg@217.119.234.214] has quit [Ping timeout: 240 seconds]15:08
-!- iglesiasg [~iglesiasg@217.119.234.214] has joined #shogun15:10
-!- mode/#shogun [+o iglesiasg] by ChanServ15:10
mikelingwiking: ok, I it's mutithreads for KRRNystrom.*, but it = 1 in CrossValidation_multithread.*15:34
@wikingmmm15:34
@wikingi think in case of crossval15:34
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has quit [Ping timeout: 246 seconds]15:34
@wikingit is forced to be 115:34
@wikingimo15:34
@wikingindeed //#pragma omp parallel for15:35
@wikingbecause it's full with bugs :)15:35
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has joined #shogun15:35
-!- mode/#shogun [+o HeikoS] by ChanServ15:35
-!- leagoetz [~leagoetz@x4db43ef6.dyn.telefonica.de] has quit [Remote host closed the connection]15:56
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has quit [Quit: Leaving.]15:57
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has joined #shogun16:06
-!- mode/#shogun [+o HeikoS] by ChanServ16:06
mikelingwiking: I got this error https://pastebin.mozilla.org/902277216:51
mikelingwith valgrind --tool=helgrind  ./bin/shogun-unit-test --gtest_filter=KRRNystrom.*16:51
mikelingis that mean we have data race for CSignal::cancel_computations()?16:51
@iglesiasgit looks like it16:53
@iglesiasgit might be a false positive16:54
-!- TingMiao [uid229534@gateway/web/irccloud.com/x-kxuqzcdhsbnztlow] has joined #shogun17:08
@iglesiasgTingMiao: hey! I just read your e-mail. Did you want to add anything else to point 3? It finishes with "After that, I should"17:15
TingMiaoLet me have a look!17:18
TingMiaoiglesiasg:17:19
TingMiaoiglesiasg: yes I see it is not complete.. sorry about that17:19
@iglesiasgno problem :)17:20
@iglesiasgif you were planning to add anything else, just send a follow-up17:20
TingMiaoiglesiasg: Do you think there is any point that I missed?17:22
@iglesiasgTingMiao: nothing missed, all good!17:25
@iglesiasgTingMiao: regarding point 3, before you start working on a notebook for this "further ideas" part, let's sync again17:27
TingMiaoiglesiasg: Will do that!17:27
@iglesiasgso, once you have the "I should pick up the most related ideas and think about how those ideas can be applied to our project.", write it down (a couple of paragraphs should be enough), we discuss about it, and then we go to the proof of concept notebook17:28
TingMiaookay ;-)17:28
@iglesiasgTingMiao: I am going through the https://github.com/tingpan/shogun-project-demos/blob/master/kmeans.ipynb17:28
@iglesiasgI will make a few comments in github17:29
TingMiaoSure ;-)17:30
TingMiaoiglesiasg: Thank you!17:32
@iglesiasgTingMiao: you are welcome17:33
@iglesiasgTingMiao: do you get notifications and can see them?17:33
TingMiaoiglesiasg: yes, I received the email notification17:34
@iglesiasgTingMiao: great! Let me know if anything is not clear17:37
TingMiaoiglesiasg: Thank you very much! I will definitely contact you if anything is unclear!17:42
-!- Positron_ [~textual@c-73-162-174-23.hsd1.ca.comcast.net] has joined #shogun17:46
-!- Positron_ [~textual@c-73-162-174-23.hsd1.ca.comcast.net] has quit [Client Quit]17:48
@iglesiasgTingMiao: just finished going through the notebook. it is interesting stuff, looking forward to continuing seeing more about the project17:59
@iglesiasgI am off now, ttyl18:00
-!- leagoetz [~leagoetz@x4db43ef6.dyn.telefonica.de] has joined #shogun18:01
-!- leagoetz [~leagoetz@x4db43ef6.dyn.telefonica.de] has quit [Remote host closed the connection]18:06
-!- iglesiasg [~iglesiasg@217.119.234.214] has quit [Quit: leaving]18:09
-!- leagoetz [~leagoetz@x4db43ef6.dyn.telefonica.de] has joined #shogun18:12
-!- leagoetz [~leagoetz@x4db43ef6.dyn.telefonica.de] has quit [Client Quit]18:14
-!- HeikoS [~heiko@x4db43ef6.dyn.telefonica.de] has quit [Ping timeout: 260 seconds]18:50
--- Log closed Sat May 27 00:00:47 2017
« Thursday, 2017-05-25 Index Saturday, 2017-05-27 »

Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!