| --- Log opened Sat Mar 17 00:00:44 2018 | ||
| -!- Farouk [81617d04@gateway/web/freenode/ip.129.97.125.4] has quit [Quit: Page closed] | 00:05 | |
| -!- witness [uid10044@gateway/web/irccloud.com/x-xddnldgkqtduxlsc] has quit [Quit: Connection closed for inactivity] | 00:53 | |
| @sukey | [https://github.com/shogun-toolbox/shogun] Pull Request https://github.com/shogun-toolbox/shogun/pull/4208 opened by FaroukY | 06:12 |
|---|---|---|
| -!- Farouk [81617d04@gateway/web/freenode/ip.129.97.125.4] has joined #shogun | 06:13 | |
| -!- Farouk [81617d04@gateway/web/freenode/ip.129.97.125.4] has quit [Client Quit] | 06:18 | |
| -!- Jinquan [72d455ad@gateway/web/freenode/ip.114.212.85.173] has joined #shogun | 08:02 | |
| -!- Jinquan [72d455ad@gateway/web/freenode/ip.114.212.85.173] has quit [Client Quit] | 08:04 | |
| -!- tctara [~quassel@irc.redcorelinux.org] has quit [Ping timeout: 240 seconds] | 10:28 | |
| @wiking | lisitsyn, ping? | 11:11 |
| lisitsyn | wiking: hey | 11:12 |
| @wiking | i have some questions | 11:12 |
| lisitsyn | sure | 11:12 |
| @wiking | so we were talking with Heiko about coreml and implementing an executore | 11:12 |
| @wiking | *executor for that | 11:12 |
| @wiking | so that we can use coreml exported models easily in shogun | 11:12 |
| lisitsyn | aha | 11:12 |
| @wiking | so i was wondering what do you think about how would you optimize the generated model? | 11:13 |
| @wiking | meaning you know that it can define a pipeling | 11:13 |
| @wiking | *pipeline | 11:13 |
| @wiking | and of course we could do simply just a vanilla implemenation | 11:13 |
| @wiking | where you implement all those modules in coreml | 11:14 |
| @wiking | and apply on the features 1-by-1 | 11:14 |
| lisitsyn | optimize like train with warm start? | 11:14 |
| @wiking | but i guess the reason why CoreML api has compileModel | 11:14 |
| @wiking | is that they actually can do optimization over the typeline | 11:14 |
| @wiking | that way there's no temporary values etc | 11:14 |
| @wiking | see what i mean? | 11:14 |
| @wiking | https://developer.apple.com/documentation/coreml/mlmodel/2921516-compilemodel | 11:15 |
| lisitsyn | sounds like apple specific | 11:15 |
| @wiking | i mean as a beginning of course we can have the whole thing vanilla | 11:15 |
| @wiking | but i guess later on it would be good to be able to do this | 11:15 |
| lisitsyn | I think coreml is just a transport for models | 11:15 |
| @wiking | that of course means that we have a sort of JIT capability | 11:16 |
| lisitsyn | and we just convert it into our vanilla models | 11:16 |
| @wiking | yeye | 11:16 |
| lisitsyn | I mean if it is a linear model we should just fill the weights | 11:16 |
| @wiking | i mean now i'm trying to finish up the mini example | 11:16 |
| @wiking | where i've trained scikit-learn LR | 11:16 |
| @wiking | on boston housing dataset | 11:16 |
| @wiking | it's a simple regression model | 11:16 |
| @wiking | it consist of 2 stages | 11:16 |
| @wiking | a) featurevectorizer | 11:16 |
| @wiking | b) glm regressor | 11:17 |
| lisitsyn | ah | 11:17 |
| lisitsyn | this thing | 11:17 |
| -!- tctara [~quassel@irc.redcorelinux.org] has joined #shogun | 11:17 | |
| @wiking | anyhow | 11:18 |
| lisitsyn | I guess then we load(somefile) | 11:18 |
| lisitsyn | and it returns a set of objects | 11:18 |
| @wiking | do we want that actually? | 11:18 |
| lisitsyn | preprocessor, machine, blabla | 11:18 |
| @wiking | i mean do we actually wanna expose the model? | 11:18 |
| lisitsyn | expose like? | 11:18 |
| @wiking | i thought this to be the simply api | 11:18 |
| @wiking | namespace nobunaga | 11:19 |
| @wiking | { | 11:19 |
| @wiking | typedef std::shared_ptr<arrow::Table> Features; | 11:19 |
| @wiking | class Model | 11:19 |
| @wiking | { | 11:19 |
| @wiking | public: | 11:19 |
| @wiking | explicit Model(std::istream& is); | 11:19 |
| @wiking | ~Model(); | 11:19 |
| @wiking | void predict(Features f) const; | 11:19 |
| @wiking | private: | 11:19 |
| @wiking | class Self; | 11:19 |
| @wiking | std::unique_ptr<Self> m_self; | 11:19 |
| @wiking | }; | 11:19 |
| @wiking | } | 11:19 |
| @wiking | this is all you can have | 11:19 |
| @wiking | note the predict will need a return value | 11:19 |
| @wiking | but that'll be handled later | 11:19 |
| @wiking | but the idea is this | 11:19 |
| @wiking | and note that we use arrow for features | 11:19 |
| @wiking | (the reason for that that way we can easily solve the problem of GPU vs CPU operations.... arrow has GPU/CUDA memorybackend) | 11:20 |
| lisitsyn | isn't it ArrowFeatures? | 11:20 |
| lisitsyn | I mean ArrowFeatures is an instance of Features | 11:20 |
| lisitsyn | I think the most straightforward is to map coreml objects into shogun objects | 11:21 |
| lisitsyn | I mean I'd treat coreml as just another transport for models | 11:23 |
| @wiking | mmm | 11:24 |
| @wiking | the reason i dont want to | 11:24 |
| @wiking | tie it to shogun is because if you just wanna have an executor | 11:24 |
| @wiking | then you dont need shogun | 11:24 |
| @wiking | we have waaaaaay tooo many stuff there | 11:24 |
| @wiking | whereas this is jsut some very simple linalg | 11:24 |
| @wiking | over arrays | 11:24 |
| @wiking | right? | 11:24 |
| @wiking | we dont need the whole linalg abstraction | 11:24 |
| @wiking | we dont need GMM etc etc | 11:24 |
| @wiking | for being able to run a pretrained coreml model | 11:25 |
| lisitsyn | ok then it might be that we need both | 11:25 |
| @wiking | this should be a lightweight package that can be fully used within shogun | 11:25 |
| lisitsyn | I mean having 'executor' of models | 11:25 |
| lisitsyn | and being able to load coreml model | 11:25 |
| @wiking | + of course being able to export | 11:25 |
| @wiking | eyey | 11:25 |
| @wiking | that's another story | 11:25 |
| @wiking | first it'd be great | 11:25 |
| lisitsyn | it could be that | 11:25 |
| @wiking | if we could just use within shogun | 11:25 |
| @wiking | tf, xgboost, sklearn, catboost models | 11:26 |
| lisitsyn | yeap | 11:26 |
| @wiking | (as all of them can be exported into coreml) | 11:26 |
| @wiking | i mean i'm happy to go and do the vanilla implementation | 11:26 |
| @wiking | the question is of course | 11:26 |
| @wiking | i'll try to push the GLM | 11:26 |
| @wiking | part that 'works' | 11:27 |
| @wiking | so we can iterate on the code | 11:27 |
| @wiking | and not just talk in the air | 11:27 |
| lisitsyn | ok sounds good | 11:27 |
| @wiking | but i was wondering | 11:27 |
| @wiking | how would you solve the fact | 11:27 |
| @wiking | of being able to "JIT" it | 11:27 |
| @wiking | meaning that convert into a more optimized version | 11:27 |
| @wiking | once you have the pipeline | 11:27 |
| @wiking | as the coreml pipeline is just a simple vanilla descriptor | 11:27 |
| @wiking | of the elements | 11:27 |
| @wiking | using protobuf | 11:28 |
| lisitsyn | not sure if there is a lot of room to optimize | 11:28 |
| lisitsyn | I mean it is obviously suboptimal to work over protobuf instance | 11:28 |
| @wiking | yeah we wouldn't work | 11:28 |
| lisitsyn | but once you load into some internal representation | 11:28 |
| lisitsyn | like array<float> blabla | 11:28 |
| @wiking | we just use it for our internal mapping | 11:28 |
| @wiking | but i wouldn't expose it | 11:28 |
| @wiking | see Self | 11:28 |
| lisitsyn | not sure what has to be optimized | 11:28 |
| lisitsyn | yeah sure | 11:28 |
| lisitsyn | I mean what're the things to be optimized? | 11:29 |
| @wiking | say that you have a normalization | 11:29 |
| @wiking | before the glm | 11:29 |
| @wiking | ok? | 11:29 |
| @wiking | that is | 11:29 |
| @wiking | g(f(x)) | 11:29 |
| @wiking | right | 11:29 |
| @wiking | ? | 11:29 |
| lisitsyn | so you mean combine them into one function? | 11:29 |
| @wiking | yes | 11:29 |
| @wiking | as you can do it | 11:29 |
| lisitsyn | uh | 11:29 |
| lisitsyn | don't know | 11:29 |
| lisitsyn | llvm yolo | 11:29 |
| @wiking | yes | 11:29 |
| @wiking | it is | 11:29 |
| @wiking | i mean this is future work of course | 11:30 |
| @wiking | question is what are the constraints for this | 11:30 |
| @wiking | in a long run | 11:30 |
| lisitsyn | I haven't touched this thing but heard some success stories | 11:30 |
| @wiking | so that we dont come with a design | 11:30 |
| lisitsyn | I see your concern | 11:30 |
| @wiking | that is totally not letting you to do this | 11:30 |
| @wiking | later | 11:30 |
| @wiking | i mean | 11:30 |
| @wiking | honestly | 11:30 |
| @wiking | there's the story of xtensor | 11:30 |
| @wiking | and eigen | 11:30 |
| lisitsyn | yeah I see | 11:30 |
| @wiking | in both case you can do | 11:30 |
| @wiking | g(f(a(....) | 11:31 |
| @wiking | and then that is only executed | 11:31 |
| @wiking | once there's a 'trigger' | 11:31 |
| @wiking | .execute() | 11:31 |
| @wiking | or whatever | 11:31 |
| @wiking | i mean we dont need to go with this design right away | 11:31 |
| @wiking | i'm just wondering what are the constraints in this case | 11:31 |
| @wiking | so that we go with the right approach | 11:31 |
| @wiking | that enables this later | 11:31 |
| lisitsyn | ok let me think | 11:32 |
| @wiking | i mean i guess | 11:32 |
| @wiking | then we need to go header only internally :P | 11:32 |
| lisitsyn | I think that has to be in pipeline | 11:32 |
| lisitsyn | so once we have some Pipeline | 11:32 |
| lisitsyn | we could have some Pipeline::compile | 11:32 |
| lisitsyn | or Pipeline::optimize | 11:32 |
| lisitsyn | that actually tries to optimize it | 11:33 |
| @wiking | https://github.com/QuantStack/xtensor/blob/master/include/xtensor/xexpression.hpp | 11:33 |
| lisitsyn | once you call it you don't have access to your models | 11:33 |
| @wiking | yeah indeed | 11:33 |
| @wiking | once you compile it | 11:33 |
| @wiking | it's just a linalg stack | 11:33 |
| @wiking | that does operations over the input | 11:33 |
| lisitsyn | and I wouldn't even bother exporting them back | 11:33 |
| @wiking | what do you mean by exporting them back? | 11:33 |
| lisitsyn | I mean it is one-way ticket | 11:34 |
| @wiking | ah yeah | 11:34 |
| @wiking | indeed | 11:34 |
| lisitsyn | you compile and then you can't analyze them | 11:34 |
| lisitsyn | check the weights | 11:34 |
| @wiking | you cannot reverse it | 11:34 |
| lisitsyn | no training anymore | 11:34 |
| lisitsyn | :) | 11:34 |
| @wiking | yeye totally | 11:34 |
| lisitsyn | such a design would not restrict (probably) | 11:35 |
| @wiking | but i mean even in case of none-optimized version | 11:35 |
| @wiking | do you want to have the user | 11:35 |
| @wiking | to have a reference | 11:35 |
| @wiking | on the models etc? | 11:35 |
| @wiking | (and their params?) | 11:35 |
| lisitsyn | in case of coreml? | 11:35 |
| @wiking | see apple's design is | 11:35 |
| @wiking | that once you exported into the protobuf coreml format | 11:35 |
| lisitsyn | I think it would be very nice thing to have | 11:35 |
| @wiking | you just can deserialize | 11:36 |
| @wiking | and apply the model | 11:36 |
| @wiking | you cannot 'observe' nor change anything | 11:36 |
| lisitsyn | but this doesn't have to be in the executor | 11:36 |
| lisitsyn | so not for executor but for IO with coreml protos | 11:36 |
| lisitsyn | but in general | 11:36 |
| lisitsyn | it would be a nice thing to have even in executor | 11:36 |
| lisitsyn | I mean in production you actually want to monitor things | 11:37 |
| lisitsyn | I mean I'd put a few graphs like weights norm, maximal weight, minimal weight etc | 11:37 |
| @wiking | mmm | 11:38 |
| lisitsyn | and if you can't access that in shogun you'd need some code to access that somewhere else | 11:38 |
| @wiking | yeah we can add these options | 11:38 |
| @wiking | i'm just wondeirng | 11:38 |
| @wiking | whether then we should actually | 11:38 |
| @wiking | expose the protobuf objects? | 11:39 |
| @wiking | i wouldn't do it personally | 11:39 |
| @wiking | but essentially you could do it :) | 11:39 |
| lisitsyn | I am not sure it is protobuf objects | 11:39 |
| lisitsyn | it is rather shogun objects or so | 11:40 |
| @wiking | what i meant | 11:40 |
| @wiking | that the easiest way to expose any param | 11:40 |
| @wiking | is to expose the protobuf objects | 11:40 |
| @wiking | :) | 11:40 |
| lisitsyn | yeah | 11:40 |
| @wiking | as they essentially contain all the info | 11:40 |
| lisitsyn | ok it might be | 11:41 |
| lisitsyn | that | 11:41 |
| lisitsyn | coreml protos -> (load) shogun objects -> (compile) optimized pipeline | 11:41 |
| @wiking | mmm | 11:42 |
| lisitsyn | would be the way | 11:42 |
| @wiking | yeah but | 11:42 |
| @wiking | i dont see why it's good to use shogun object | 11:42 |
| @wiking | s | 11:42 |
| @wiking | they are super clunky | 11:42 |
| @wiking | and way too much overhead | 11:42 |
| @wiking | for a simple executore | 11:42 |
| @wiking | right? | 11:42 |
| lisitsyn | to let users re-train them etc | 11:42 |
| @wiking | yeah but you can do that | 11:42 |
| @wiking | in the place | 11:42 |
| @wiking | where you did it already | 11:42 |
| @wiking | :) | 11:42 |
| lisitsyn | so you mean it should be two things | 11:42 |
| lisitsyn | 1) coreml proto -> shogun objects | 11:43 |
| lisitsyn | 2) coreml proto -> optimized pipeline | 11:43 |
| lisitsyn | and third one is actually | 11:43 |
| @wiking | yeah | 11:43 |
| lisitsyn | shogun objects -> optimized pipeline | 11:43 |
| @wiking | yeye | 11:43 |
| @wiking | first i would do 2) | 11:43 |
| @wiking | then 3) | 11:43 |
| @wiking | and then 1) | 11:43 |
| lisitsyn | sounds good also | 11:43 |
| @wiking | with 2) we could integrate into shogun easily | 11:43 |
| @wiking | so we can use models | 11:43 |
| @wiking | from other libs | 11:43 |
| @wiking | 3) we want to be able to serialize our models :P | 11:44 |
| @wiking | and the 2) would be actually using all our infra | 11:44 |
| @wiking | for how to make this multilang | 11:44 |
| @wiking | because we could make the whole thing like shogun | 11:44 |
| @wiking | that you can actually use from all the swigable languages | 11:44 |
| lisitsyn | sounds good | 11:45 |
| @wiking | with arrow you have zero copy | 11:45 |
| @wiking | and then you could actually run a coreml | 11:45 |
| @wiking | model in jvm | 11:45 |
| @wiking | or python | 11:45 |
| @wiking | or ruby | 11:45 |
| @wiking | on none-apple systems as well | 11:45 |
| @wiking | imo that'd be good | 11:45 |
| lisitsyn | yeah I agree | 11:46 |
| lisitsyn | sounds reasonable | 11:46 |
| @wiking | ok so i'll try to finish this simple glm object | 11:46 |
| @wiking | and push it | 11:46 |
| @wiking | and see what you think about it | 11:46 |
| @wiking | and then slowly we can actually do this for all the things in the coreml definition | 11:46 |
| lisitsyn | I wonder if google is to come with similar definitions for android | 11:50 |
| lisitsyn | they still didn't release anything like that | 11:51 |
| @wiking | i mean they have serving | 11:51 |
| lisitsyn | yeah but it is a bit bs | 11:51 |
| @wiking | it is in a way similar format | 11:51 |
| lisitsyn | I mean tf is not the only way to serve models | 11:51 |
| @wiking | but has an grpc stack over it | 11:51 |
| lisitsyn | it doesn't solve the problem of exporting a model from your jupyter lab notebook | 11:52 |
| lisitsyn | that's why I wonder | 11:52 |
| @wiking | yeah | 11:52 |
| @wiking | good question | 11:52 |
| @wiking | https://github.com/tf-coreml/tf-coreml | 11:52 |
| @wiking | :) | 11:52 |
| @wiking | but there's already keras -> coreml | 11:53 |
| lisitsyn | yeah true | 11:53 |
| lisitsyn | but no similar executor in android still! | 11:53 |
| lisitsyn | or not? | 11:53 |
| @wiking | dunno | 11:53 |
| @wiking | have not heard about it | 11:54 |
| @wiking | https://www.quora.com/What-is-the-counterpart-of-Apples-CoreML-in-Android | 11:54 |
| @wiking | https://www.quora.com/What-is-the-counterpart-of-Apples-CoreML-in-Android;) | 11:54 |
| @wiking | :) | 11:54 |
| lisitsyn | yeah some tf lite but that's bs | 11:54 |
| lisitsyn | :) | 11:54 |
| lisitsyn | crazy | 11:54 |
| @wiking | https://techcrunch.com/2017/05/17/googles-tensorflow-lite-brings-machine-learning-to-android-devices/ | 11:54 |
| @wiking | :) | 11:54 |
| lisitsyn | and nobody started a startup doing this?! | 11:55 |
| lisitsyn | crazy | 11:55 |
| @wiking | :)))))))))))))) | 11:55 |
| lisitsyn | the idea is so obvious I could cry | 11:55 |
| @wiking | raise 10M! | 11:55 |
| lisitsyn | I bet you get 100M from nowhere | 11:55 |
| @wiking | yeah let's raise it then | 11:55 |
| @wiking | nobunaga :) | 11:55 |
| lisitsyn | billions of android devices and no general solution | 11:56 |
| @wiking | anyhow lemme see how this works with arrow | 11:56 |
| @wiking | and a simple weight vector :) | 11:56 |
| -!- baladinha_top [55da31af@gateway/web/freenode/ip.85.218.49.175] has joined #shogun | 12:08 | |
| -!- baladinha_top [55da31af@gateway/web/freenode/ip.85.218.49.175] has quit [Quit: Page closed] | 12:16 | |
| -!- Farouk [81617d04@gateway/web/freenode/ip.129.97.125.4] has joined #shogun | 16:17 | |
| Farouk | Hi everyone. So I just wanted to finalize my idea for the GSOC project. So I want it to be a two parts project. The first part would be to add to the NN component of Shogun. Right now, Shogun only supports training according to the Minimum squared error loss function, by hard coding the actual derivatives and the back propagation algorithm. | 16:25 |
| Farouk | I want to change that to use some auto differentiation library like Adept that would allow to define an arbitrary error function and then automatically differentiate it and back propagate. We can also add some extra ready made functions like categorical cross entropy, etc. | 16:26 |
| Farouk | The second part of the project would be to use the new improved NN features to build a NN that plays an Atari game from scratch. That would be a notebook and can showcase the new features of the NN component. | 16:26 |
| Farouk | So any feedback? | 16:26 |
| @wiking | Farouk sounds good | 16:38 |
| @wiking | make sure that you have detailed weekly schedule | 16:39 |
| @wiking | for those weeks of gsoc | 16:40 |
| @wiking | ands till then u have couple of prs already merged (by mid april( | 16:40 |
| Farouk | Sure. No Problem. I have 2 pull requests in Review right now. | 16:41 |
| @wiking | regardung | 16:41 |
| @wiking | autodiff | 16:41 |
| @wiking | there were couple of discussions about this.... we were looking into stan | 16:41 |
| Farouk | Is that a new autodiff library? I am open to any one really, It's just I worked a bit before on Adept before so had more experience in it. But I think it shouldn't be too hard to switch to stan? | 16:43 |
| @wiking | well | 16:43 |
| @wiking | the problem is that autodiff in general would only make sense with wrapping the code | 16:43 |
| @wiking | because we wanna avoid | 16:44 |
| @wiking | direct dependencies | 16:44 |
| @wiking | as what if one of the lib stops being developed | 16:44 |
| Farouk | Ahh I see. | 16:44 |
| Farouk | So any additional new library would need to be wrapped first | 16:45 |
| @wiking | ideally :) | 16:45 |
| Farouk | Hmm so how is the discussion on Stan going. Any updates on it? | 16:46 |
| Farouk | Ohh i see that stan is a header-only library. That seems convenient. Had a look at some examples and they look similar to Adept. | 17:02 |
| @wiking | Farouk, yeah stan is pretty cool library :) | 18:05 |
| Farouk | So is it okay to assume that I can insert the stan library into my program, or is that still under discussion? | 18:15 |
| -!- iglesias [~iglesias@f119189.upc-f.chello.nl] has joined #shogun | 18:16 | |
| -!- syashakash [0e8becd2@gateway/web/freenode/ip.14.139.236.210] has joined #shogun | 18:26 | |
| @wiking | Farouk, in a way yes | 18:30 |
| @wiking | it's not an easy task :) | 18:30 |
| @wiking | but yeah | 18:30 |
| @wiking | let's have the assumption that it can be done | 18:30 |
| Farouk | Okay great then. Thanks for the feedback and the information :) | 18:31 |
| -!- syashakash [0e8becd2@gateway/web/freenode/ip.14.139.236.210] has quit [Quit: Page closed] | 18:49 | |
| -!- syashakash [0e8becd2@gateway/web/freenode/ip.14.139.236.210] has joined #shogun | 18:50 | |
| -!- HeikoS [~heiko@host86-132-201-109.range86-132.btcentralplus.com] has joined #shogun | 20:01 | |
| -!- mode/#shogun [+o HeikoS] by ChanServ | 20:01 | |
| -!- HeikoS [~heiko@host86-132-201-109.range86-132.btcentralplus.com] has quit [Ping timeout: 260 seconds] | 20:27 | |
| -!- HeikoS [~heiko@host86-132-201-109.range86-132.btcentralplus.com] has joined #shogun | 20:29 | |
| -!- mode/#shogun [+o HeikoS] by ChanServ | 20:29 | |
| -!- Farouk [81617d04@gateway/web/freenode/ip.129.97.125.4] has quit [Quit: Page closed] | 21:43 | |
| -!- HeikoS [~heiko@host86-132-201-109.range86-132.btcentralplus.com] has quit [Ping timeout: 260 seconds] | 22:22 | |
| -!- iglesias [~iglesias@f119189.upc-f.chello.nl] has quit [Quit: leaving] | 23:09 | |
| -!- HeikoS [~heiko@host86-132-201-109.range86-132.btcentralplus.com] has joined #shogun | 23:25 | |
| -!- mode/#shogun [+o HeikoS] by ChanServ | 23:26 | |
| --- Log closed Sun Mar 18 00:00:46 2018 | ||
Generated by irclog2html.py 2.10.0 by Marius Gedminas - find it at mg.pov.lt!