All right. 1
So we've been talking about all the fantastic things that you can do with machine learning, 2
but now it's time to get hands on with CoreML. 3
So what exactly is CoreML and why is it so exciting? 4
So CoreML essentially allows you to do two things that makes it easy to integrate machine learning 5
into your IOs project. 6
So the first thing is that it alows you to load a pre-trained machine learning model and they include 7
an easy way of converting your pre-trained model, whether if it was trained on Caffe or Keras, or wherever 8
it may be. You can take that model and convert it into a class that can be used within Xcode. 9
For those of you who are already familiar with machine learning, your pre-trained model gets converted into 10
a .mlmodel file which is a open-file format, and includes all of your input-output layers, as well 11
as all your training weights. 12
Now, the second thing that CoreML allows you to do is to make predictions to have the model loaded 13
up locally on the device 14
once the user download your app and your app is able to use it to make the predictions, be it 15
image recognition, speech recognition, or whatever else it is that you trained your model to do. 16
Now, if you go to developer.apple.com/machine-learning, you can see that they compiled a 17
website that includes an overview on how you can use machine learning in your iOS apps. 18
But the most interesting part of this website are the ready-to-use models down here. 19
So these are essentially plug-and-play models which already been pre-trained. But the one that we're 20
going to be using in our project is this one from Google which is Inception v3 and it's been trained 21
on over a thousand categories on loads and loads of images, and it's learned to recognize these images 22
and be able to classify them. 23
So we're going to be using this to identify what our user takes a photo of in the app and we're going 24
to check to see if the classification was a Hotdog or if it was not a Hotdog. 25
Now, in WWDC, they announced all of the things that CoreML is able to do, but essentially, it boils 26
down to just two things. 27
It's able to do classification or regression. 28
And at the moment, it's pretty much limited to just those two types of tasks, but I'm sure in the future 29
this will be extended. 30
And besides in most cases, when you're implementing machine learning, you'll only be using classification 31
or regression anyways. 32
Now, we've spoken about some of the things that CoreML can do, but one of the most important things that 33
it can do is that you're not able to use any of your app data or user generated data while they're using 34
your app to train the model further. 35
So the model is loaded up as a pre-trained model and it stays as it is. So static. 36
And although there are workarounds in most cases, your model will just stay as it is and you'll only 37
be able to update it when you push a new update to your app. 38
So for the moment, anyways, you won't be able to use the user-generated data in the app to train the model 39
on the fly. 40
Now, the third thing is that it's not encrypted. 41
So if you're using a proprietary model or if anything within your package contains sensitive data, then 42
you have to be aware that none of this is encrypted. And in fact, when you convert your pre-trained model 43
into a .mlmodel file, if you have a look within the file structure, you'll actually see, in most 44
cases, it's split into a whole bunch of JSONS that contain the training weights and the inputs and the 45
outputs, et cetera. 46
So be aware that if you have, say, a financial model that has a proprietary way of predicting the stock 47
market, for example, you might not want to use it with CoreML just because you might be giving your 48
secret source away for anyone who cares to look into the source code of your app. 49
So those are some of the limitations to CoreML. 50
But remember that the most exciting thing about CoreML is that it just works out of the package 51
like many other Apple products. 52
And even if you don't know a whole lot about machine learning, you can use it straight away without a 53
lot of code and without a lot of complexity. 54
And that's exactly what we're going to show you in this tutorial. 55
So you'll be able to take something like a pre-trained model on recognizing plants and build it into 56
your app and start feeding photos to it and they'll be able to send you outputs that you can then display 57
to the user all without you having to understand anything that's happening in the background. 58
So, for example, one of my favorite apps is called Garden Answers and it's really cool because you can 59
take a photo of a plant and it's able to classify that plant based on the way that the flower or the 60
leaf looks. 61
Now, my one gripe with it though is that it requires an internet connection. 62
So very often when you're wanting to identify plants, you're either abroad or you're somewhere without 63
internet connection, 64
you know, really in the sticks. And at the moment, what they're doing is that they're querying an API and 65
their model exists somewhere in the cloud which they send the image data to, 66
and then they can get an answer back as an interpretation. 67
But with CoreML, you can actually put that entire classifier within your app on your device, so your 68
users won't even need an internet connection 69
and it will be able to do it on the local device. 70
So if you're not using one of the ready-to-use models that Apple has provided on their website, then 71
you'll need to convert your pre-trained model into a .mlmodel file, and Apple provides a relatively 72
easy way of doing that for all of the major frameworks, including Caffe, including Keras scikit-learn, et cetera. 73
One of the most notable ones are missing from this, however, is TensorFlow which is something I really 74
like. But there are open-source converters out there and it's actually not very difficult to convert 75
your TensorFlow into a mlmodel file either. 76
So, essentially, what I'm trying to say is that there's two routes for you to go down using CoreML. 77
One is if you're a complete beginner to machine learning, then you can just use one of the plug-and-play 78
models and you can start building it into your app with relative ease. 79
Now, if you're somebody who is very serious about machine learning and you've already got your model 80
train on Keras, or train on Caffe, wherever it may be, you can then convert it into an mlmodel file being 81
aware of the limitations of CoreML and implementing it in your app relatively easily as well. 82
So enough talking. We've spoken a lot about CoreML and machine learning. 83
It's really time to get stuck in and start creating our very own CoreML app. So I'll see you on the next 84
lesson.