All right. 1
So, now that we've gotten our UIImagePickerController set up and ready to go, 2
and we've also incorporated our Inception v3 machine learning model, 3
we're ready to start getting some interpretations using that model. 4
So the first thing I'm going to do is I'm going to go just below where we set the image views image 5
property to the image that the user picked, 6
we're going to convert this UIImage into a CIImage which stands for Core Image image and that's a special 7
type of image that allow us to use the Vision framework and the CoreML framework in order to get 8
an interpretation from it. To do that, all we need to write is let ciimage, 9
so we'll just call it ciimage, 10
you can call it whichever you like, 11
equal Core Image image, open bracket, and the image that we're going to be using to process is that 12
userPickedImage that we got from earlier on. 13
So the code as it is is fine and it will work, but I'm going to add a little security feature in order 14
to make it a little bit safer. 15
So what I'm going to do is I'm going to add a "guard" statement in front of the "let," and then I'm going 16
to write an "else" statement that will trip if this part fails, i.e., if it's unable to convert our UIImage 17
that the user picked into a CIImage. 18
Now, that shouldn't really happen, but if it does, we want to trigger a fatalError 19
and we also want to write a message saying something like, "Could not convert to CIImage," so that we 20
know exactly what's happening. 21
All right. So, now that we've converted our UIImage into a CIImage, we're going to create a method 22
that will process that CIImage and get an interpretation or classification out of it. 23
So I'm going to write this just above my cameraTapped IBAction. 24
I'm going to create a new method and the method is going to be called detect and it's going to take 25
only one parameter and the parameter is going to be called image and the data type will be a CIImage. 26
And what I'm going to do inside this method is I'm going to use the Inception v3 model. 27
So all you have to do is to create a new object, going to be called model, and we're going to use the 28
and VNCoreMLModel as a container for our mlmodel and our model is called Inceptionv3. 29
So in this part where you've got a placeholder for mlmodel, you're going to write the model that was 30
generated when we dragged and dropped in this Inceptionv3 mlmodel. 31
So you can see it's already autosuggesting, but you can also find out if you just click on this little 32
arrow here and you can see that our class that was generated is called Inceptionv3 and it's got a property 33
called model. 34
So, now we've got Inceptionv3 creating a new object and we're tapping into its model property. So, now 35
we have created a object called model using the VNCoreMLModel container and creating a new 36
object of Inceptionv3 and getting its model property loaded up. 37
So, basically, this model is now what we're going to be using to classify our image. 38
Now, you'll see that if you write this line, you're going to get an error, and it's saying that this line 39
of code can throw an error and we're not handling it at all. 40
So let's go ahead and handle that by typing 41
"try?" in front of this. 42
So when we add "try?" it's going to attempt to perform this operation that might throw an error. If this 43
operation succeeds, then the result is going to be wrapped as a optional. 44
But if it fails, i.e., if there was an error that was thrown, then the result of this line will be nil. 45
So to guard against those situations where this fails and we end up getting a model that is nil, we don't 46
want to do any more image processing with something that with a nil model. 47
So if our app does crash under those conditions, we want to know exactly why it did, so we want to send 48
an error message to our debug console. 49
So to do this, I'm going to wrap everything inside a "guard" statement. 50
So, now instead of "if let," I've got "guard let." And this means that if my model is nil, then I'm going to 51
trigger an "else" statement that is going to throw a fatalError, giving me the message that, maybe, "Loading 52
CoreML Model failed." That way I'll get it printed in my deep console and I'll know exactly which part 53
of my code failed. 54
So, it's basically a way of allowing your app to exit, giving you plenty of information about what caused 55
it to exit or to crash. 56
And in fact, if you think about it, in this case, we could probably also benefit from wrapping it inside 57
a "guard" statement because in the cases where we can actually convert our UIImage into a CIImage successfully, 58
then we could end up crashing our app without knowing what caused it. 59
So let's do the same. 60
So if you remember how to add the "guard" statement and how it worked, then pause the video and add it now. 61
So we want to throw a fatalError telling us in the debug console that we couldn't convert the UIImage 62
into a CIImage. 63
Now, if you don't remember how to do that, 64
I'm going to show it to you now. 65
So instead of simply "let," we're going to call it "guard let," 66
and it's going to evaluate this statement. If it's true, then it's simply going to do nothing. But if it's false, 67
then we're going to trigger this "else" statement. 68
So instead of an is-- 69
So instead of an "if else," it's a "guard else. And in here, we're going to throw a fatalError 70
that tells us that we could not convert UIImage into a CIImage. 71
So, now if our app crashes because this failed, we'll know exactly what happens because we have that message 72
in there. 73
So you might be wondering where this VNCoreMLModel comes from. 74
Well, it actually comes from the vision framework. It allows us to perform an image analysis requests 75
that uses our CoreMLModel to process images. 76
So the next thing that we're going to do is we're actually going to use this model that we're certain 77
exists because this part didn't trip and our app didn't exit. 78
So the next part will only happen if this was not nil. 79
So here, we're going to create a Vision CoreMLRequest. 80
So we're going to call it request and it's going to equal VNCoreMLRequest 81
and we're going to use the one that has a completion handler. 82
We're going to hit into the model that we're going to be using is, of course, this model that we created 83
earlier on. 84
And if you now just click on this VNRequestCompletionHandler? placeholder and hit enter, 85
it'll insert the code automatically for you. 86
So, now as a part of our completion handler, we're going to get two things back. 87
One is the request. 88
So we're going to call it simply request. 89
The next thing is an error, 90
so we're going to call this error. And the code that we want to happen inside this completion block, i.e., 91
when that request has completed is to process the results of that request. 92
So to do that, we're going to create an object called results, 93
so let results = request. 94
So that comes from the completion handler callback .results. 95
And as you can see at the moment, its data type is an array of any objects. 96
So we're going to make that little bit more specific 97
by, again, using the "as?" and instead of being an array of any data type, we're going to make 98
it an array of VNClassificationObservations. 99
And this is a class that holds classification observations after our models been processed. 100
So, again, we want to use a guard to make sure that if this fails, our code exits, and we know why it failed. 101
So go ahead and wrap this line inside a "guard" statement as we did before. 102
Pause the video and try it out. 103
All right. 104
So as we did before, all we need to do is add the "guard" in front of the "let." 105
I'm going to add an "else" statement that addresses what should happen when that fails. 106
So, again, we're going to trigger a fatalError and the message is going to say, maybe, "Model failed to 107
process 108
image." 109
But if it succeeds, then we should have this object called results which we should be up to print out 110
into the console in order to see what results we got back. 111
The last thing that we need to do is to actually perform this request. 112
So you might notice that this request has a model associated with it, but it doesn't actually know which 113
image to perform that classification request on. 114
So this is what we're going to address now. 115
We're going to create a handler that specifies the image we want to classify. 116
So I'm going to write let händler = VNImageRequestHandler and it's going to take a ciimage, 117
and the image that we're going to input in here is the one that was passed into this method as a parameter, 118
so the CIImage that we're going to pass in, and that image is going to come from, of course, over here. 119
So we're going to call that method in here a little bit later on. 120
So the image that the user picked gets converted into ciimage, then gets passed into this method 121
as a CIImage, and then gets put into this handler to specify that it's the one that we want to classify 122
using our machine learning model. 123
So I'm going to use the parameter name, of course, which is image, and there is my image handler, 124
and now all I need to do is just to write try using my handler to perform the request which is, of course, 125
this request that we created early on. 126
So this line of code will work as long as you don't get any errors here. 127
Now, as you can see, we've got the exclamation mark after the try 128
which means that we're forcing it to execute this line. 129
But instead, the safer way to do this would be to wrap this execution inside a "do catch" block. 130
So if you don't remember from previously, 131
all you need to do is write the keyword "do," and then enclose whatever you want to try 132
inside the do block. And then you have another keyword called "catch" which catches any errors that are thrown. 133
And if there are any errors, then we want to print them into the console. And that's all you need to do 134
in order to use a machine learning model to classify your image. 135
So the last thing we need to do before we can run our app and test it out is that we actually need to 136
call this beautiful method that we've just created. At the moment, 137
it's not called anywhere, 138
and we won't get any results back if we run it now. 139
And in fact, you can see that this warning here is sort of giving you a hint as to the fact that maybe 140
you have a bug in your code, because it's saying that this CIImage that you created was never used 141
anywhere. It says, here's a fix, 142
instead of giving it a name, just make it a Boolean test. 143
But that's actually not what we want, 144
it's onto the right problem, 145
we haven't used this anywhere in our code, but it suggested fix is not always the right one to go for. 146
Instead, what we want to do is we want to pass the ciimage into this method detect so that the image 147
can be used to be classified by our model. 148
So very simply, all we need to do is detect and the image that we want to detect is of type ciimage 149
and that, of course, is this image that we converted from our userPickedImage. 150
So, now if we run our app, see what happens. 151
All right. 152
So here, I've got the app and I'm going to tap on the camera button. 153
I'm just going to take a picture of my keyboard and I'm going to click use photo. 154
So this is the data that we got back. 155
And as you can see, it looks like a pretty big mess. 156
But if you dig into it in detail, you can see that we've got a whole bunch of these 157
VNClassificationObservationObjects. 158
And if we go from the top, you can see that the first one which was rated with 82 percent confidence 159
is computer keyboard or keypad 160
which is exactly what it is. 161
Now the next one with low percentage confidence is maybe a typewriter keyboard which is also I can see 162
where it's coming from. 163
And then you've got this one with temps and confidence which is a notebook or notebook computer. 164
So as you can see this Inceptionv3 model is incredibly accurate in predicting what that image shows. 165
Now, the only problem is that this is a tangle of data. 166
So we have to make some sense out of it. 167
So, in the next lesson, that's exactly what we're going to be doing. 168
We're going to be focusing on the task that we've set for ourselves in the beginning which is being 169
able to take an image of anything and classify it as whether if it's a hot dog or whether if it's not 170
a hot dog. 171
So we're going to do that in the next lesson.