All right. 1
So in the last lesson, we managed to get our 3D dice to get rendered and displayed in augmented reality. 2
Now, the only problem is that it looks a bit weird because it's kind of just free floating in midair 3
which is not very realistic for a solid object. 4
So in this lesson, we're going to focus on plane detection. 5
So we're going to be using a structure within ARKit that allows us to detect horizontal planes in 6
the camera image that's captured. 7
So at the moment, we can only detect horizontal planes because, as you can see, the options only include 8
horizontal. 9
Now, we think that in the near future, vertical plane detection is going to be added in here as an option. 10
And I expect that the Apple engineers are probably just working on it as we speak. 11
And that was certainly the indication that we got at WWDC. 12
So talking to some of the engineers, they did seem to hint that this was something that was going to 13
be enabled within a short period of time, and not something that we would have to wait for, say, iOS 14
12 to appear. 15
So that's really encouraging. 16
But it does mean that, in the meantime, we won't be able to do any vertical plane detection. 17
So all of you guys who have ideas of, you know, hanging paintings onto the wall and seeing how it looks 18
in AR, you're going to have to hold your horses for now until they update the plane detection options 19
to include vertical. But horizontal is good enough for our dice app because what we want to do is we 20
want to be able to place our dice on the horizontal surface that we detect in augmented reality. So, say, 21
if we're looking at a table or the floor, 22
I want to be able to place my dice onto that horizontal surface and give the illusion that it is a part 23
of the real world. 24
So it's like when you play Pokémon GO and all the Pokemon appear on the ground. 25
This is what we're going to be doing in this lesson. 26
All right. 27
So the first thing we need to do is I'm going to comment out the part where we created our dice. 28
I'm going to come back to that later, but for now I just want to focus on detecting the horizontal plane. 29
So if you go into your viewWillAppear where you created your session configuration, 30
I'm going to add another property to that configuration which is, as we saw earlier on, the plane detection 31
property. 32
And I'm going to set that to equal .horizontal. 33
So as you can see, this is a enum. 34
And if you don't remember what enums are, then have a look back at the intermediate Swift lessons where 35
we went into that in more detail. 36
But, again, having the .horizontal as an enum, rather than having something like configuration.horizontal 37
planeDetection = true is another good indication, code wise that vertical is coming. 38
All right, cool. 39
So we've done that. We've said that let our configuration be our ARWWorldTrackingSessionConfiguration 40
and enable horizontal plane detection on that configuration, and then run it. 41
So the next step is we need to use one of the delegate methods from the ARSCNViewDelegate. And the 42
one that we're going to be using is a method called didAdd. 43
So if you just start typing didAdd with the ARSCNViewDelegate adopted, then you will see this delegate 44
method show up. And as it says, it tells the delegate, which is this current view controller, that a SceneKit 45
node corresponding to a new AR anchor has been added to the scene. 46
Okay. 47
So a lot of new words here. 48
But it basically means that it's detected a horizontal surface and it's given that detected surface 49
a width and a height which is an AR anchor 50
so that we'll be able to use it to place things or to visualize it. 51
And I'm going to hit enter to insert that piece of code. And inside this didAdd method is where we're going 52
to setup our horizontal plane. 53
So the most important thing that we get from this method is that when it detects a new horizontal plane, 54
it's going to, firstly, call this method and trigger the code that's inside. 55
But also we're going to receive this object called anchor and the anchor is of data type ARAnchor, 56
so it's a real world position orientation that can be used for placing objects in an ARScene. 57
So, this, you can think of as almost like a tile on the floor or on your horizontal plane, and you can 58
use that tile to place objects. It has a dimension. It has a position. It has a rotation. It has a whole bunch 59
of coordinates. And we're going to use those real world coordinates of that plane in order to place our 60
object onto it. 61
So this anchor is where we're going to be working with. 62
Now, ARAnchor is a broad category of anchors. 63
We want to specifically check to see if the anchor was of the type ARPlaneAnchor, i.e., that that anchor 64
corresponds to a plane, rather than some other sort of 3D object in the scene. 65
So to do that, we can say if anchor is ARPlaneAnchor, and this line of code basically checks to see 66
if this new anchor that was added is of the type ARPlaneAnchor, i.e., it is from our plane detection. 67
Now, if that was true, then we're going to print "plane detected," and if it's false or else, we're simply 68
going to return which means that we're just going to exit from this method call. 69
So let's give it a go and see what happens when we try to detect some planes. 70
Now, with ARKit, 71
you notice that there's usually a bit of delay before it detects a plane. 72
So we can enable one of the debug options which will show us as it's trying to look for points in the 73
view that are lining up to form a plane. 74
So to do that, we're going to go right to the top of our viewDidLoad and we're gonna write 75
self.sceneView.debugOptions, and this, again, is an array, but we're only gonna have one item in it ,and 76
we're gonna use ARSCNDebugOptions.showFeaturedPoints. 77
All right. So let's run the app again and you can see the difference. 78
All right. 79
So that made it a little bit easier to see what was actually happening behind the scenes in order to 80
enable plane detection. 81
So it was looking for all of these feature points. 82
And you'll notice that feature points don't get generated very easily when you have a shiny metallic 83
object. 84
So if you go to a mirror and you try to use the app to detect feature points, you won't actually get 85
all that far. 86
But if you have a soft fabric that has maybe some creases in it that casts a little bit of shadow and 87
gives a little bit texture to that plane, then your plane detection will work so much quicker and so 88
much easier. 89
But before the feature points actually appear on screen, your plane detection isn't taking place. 90
So it's just a good way of being able to debug what's going on. 91
All right. 92
So, now instead of printing that the plane is detected, I'm going to downcast our anchor into the type 93
ARPlaneAnchor. 94
So I will say let planeAnchor = anchor as ARPlaneAnchor. 95
So our code basically checks to see if this anchor that was added or detected is of type ARPlaneAnchor. 96
And if it is, then we are going to change its type from ARAnchor to ARPlaneAnchor. 97
Now, you'll notice that this is pretty wordy and there's certainly ways of making this more succinct 98
and much, much shorter on the number of lines of code. 99
But because there's going to be quite a lot of things going on in this block of code, some of which is 100
quite abstract and quite difficult to understand, 101
I'm going to try and keep my code as expressive as possible for now, and then later on, we can refactor 102
it and make it shorter and simpler. 103
So, now we have this constant called planeAnchor that is equal to the anchor that we got back from the 104
delegate method which detected a new area that corresponds to a horizontal plane, 105
and we have changed its data type to ARPlaneAnchor. 106
All right. 107
So the next thing that we need to do, we're going to convert the dimensions of our anchor into what's 108
called a ScenePlane. 109
So similar to how we created spheres and boxes using SceneBox or SceneSphere, 110
there's something called SCNPlane that allows us to create a plane in SceneKit. And we're going to 111
call ours plane and it's going to be created using ScenePlane and SCNPlane only requires two things: 112
a width and height. 113
So luckily, the anchor that we've got, which is of type ARPlaneAnchor has a property called extent. And 114
the extent comprises of a width and a height. So you can think of an anchor as simply just like a tile 115
on the ground, and it has a width, it has a height, 116
and we're going to use that tile that's been detected and convert it into something that we can use with 117
SceneKit and that we can render onto our sceneView. 118
So the width is going to be planeAnchor.extent. x. And the height is going to be 119
planeAnchor.extent.z. 120
So, a really important thing is that this is x and this is z, a really common error is that people will 121
write y here. 122
But if you actually read the documentation by option clicking on extent, you can see that this is the 123
estimated width and length of the detected plane. 124
And most importantly, although, what type of this property is vector_float3, i.e., a three-dimensional 125
object, a planeAnchor is always two-dimensional, and is always position size in only the x and z directions. 126
So make sure that you don't put y in here because, otherwise, your code is not going to work 127
going forth. Now, the next thing you might notice is that Xcode is giving us this error here, and it's 128
telling us that it can't convert the type 'Float,' which this extent.x is, into the expected type which 129
is 'CGFloat' which this function requires of its parameters. 130
So we can go ahead and cast it into a CGFloat by just clicking Fix. And we fixed the x, 131
so now we need to fix the extent.z. 132
So the next step is to create a planeNode. So you remember when we created spheres? We first created 133
the geometry using SCNSphere and then we created a node to attach that geometry, too. let planeNode equals 134
SCNNode. 135
And then I want to set the planeNodes position property, planeNode.position = 136
SCNVector3. 137
So the x is going to be planeAnchor.center.x 138
and y is going to be zero. 139
The y position is going to be zero because, remember, it's a flat horizontal plane, 140
so there is no y element. 141
We don't want it to be above the horizontal plane that was detected. And z is going to be 142
planeAnchor.center.z. 143
All right. 144
So up till now, this is all pretty similar to what we did previously with the boxes and the spheres. 145
Now, the next thing that we have to understand is a bit of a tricky concept. 146
Now, the idea here is that we're creating this thing called a SCNPlane which is a rectangular one-sided 147
plain geometry of a specified width and height. 148
So we specified its width and height using the anchor that was detected. 149
And then we might think that we're done but, actually, there's a slight problem because SCNPlanes are 150
created as a vertical plane. 151
So if you have a look at this graphic here and the 3D key, the red is the x, the green is the y, and the 152
z component is non-existent because this is a 2D object. 153
Now, as you might remember, in ARKit on our sceneView, everything is 3D, and also our plane is horizontal. 154
So we need to convert this vertical plane 155
that's got a component in the x and y into a flat plane that's got a component in the x and the 156
z. 157
So to do that, we're going to have to transform our planeNode and we're going to have to rotate it by 158
90 degrees in order to make it horizontal. So planeNode.transform. 159
And we're going to specify the amount that we want to transform it using something called a 160
SCNMatrix4MakeRotation. 161
And this takes four parameters, 162
first of which, is the angle that you want to rotate it by, and then the next three parameters, you specify 163
along which axis you want to rotate your object. 164
So the angle that we want to rotate our plane by is 90 degrees. 165
We want to change it from vertical to horizontal. 166
Now, this is in radians. 167
So just a bit of recap on high school math, 168
1 pi radian is equivalent to 180 degrees. 169
So in order to rotate this by 90 degrees, we need half pi radians. 170
So I'm going to grab pi by writing float.pi, and this is something that's available in UIKit because 171
if you're working with spheres, if you're working with angles, and you tend to need to use pi, divide it by 172
2, and that's specifying that we're rotating by 90 degrees. 173
However, if you have a look at this function, you can see that the angle is the amount of rotation, in 174
radians. 175
We got all of that right. Measured counterclockwise around the rotation axis. 176
So we actually want it to be rotated clockwise. So because when it's positive, it's counterclockwise. 177
So to cancel that, we just need to add a minus in front of it. 178
So two minuses cancel out and this is now rotating our vertical plane 90 degrees clockwise, and then 179
we are going to specify the axis that we want to rotate around which is, of course, the x axis. Snd for 180
the other two, we're going to leave it alone. 181
So our rotation only has an x component. 182
All right. 183
So that is our plane node created and transformed. 184
Now, for our human eyes to be able to see it and evaluate whether if our plane was created in the correct 185
place, we're going to need to give it a material. 186
So in this lesson, there is a download for a grid.png file. And this is the asset that Apple provided 187
in their example project for us to be able to visualize these planes. 188
So we're going to use the same image. 189
And the important thing about this image that you might notice is that it's a .png file type. 190
And you may or may not know, but PNG file types have transparencies, 191
so we'll be able to see through our grid and see the material underneath. 192
And you're going to see really soon what that looks like. 193
All right. 194
So I'm going to drag my grid and, once again, into the art.scnassets folder, and then I'm going 195
to create something called gridMaterial. 196
And, again, it's going to be created using SCNMaterial. 197
So we've done this before and we're going to set the gridMaterial's diffuse.contents to a UIImage 198
which is the one that we dragged in just now. 199
So we're going to have to specify the folder. 200
So art.scnassets/grid.png. 201
Okay. Cool. 202
And we're going to assign this material to our planeGeometry that we created earlier on. 203
So plane. materials = gridMaterial. 204
And finally, we're going to set the geometry of our planeNode to our plane. 205
All right. 206
So the only last thing we need to do as we did before with our spheres and our boxes is to add our 207
childNode into the rootNode. So you can see that as a part of this delegate method. We also have this parameter 208
called node which gives us a blank node to work with. 209
So you can either, as we did before, tap into the sceneView.SCNRootNode and childNode, 210
or we can just simply use this node that got created when the didAdd delegate method got called. 211
node.addChildNode planeNode onto the scene. 212
So let's give it a run and see what it looks like. 213
All right. 214
So there you have it. 215
Just with a few lines of code and the assistance of ARKit, we've been able to implement an immensely 216
complicated computational task which is detecting a horizontal plane in the real world based on the 217
image that's coming in through the camera. So you can pat yourself on the back and we can thank ARKit 218
for making our lives easier. 219
So in the next lesson, we're going to be looking at how we can detect touch in the scene because, ultimately, 220
we want to be able to touch a plane that we've detected, say, a table or the floor and be able to place 221
our dice at the position that we touched in the 3D world and scaled, 222
so it looks like it's a realistic dice that's been placed in the 3D world. 223
So we're going to do that in the next lesson. 224
So I'll see you there.