Welcome to this selector.
In this lecture we learn about web crawler will make this web crawler with the help of the python from
the next 6 and onwards.
We have to learn about the javascript is basic and about the birds.
So bird suit is a advanced tool that is going to be used while we just spent days some of the Web sites
on a web application.
So we have to learn about that bird suit application in the next upcoming lectures.
But for now let me just make a simple web crawler in these Alexa so you may asks why do we have to just
make our own web crawler instead of just using.
That is the application that is available here.
So and so here is that the most of the case.
Those Web applications are those spent testing application like Bob suits are professionals.
So that means they have some at said Montes while talking about the smaller Web sites but they're very
eight months and they can have their price too.
So they cannot be affordable by natural worse.
They are for professionals.
So it's better to learn how to make your own web crawler because this test some basic web sites but
using the buzzword we have to test some big Web sites and that can be covered in the upcoming looks.
But for now we'll just see how to make these web crawler which we are going to covered in this lecture
with our simple programming language that is Python and you will see how easy it is to make the web
crawler with the help of Python.
So let's start from the basics before us.
Something with the code.
So first of all when I just type your Google dot com when it is hit into this you can see this web page
just surrendering over here.
So this web is has in the form of the markup language that we normally call the yes daemon language.
So what do you see here it just surrender.
So couple of images but inside here you can see the source code containing the SDM and language's markup
language scored.
So that means the computer will understand only that code but not this image is not this layout and
not this navigation bar.
So if you want to communicate through python with the computer you have to just work with this as the
male parser.
So we'll learn about that in the next lecture.
But for now let's start from the basics.
So first of all we learn about the request how to get the request how to get the response from the Python
and basically to work with that we have to import some library first.
So first of all I'm going to open my jaw by some.
That is this idea.
And next is to create the new Python fine I'll just say this is demo derby why is my python file.
I'll just import this request back is it d request.
And if you get any idea that is saying that this is not the module are basically no named module you
have to first imported or basically have to install it.
So first of all open your settings go in your interpreter and button over here and type a request.
You'll get this request install this back is right here apply here and just press okay and you'll get
this request easily imported on your project.
So next thing what I want to do is simple you at all.
So let's test this for you earlier which is Twitter dot com.
So let's do this.
So we see the Twitter dot com over here and now I have to make use of this strike as blog.
So if you wonder why Drake Platt's blog because if he does see that if you just open any Web site you
are going to get some Cordoba here which is estimable code and if you get 200 status scored that means
this is the page or this is the available page.
But if we get any either that will just throw you an exception that means we have two cats that exist
and so we have to cast that exception before this court fails.
So first of all we have to get the response.
So I'll just say good response is going to be called from these requests.
So this is the model that we have just imported and I have to call this get a method.
So this get method is a very important method and this is going to get us the UL and parameters and
basically this will just give us the response object.
So here I'll just pass that up.
So instead of just passing the letter I will do something like let me say as TTP which is obviously
there's TTP the website you are.
And next I want to concatenate this with our you are.
So if you see this you are released to third outcome.
So I am just concatenate thing with this new URL.
So next thing I want to do is I want to print this response.
So we have no response object.
So the next thing is to thus get a D block which is except.
And here we have to cast that exception.
But for now let me just do something like Let me show you what this exhibition will be.
So before that let me just type this exception.
So this exhibition will be request dot exception dot connect senator.
So this will be your exhibition if this is the case I'll just pretend that there is no Web site.
So this will be massive.
If there is any exception.
So first of all let me just sit on this and show you what is happening over here.
So this is demo derby way I'm just running over here and to this tick and it is saying that there is
a response to and that means.
OK.
There is this Web site.
So there is the presence of this Web site that is Twitter dot com.
So you can see how it is easy to see if there is a website or not from this request module.
So it is pretty simple script but we see now from the next lecture how do this again the subdomain directories
and crawl the web is from our work ruler.
For now you can see if I just say something like OK let me type injuries and if I see a dot com and
let me says something that is w w w dot some gibberish dot com let me just sit on and saw you and it
is saying there is no Web site because we just get to the exception.
So if I just remove the X exception.
So for now let's see that if I just remove this exception and now if I just print this again.
So let me just run it.
Then it will see there is a saying that there is exception.
Call is a request dot exception dot connection.
So these were the same exception that we just handled before in the strike as lucky we just remember
we have this code in this Excel block that is saying a request exception connection Ito.
So this was the same exception that we just handle from this strike as blog.
So if there is no Web site that is going to print there is no offset except it won't be something like
this.
So if you get any other like this that means your code will just get fill and you will just get your
application that is stopped.
So instead of doing that we want this to be cast over here.
If there is no upside.
And that will just print that this is did our application will work fine.
It want to basically stop at that point.
It will just run showing us that there is an oil set so you can see that if there is any Web site let
me see Facebook then there will be.
Let me just bring this run them home and you will just get again.
They start to score saying this points to hundred.
That means there is a Web site.
So this response is just to press ones.
We'll see about this response cord in upcoming lecture where we just spent test some of the applications
using the javascript payload.
But for now just remember this is the way in the python to get there you are take for this response.
If there is then you do this to get the response of 200 instead of scored.
If there is not will just gas description it will just print something or you can simply pass over here
instead of printing anything.
You can simply do pass.
So this is the way from this request better.
So next thing we want to do is to get every subdomain of this Web site.
So that will be covered in the next lecture.
See you in the next.