Welcome to this Celexa.
So in this lecture we're going to make another web crawler.
But in much more efficient and much easier way.
So this will be using the next to module.
And that is beautiful soup.
So it is very popular and powerful.
So if you have ever harvested the URL from the Lings then you should have heard this name that is beautiful
soup.
So this beautiful soup automatically converts our incoming document to the unicorn so that this decoding
techniques should not be used and that this you can see the pattern over here.
So this would not be used if you use that beautiful soup.
So it is much more easy if you just make the web crawler using the beautiful soup.
So let's see how this works.
So first of all we have to this import this beautiful soup so for that we have to first go to the settings
and this import that module on install that pack is maybe.
So I'll just say I have to install B As for and that is a beautiful soup.
And let me just install this package it in Starling so let it give it time to just install that packages.
So it has successfully installed our packages in this machine.
So let me just press.
OK so let me just create a new project over here and let me say this is the new project new python file
and this will be day more too.
And it will be deemed to be right.
So first of all use that to be as for which is beautiful so from B As for we want to import the beautiful
soup module or this is a plus beautiful soup is a glass.
So next year to import the request.
So we want to get that you at a request are basically get the response from that you order.
So next thing I want to get the URL from the user.
That means we can get the input from the user.
So I'll just say inter web site for x frac sand okay extracts and process or something like this.
This will be message to the user.
And the next thing what I want to do is get the response.
So you know how to get the response it is similar that we did in the previous lecture getting the request
to use the get method pass to you at all.
So here I'll just formatted with the SDP and now I'll say add this with how are you.
Or you can simply get the other in the palm of his CTP.
But I want the user to enter only the URL but not this year's GDP part.
So next thing is to get the data.
It is important so data should be given in the form of text.
So I'll just say response dot text.
So this is the way.
OK.
So text.
So just get the text.
Now we sort of use the beautiful soup.
So let's see beautiful soup which is the class in here.
We have to pass the data that is in the form of text.
So will this face this and will this make the object basically in instances or instances of this beautiful
soup over here.
And now this harvest this link so you can see how easy it is.
So we'll just see what these beautiful soup instances.
And we'll just use this find all method same.
Is that of regular expression method but for now we have to only harvest this ankle tag.
So what this is going to do is it is going to the source in that document estimate document and it will
just find this anchor tag.
So if you just remember what we just have taught over here that is in the space.
If I go to the view page source you could see that it's area of tag is in these encode.
So this sets area of attribute is in this anchor tag.
That means we are just finding these anchor tags now.
So instead of using this you could see that this pattern which we just did over here we can simply source
that encoded and print.
Now that is the link so that links should be link the dog get so inside of that area.
We want to get that link.
So this is so easy to get the link from the beautiful soup.
So what we just did here we just get the response.
We just convert it into the text data and first into the beautiful soup with this instance it that soup
with this soup object and we just find from that with this final method that is a theory of tag or basically
anchor tag and after that we get the content of that its idea with this get method.
So if you see this link will be in the form of this name you can see this link also.
This will be in the form of dysentery and we want to get the link.
So this DNA will contain the IDF of is a key and the link is a value.
So we are just getting the yes area.
That means we want only the link to be printed so that there is no encoding to be done.
So if you see what we did in the previous lecture we had to decode with this duty of it.
But in the case of you beautiful soup it is done automatically to just convert incoming document to
the unicode an outgoing document to UTF 8 decoding.
So let's do this and let's go to the demo derby way.
Let's run this file now and it should be demo to that boy.
Now let me this into the Web site.
So this Web site will be this Web site.
So let me just grab this copy this and let it go.
Well here let me just press.
So we don't want to maybe desisted to be over here.
So let's go over here and slowly move this.
Let's see if this works.
So what happened is with this put here everything.
So instead of that as the set test be a speed out three you will in a wave.
Dot com is my oversight if I just sit into it is going to give us a link so you can see how easy it
is to make this link.
So next thing what you can do is similar to that what we just did in the previous LEXA.
So let's go over here now you can make use of this you are a join in order to just convert that relatable
link to the beautiful thing.
So let's go right over here and paste it down here.
And after you get that it's idea of you can simply join that link too so you can see you can do something
like this.
You can just copy that link and update this.
Get that.
You can now print that link.
So next thing is to this target you are also this won't be the target you will.
This will be your you are all.
And in this case it is only the order.
And we'll just join with that link.
We can also print that link.
Now after we just get that.
So basically you can store this yes idea for some way.
So instead of just doing this you can do something like OK let me just say get this and let me just
restore in the test link if I say let me just put this in the test link.
And after we just get that this link I want to join that is with the test link and let's bring that
link now.
So you can simply do this.
And now let's see if this works or not.
So you can also simply print over here.
So let me just run this demo again.
You'll see.
Now this would have now again type that this to be a speed or b roll in the wave dot come so this will
be a for this let.
So we saw how we can do this way.
That is using these you are all request and regular expression way and the next is easy way that is
using this beautiful soup.
So this will be there for this lecture soon the next one.