Welcome to dyslexia.
So in this letter we're going to make a simple way of crawler so what we are going to do here is we
will just face the content of the websites.
So instead of just getting some of these images or everything will just get only they you add all of
the particular Web site which is the basic uses of a wave crawler.
So if you have learned about a spider if you have heard about this spider from anywhere that is going
to do same is this.
So it is going to just crawl in the web is and it will just phase the other so that we can attack to
that we you URL and see if it has a gate or post meter.
We'll see all about that in the zombie scream section which will be dedicated six and four day exercise
attack but for now let's see how we can make this simple web crawler with the help of the Python language.
So first of all we'll just skip this request method from the previous lecture which will just get this
you idle response and see if that exists or not.
It does particularly send us a request object with a static scored of 200.
That means that the Web site has been there.
That means it is okay.
That is a status 200 means the Okay signal.
That means there is the presence of that Web site.
Now in this target you are all I want to include this your.
So this is our test PSP ville anyway.
Dot com will you order.
It is a test Web site.
So let me just insert it here.
So let me just remove this because we have already give.
That is TB over here in the request to get.
So we have.
Yes the TB over here.
So I have to make this column right here.
This is the part of the part.
So now we have our target.
You are already.
So let's get the response now first.
So let me just get the response is in request.
So we have to response to this requires the method for us.
And it will just pass this target to you.
So we have now passed this target to you all.
Let's print that response.
So what we're gonna do is we have to now see the response in the form of markup language.
So if I just go over here and if I just see the best source you are going to get this content.
So this is the you are a lot.
This is the basic content that we want to fits.
So this is made by the U.S. de Mello which is a markup language.
So we have to now fits the you at all from these Markup Language or basically from the estimate.
So we want to target only those links which have this you are all because these are important what we
don't care is about these e-mails is so web crawler only acts in a way that it has to fit only the other.
It doesn't care about these e-mails it doesn't care about this stable border anything it was just scary
about this in Kotak yet Diary of a tribute.
So let's go to our buy some again and we have to now see the content.
So if you want to just get the content it is done by this content attribute.
So this is basically to get the SD malcontent out basically this will give you the content of the webpage.
So let's see if this can do or not.
So if I just sit on this you will see that.
Let me just run this and I'll just go here then you could see we have phase the content of this website
so you could see everything over here.
So it is the single line though.
But will this make or modify this in a minute but you can see we have fears the content.
This is a markup and we have this phase every content over here from this Web site.
So it has this yes language is made of the content.
It has the way to the table and everything over here.
So we have successfully phased the Web site content with the simple response.
And we just phase the content from the Web site in this script.
So next thing what you need to do is just modify.
This is group so as to get the link from those set.
So we have already learned about one thing that these the regular expression.
So if you just remember in the previous lecture we have learned about the regular expression and how
to use it in our programming language.
Basically now it script.
So first of all what you need to do is you have to make use of that regular expression and to do that
you have to import that regular expression.
And I'll just say import Ari which is for the regular expression after we just do that.
Let me just define one new method which will just some extract the links.
So this method will just extract link because if you see in our output there isn't everything over here
which is unnecessary information.
So there is a re listener see.
There is basically this template PSC file everything is there.
This is unnecessary what we really want is only the link because web crawler is only interested on the
link it will just harvest those links.
So that is the main reason we are making these extract link.
So from that we've said we want to extract only the links so far that we want to return by just refining
with these regular expressions.
So we want to make use of this final method.
We have only learned about the match in the source method in the regular expression section where we
just give the pattern.
So it does seem is the match and SaaS but here what we do is we pass and we start in that sentence or
in that content.
So here I want to just pass now the pattern.
So first argument to this final is the pattern so patterns would be inside these single code.
Now I'll just put the pattern over here.
So I'll say yeah sorry of because if you just go over here all the links are inside this idea.
So let me just.
So over here the new coal seam it is here that is the link is inserted inside this sets idea of attribute
of the anchor that that means we're interested to source for these sets idea of tag.
So let's go for this areas that we have to add other wildcard characters too.
So first wildcard character would be this column and the next one is this question mark.
So this is very important so we are searching for this yet idea of now inside this pattern.
So it should be inside now parentheses because this is not exception.
This should be compulsory whereas the saying that this person should be compulsory.
So what is this did overhear was we are just sourcing for its idea of equal to an opening code.
That means we are just making a pattern which rules us get this part which is that idea of equal and
thus double code.
So we are just matching this pattern.
We're finding this pattern in every part of the unitary content of that document.
So we're just saying that get me that idea of equal sign and this double code in the document after
this pattern is finished.
We want the next pattern.
And if we just remember this dot was a wild character.
That means anything.
So it can be that is it.
It can be maybe your alphabetical character.
So this dot means every alphabetical character.
And if you just remember this strict was any number of character.
So this dot was for alphabetical or digit or any character.
This streak was for one or more character or it can be also zero character so it can be any number of
character for this district and we'll just give the.
Again this question mark.
Maybe this is for absence or present at once we have learned about this question mark in the regular
expression if we haven't gone through it makes you do this take that section and we'll just get understanding
about this wildcard characters.
So it is very much easier way.
So after users get this pattern you have to close with the double code because if you see this your
idea of tag is closed with this double code.
So we're just searching for this pattern that it calls with the idea of equal to an double quote starting
double caught.
We don't care whatever is inside it.
After this double court that is what we just saw here that is anything can be.
After this single after this double court and after that character is indeed there should be ending
with this double court.
So that we just get the Udell from that.
So we just find that we just return that from this extract link.
So one thing what you can do here is instead of just making this request over in a good you can simply
remove this.
You can simply get a response here.
OK.
We are just returning.
That means we have to first get the response.
So we'll just get a response from this request module and we'll just use this get method.
We'll just pass it here.
That is your you are all so that you all should be sent.
Well we call this extract link.
So instead of this request method now we have to parse that is we have to call basically this extract
link method.
So now we have simply called this so instead of this printing this now we have to now format this first.
So we'll just remove this so if you see here here is something that we are finding so we haven't put
any another argument in this final.
So if you ever remember in that case where we just saw these are matters for the pattern in the sentence
the next argument used to be the sentence but it is not the sentence in this case we have to source
in the document which is this dismal document.
So if you want to estimate document it should be inside this response and contained.
So we are sorting this pattern inside this document which was just printed over here which contains
everything from this ankle tag or basically division tag paragraph tag everything.
So next thing is just to go over here.
Will this now print our yes area of link.
So instead of this response.
So this is no further the response it is the link which is this area of link.
So now we have to disprove that this idea of link.
So we have completed our web crawler over here.
So if you see what you just did here what we just imported the regular expression to the same use this
method which is called Final which will just see this pattern in this document because this pattern
is to the sick face that you know from the document estimate document.
We don't care any any other thing in this document except the you.
So we are just so going for the euro and after this we completed everything fitting from this.
Well let me just close this and now we just on that everything from you earlier in this is struggling
we just get that the link over here we just print that link.
So let me just run this real quick OK.
It is giving us either because obviously we haven't have here digested TB.
So first of all you have to have the ESTA TB and let it be the actual Web site.
So just make sure you include this has to TB otherwise it is not going to work.
Let me run it one more time.
And again it is saying something is missing.
So okay maybe about here we missed maybe so it comes because we have to now just decoded.
So first of all let me just decode this with maybe you deserve it and let's see if this works.
This is a decoding technique for the content of their statement.
So let me just run and see if this works.
So OK it is working.
So this is giving us the yes idea of links so you can see bunds of a PSU file over here so you can see
what's more Piers you file so much obviously file you can also see the link over here.
So our web crawler is working perfectly now.
There was a small problem because of a DC decoding technique which you to add in this content because
this is the duty of a decoding technique for the estimable parser.
So you have to add to this response and we are just matching with this pattern now and I have a little
about your pattern that is basically we sorting through your tired your pattern and any character after
this double code and we have to close with this closing double chord at the end.
We are just searching for its idea attribute in the tag and we just started getting this you Adam.
So next thing what you can do is we can just make a use of the for loop so that we can just make one
link in a one line.
So what you need to do is instead of just printing this let me just comment this out.
So if this seed our output all of the link is in the single line.
So if you want it's a line having a single link you have to make use of this for loop.
So I'll say for link in the idea of link I want to print a link.
So now if I just print again run the demo you will see nothing okay.
So should give you now.
So now it has every link in the single line.
You can see it has a CSI style too so you can see the link over here.
So we have successfully harvested the link from our web crawler but this is not actually a perfect web
crawler because you can see it has some of the links but in the form of PSP.
So this is not a complete link.
It is a related link so this link alcohol related link this is not the complete link if you just copy
this and go.
And basically if you just put it over here this is not going to work because this is nothing.
And this will work if there is a related path for this.
So if you do something like If you copy this and put this is that directly after this forward less.
This is going to work.
So that can be done with these suppressor.
So we have to make use of now parser to join this you at all with the link so we can do that now.
So first of all after we just get this idea of link we can use the you will pass.
So in order to this do that we have to import.
Now you will pass.
So if you do something that import me you will pass.
This is not going to work because we do not have your will pass in the version of a python that is greater
than three.
So we have to first import or I can simply say from you are real library.
So if you want to work with this you it'll pass.
To make this you look nice or become more like the other rather than having the ability but you have
to use that you all pass module.
So if you do something like import me you are all pass.
This is not going to work in the python newer version so all you have to do is you have to import first
you are a library and from that you have to just get this pass and from that you have to import now
your usual pass.
So this is the way to import the URL pass.
Now you can use your you are a pass so what are you gonna do here is we are going to make a new link
for those type of related links which have no meaning whatsoever over here.
So if you see this log in that PSP is alone.
So we have to make something like this link for these logging that PSP too.
So let's make that.
So first of all I'll just make use of this you it'll pass.
So we have something that is you will join over here which is the meter.
So in the near version you don't have this you also in maybe.
So what we can do is sic for the documentation maybe.
So if you do this okay we can simply use you a joint.
So instead of you it'll pass.
I'll just make use of this you are a joint.
So in this you're a joint what do we want.
Is our target you all which is this you Earl if you see this with a joint with this link that we just
have you.
So you could wonder what happens for this type of link.
So what happens if this link is thus explored for this type of link.
Okay.
You can join.
You can join with these target links.
But what if we just find this.
So is it going to this combine the tooling and make us again the link that is the concatenation of these
tooling.
No.
So that is not in the case of this Euro zone.
So it is only going to be active this part or this statement is going to be active only in the case
of the relative link.
So it will be active in this part of link but when it sees that the link start from its GDP and all
of this then this will just ignore and it will just print this link.
So let's see this in the action.
So
now you can completely see we have concatenate over here so we have now.
That is that was.
So let me just remove this again and let me show you.
So it was the case where we have these relative links.
Now we have added from this year are join.
And now let me run it then you'll see completely the new link for each of these relative link.
We have added this test PSP which is our target website from this parser.
So the new way to do this joining is not from the other parts in the newer version of the python.
It is from the you are all joint.
So we used to do something like you Earl pass before.
So we used to do do something like you at a pass and if such something like you pass and used to do
it you are doing.
But this is not the case in the newer version.
You can simply use your joint.
So here you have to first import you are a join if you want to use that the eurozone.
So we have now made our simple web crawler but a powerful web crawler which will just get every test
a link of the website so that we can just test for the wonder abilities in each of these link.
We have seen how to get the subdomain of the link in the previous letter which was the important links
it to.
So this lecture was also important because we use our regular expression to match the patterns to find
the link are basically the thing we want to match in the document that makes our link are basically
web crawler much more efficient than before.
We just so join that link.
So that reality part becomes the complete link.
So this will be there for this Lexis.
So in the upcoming lecture we'll see about the Java script and how we can test the use of a script to
attack on these type of applications waiver applications and websites.
So see you in the.