[Psycho_Coder]

Hello all,

I have a web crawler for personal use mainly. This is a site http://www.geeksforgeeks.org/ for CS students to learn Algorithms, Data Structures, C, C++ and various other topics. Its a very very good site and has a lot of well known CS articles. So, I wanted an offline version of the articles, just in case the site is down or offline for maintenance. So I made this.

Basically, reading from GeeksforGeeks was tedious as every time you to need to search and wait for it to load. This site is a great resource for each and every CS guy or girl. Sometimes the site gets down and hence I can't risk losing the content, and hence getting all of it today .

So I decided that I will download all the content of that site. So of course I had to crawl and get the content. So here's what I did.

1. I viewed the source and searched for how the contents are arranged and found that all the contents were posted in the form or either category or tag. For every category we have a number of thread and articles with pagination, which means that there are several pages of content and hence you cannot focus on the first.

2. I used python and quickly setup an http connection using urllib2 and then used BeautifulSoup to process the html source.

Next I had to get the number of pagination pages for each category (The category list will be give as input). It was fuckingly easy.

It was written within span tag and class = "pages" and hence used BS to get the number.

Then the page url structure for each pagination is

<base-url> / <category> /page/<page number>

Hence iterated and got all the content links and then iterated over the links and saved the pages.

3. Then I though What if I can get them in pdf format. Okay then, I got a lib xhtml2pdf and used it to convert all the html to pdf files. You can get a markdown of them if you use html2text lib.

After you have given all the categories, just sit sit back for an hour or or two and this will download almost the complete site for you.

The code is very user friendly and easy to read and fully commented. Almost all kinds of error checking has been done.

So Go and check it out yourself.

Sample Test File:

Code:

from G4GExtractor import G4GExtractor d = G4GExtractor() #You can set the path by calling a method or passing the path as constructor d.set_filesave_path("/root/PycharmProjects/GeekForGeeks-Spider/") #You can set the base url path for tag or category. Here we show an example with tag. #Of Course tags much match the tags of the site else you will receive error. d.set_baseweburl_path("http://www.geeksforgeeks.org/tag/") #Set the tag list tag_list = ["pattern-searching"] #call this function to save the files and if yoy send a second parameter as True # then files will be saved as pdf. totallinks = len(d.extract_content_and_save(tag_list, True)) print("Number of links crawled and saved is %d" % totallinks)

Help Docs and usage instructions will be updated on Github soon.

Github Link : https://github.com/PsychoCoderHC/GeeksFo...-Extractor

[kingpoop]

Hello P_C
I cannot get your geeksforgeeks crawler working. Can you check out your code and help me out?
much appreciated,
kp