[L0aD1nG]

Hello [username], this is a simple web crawling related function. I find it very handy when it comes to html code manipulations so I decided to share it with you. I coded and tested this on python2.7 it maybe works on other versions too, but make sure the version is able to install and use BeautifulSoup4 module properly.


Requirements : python2.7, bs4(BeautifulSoup4)

Code :

Code:

## Importing from bs4 import BeautifulSoup from urllib2 import Request, urlopen ## Predifined Header Constant HEADERS = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64)', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3', 'Accept-Encoding': 'none', 'Accept-Language': 'en-US,en;q=0.8', 'Connection': 'keep-alive'} def tag_list_create(url, tag): """This functions will create and return a list of all the tags from the given url's html code each list item as a str object. """ ## Setting up the request request = Request('http://' + url, headers=HEADERS) ## Opening the site site_open = urlopen(request) ## Reading the html code site = site_open.read() ## Setting the soup soup = BeautifulSoup(site) ## Getting&converting the tags taglist = map(str, soup.findAll(tag)) return taglist

Sample Run :

Code:

from pprint import pprint pprint(tag_list_create('www.nytimes.com', 'p'))

Results :

Spoiler: Double Image

Replies/thoughts will be appreciated as always.
I hope you will find it interesting. Kappa
Thanks you for you time.
Sincerely,
L0aD1nG


EDIT : Forgotted the HEADERS constant added now. If it didn't work that was the problem, my bad.

[L0aD1nG]

what is python? please i dont really get you. and for the result of your codes is that the siteyou hacked or what. if its the site that's just the view source options,

LOL.
I didn't "hacked" anything, get reasonable please.
You entered on a Coding's section subsection... so probably Python would be a programming language like almost all the other subsections in that section.

Also the test has been taken in www.nytimes.com so if you do the same you will get the same results (or with some you advertisments, articles etc the uploaded lately after my ss).

[Psycho_Coder]

It would be better if you use lxml, which is much faster than beautifilsoup.

[L0aD1nG]

Thanks for sure, bro!

You are welcome. Thanks for the reply.

It would be better if you use lxml, which is much faster than beautifilsoup.

I didn't really know that this can be achieved with lxml, I will check it out dude.