HTML taglist creator - Printable Version

HTML taglist creator - Printable Version

+- Sinisterly (https://sinister.li)
+-- Forum: Coding (https://sinister.li/Forum-Coding)
+--- Forum: Python (https://sinister.li/Forum-Python)
+--- Thread: HTML taglist creator (/Thread-HTML-taglist-creator)

HTML taglist creator - L0aD1nG - 08-28-2014

Hello [username], this is a simple web crawling related function. I find it very handy when it comes to html code manipulations so I decided to share it with you. I coded and tested this on python2.7 it maybe works on other versions too, but make sure the version is able to install and use BeautifulSoup4 module properly.

Requirements : python2.7, bs4(BeautifulSoup4)

Code :

Code:
## Importing
from bs4 import BeautifulSoup
from urllib2 import Request, urlopen

## Predifined Header Constant
HEADERS = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64)',
       'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
       'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
       'Accept-Encoding': 'none',
       'Accept-Language': 'en-US,en;q=0.8',
       'Connection': 'keep-alive'}

def tag_list_create(url, tag):
    """This functions will create and return
    a list of all the tags from the given url's 
    html code each list item as a str object.
    """
    ## Setting up the request
    request = Request('http://' + url, headers=HEADERS)
    ## Opening the site
    site_open = urlopen(request)
    ## Reading the html code
    site = site_open.read()
    ## Setting the soup
    soup = BeautifulSoup(site)
    ## Getting&converting the tags
    taglist = map(str, soup.findAll(tag))
    return taglist

Sample Run :

Code:
from pprint import pprint
pprint(tag_list_create('www.nytimes.com', 'p'))

Results :

Spoiler: Double Image

Replies/thoughts will be appreciated as always.
I hope you will find it interesting. Kappa
Thanks you for you time.
Sincerely,
L0aD1nG

EDIT : Forgotted the HEADERS constant added now. If it didn't work that was the problem, my bad.

RE: HTML taglist creator - L0aD1nG - 08-28-2014

(08-28-2014, 03:17 PM)daviesadeleye Wrote: what is python? please i dont really get you. and for the result of your codes is that the siteyou hacked or what. if its the site that's just the view source options,

LOL.
I didn't "hacked" anything, get reasonable please.
You entered on a Coding's section subsection... so probably Python would be a programming language like almost all the other subsections in that section.

Also the test has been taken in www.nytimes.com so if you do the same you will get the same results (or with some you advertisments, articles etc the uploaded lately after my ss).

RE: HTML taglist creator - Psycho_Coder - 08-29-2014

It would be better if you use lxml, which is much faster than beautifilsoup.

RE: HTML taglist creator - L0aD1nG - 08-29-2014

(08-29-2014, 02:40 PM)Spell Wrote: Thanks for sure, bro!

You are welcome. Thanks for the reply.

(08-29-2014, 02:38 PM)Psycho_Coder Wrote: It would be better if you use lxml, which is much faster than beautifilsoup.

I didn't really know that this can be achieved with lxml, I will check it out dude.