[Ligeti]

Hello
(you may skip the talking/story and just go directly to project description)

This is not a python tutorial, this is a study case!

Yesterday night I made this very simple demo project as problem solving challenge and as a project manager (which I used to be) but anyway, the demo was made for a company that I am looking forward to work in/with (wish me good luck)... the idea is that this was not to show my programming skills.

OK enough of my rather boring autobiography!

Motivation
The company has websites (social media and eShops), the spend lot's of time gathering the websites content and materials everyday, from news, sport, weather, horoscope... etc., they visit hundreds of websites everyday to collect all this information, then they summarize and edit, then compile everything in new content (original)... I immediately saw an opportunity there, and asked the manager:

Why not make a platform/solution that gathers all this information for you and save them in one database, and build an interface to show/list the data?

Well... my idea is much more complex than this, but I want to simplify it (I don't want to make this post more boring)

Project Description
So I will demonstrate how I scraped three websites to read the horoscope and save it in the database, the three (random) website I picked are:

• elle.com
  
• astrocenter.com
  
• huffingtonpost.com
  

I made a python script for each, the three scripts are similar.

The database
I used the "test" DB and created a new table, called it horoscope:
[

Code:

sql]create table horoscope(id serial, source varchar(255), readingTime DATETIME, zodiac varchar(255), horoscope TEXT);

Please don't flame at my poor coding and optimization, and remember that this is not a programming challenge as it is problem solving!

The Scrapers
This code I used for elle.com

Code:

import re import urllib import MySQLdb db=MySQLdb.connect(host="localhost",user="root", passwd="",db="test") myCursor = db.cursor() zodiac = ['aries','taurus', 'gemini', 'cancer', 'leo', 'virgo', 'libra', 'scorpio', 'sagittarius', 'capricorn', 'aquarius', 'pisces'] regex = '<div class="body bodySign">\s*(.+?)\s*</div>' patter = re.compile(regex, re.DOTALL) for sign in zodiac: htmlfile = urllib.urlopen('http://www.elle.com/horoscopes/daily/' + sign + '-daily-horoscope/') htmltext = htmlfile.read() horoscope = re.findall(patter, htmltext) print sign print str(horoscope[0]) print '-----------------------------------------' query = 'INSERT INTO horoscope(source, readingTime, zodiac, horoscope) VALUES ("elle.com", NOW(), "' + sign + '", "' + str(horoscope[0]) + '")' myCursor.execute(query) db.commit()

As you can see I used urllib to download the page, then filtered the text using regex, this step is the heart of scraping process, after that I stored the result in the database.

This is the most important part if my code, which actually to set the regular expression!

Code:

regex = '<div class="body bodySign">\s*(.+?)\s*</div>'

For advanced programmers this may look bad (although I really don't see why), but I just have the feeling that @shp0ngl3 will give me hell!

Unfortunately I can't post the web code here (I think it is illegal/not nice), so you will have to go and check the code of the page yourself... you can extract the links from the code (you don't need to be a genius to figure it out).

Spoiler: Code for astrocenter.com

Code:

import re import urllib import MySQLdb db=MySQLdb.connect(host="localhost",user="root", passwd="",db="test") myCursor = db.cursor() zodiac = ['aries','taurus', 'gemini', 'cancer', 'leo', 'virgo', 'libra', 'scorpio', 'sagittarius', 'capricorn', 'aquarius', 'pisces'] regex = '<div id=\"textline\" class=\"fontdef1\">\s*(.+?)\s*</div>' patter = re.compile(regex, re.DOTALL) for sign in range(0,12): htmlfile = urllib.urlopen('http://www.astrocenter.com/us/horoscope-daily.aspx?When=0&ZSign=' + str(sign) + '&Af=0') htmltext = htmlfile.read() horoscope = re.findall(patter, htmltext) print zodiac[sign] print str(horoscope[0]) print '-----------------------------------------' query = 'INSERT INTO horoscope(source, readingTime, zodiac, horoscope) VALUES ("astrocenter.com", NOW(), "' + zodiac[sign] + '", "' + str(horoscope[0]) + '")' myCursor.execute(query) db.commit()

You can see that the code is really similar to the one for elle.com, the only thing that changes is the source, the links and the regular expression!

Spoiler: Code for huffingtonpost.com

Code:

import re import urllib import MySQLdb db=MySQLdb.connect(host="localhost",user="root", passwd="",db="test") myCursor = db.cursor() zodiac = ['aries','taurus', 'gemini', 'cancer', 'leo', 'virgo', 'libra', 'scorpio', 'sagittarius', 'capricorn', 'aquarius', 'pisces'] regex = '<div class=\"aboutimg .*BigThumb\"></div>\s*<p>(.+?)</p>' patter = re.compile(regex, re.DOTALL) for sign in zodiac: htmlfile = urllib.urlopen('http://www.huffingtonpost.com/horoscopes/astrology/'+sign+'/') htmltext = htmlfile.read() horoscope = re.findall(patter, htmltext) print sign print str(horoscope[0]) print '-----------------------------------------' query = 'INSERT INTO horoscope(source, readingTime, zodiac, horoscope) VALUES ("huffingtonpost.com", NOW(), "' + sign + '", "' + str(horoscope[0]) + '")' myCursor.execute(query) db.commit()

Ok now the PHP code, as I said it is very simple code that I use just to view the information.

PHP Code:

<!DOCTYPE html> <?php try { $con=mysqli_connect("localhost","root","","test"); if (mysqli_connect_errno()){echo "Failed to connect to MySQL: " . mysqli_connect_error();} $query = "select * from horoscope order by zodiac, readingTime desc"; $result = mysqli_query($con, $query); while($row = mysqli_fetch_array($result)) { echo "Zodiac: ".$row['zodiac']."<br />"; echo "Source: ".$row['source']."<br />"; echo "Time: ".$row['readingTime']."<br />"; echo "Horoscope: <br/>".$row['horoscope']."<br />"; echo "<hr />"; } mysqli_close($con); } catch(Exception $e){echo 'Message: ' .$e->getMessage();} ?>

And here is the output:



OK, as I said (and again) this is a study case, please let me know what you think about this spaghetti solution!

Thanks

Ligeti

[note] this thread is exclusive for HC.