+- Sinisterly (https://sinister.li)
+-- Forum: Coding (https://sinister.li/Forum-Coding)
+--- Forum: Coding (https://sinister.li/Forum-Coding--71)
+--- Thread: [Help] Web Crawlers and Bots (/Thread-Help-Web-Crawlers-and-Bots)
[Help] Web Crawlers and Bots - Psycho_Coder - 08-19-2014
Hello,
I want to increase my knowledge more on web content extraction and web crawling. I have run out of ideas about what to crawl and what kinds of bots I can make.
I am not sure if I should used the term bot here.
If you have some idea do throw them here.
Thank you,
Sincerely,
Psycho_Coder.
RE: [Help] Web Crawlers and Bots - Deque - 08-19-2014
Have a look at these:
http://www.engpaper.com/web-crawler-2012-research-papers.htm
http://research.microsoft.com/pubs/102936/eds-webcrawlerarchitecture.pdf
https://evilzone.org/tutorials/analysis-and-construction-of-spambots/
http://cis.poly.edu/suel/papers/crawl.pdf
http://www.bagualu.net/linux/crawler.pdf
http://support.sas.com/resources/papers/proceedings10/053-2010.pdf
http://www.ldodds.com/projects/slug/slug-a-semantic-web-crawler.pdf
http://www.econstor.eu/bitstream/10419/52173/1/67255853X.pdf
http://ijcsi.org/papers/IJCSI-8-6-1-309-313.pdf
RE: [Help] Web Crawlers and Bots - Psycho_Coder - 08-19-2014
(08-19-2014, 03:14 PM)Deque Wrote: Have a look at these:
http://www.engpaper.com/web-crawler-2012-research-papers.htm
http://research.microsoft.com/pubs/102936/eds-webcrawlerarchitecture.pdf
https://evilzone.org/tutorials/analysis-and-construction-of-spambots/
http://cis.poly.edu/suel/papers/crawl.pdf
http://www.bagualu.net/linux/crawler.pdf
http://support.sas.com/resources/papers/proceedings10/053-2010.pdf
http://www.ldodds.com/projects/slug/slug-a-semantic-web-crawler.pdf
http://www.econstor.eu/bitstream/10419/52173/1/67255853X.pdf
http://ijcsi.org/papers/IJCSI-8-6-1-309-313.pdf
Now thats what I call a great feedback
RE: [Help] Web Crawlers and Bots - dropzon3 - 08-19-2014
You can always move into building game bots for browser based mmorpgs. It combines both the scraping aspect and logic aspects as you try to play like a normal player.
RE: [Help] Web Crawlers and Bots - dropzon3 - 08-19-2014
You can always move into building game bots for browser based mmorpgs. It combines both the scraping aspect and logic aspects as you try to play like a normal player.
RE: [Help] Web Crawlers and Bots - Singularity_mybb_import13102 - 08-20-2014
I was bored and I created a script that was scrapping entire PHPBB forums and inserting the content into a local database.
I later realized that something went wrong in it, so I have to fix some thing, but it was sort of fun.
I'm thankful for @Deque's feedback, as I'm also looking for new things to do in this field of programming.
RE: [Help] Web Crawlers and Bots - Singularity_mybb_import13102 - 08-20-2014
I was bored and I created a script that was scrapping entire PHPBB forums and inserting the content into a local database.
I later realized that something went wrong in it, so I have to fix some thing, but it was sort of fun.
I'm thankful for @Deque's feedback, as I'm also looking for new things to do in this field of programming.