Login Register






Web Scraping Platform filter_list
Author
Message
Web Scraping Platform #1
Project Orville



Past Project

Well, as most of you know I was working on a project, Authize, which was a license management system I was going to market here and other forums. Unfortunately there was too much competition and I can no longer see the project as being viable for income. Hence, project Orville.



What is Orville?

Orville will be a GUI-based (currently CLI) web scraping utility with API endpoints and the ability to embed if desired. The goal is to make web scraping faster than is currently possible with Selenium and similar frameworks that actually render a page in a browser. It will also make it easier than the mishmash of libraries people use for scraping in languages such as Python.

I originally started this project because I was working on email scraping for a few investment companies, but decided that other people might also enjoy using what I've created.

Notable features include:

• Scriping support (currently lua, maybe JS later)
• HTTP request building
• Easy cookie and crumb scraping




How much will it cost?

Price is still very much undecided, but there will be a limited free version for trials.




Technical Details

The Orville Framework and application will be written mostly in Go with certain parts done in C/ASM. The API will be accessible through a RESTful API and possible an RPC API.

[+] 1 user Likes Hoss's post
Reply

RE: Web Scraping Platform #2
Sounds like a cool idea. If you do implement JS scripting, you should consider including a jquery-like DOM manipulation framework (e.g. cheerio for node).
It's often the outcasts, the iconoclasts ... those who have the least to lose because they
don't have much in the first place, who feel the new currents and ride them the farthest.

Reply

RE: Web Scraping Platform #3
(07-23-2017, 08:31 PM)Inori Wrote: Sounds like a cool idea. If you do implement JS scripting, you should consider including a jquery-like DOM manipulation framework (e.g. cheerio for node).

Yes, I've actually already designed jQuery-like libraries for both Lua and Go that will be included, as well as other libraries like a robots.txt parser.

[+] 1 user Likes Hoss's post
Reply

RE: Web Scraping Platform #4
I believe you've made a very good decision with your new project. From what I've gathered during my travels on the net, I do think there Is a market for It.

The free trial version Is also a good Idea. It'll allow you to get all sorts of feedback, Inclusive of bugs, Improvements etc.
[Image: AD83g1A.png]

Reply