[chuck19]

hi im trying to scrape a website but alot of the site is in java script imm programing in java but cant find a way to do this. i have tryed jsooup,text oly browses and automation but nothing seems to work does anyone know of a good tutorial on scraping ALL text from a site in cluding those wrped in javascript r a text only browser that can support javascript apart from elinks cos it has crap dom support with spidermonkey and csnt grab the text


thanks alot for any replys this is my first time scraping and cant bleave its such a hasstle all i want to do is open a page copy and paste all text into a document but from a programing [erspective this is a reql real pain any help would be great

[Deque]

You should use the library Cobra. It applies changes made by JavaScript to the DOM.

From the website: http://lobobrowser.org/cobra/java-html-parser.jsp

It is Javascript-aware. DOM modifications that occur during parsing will be reflected in the resulting DOM.

Write again if you need more help with this.

[chuck19]

that seems to be what im looking for thanks but i am having a few probs i installed the complete lobo package and download the cobra library and impoted it to ide i got some code that should work from another forum

Code:

package cobtest; import com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl; import java.net.URL; import org.omg.CORBA_2_3.portable.InputStream; import org.w3c.dom.Document; import org.w3c.dom.Element; public class Cobtest { public static void main(String[] args) { UserAgentContext uacontext = new SimpleUserAgentContext(); DocumentBuilderImpl builder = new DocumentBuilderImpl(rcontext); URL url = new URL(TEST_URI); InputStream in = url.openConnection().getInputStream(); try { Document d = builder.parse(new InputSourceImpl(in, TEST_URI, "utf-8")); } finally { System.out.println("blah"); } in.close(); Element ele = d.getElementById("name"); System.out.println(ele.getAttribute("value")); } }

the problem is UserAgentConteext still has redlines under and i dont have write package to fix this any ideas on what its looking for as cobras support aint great
thanks

the problem is

[Deque]

that seems to be what im looking for thanks but i am having a few probs i installed the complete lobo package and download the cobra library and impoted it to ide i got some code that should work from another forum

Code:

package cobtest; import com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl; import java.net.URL; import org.omg.CORBA_2_3.portable.InputStream; import org.w3c.dom.Document; import org.w3c.dom.Element; public class Cobtest { public static void main(String[] args) { UserAgentContext uacontext = new SimpleUserAgentContext(); DocumentBuilderImpl builder = new DocumentBuilderImpl(rcontext); URL url = new URL(TEST_URI); InputStream in = url.openConnection().getInputStream(); try { Document d = builder.parse(new InputSourceImpl(in, TEST_URI, "utf-8")); } finally { System.out.println("blah"); } in.close(); Element ele = d.getElementById("name"); System.out.println(ele.getAttribute("value")); } }

the problem is UserAgentConteext still has redlines under and i dont have write package to fix this any ideas on what its looking for as cobras support aint great
thanks

the problem is

What is the error message of your IDE?
You probably have to import it.

Code:

import org.lobobrowser.html.UserAgentContext;

[chuck19]

it seems i dont have lobo package lobo is a brwser which i have installed and can use to surf the net lobo uses cobra from parsing and can be impoorted to ide cant find a library for lobo anywhere i tryed just importing the jar from the browser but this didiint work as it aint a library.

am i completely missunderstanding this? i have been searching for what i f-ed up all night thanks for pointing me in right direction though the is very little knwon about cobra on ohter forums thanks

---

sorry the error states that
package org.lobobrowser.html does not exist
but
import org.omg.CORBA_2_3.portable.InputStream;
is fine
but lobo aint a package to be imported its a full on java browseer using cobra??
i cant find anything in lobo folder that can be imported as a library but defo have cobra imported and lobo does work cos i can surf with it

---

these are all of the stuff i have imported to package manually
/home/jyt/.Desktop/jsoup-1.7.2.jar
/home/jyt/NetBeansProjects/jsoup-1.7.2-javadoc.jar
/home/jyt/NetBeansProjects/cobra-0.98.4.zip
/home/jyt/NetBeansProjects/install-lobo-0.98.4.jar
/home/jyt/NetBeansProjects/htmlunit-2.12-src.zip
/home/jyt/NetBeansProjects/Lobo/lobo-pub.jar//this is just from lobo brwser i dont think its doing anything and doeint fix any import issues
/home/jyt/NetBeansProjects/Lobo/lobo.jar//this is just from lobo brwser i dont think its doing anything and doeint fix any import issues

[Deque]

it seems i dont have lobo package lobo is a brwser which i have installed and can use to surf the net lobo uses cobra from parsing and can be impoorted to ide cant find a library for lobo anywhere i tryed just importing the jar from the browser but this didiint work as it aint a library.

am i completely missunderstanding this? i have been searching for what i f-ed up all night thanks for pointing me in right direction though the is very little knwon about cobra on ohter forums thanks

---

sorry the error states that
package org.lobobrowser.html does not exist
but
import org.omg.CORBA_2_3.portable.InputStream;
is fine
but lobo aint a package to be imported its a full on java browseer using cobra??
i cant find anything in lobo folder that can be imported as a library but defo have cobra imported and lobo does work cos i can surf with it

---

these are all of the stuff i have imported to package manually
/home/jyt/.Desktop/jsoup-1.7.2.jar
/home/jyt/NetBeansProjects/jsoup-1.7.2-javadoc.jar
/home/jyt/NetBeansProjects/cobra-0.98.4.zip
/home/jyt/NetBeansProjects/install-lobo-0.98.4.jar
/home/jyt/NetBeansProjects/htmlunit-2.12-src.zip
/home/jyt/NetBeansProjects/Lobo/lobo-pub.jar//this is just from lobo brwser i dont think its doing anything and doeint fix any import issues
/home/jyt/NetBeansProjects/Lobo/lobo.jar//this is just from lobo brwser i dont think its doing anything and doeint fix any import issues

The install-lobo.jar is an installer for the lobo browser and not an API.
It seems you need the Lobo API instead: http://lobobrowser.org/browser/api-info.jsp

Run or extract install-lobo.jar and then go into the installation directory and add the .jars as described here to your project:

lobo.jar, lobo-pub.jar and any other JAR files in the main installation directory should be added to the CLASSPATH. JAR files under the ext directory should not, as they are loaded dynamically. But they are expected to retain their placement relative to lobo.jar.

[chuck19]

in the end selenium done the job thanks for help

[chuck19]

in the end selenium done the job thanks for help

[chuck19]

in the end selenium done the job thanks for help