Tor_Crawler. - Printable Version

Tor_Crawler. - Printable Version

+- Sinisterly (https://sinister.li)
+-- Forum: Coding (https://sinister.li/Forum-Coding)
+--- Forum: Python (https://sinister.li/Forum-Python)
+--- Thread: Tor_Crawler. (/Thread-Tor-Crawler)

Tor_Crawler. - Hack3rcon - 02-23-2020

Hello,
I downloaded the Tor_Crawler from below URL:

Code:
https://github.com/alex-miller-0/Tor_Crawler

But I can't use it and got below error:

Code:
$ python3.7m
Python 3.7.3 (default, Dec 20 2019, 18:57:59) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from TorCrawler import TorCrawler
>>> my = TorCrawler()
Traceback (most recent call last):
  File "/home/jason/.local/lib/python3.7/site-packages/socks.py", line 809, in connect
    negotiate(self, dest_addr, dest_port)
  File "/home/jason/.local/lib/python3.7/site-packages/socks.py", line 444, in _negotiate_SOCKS5
    self, CONNECT, dest_addr)
  File "/home/jason/.local/lib/python3.7/site-packages/socks.py", line 533, in _SOCKS5_request
    raise SOCKS5Error("{:#04x}: {}".format(status, error))
socks.SOCKS5Error: 0x05: Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
  File "/home/jason/.local/lib/python3.7/site-packages/socks.py", line 47, in wrapper
    return function(*args, **kwargs)
  File "/home/jason/.local/lib/python3.7/site-packages/socks.py", line 814, in connect
    raise GeneralProxyError("Socket error", error)
socks.GeneralProxyError: Socket error: 0x05: Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f02e10a5278>: Failed to establish a new connection: Socket error: 0x05: Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jason/.local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='www.icanhazip.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f02e10a5278>: Failed to establish a new connection: Socket error: 0x05: Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jason/.local/lib/python3.7/site-packages/TorCrawler.py", line 139, in __init__
    self.ip = self.check_ip()
  File "/home/jason/.local/lib/python3.7/site-packages/TorCrawler.py", line 256, in check_ip
    return requests.get("http://www.icanhazip.com").text[:-2]
  File "/home/jason/.local/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/home/jason/.local/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/jason/.local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/jason/.local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/jason/.local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='www.icanhazip.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f02e10a5278>: Failed to establish a new connection: Socket error: 0x05: Connection refused'))

Tor is running:

Code:
# service tor status tor
● tor.service - Anonymizing overlay network for TCP (multi-instance-master)
   Loaded: loaded (/lib/systemd/system/tor.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sun 2020-02-23 12:00:45 +0330; 10min ago
  Process: 17252 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 17252 (code=exited, status=0/SUCCESS)

Feb 23 12:00:45 Xen-aka-X3n systemd[1]: Starting Anonymizing overlay network for TCP (multi-instance-mast
Feb 23 12:00:45 Xen-aka-X3n systemd[1]: Started Anonymizing overlay network for TCP (multi-instance-maste

How can I solve it?

Thank you.

RE: Tor_Crawler. - sleaze - 02-23-2020

iirc tor isn't configured as a socks proxy out of the box
https://tor.stackexchange.com/questions/3578/how-do-i-setup-tor-on-debian-for-secure-use-as-a-socks5-proxy

RE: Tor_Crawler. - Hack3rcon - 02-24-2020

No, it is configured.
Can I share "torrc" content?

RE: Tor_Crawler. - Darkbyte - 03-23-2020

Yeah im late to this party apparently , but.. tor is a socks proxy out of the box yes on port 9050 by default. I have been using it lately and auto switching ip every 10 seconds rather than using proxys for some cracking configs.

Requests does not come with out of the box socks support so:
pip install pysocks

if all else fails , its a tiny script and its4 years old so it is probably made for a early version of python 3 , I would just re-code the web request code to use requests over tor heres a little wrapper I made:

Code:
import requests

import shutil


requests.packages.urllib3.disable_warnings()


class HttpWrapper:
    def __init__(self,proxy = "localhost:9050"):

        self.headers =    headers = {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate, sdch, br",
        "Accept-Language": "it-IT,it;q=0.8,en-US;q=0.6,en;q=0.4",
        "Connection": "keep-alive",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36",
    }
        self.s = requests.Session()
        # self.s.cookies=thecookie
        self.useproxy = useproxy
        self.proxyDict = {
            "http": "socks5://" + proxy,
            "https": "socks5://" + proxy
        }


    def load_cookies_from_lwp(self, filename):
        lwp_cookiejar = cookielib.LWPCookieJar()
        lwp_cookiejar.load(filename, ignore_discard=True)
        return lwp_cookiejar

    def setuseragent(self, theagent):

        self.headers = {'user-agent': theagent}

    def save_cookies_lwp(self, filename):
        lwp_cookiejar = cookielib.LWPCookieJar()
        for c in self.s.cookies:
            args = dict(vars(c).items())
            args['rest'] = args['_rest']
            del args['_rest']
            c = cookielib.Cookie(**args)
            lwp_cookiejar.set_cookie(c)
        lwp_cookiejar.save(filename, ignore_discard=True)

    def get(self, theurl, refferer=""):

        if not refferer == "":
            self.headers["referer"] = refferer
        if self.useproxy == True:
            r = self.s.get(theurl, headers=self.headers, proxies=self.proxyDict, verify=False, timeout=15.000)
        else:

            r = self.s.get(theurl, headers=self.headers, verify=False, timeout=15.000)
        return r.text

    def post(self, theurl, data, refferer=""):
        if not refferer == "":
            self.headers["referer"] = refferer
        if self.useproxy == True:

            r = self.s.post(theurl, data=data, headers=self.headers, proxies=self.proxyDict, verify=False,
                            timeout=10.000)
        else:
            r = self.s.post(theurl, data=data, headers=self.headers, verify=False, timeout=10.000)

        return r.text


    def download(self,url,file):
        response = requests.get(url, stream=True)
        with open(file, 'wb') as out_file:
            shutil.copyfileobj(response.raw, out_file)
        del response

Then we can just use something like

get:

Code:
from HttpWrapper import HttpWrapper
wrapper = HttpWrapper()
html = wrapper.get("https://www.somelink.onion")
print(html)

post

Code:
postdata = {"username" : "MyUsername",
	"password": "Mypassword"}

html = wrapper.post("https://www.somelink.onion",postdata)