+- Sinisterly (https://sinister.li)
+-- Forum: Coding (https://sinister.li/Forum-Coding)
+--- Forum: Python (https://sinister.li/Forum-Python)
+--- Thread: Scraping target Twitters for evidence to import into EnCase or similar (/Thread-Scraping-target-Twitters-for-evidence-to-import-into-EnCase-or-similar)
Scraping target Twitters for evidence to import into EnCase or similar - ConcernedCitizen - 06-30-2021
Code below. I was following a Twitter profile of a target and just got annoyed by the amount of links I had to follow. I started automating the process and this was the outcome of my efforts.
It utilizes a crawler to scrape the target profiles for evidence that I use to feed a case-file, but doesn't directly import anything to it, unless you know what you're doing. That's beyond the scope of this small program so figure that out if you are interested. Feedback is welcome.
Code:
#!/usr/bin/python3
# -*- coding: utf-8 -*-
#
# This file is part of twitEvidence.
# Only change this if you know what exactly it does.
# It is obviously traceable without torifying/tsocks.
import subprocess
import shlex
import os
CMD_HTTrack = "httrack {URLS} +* -r1 -O twitter/httrack"
CMD_YDL = "youtube-dl -i {URLS} -o ./twitter/videos/%(title)s.%(ext)s"
# create directories
os.makedirs('./twitter/videos', exist_ok=True)
os.makedirs('./twitter/httrack', exist_ok=True)
# place all links to scrape in file "links"
with open('links', 'r') as f:
# concatenate all urls together
line = f.readline().rstrip('\n')
urls = ''
while line:
urls += line + ' '
line = f.readline().rstrip('\n')
# start scraping using HTTrack
run = subprocess.Popen(shlex.split(CMD_HTTrack.format(
URLS=urls
run.wait()
# also scrape using youtube-dl (videos only)
run = subprocess.Popen(shlex.split(CMD_YDL.format(
URLS=urls
run.wait()