Split large text files into smaller files 05-17-2013, 04:46 PM
#1
I made this script because I have some very large wordlists, one with over 14 million lines. So I wanted to split this into smaller files.
Syntax
Example
This will generate files with 500000 lines in each
Like always, any feedback and suggestions for improvement is very much appreciated
Syntax
Code:
python splitfile.py [file] [chunk size]Example
Code:
python splitfile.py rockyou.txt 500000Code:
#!/usr/bin/python
import sys
import math
import re
def makeFilename(number, chunks):
length = str(len(str(chunks)))
format = 'output-%0' + length + 'd'
return format % (number)
if len(sys.argv) != 3:
sys.exit('Syntax error: ./splitfile.py <filename> [lines per file]')
else:
filename = sys.argv[1]
if not re.match('^\d+$', sys.argv[2]):
sys.exit('Chunk size must be a number')
try:
with open(filename) as f:
lines = f.readlines()
total = len(lines)
chunk_size = int(sys.argv[2])
chunks = (total / chunk_size) + 1
i = 0
j = 1
fout = open(makeFilename(j, chunks), 'wb')
print 'Writing file #' + str(j) + ' of ' + str(chunks)
for line in lines:
fout.write(line)
if i % chunk_size == 0:
fout.close()
fout = open(makeFilename(j, chunks), 'wb')
print 'Writing file #' + str(j) + ' of ' + str(chunks)
j += 1
i += 1
fout.close()
except IOError:
sys.exit('IOError: Unable to read file')Like always, any feedback and suggestions for improvement is very much appreciated
![[+]](https://sinister.li/images/modern/collapse_collapsed.png)
