Split large text files into smaller files

Split large text files into smaller files - Printable Version

+- Sinisterly (https://sinister.li)
+-- Forum: Coding (https://sinister.li/Forum-Coding)
+--- Forum: Python (https://sinister.li/Forum-Python)
+--- Thread: Split large text files into smaller files (/Thread-Split-large-text-files-into-smaller-files)

Split large text files into smaller files - RogueCoder - 05-17-2013

I made this script because I have some very large wordlists, one with over 14 million lines. So I wanted to split this into smaller files.

Syntax

Code:
python splitfile.py [file] [chunk size]

Example

Code:
python splitfile.py rockyou.txt 500000

This will generate files with 500000 lines in each

Code:
#!/usr/bin/python

import sys
import math
import re

def makeFilename(number, chunks):
    length = str(len(str(chunks)))
    format = 'output-%0' + length + 'd'
    return format % (number)

if len(sys.argv) != 3:
    sys.exit('Syntax error: ./splitfile.py <filename> [lines per file]')
else:
    filename = sys.argv[1]
    if not re.match('^\d+$', sys.argv[2]):
        sys.exit('Chunk size must be a number')
    try:
        with open(filename) as f:
            lines = f.readlines()
	    total = len(lines)
            chunk_size = int(sys.argv[2])
	    chunks = (total / chunk_size) + 1
            
            i = 0
            j = 1
            fout = open(makeFilename(j, chunks), 'wb')
            print 'Writing file #' + str(j) + ' of ' + str(chunks)
            for line in lines:
                fout.write(line)
                if i % chunk_size == 0:
                    fout.close()
                    fout = open(makeFilename(j, chunks), 'wb')
                    print 'Writing file #' + str(j) + ' of ' + str(chunks)
                    j += 1
                i += 1
            fout.close()


    except IOError:
        sys.exit('IOError: Unable to read file')

Like always, any feedback and suggestions for improvement is very much appreciated Smile

Code:
python splitfile.py [file] [chunk size]

Example

Code:
python splitfile.py rockyou.txt 500000

This will generate files with 500000 lines in each

Code:
#!/usr/bin/python

import sys
import math
import re

def makeFilename(number, chunks):
    length = str(len(str(chunks)))
    format = 'output-%0' + length + 'd'
    return format % (number)

if len(sys.argv) != 3:
    sys.exit('Syntax error: ./splitfile.py <filename> [lines per file]')
else:
    filename = sys.argv[1]
    if not re.match('^\d+$', sys.argv[2]):
        sys.exit('Chunk size must be a number')
    try:
        with open(filename) as f:
            lines = f.readlines()
	    total = len(lines)
            chunk_size = int(sys.argv[2])
	    chunks = (total / chunk_size) + 1
            
            i = 0
            j = 1
            fout = open(makeFilename(j, chunks), 'wb')
            print 'Writing file #' + str(j) + ' of ' + str(chunks)
            for line in lines:
                fout.write(line)
                if i % chunk_size == 0:
                    fout.close()
                    fout = open(makeFilename(j, chunks), 'wb')
                    print 'Writing file #' + str(j) + ' of ' + str(chunks)
                    j += 1
                i += 1
            fout.close()


    except IOError:
        sys.exit('IOError: Unable to read file')

Like always, any feedback and suggestions for improvement is very much appreciated Smile

RE: Split large text files into smaller files - Ex094 - 05-17-2013

You could have used itertools http://docs.python.org/2/library/itertools.html#recipes It makes your work a little bit easier Smile

RE: Split large text files into smaller files - RogueCoder - 05-17-2013

Thanks @Ex094 Smile

I'll have look into that

RE: Split large text files into smaller files - RogueCoder - 05-17-2013

Thanks @Ex094 Smile

I'll have look into that

RE: Split large text files into smaller files - MrGeek - 05-17-2013

Nice dude Biggrin

I will add this to index.

RE: Split large text files into smaller files - MrGeek - 05-17-2013

Nice dude Biggrin

I will add this to index.

RE: Split large text files into smaller files - RogueCoder - 05-17-2013

(05-17-2013, 05:32 PM)MrGeek Wrote: Nice dude
I will add this to index.

Thanks @MrGeek Smile

I'm feel honored being on that list Cool

RE: Split large text files into smaller files - RogueCoder - 05-17-2013

(05-17-2013, 05:32 PM)MrGeek Wrote: Nice dude
I will add this to index.

Thanks @MrGeek Smile

I'm feel honored being on that list Cool