[Eclipse]

Here's the scenario:

There are a variable number of .txt files and each file has a large number of lines. However, there may be duplicate lines across the files. How would I go through the files and delete all but one occurrence of duplicate lines?

Eclipse

[Dyme]

http://pastebin.com/PbbKmZ9h

Code:

#!/usr/bin/python import os files = [] flines = [] for fname in os.listdir('.'): if fname[-3:] == 'txt': files.append(fname) for f in files: for line in open(f, 'r').readlines(): if line not in flines: flines.append(line) cfile = open(f, 'w') for line in flines: cfile.write(line) cfile.close() flines = []

[Eclipse]

http://pastebin.com/PbbKmZ9h

Code:

#!/usr/bin/python import os files = [] flines = [] for fname in os.listdir('.'): if fname[-3:] == 'txt': files.append(fname) for f in files: for line in open(f, 'r').readlines(): if line not in flines: flines.append(line) cfile = open(f, 'w') for line in flines: cfile.write(line) cfile.close() flines = []

Surprisingly simple solution, but not really what I was looking for. I'm sorry for not explaining it too well, but the program needs to remove duplicates across files. The program above looks through all the files there and removes any duplicates from each one. Say there's 'onilikesdongs' twice in one file and three times in the other. After the program is run, it should only be written once, in one of the files. Not both.

I'm not looking for a completed program as such, just an idea or a solution that I could then code myself.

[phyrrus9]

heres what you do...pseudocode

open each file in order
read each line
check list to see if it is in there, if it is, delete the line, if not, add it to the list
close each file when EOF is read

It shouldn't be too hard, if you use C++ you could use a vector and strings, but it would be a pain and would use other files to delete a line... maybe

open each file in order
read each line
check list to see if it is in there, if it is, delete the line, if not, add it to the list, then add it to the list for that file
close each file when EOF is read
zero each file
print the contents of each file's list to their respective files

[Adorapuff]

http://pastebin.com/PbbKmZ9h

Code:

#!/usr/bin/python import os files = [] flines = [] for fname in os.listdir('.'): if fname[-3:] == 'txt': files.append(fname) for f in files: for line in open(f, 'r').readlines(): if line not in flines: flines.append(line) cfile = open(f, 'w') for line in flines: cfile.write(line) cfile.close() flines = []

Is that sublime, and if so, what theme is that?
Also, why you always put semi-transparent shell over code?

[Dyme]

Surprisingly simple solution, but not really what I was looking for. I'm sorry for not explaining it too well, but the program needs to remove duplicates across files. The program above looks through all the files there and removes any duplicates from each one. Say there's 'onilikesdongs' twice in one file and three times in the other. After the program is run, it should only be written once, in one of the files. Not both.

I'm not looking for a completed program as such, just an idea or a solution that I could then code myself.

The code I gave you can be easily modified to give your desired results. Add a new list that doesn't get reset with every file, and use it for the line duplicate check. If you still can't get it, lmk and I'll send you the code.

Is that sublime, and if so, what theme is that?
Also, why you always put semi-transparent shell over code?

1. Yes, and it's just the default theme.
2. My terminal is always semi-transparent, and I put the source of whatever I'm demonstrating behind it for artistic styling.

[Eclipse]

heres what you do...pseudocode

open each file in order
read each line
check list to see if it is in there, if it is, delete the line, if not, add it to the list
close each file when EOF is read

It shouldn't be too hard, if you use C++ you could use a vector and strings, but it would be a pain and would use other files to delete a line... maybe

open each file in order
read each line
check list to see if it is in there, if it is, delete the line, if not, add it to the list, then add it to the list for that file
close each file when EOF is read
zero each file
print the contents of each file's list to their respective files

Wouldn't using lists be useless for large text files?

The code I gave you can be easily modified to give your desired results. Add a new list that doesn't get reset with every file, and use it for the line duplicate check. If you still can't get it, lmk and I'll send you the code.

Same as above; isn't using lists limited if, say, you have three 4GB files?

[Dyme]

Wouldn't using lists be useless for large text files?


Same as above; isn't using lists limited if, say, you have three 4GB files?

Limited in what regard? Unless these files have over 500 million lines, Python will be able to store them correctly.

Why take my word for it, though, when you can try it yourself.

[phyrrus9]

Wouldn't using lists be useless for large text files?


Same as above; isn't using lists limited if, say, you have three 4GB files?

It will consume a fuckload of memory, yes. Instead of using lists, you can just use temporary files though. That way they are written to disk and not stored in RAM. When you check to see if the line already exists, read the file in predetermined chunk sizes to save time (powers of two work nicely).

[Eclipse]

Limited in what regard? Unless these files have over 500 million lines, Python will be able to store them correctly.

Why take my word for it, though, when you can try it yourself.

That's reassuring.

It will consume a fuckload of memory, yes. Instead of using lists, you can just use temporary files though. That way they are written to disk and not stored in RAM. When you check to see if the line already exists, read the file in predetermined chunk sizes to save time (powers of two work nicely).

I was thinking of doing this; writing the lists into .txt files and then iterating through them.