Checking for Duplicates Across Multiple Files

Checking for Duplicates Across Multiple Files - Printable Version

+- Sinisterly (https://sinister.li)
+-- Forum: Coding (https://sinister.li/Forum-Coding)
+--- Forum: Python (https://sinister.li/Forum-Python)
+--- Thread: Checking for Duplicates Across Multiple Files (/Thread-Checking-for-Duplicates-Across-Multiple-Files)

Pages: 1 2

Checking for Duplicates Across Multiple Files - Eclipse - 01-24-2015

Here's the scenario:

There are a variable number of .txt files and each file has a large number of lines. However, there may be duplicate lines across the files. How would I go through the files and delete all but one occurrence of duplicate lines?

Eclipse

RE: Checking for Duplicates Across Multiple Files - Dyme - 01-24-2015

http://pastebin.com/PbbKmZ9h

Code:
#!/usr/bin/python
import os

files = []
flines = []

for fname in os.listdir('.'):
	if fname[-3:] == 'txt':
		files.append(fname)

for f in files:
	for line in open(f, 'r').readlines():
		if line not in flines:
			flines.append(line)
	cfile = open(f, 'w')
	for line in flines:
		cfile.write(line)
	cfile.close()
	flines = []

RE: Checking for Duplicates Across Multiple Files - Eclipse - 01-25-2015

(01-24-2015, 10:46 PM)Dyme Wrote: http://pastebin.com/PbbKmZ9h

Code:
#!/usr/bin/python import os files = [] flines = [] for fname in os.listdir('.'): if fname[-3:] == 'txt': files.append(fname) for f in files: for line in open(f, 'r').readlines(): if line not in flines: flines.append(line) cfile = open(f, 'w') for line in flines: cfile.write(line) cfile.close() flines = []

Surprisingly simple solution, but not really what I was looking for. I'm sorry for not explaining it too well, but the program needs to remove duplicates across files. The program above looks through all the files there and removes any duplicates from each one. Say there's 'onilikesdongs' twice in one file and three times in the other. After the program is run, it should only be written once, in one of the files. Not both.

I'm not looking for a completed program as such, just an idea or a solution that I could then code myself.

RE: Checking for Duplicates Across Multiple Files - phyrrus9 - 01-25-2015

heres what you do...pseudocode

open each file in order
read each line
check list to see if it is in there, if it is, delete the line, if not, add it to the list
close each file when EOF is read

It shouldn't be too hard, if you use C++ you could use a vector and strings, but it would be a pain and would use other files to delete a line... maybe

open each file in order
read each line
check list to see if it is in there, if it is, delete the line, if not, add it to the list, then add it to the list for that file
close each file when EOF is read
zero each file
print the contents of each file's list to their respective files

RE: Checking for Duplicates Across Multiple Files - Adorapuff - 01-26-2015

(01-24-2015, 10:46 PM)Dyme Wrote: http://pastebin.com/PbbKmZ9h

Code:
#!/usr/bin/python import os files = [] flines = [] for fname in os.listdir('.'): if fname[-3:] == 'txt': files.append(fname) for f in files: for line in open(f, 'r').readlines(): if line not in flines: flines.append(line) cfile = open(f, 'w') for line in flines: cfile.write(line) cfile.close() flines = []

Is that sublime, and if so, what theme is that?
Also, why you always put semi-transparent shell over code?

RE: Checking for Duplicates Across Multiple Files - Dyme - 01-26-2015

(01-25-2015, 09:59 AM)Eclipse Wrote: Surprisingly simple solution, but not really what I was looking for. I'm sorry for not explaining it too well, but the program needs to remove duplicates across files. The program above looks through all the files there and removes any duplicates from each one. Say there's 'onilikesdongs' twice in one file and three times in the other. After the program is run, it should only be written once, in one of the files. Not both.

I'm not looking for a completed program as such, just an idea or a solution that I could then code myself.

The code I gave you can be easily modified to give your desired results. Add a new list that doesn't get reset with every file, and use it for the line duplicate check. If you still can't get it, lmk and I'll send you the code.

(01-26-2015, 04:14 AM)Adorapuff Wrote: Is that sublime, and if so, what theme is that?
Also, why you always put semi-transparent shell over code?

1. Yes, and it's just the default theme.
2. My terminal is always semi-transparent, and I put the source of whatever I'm demonstrating behind it for artistic styling.

RE: Checking for Duplicates Across Multiple Files - Eclipse - 01-26-2015

(01-25-2015, 08:06 PM)phyrrus9 Wrote: heres what you do...pseudocode

open each file in order
read each line
check list to see if it is in there, if it is, delete the line, if not, add it to the list
close each file when EOF is read

It shouldn't be too hard, if you use C++ you could use a vector and strings, but it would be a pain and would use other files to delete a line... maybe

open each file in order
read each line
check list to see if it is in there, if it is, delete the line, if not, add it to the list, then add it to the list for that file
close each file when EOF is read
zero each file
print the contents of each file's list to their respective files

Wouldn't using lists be useless for large text files?

(01-26-2015, 04:23 AM)Dyme Wrote: The code I gave you can be easily modified to give your desired results. Add a new list that doesn't get reset with every file, and use it for the line duplicate check. If you still can't get it, lmk and I'll send you the code.

Same as above; isn't using lists limited if, say, you have three 4GB files?

RE: Checking for Duplicates Across Multiple Files - Dyme - 01-26-2015

(01-26-2015, 02:27 PM)Eclipse Wrote: Wouldn't using lists be useless for large text files?

Same as above; isn't using lists limited if, say, you have three 4GB files?

Limited in what regard? Unless these files have over 500 million lines, Python will be able to store them correctly.

Why take my word for it, though, when you can try it yourself.

RE: Checking for Duplicates Across Multiple Files - phyrrus9 - 01-26-2015

(01-26-2015, 02:27 PM)Eclipse Wrote: Wouldn't using lists be useless for large text files?

Same as above; isn't using lists limited if, say, you have three 4GB files?

It will consume a fuckload of memory, yes. Instead of using lists, you can just use temporary files though. That way they are written to disk and not stored in RAM. When you check to see if the line already exists, read the file in predetermined chunk sizes to save time (powers of two work nicely).

RE: Checking for Duplicates Across Multiple Files - Eclipse - 01-26-2015

(01-26-2015, 03:05 PM)Dyme Wrote: Limited in what regard? Unless these files have over 500 million lines, Python will be able to store them correctly.

Why take my word for it, though, when you can try it yourself.

That's reassuring.

(01-26-2015, 03:18 PM)phyrrus9 Wrote: It will consume a fuckload of memory, yes. Instead of using lists, you can just use temporary files though. That way they are written to disk and not stored in RAM. When you check to see if the line already exists, read the file in predetermined chunk sizes to save time (powers of two work nicely).

I was thinking of doing this; writing the lists into .txt files and then iterating through them.