[phyrrus9]

Wouldn't using lists be useless for large text files?


Same as above; isn't using lists limited if, say, you have three 4GB files?

It will consume a fuckload of memory, yes. Instead of using lists, you can just use temporary files though. That way they are written to disk and not stored in RAM. When you check to see if the line already exists, read the file in predetermined chunk sizes to save time (powers of two work nicely).