[Eclipse]

Limited in what regard? Unless these files have over 500 million lines, Python will be able to store them correctly.

Why take my word for it, though, when you can try it yourself.

That's reassuring.

It will consume a fuckload of memory, yes. Instead of using lists, you can just use temporary files though. That way they are written to disk and not stored in RAM. When you check to see if the line already exists, read the file in predetermined chunk sizes to save time (powers of two work nicely).

I was thinking of doing this; writing the lists into .txt files and then iterating through them.