+- Sinisterly (https://sinister.li)
+-- Forum: Coding (https://sinister.li/Forum-Coding)
+--- Forum: Visual Basic & .NET Framework (https://sinister.li/Forum-Visual-Basic-NET-Framework)
+--- Thread: Handling huge wordlists (/Thread-Handling-huge-wordlists)
Pages:
1
2
Handling huge wordlists - Coder-san - 04-19-2013
How to handle huge wordlists? Like this one here:-
[link=http://www.mediafire.com/?xs4tubdach8q0hr]26.6GB_wordlist_4m_code46342.exe (33MB)[/link]
I'm thinking of something like reading around 500,000 words at a time. But how can I do something like that?
Code:
Dim words As String() = loadWords("C:\wordlist.txt",3500000) 'Arguments being - location, starting point)I'm a self-taught casual programmer, so I may have missed some important basic file I/O operations. So pardon me on that.
Handling huge wordlists - Coder-san - 04-19-2013
How to handle huge wordlists? Like this one here:-
[link=http://www.mediafire.com/?xs4tubdach8q0hr]26.6GB_wordlist_4m_code46342.exe (33MB)[/link]
I'm thinking of something like reading around 500,000 words at a time. But how can I do something like that?
Code:
Dim words As String() = loadWords("C:\wordlist.txt",3500000) 'Arguments being - location, starting point)I'm a self-taught casual programmer, so I may have missed some important basic file I/O operations. So pardon me on that.
RE: Handling huge wordlists - Deque - 04-19-2013
A stream should do fine. I can't help you with the details as I am not familiar with VB, but streams load the files in chunks whose size you can usually specify.
RE: Handling huge wordlists - Deque - 04-19-2013
A stream should do fine. I can't help you with the details as I am not familiar with VB, but streams load the files in chunks whose size you can usually specify.
RE: Handling huge wordlists - Coder-san - 04-19-2013
Hmm I'll have to read about stream then. I'll bump this thread if I get anything useful.
RE: Handling huge wordlists - Coder-san - 04-19-2013
Hmm I'll have to read about stream then. I'll bump this thread if I get anything useful.
RE: Handling huge wordlists - Fredjes - 04-19-2013
Yes a FileStream should work fine, you can visit the Microsoft-website for more details. (http://msdn.microsoft.com/en-us/library/system.io.filestream.aspx)
RE: Handling huge wordlists - Fredjes - 04-19-2013
Yes a FileStream should work fine, you can visit the Microsoft-website for more details. (http://msdn.microsoft.com/en-us/library/system.io.filestream.aspx)
RE: Handling huge wordlists - ArkPhaze - 04-22-2013
You have to deal with streams on your own here. For instance, the StreamReader, and not a wrapper for it like the File methods from the System.IO namespace. It's not hard to keep a buffer though. If a word per line is how the file is formatted, then use a for loop to call ReadLine() x number of times, and append the data to a StringBuilder or a generic List<string> if you prefer. If it's fixed, you could just use an array as well for speed. After the data is collected, deal with it, and repeat the process by clearing out the buffer and appending x number of lines to your storage, whatever that may be.
RE: Handling huge wordlists - Coder-san - 04-22-2013
I can now read a huge file in chunks, but the problem is I can't seem to integrate this into my multithreaded bruteforcer.
To fasten the task I loop 10 (user-input) threads which perform the dictionary attack, and for that I'm storing words in Queues.
But now the queue needs to be filled in chunks and hence is causing problems. I got myself confused.
I can integrate both the "loading in chunks" function and dictionary-attack function, and run those in multiple threads but not sure if that would be fast.