[Coder-san]

Hello everyone, in this tutorial we will be touching the wonderful world of Regular expressions (Regex) to create a basic "Binary reader" like McAfee's BinText in just a few lines of code.

Introduction to Regex
Lets have a brief introduction on Regex:-

Basically, a regular expression is a pattern describing a certain amount of text.

In computing, a regular expression is a specific pattern that provides concise and flexible means to "match" (specify and recognize) strings of text, such as particular characters, words, or patterns of characters. Common abbreviations for "regular expression" include regex and regexp.

Patterns can be something like:-

Code:

[A-Z] 'This pattern matches any uppercase letter from A to Z

And even:-

Code:

\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b 'This is a more complex pattern to match any text which looks like an email address [Source: regular-expressions.info] 'Lets skip this one for now

If you are a fast learner, for more information about patterns and their usage refer to Regular-expressions.info

Objective and purpose

Objective:
To read only the text which we require, and skip all which we not.

Purpose:
To display textual EOF-data and in between if present

Concept of our program

Issue:
As most of us have seen that a PE file, or most file-types in this matter cannot be read directly in a Text-editor like Notepad.
Why? Because they are not meant to be. The text-editor tries to forcefully read the files as text in ANSI encoding.

Our honest effort:
In this tutorial we will be trying to filter out the letters we understand i.e. A-Z, a-z, and 0-9. Hence remove all other characters from the text obtained from a file.
For this we will prepare a function to input the location of the file and return a filtered text.

Requirements

1. Microsoft Visual Studio
   
2. Basic VB.Net knowledge
   
3. Any executable file
   

Programming

Firstly we will have to import namespace RegularExpressions

Code:

Imports System.Text.RegularExpressions 'at the very top of your code

Next is the use of Regex to remove all unwanted characters:

Code:

Dim TextInFile As String = Regex.Replace(IO.File.ReadAllText(Location), "[^A-Za-z0-9]", " ", RegexOptions.None)

Lets break this in parts:

1. TextInFile is declared as a string variable
   
2. We use the Replace function of Regex. Its arguments are input, pattern, replacement, and RegexOptions
   [table]
   [row]
   [cell] 1 [/cell]
   [cell] input [/cell]
   [cell] Simply the input text, the text to be filtered [/cell]
   [/row]
   [row]
   [cell] 2 [/cell]
   [cell] pattern [/cell]
   [cell] This is where we put our Regex pattern [/cell]
   [/row]
   [row]
   [cell] 3 [/cell]
   [cell] replacement [/cell]
   [cell] The replacement to be done in case of a match [/cell]
   [/row]
   [row]
   [cell] 4 [/cell]
   [cell] RegexOptions[/cell]
   [cell] Some more options for searching and matching. It is an optional argument, by default it is None[/cell]
   [/row]
   [/table]
   

Now that we know about the Regex.Replace function, we'll use it.
[table]
[row]
[cell] 1 [/cell]
[cell] input[/cell]
[cell] IO.File.ReadAllText(Location) 'this will read the file at the "Location" as Text [/cell]
[/row]
[row]
[cell] 2 [/cell]
[cell] pattern[/cell]
[cell] "[^A-Za-z0-9]" 'we come across two symbols here; ^ (caret) and - (hyphen)
- simply means range of characters
^ just after [ matches any character except the ones mentioned afterwards.
"[^A-Za-z0-9]" combined means to match every character except A-Z, a-z, and 0-9[/cell]
[/row]
[row]
[cell] 3 [/cell]
[cell] replacement [/cell]
[cell] " " 'Replacement is a space, to prevent the remaining text from getting merged with each other and to prevent compromising the readability [/cell]
[/row]
[row]
[cell] 4[/cell]
[cell] RegexOptions[/cell]
[cell] RegexOptions.None 'We do not require this currently.[/cell]
[/row]
[/table]

If you'll return the TextInFile right now you'll get tons of unwanted space characters " ". To solve this issue, we can use:-

1. another Regex, or
   
2. Do While Loop to replace all two spaces with 1 spaces till there is no 2 space left.
   

As pointed out by ArkPhaze, Regex can be and should be used here as well.

Code:

TextInFile = Regex.Replace(TextInFile, "\s+", " ")

Here we come across two more symbols \ (backslash) and + (plus)
\ (backslash) followed by s matches a whitespace
But it is not enough on its own. \s will just search for 1 whitespace, and since we won't be knowing how many there are, we'll use the second symbol in this regex + (plus)
+ (plus) repeats the previous item once or more. So when used as \s+, the regex will search for one or more whitespaces.
Altogether, the code will replace one or more whitespaces with a single whitespace.

We can also do this by traditional Do While Loop

Spoiler: Do While Loop

Code:

'Perform this action till a space followed by another space remains Do While TextInFile.Contains(" ") 'Replace all 2 spaces with 1 space, so no consecutive space character remains TextInFile = TextInFile.Replace(" ", " ") Loop

Lastly, we have what we needed and we'll return the variable TextInFile.

Code:

Return TextInFile

The full code:-



And with this we complete the tutorial. Try it on any executable file.
Thanks for reading.

Conclusion / Comments of Author

Regular expressions is a very interesting and useful part of programming according to me, and is a must for everyone.

Bibliography

Websites referred:

1. http://www.regular-expressions.info
   
2. http://en.wikipedia.org/wiki/Regular_expression
   
3. http://stackoverflow.com/questions/70188...nsi-format
   

Special thanks to ArkPhaze for the second Regex

[Coder-san]

Hello everyone, in this tutorial we will be touching the wonderful world of Regular expressions (Regex) to create a basic "Binary reader" like McAfee's BinText in just a few lines of code.

Introduction to Regex
Lets have a brief introduction on Regex:-

Basically, a regular expression is a pattern describing a certain amount of text.

In computing, a regular expression is a specific pattern that provides concise and flexible means to "match" (specify and recognize) strings of text, such as particular characters, words, or patterns of characters. Common abbreviations for "regular expression" include regex and regexp.

Patterns can be something like:-

Code:

[A-Z] 'This pattern matches any uppercase letter from A to Z

And even:-

Code:

\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b 'This is a more complex pattern to match any text which looks like an email address [Source: regular-expressions.info] 'Lets skip this one for now

If you are a fast learner, for more information about patterns and their usage refer to Regular-expressions.info

Objective and purpose

Objective:
To read only the text which we require, and skip all which we not.

Purpose:
To display textual EOF-data and in between if present

Concept of our program

Issue:
As most of us have seen that a PE file, or most file-types in this matter cannot be read directly in a Text-editor like Notepad.
Why? Because they are not meant to be. The text-editor tries to forcefully read the files as text in ANSI encoding.

Our honest effort:
In this tutorial we will be trying to filter out the letters we understand i.e. A-Z, a-z, and 0-9. Hence remove all other characters from the text obtained from a file.
For this we will prepare a function to input the location of the file and return a filtered text.

Requirements

1. Microsoft Visual Studio
   
2. Basic VB.Net knowledge
   
3. Any executable file
   

Programming

Firstly we will have to import namespace RegularExpressions

Code:

Imports System.Text.RegularExpressions 'at the very top of your code

Next is the use of Regex to remove all unwanted characters:

Code:

Dim TextInFile As String = Regex.Replace(IO.File.ReadAllText(Location), "[^A-Za-z0-9]", " ", RegexOptions.None)

Lets break this in parts:

1. TextInFile is declared as a string variable
   
2. We use the Replace function of Regex. Its arguments are input, pattern, replacement, and RegexOptions
   [table]
   [row]
   [cell] 1 [/cell]
   [cell] input [/cell]
   [cell] Simply the input text, the text to be filtered [/cell]
   [/row]
   [row]
   [cell] 2 [/cell]
   [cell] pattern [/cell]
   [cell] This is where we put our Regex pattern [/cell]
   [/row]
   [row]
   [cell] 3 [/cell]
   [cell] replacement [/cell]
   [cell] The replacement to be done in case of a match [/cell]
   [/row]
   [row]
   [cell] 4 [/cell]
   [cell] RegexOptions[/cell]
   [cell] Some more options for searching and matching. It is an optional argument, by default it is None[/cell]
   [/row]
   [/table]
   

Now that we know about the Regex.Replace function, we'll use it.
[table]
[row]
[cell] 1 [/cell]
[cell] input[/cell]
[cell] IO.File.ReadAllText(Location) 'this will read the file at the "Location" as Text [/cell]
[/row]
[row]
[cell] 2 [/cell]
[cell] pattern[/cell]
[cell] "[^A-Za-z0-9]" 'we come across two symbols here; ^ (caret) and - (hyphen)
- simply means range of characters
^ just after [ matches any character except the ones mentioned afterwards.
"[^A-Za-z0-9]" combined means to match every character except A-Z, a-z, and 0-9[/cell]
[/row]
[row]
[cell] 3 [/cell]
[cell] replacement [/cell]
[cell] " " 'Replacement is a space, to prevent the remaining text from getting merged with each other and to prevent compromising the readability [/cell]
[/row]
[row]
[cell] 4[/cell]
[cell] RegexOptions[/cell]
[cell] RegexOptions.None 'We do not require this currently.[/cell]
[/row]
[/table]

If you'll return the TextInFile right now you'll get tons of unwanted space characters " ". To solve this issue, we can use:-

1. another Regex, or
   
2. Do While Loop to replace all two spaces with 1 spaces till there is no 2 space left.
   

As pointed out by ArkPhaze, Regex can be and should be used here as well.

Code:

TextInFile = Regex.Replace(TextInFile, "\s+", " ")

Here we come across two more symbols \ (backslash) and + (plus)
\ (backslash) followed by s matches a whitespace
But it is not enough on its own. \s will just search for 1 whitespace, and since we won't be knowing how many there are, we'll use the second symbol in this regex + (plus)
+ (plus) repeats the previous item once or more. So when used as \s+, the regex will search for one or more whitespaces.
Altogether, the code will replace one or more whitespaces with a single whitespace.

We can also do this by traditional Do While Loop

Spoiler: Do While Loop

Code:

'Perform this action till a space followed by another space remains Do While TextInFile.Contains(" ") 'Replace all 2 spaces with 1 space, so no consecutive space character remains TextInFile = TextInFile.Replace(" ", " ") Loop

Lastly, we have what we needed and we'll return the variable TextInFile.

Code:

Return TextInFile

The full code:-



And with this we complete the tutorial. Try it on any executable file.
Thanks for reading.

Conclusion / Comments of Author

Regular expressions is a very interesting and useful part of programming according to me, and is a must for everyone.

Bibliography

Websites referred:

1. http://www.regular-expressions.info
   
2. http://en.wikipedia.org/wiki/Regular_expression
   
3. http://stackoverflow.com/questions/70188...nsi-format
   

Special thanks to ArkPhaze for the second Regex

[Coder-san]

Hello everyone, in this tutorial we will be touching the wonderful world of Regular expressions (Regex) to create a basic "Binary reader" like McAfee's BinText in just a few lines of code.

Introduction to Regex
Lets have a brief introduction on Regex:-

Basically, a regular expression is a pattern describing a certain amount of text.

In computing, a regular expression is a specific pattern that provides concise and flexible means to "match" (specify and recognize) strings of text, such as particular characters, words, or patterns of characters. Common abbreviations for "regular expression" include regex and regexp.

Patterns can be something like:-

Code:

[A-Z] 'This pattern matches any uppercase letter from A to Z

And even:-

Code:

\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b 'This is a more complex pattern to match any text which looks like an email address [Source: regular-expressions.info] 'Lets skip this one for now

If you are a fast learner, for more information about patterns and their usage refer to Regular-expressions.info

Objective and purpose

Objective:
To read only the text which we require, and skip all which we not.

Purpose:
To display textual EOF-data and in between if present

Concept of our program

Issue:
As most of us have seen that a PE file, or most file-types in this matter cannot be read directly in a Text-editor like Notepad.
Why? Because they are not meant to be. The text-editor tries to forcefully read the files as text in ANSI encoding.

Our honest effort:
In this tutorial we will be trying to filter out the letters we understand i.e. A-Z, a-z, and 0-9. Hence remove all other characters from the text obtained from a file.
For this we will prepare a function to input the location of the file and return a filtered text.

Requirements

1. Microsoft Visual Studio
   
2. Basic VB.Net knowledge
   
3. Any executable file
   

Programming

Firstly we will have to import namespace RegularExpressions

Code:

Imports System.Text.RegularExpressions 'at the very top of your code

Next is the use of Regex to remove all unwanted characters:

Code:

Dim TextInFile As String = Regex.Replace(IO.File.ReadAllText(Location), "[^A-Za-z0-9]", " ", RegexOptions.None)

Lets break this in parts:

1. TextInFile is declared as a string variable
   
2. We use the Replace function of Regex. Its arguments are input, pattern, replacement, and RegexOptions
   [table]
   [row]
   [cell] 1 [/cell]
   [cell] input [/cell]
   [cell] Simply the input text, the text to be filtered [/cell]
   [/row]
   [row]
   [cell] 2 [/cell]
   [cell] pattern [/cell]
   [cell] This is where we put our Regex pattern [/cell]
   [/row]
   [row]
   [cell] 3 [/cell]
   [cell] replacement [/cell]
   [cell] The replacement to be done in case of a match [/cell]
   [/row]
   [row]
   [cell] 4 [/cell]
   [cell] RegexOptions[/cell]
   [cell] Some more options for searching and matching. It is an optional argument, by default it is None[/cell]
   [/row]
   [/table]
   

Now that we know about the Regex.Replace function, we'll use it.
[table]
[row]
[cell] 1 [/cell]
[cell] input[/cell]
[cell] IO.File.ReadAllText(Location) 'this will read the file at the "Location" as Text [/cell]
[/row]
[row]
[cell] 2 [/cell]
[cell] pattern[/cell]
[cell] "[^A-Za-z0-9]" 'we come across two symbols here; ^ (caret) and - (hyphen)
- simply means range of characters
^ just after [ matches any character except the ones mentioned afterwards.
"[^A-Za-z0-9]" combined means to match every character except A-Z, a-z, and 0-9[/cell]
[/row]
[row]
[cell] 3 [/cell]
[cell] replacement [/cell]
[cell] " " 'Replacement is a space, to prevent the remaining text from getting merged with each other and to prevent compromising the readability [/cell]
[/row]
[row]
[cell] 4[/cell]
[cell] RegexOptions[/cell]
[cell] RegexOptions.None 'We do not require this currently.[/cell]
[/row]
[/table]

If you'll return the TextInFile right now you'll get tons of unwanted space characters " ". To solve this issue, we can use:-

1. another Regex, or
   
2. Do While Loop to replace all two spaces with 1 spaces till there is no 2 space left.
   

As pointed out by ArkPhaze, Regex can be and should be used here as well.

Code:

TextInFile = Regex.Replace(TextInFile, "\s+", " ")

Here we come across two more symbols \ (backslash) and + (plus)
\ (backslash) followed by s matches a whitespace
But it is not enough on its own. \s will just search for 1 whitespace, and since we won't be knowing how many there are, we'll use the second symbol in this regex + (plus)
+ (plus) repeats the previous item once or more. So when used as \s+, the regex will search for one or more whitespaces.
Altogether, the code will replace one or more whitespaces with a single whitespace.

We can also do this by traditional Do While Loop

Spoiler: Do While Loop

Code:

'Perform this action till a space followed by another space remains Do While TextInFile.Contains(" ") 'Replace all 2 spaces with 1 space, so no consecutive space character remains TextInFile = TextInFile.Replace(" ", " ") Loop

Lastly, we have what we needed and we'll return the variable TextInFile.

Code:

Return TextInFile

The full code:-



And with this we complete the tutorial. Try it on any executable file.
Thanks for reading.

Conclusion / Comments of Author

Regular expressions is a very interesting and useful part of programming according to me, and is a must for everyone.

Bibliography

Websites referred:

1. http://www.regular-expressions.info
   
2. http://en.wikipedia.org/wiki/Regular_expression
   
3. http://stackoverflow.com/questions/70188...nsi-format
   

Special thanks to ArkPhaze for the second Regex

[ArkPhaze]

Why would you use a Do While loop when you can use the Regex.Replace() function. This tutorial is about Regex isn't it? :whistle:

Regex:

Code:

\s+

Example:

Code:

Dim s As String = "This is a test..." s = Regex.Replace(s, "\s+", " ") Console.WriteLine(s)

[Coder-san]

Nice. Didn't try to look for a way via Regex for that.

[Deque]

It is a good idea to combine teaching regex with creating a basic program for static malware analysis.
Well structured tutorial, the explanations are very good. I really like it.