[noize]

Introduction to AWK

This is a simple introduction to the AWK programming language. If you already have some experience in other programming languages, it would be better, as long as in this tutorial some concepts (what are functions, void functions, variables, strings, etc) are given for already-known.

- noize at www.hackcommunity.com -

AWK is an interpretated language developed at the Bell Labs in the 1970s (though it's been released only in 1979) by Alfred Aho, Peter Weinberger and Brian Kernighan (AWK is an acronym of the last names of its authors).

AWK is one of the languages that mostly inspired Perl.

Program and subroutines boundaries

A BEGIN instruction denotes the first of the three main structures of the script (though it could also be the only one or even missing).

Code:

BEGIN { # first main pattern of the program # (sharp character is a comment definer) }

Printing strings, variables and algebrical results

To display strings, variables, algebrical operations' results and such you can use the print command.

Here's the Hello, World! program in AWK:

Code:

BEGIN { print "Hello, world!" # identation is not a must }

or also:

Code:

BEGIN { print "Hello, world!" }

For defining and printing variables and mathematical operating:

Code:

BEGIN { print "print a string."; } { myVar = "myString"; print myVar; # prints "myString" myRes = 7+3; print myRes; # prints 10 } END { print 5 - 2; # prints 3 }

As you can see from the above script, AWK divides the code structure in 3 main blocks:
- the BEGIN pattern;
- the action pattern (the main section);
- the END pattern.

Please, note that semicolons (;) as command delimiters are not needed, but are accepted; they're mostly useful to put more commands on the same line.

Awk concatenates strings with variables with no special characters:

Code:

print "string" variable

Though, you might do for instance:

Code:

print "myStr", myVar

and it would be read the same way.

Algebrical operators

[table]
[row]
[cell]
+
[/cell]
[cell]
sum
[/cell]
[/row]
[row]
[cell]
-
[/cell]
[cell]
subtraction
[/cell]
[/row]
[row]
[cell]
*
[/cell]
[cell]
multiplication
[/cell]
[/row]
[row]
[cell]
/
[/cell]
[cell]
division
[/cell]
[/row]
[row]
[cell]
%
[/cell]
[cell]
modulo
[/cell]
[/row]
[row]
[cell]
++
[/cell]
[cell]
increment
[/cell]
[/row]
[row]
[cell]
--
[/cell]
[cell]
decrease
[/cell]
[/row]
[/table]

The only string operation in AWK is concatenation, and we already saw, it is used with a whitespace.

AWK supports C-like increase and decrease operators:

Code:

BEGIN { x=1 } { x++ } END { print x # prints 2 }

And the same goes for the decrease operator (--). Else than as standalone assigners, they can be also used within other commands:

Code:

x=10 print x++

The above code will print 10 as, when the operator follows the variable name, we first print its value and then assign it to the new value. If we wanted instead to first assign it to a new value, we just add to put the operator before the variable name:

Code:

x=5 print x++ # prints 5 print --x # prints 5 print ++x # prints 6 print x-- # prints 6 print x # prints 5

Now, let's take the code bit by bit.

x is 5, we print x++ and we output 5 as long as the operator was following the x.
Now x is 6, we print --x but we still get 5 'cause we first decrease the value of x and then print it.
Now x is 5 again, we first increase the value of x and then print 6.
Then print x and then decrease the value of x.
Now x is 5 again.

Functions

Simple sum and subtraction void functions:

Code:

BEGIN { function sum(a,b) { print a + b } } { function sub(x,j) { print x - j } sum(4,9) # echoes 13 } END { sub(13,9) # echoes 4 }

Non-void functions:

Code:

BEGIN { function myFunc() { return "myStr" } } END { print myFunc() # prints "myStr" }

Function value to variable value:

Code:

BEGIN { function do() { return 1+3; } } { done=do(); # setting done variable's value to 4 } END { print done; # printing 4 }

Loops

The for loop is pretty much like in C, C++, Lua, PHP, etc.

The syntax is:

Code:

for (startingDefinition;loopCondition;increment) { # looped }

"startingDefinition" is a variable definition (e.g: i=1).
"loopCondition" is the condition for the loop to exist (e.g: i<10).
"increment" is the variation of the variable's value (e.g: i++).

An example loop:

Code:

BEGIN { for (i=10;i<=100;i++) print i }

Code:

BEGIN { for (i=10;i<101;i=i+1) print i }

Both of the above scripts will output all integer numbers between 9 and 101.

For the while loop it's all just the same.

Code:

BEGIN { while (a >= 0) print "a is greater or equal to 0." }

For both the for and the while loops, if there is more than one statement to be looped, you should either put it all on one line using semicolons as delims like this:

Code:

while (1==1) print "Hello,"; print "world!";

Or you shall use curly brackets:

Code:

while (i <= 100) { print "Hello," print "world!" }

There's one more form for the while loop, which is do while.

Code:

do print "Hello, world!" print "something more" while ( 1 + 1 == 2 )

To break a loop you can use break.

Code:

for (i=1;i=500;i++) { print i if (i >= 100) break }

This will print all numbers from 1 to 100 and then break the for loop.

The IF statement

Code:

BEGIN { if (a <= 100) { # do something print "something" } }

This is the simplest form for "if", which could be put on one line just like "for" or "while", e.g:

Code:

if (1==1) print "this"; print "and also this";

"If, else-if, else" example:

Code:

if (a >= 0) print "a is positive."; else if (a < -100) print "a is less than -100." else if (a <= -50) print "a is less or equal to 50 and greater or equal to -100." else print "a is less than 0 and greater than -50."

AWK uses the double pipe character for the "if this or if that" statement:

Code:

if (i<0 || j<0) { # if either i or j is less than zero print "this" }

And the double "&" for the "if this and if that":

Code:

if (i<0 && j<0) # they're both negative

For disequality checks, you can use !=:

Code:

if (1 != 0) # yes it is

Records handling

The main use of AWK is to handle records. Records are defined by the content of the input files.

Command line example invocation:

Code:

$ cat myinputfile Hello, world! line2 line 3 $ awk -f myAwkScript.awk myinputfile

Now, the default record separator (RS) in AWK is "\n". This means that if myAwkScript.awk was:

Code:

BEGIN { print $2 }

The output would have been:

Code:

world! 3

Now, what is "$2"? It might look as a variable as long as in Perl and PHP variables are denoted by a dollar sign. It is indeed a variable defined by Awk, it is the second field of the each record got from the input file, in this case, the second word of each line of the file. AWK reads input files line by line by default, if we change the RS, we change the record delimitation.

Hi-speed recap:
- $ is the field operator;
- records are the parts in which the input file is divided to be handled;
- fields are the parts in which each record is divided to be handled;
- default RS (record separator) is a newline;
- default FS (field separator) is a whitespace/tab.

Just as the RS variable, we have the FS variable, which is the field separator (default is whitespace/tab).

Code:

BEGIN { RS="\""; # characters are escaped with a backslash in AWK, so we're setting the record separator to a double quote } { print $0, ": ", $1; }

$0 is a special field variable that means the whole record.

So, if the input file was:

Code:

1 John the Ripper 2 Cain and Abel 3 Hack me hard 4 Please, please, please me 5 Yes!

The output would have been:

Code:

1 John the Ripper : 1 2 Cain and Abel : 2 3 Hack me hard : 3 4 Please, please, please me : 4 5 Yes! : 5

An example of user-defined field separation:

Code:

BEGIN { FS="=" } { print "Results:" } END { print $2 }

Where the input file is:

Code:

1+2=3 2 * 4 === 8 3 - 2 == 5 what the fuck, above operation was wrong! 1 ==== 1 never mind me

The output will be:

Code:

Results: 3 8 5 1

There is also an ORS (output record separator) variable, which defines the separator to use between records in the output (default is "\n" - each record on a newline).

Code:

BEGIN { ORS=";\n" } { print $0 }

Let's pretend the input file is:

Code:

- eggs - tomatoes - fucking potatoes

Output will be:

Code:

- eggs; - tomatoes; - fucking potatoes;

If we used this code instead:

Code:

BEGIN { ORS="; " # note there is a space after the semicolon } { print $0 }

We would have got:

Code:

- eggs; - tomatoes; - fucking potatoes;

Let's have a look at the case where we use a non-default RS, FS and ORS:

Code:

BEGIN { RS = ""; } { ORS="." FS=", "; } END { print $0 $1, ", " $2 }

As you might have also noticed from other sample codes, I'm letting you see that you're enough liberty in adding commas, semicolons and spaces as delimiters in the code.
Here, as we set the record separator to an empty string, all the input file will be read as one single record, so we will first output the whole file, then just the first field delimited by a comma followed by a whitespace and then the second field delimited again by a comma and a whitespace.

Let the input file be:

Code:

Once upon a time, there was a story, the end; The very end,.. Now.

First field will be "Once upon a time", second field will be "there was a story" and third field will be "the end;\nThe very end,..\nNow." (the comma followed by the dot won't count as a field separator, as long as we specified ", " and not ",").

Hence the output will be:

Code:

Once upon a time, there was a story, the end; The very end,.. Now. Once upon a time, there was a story.

We said that there shall be a ", " between $1 and $2 in the code, that's not part of the field itself, while the ending dot is the ORS.

Good-to-know: "print" with no argument, prints the current record in its entirety (equal to "print $0").

Arrays and tables

Arrays in AWK are very similar to arrays in many other programming languages.

Code:

arrayName[myIndex]="myString"

Let's see an example:

Code:

BEGIN { myArr[""]=0 # we first need to initialize our array } { myArr[1]="Number 1" myArr[2]="Number 2" myArr[3]="Number 3" } END { print myArr[2] # prints "Number 3" }

For tables it's just the same thing as long as AWK supports associative arrays as a primitive data structure. Example code:

Code:

someTable["foo"]="bar" print someTable["foo"]

If you wanted to print all the values of a complete array with 10 values you could do:

Code:

BEGIN { people[""]=0 } { people[1]="John" people[2]="Mary" people[3]="Billie" people[4]="Jane" people[5]="Jimmy" people[6]="Joe" people[7]="Luke" people[8]="Ringo" people[9]="Stephen" people[10]="Jay" } END { for (i=1; i<=10; i++) { print people[i] } }

Or you could use a better and easier code that works with any kind array or table:

Code:

for (variable in array) { print variable }

This is a use of for we still hadn't explored. Let's pretend we've got our table:

Code:

t["foo"]="bar" t[1]="Hello, world!" t["h"]="h"

This is our code:

Code:

for (v in t) { # t is our table, look at the above code if (v=="Hello, world!") { print v; } } for (v in t) { print v; }

With the above code we could print "Hello, world!" if it is equal to any value in the array, and then print all the values of the array.

Exit statement

The exit statement might act in different ways according to its position in the script.

If it is located in the BEGIN pattern it stops executing any input before even reading any record. If there is any END rule it is executed before closing.

Code:

BEGIN { exit } { print "This code is never executed." } END { print "This code is executed." }

If it is located in the END pattern it quits immediately with no further execution of any command.

Code:

BEGIN {     print "This code is executed." } {     print "This code is executed." } END {     print "This code is executed." exit print "This code is never executed." }

If it is located in any middle pattern that is not either the BEGIN or the END pattern it executes the END pattern (if there is any) and then quits.

Code:

BEGIN {     print "This code is executed." } {     print "This code is executed." exit 0 print "This code is never executed." } END {     print "This code is executed." }

It can be followed by an integer which is the exit code to be returned (default is 0 (the above "exit 0" is equal to "exit")).

---

I might be adding more, as at the very time as I am publishing this, I've learnt of AWK since this afternoon, and so, if anyone has got anything to say or suggest, please, let me know (in the hope that there's someone here who knows (or is at least any interested in) AWK).

Let me know if there's any syntax error (this includes the english part) I didn't notice.

[Deque]

What you described here very well is actually not the main use of awk.

awk is a command line tool to extract data and scripts initially where only written to summarize some command line tasks, but not to have an interactive application that works on its own. The latter is just a side effect, because awk happens to be a turing complete language, but I wouldn't advice learning it to write applications.

This is one example usage on the command line to get the battery status as percentage (the pathes might be different):

Code:

awk '/last full capacity/ {c = $4} /remaining capacity/ {print int( $3/c * 100 )}' /proc/acpi/battery/BAT1/info /proc/acpi/battery/BAT1/state

Here you see the typical structure. First comes the regexp, afterwards the block where you state what to do with the data. New statements are on a new line and the input file(s) come last. So here you have two statements, the first reads the maximum capacity of the battery in the info file and the second reads the remaining capacity and calculates and prints the percentage.

The manual summarizes the general structure like this:

Syntactically, a rule consists of a pattern followed by an action. The action is enclosed in curly braces to separate it from the pattern. Newlines usually separate rules. Therefore, an awk program looks like this:

pattern { action }
pattern { action }
...

That's what actually would be useful to explain in my opinion, how to utilize awk for the command line to extract data.

[noize]

What you described here very well is actually not the main use of awk.

awk is a command line tool to extract data and scripts initially where only written to summarize some command line tasks, but not to have an interactive application that works on its own. The latter is just a side effect, because awk happens to be a turing complete language, but I wouldn't advice learning it to write applications.

This is one example usage on the command line to get the battery status as percentage (the pathes might be different):

Code:

awk '/last full capacity/ {c = $4} /remaining capacity/ {print int( $3/c * 100 )}' /proc/acpi/battery/BAT1/info /proc/acpi/battery/BAT1/state

Here you see the typical structure. First comes the regexp, afterwards the block where you state what to do with the data. New statements are on a new line and the input file(s) come last. So here you have two statements, the first reads the maximum capacity of the battery in the info file and the second reads the remaining capacity and calculates and prints the percentage.

The manual summarizes the general structure like this:

Syntactically, a rule consists of a pattern followed by an action. The action is enclosed in curly braces to separate it from the pattern. Newlines usually separate rules. Therefore, an awk program looks like this:

pattern { action }
pattern { action }
...

That's what actually would be useful to explain in my opinion, how to utilize awk for the command line to extract data.

I've seen pretty much about it, and I didn't know that it was thought first as a command line tool than as a language, I just knew that it is also a command line tool (usually the interpreter (command line tool) is referred to as Awk and the language as AWK), Awk. I also noticed about its ability in handling file contents, which is addressed as the record in AWK, I just wanted to learn more about it before putting it in the paper. I'll soon get this more in depth with it, of course.

Also, the manual you're referring to is the manual for GAWK (GNU AWK), which is a very similar dialect of the AWK language, but not always analog.

[Deque]

Also, the manual you're referring to is the manual for GAWK (GNU AWK), which is a very similar dialect of the AWK language, but not always analog.

GNU AWK is not a dialect, it's one of the main language implementations of AWK. It just provides some more features than the AWK specification requires. That means everything that is AWK works with GNU AWK. The guide covers both: AWK as specified in POSIX and GAWK specific features.

[Psycho_Coder]

@Deque Should you include this in the programming language introduction thread of yours ?
I think it would be convenient to include it in that thread of yours miling:

[noize]

Also, the manual you're referring to is the manual for GAWK (GNU AWK), which is a very similar dialect of the AWK language, but not always analog.

GNU AWK is not a dialect, it's one of the main language implementations of AWK. It just provides some more features than the AWK specification requires. That means everything that is AWK works with GNU AWK. The guide covers both: AWK as specified in POSIX and GAWK specific features.

Yess, you're right again. Still, back to roots, not everything you see in the GAWK manual is valid for AWK just as well.

@Deque Should you include this in the programming language introduction thread of yours ?
I think it would be convenient to include it in that thread of yours miling:

Eh, I don't know, I mean, this is pretty much of a shoddy tutorial, if this goes in that list it would be just because no one else on HC probably ever made an AWK tutorial. At least she might want to wait until I add the part about records handling.

[Deque]

That's right, I will wait for the other part.

[The Alchemist]

Very well made tutorial... You're HQ...

[noize]

Update: added some info in different chapters (like algebrical operators et cetera), but first of all added the part about records and fields.

Edit: will soon add something about arrays and a few minor commands I still haven't talked about.

[noize]

Update: added part about arrays and tables and the exit statement.

Will probably soon add something about some commands to handle user input (interactive scripts) and more commands for records handling.

Edit: @Deque, I added the part about records the 6/20, if you were waiting for that.