"httpd.conf" Parsing on Bash 06-26-2013, 12:15 AM
#1
"httpd.conf" Parsing on Bash
In this tutorial, we're going to parse the file which contains informations about domain names, web root paths of websites stored in the server to extract these information cleaner. To do this, i use AWK parsing tool.
What is AWK? How does it work?
Let's start with its definition:
AWK, generally, finds the data you want in data given and format it as how you want it to be.
AWK accepts the input as a "data table". So, we can call every line of the file "a row" of that table and every column as a data-field. "Field Seperators" (FS) are used to seperate these columns into "data-field". It's the "space character" unless you specify another char.
Let's get an output to parse with command "ls -l"
If we use that result in AWK as input, we analyze a data-table that has 4 rows.
While using AWK to parse; in every row, the "Number of Fields" (NS) is re-calculated. So, in this example;
In first row, the value "Number of Row" (NR) is 1, NF must be 2.
NOTE: One-based numbering.
So:
NR = 1, NF = 2
NR = 2, NF = 9
AWK Usage:
We can use AWK in two-way.
The first one is the usage on command-line like " cat 1.txt | awk '{print $1}' ". It give an output with the first column of every line in 1.txt.
And the second usage is putting the AWK codes into a file. It provides us more readable coding.
For example, put the codes below into "hc.awk" file.
Then, run it with "cat 1.txt | awk -f hc.awk".
As you see, the output is same as the first one.
The File "httpd.conf"
This file holds Apache server's settings. So options of the websites stored in server could be seen in this file under the "Virtual Host" tags.
A sample of "httpd.conf"
We may say that "DocumentRoot" refers the web-root path and "ServerAlias" does domains.
Parsing the "httpd.conf":
We want to get web-root paths of all websites in "httpd.conf". Before using AWK, to find rows that contains "DocumentRoot", we can use "grep" Linux-command. I copied "httpd.conf" file into "1.txt" file.
And run:
Let's explain these commands:
Firstly, we read the file with "cat 1.txt" and "grep DocumentRoot" to select rows we want. The last command "head" is used to limit output up to 10.
"|" This provides commands using output of previous command.
Yeah, we haven't used AWK yet. We use it only to extract paths.
Our new commands:
Commands are the same except the last one.
" awk '{print $2}' ": We want AWK to print second column namely "paths".
Yeah, as you can see we got all them. You can remove the quotation marks with your editor. Or use "sed" command.
Thanks.
In this tutorial, we're going to parse the file which contains informations about domain names, web root paths of websites stored in the server to extract these information cleaner. To do this, i use AWK parsing tool.
What is AWK? How does it work?
Let's start with its definition:
AWK, generally, finds the data you want in data given and format it as how you want it to be.
AWK accepts the input as a "data table". So, we can call every line of the file "a row" of that table and every column as a data-field. "Field Seperators" (FS) are used to seperate these columns into "data-field". It's the "space character" unless you specify another char.
Let's get an output to parse with command "ls -l"
Code:
# ls -l
total 6
-rw-r--r-- 1 ryland wheel 767 Jun 6 00:04 .cshrc
-rw------- 1 ryland wheel 276 Nov 23 10:16 .history
-rw-r--r-- 1 ryland wheel 248 Jun 6 00:04 .loginIf we use that result in AWK as input, we analyze a data-table that has 4 rows.
While using AWK to parse; in every row, the "Number of Fields" (NS) is re-calculated. So, in this example;
In first row, the value "Number of Row" (NR) is 1, NF must be 2.
NOTE: One-based numbering.
So:
NR = 1, NF = 2
NR = 2, NF = 9
AWK Usage:
We can use AWK in two-way.
The first one is the usage on command-line like " cat 1.txt | awk '{print $1}' ". It give an output with the first column of every line in 1.txt.
And the second usage is putting the AWK codes into a file. It provides us more readable coding.
For example, put the codes below into "hc.awk" file.
Code:
{
print $1
}Then, run it with "cat 1.txt | awk -f hc.awk".
As you see, the output is same as the first one.
The File "httpd.conf"
This file holds Apache server's settings. So options of the websites stored in server could be seen in this file under the "Virtual Host" tags.
A sample of "httpd.conf"
Code:
<VirtualHost 13.1.1.30>
ServerName foo.net
ServerAdmin foo@foo.net
DocumentRoot "/webspace/hc8resadmin/foo/foo.net/www/html"
ServerAlias www.foo.net
ErrorLog /webspace/hc8resadmin/foo/foo.net/log/httpd/error_log
CustomLog /webspace/hc8resadmin/foo/foo.net/log/httpd/access_log combined
<IfModule mod_php5.c>
AddHandler application/x-httpd-php .php
</IfModule>
<Directory "/webspace/hc8resadmin/foo/foo.net">
AllowOverride All
php_admin_value open_basedir "/webspace/hc8resadmin/foo/foo.net/:/tmp/"
</Directory>
<Directory "/webspace/hc8resadmin/foo/foo.net/www">
</Directory>
<Directory "/webspace/hc8resadmin/foo/foo.net/www/html">
AddHandler cgi-script cgi pl
Order Allow,Deny
Allow from "All"
Options -Indexes +ExecCGI
</Directory>
<Directory "/webspace/hc8resadmin/foo/foo.net/www/cgi-bin/">
AddHandler cgi-script cgi pl
Order Allow,Deny
Allow from "All"
Options -Indexes +ExecCGI
</Directory>
ScriptAlias /cgi-bin/ "/webspace/hc8resadmin/foo/foo.net/www/cgi-bin/"
DirectoryIndex index.php default.html default.htm default.asp default.aspx index.htm index.html index.cfm index.asp index.aspx awstats.pl
Redirect "/admin" "http://202.88.238.242:8787"
<Directory "/webspace/hc8resadmin/foo/foo.net/special/phpMyAdmin">
AddHandler cgi-script cgi pl
Order Allow,Deny
Allow from "All"
Options -Indexes +ExecCGI
</Directory>
Alias /MySQLAdmin "/webspace/hc8resadmin/foo/foo.net/special/phpMyAdmin"
</VirtualHost>We may say that "DocumentRoot" refers the web-root path and "ServerAlias" does domains.
Parsing the "httpd.conf":
We want to get web-root paths of all websites in "httpd.conf". Before using AWK, to find rows that contains "DocumentRoot", we can use "grep" Linux-command. I copied "httpd.conf" file into "1.txt" file.
And run:
Code:
ersin@kandemir ~/JoaK/Temp $ grep DocumentRoot 1.txt | head -10
DocumentRoot "/var/www/html"
DocumentRoot "/asianet/www/html"
DocumentRoot "/webspace/hc8resadmin/bl**/st**.in/www/html"
DocumentRoot "/webspace/hc8resadmin/ra**/ra**.com/www/html"
DocumentRoot "/webspace/hc8resadmin/ka**/ka**.edu.in/www/html"
DocumentRoot "/webspace/hc8resadmin/jo**/di**.net/www/html"
DocumentRoot "/webspace/hc8resadmin/bl**/fo**.in/www/html"
DocumentRoot "/webspace/hc8resadmin/bi**/k*.biz/www/html"
DocumentRoot "/webspace/hc8resadmin/ma**/pi**.org/www/html"
DocumentRoot "/webspace/hc8resadmin/ja**/sa**.com/www/html"Let's explain these commands:
Firstly, we read the file with "cat 1.txt" and "grep DocumentRoot" to select rows we want. The last command "head" is used to limit output up to 10.
"|" This provides commands using output of previous command.
Yeah, we haven't used AWK yet. We use it only to extract paths.
Our new commands:
Code:
ersin@kandemir ~/JoaK/Temp $ grep DocumentRoot 1.txt | head -10 | awk '{print $2}'
"/var/www/html"
"/asianet/www/html"
"/webspace/hc8resadmin/bl**/st**.in/www/html"
"/webspace/hc8resadmin/ra**/ra**.com/www/html"
"/webspace/hc8resadmin/ka**/ka**.edu.in/www/html"
"/webspace/hc8resadmin/jo**/di**.net/www/html"
"/webspace/hc8resadmin/bl**/fo**.in/www/html"
"/webspace/hc8resadmin/bi**/k*.biz/www/html"
"/webspace/hc8resadmin/ma**/pi**.org/www/html"
"/webspace/hc8resadmin/ja**/sa**.com/www/html"Commands are the same except the last one.
" awk '{print $2}' ": We want AWK to print second column namely "paths".
Yeah, as you can see we got all them. You can remove the quotation marks with your editor. Or use "sed" command.
Code:
grep DocumentRoot 1.txt | head -10 | awk '{print $2}' | sed -e 's/"//g'Thanks.
![[+]](https://sinister.li/images/modern/collapse_collapsed.png)
![[Image: 2YpkRjy.png]](http://i.imgur.com/2YpkRjy.png)
