import>Khs26: Formatting and info awk

2008-07-30T15:44:13Z

Formatting and info awk

import>Khs26: Created page

2008-07-30T15:35:58Z

Created page

New page

== Introduction ==

awk is useful for combining regular expressions with editing and other functions so that you can easily modify text files with conditional flow control.

The general syntax for invoking awk is:

awk -f <script-file> <target-file> - where script-file contains awk instructions
awk 'pattern {commands}' <target-file> - for command line scripting

e.g. awk '/alias/' ~/.bashrc - prints lines containing alias in your .bashrc file
e.g. awk '/export PATH/ { print $2 }' ~/.bashrc - prints your path as defined in your .bashrc file

Taking the command line example, pattern is an optional pattern (typically a regular expression or some kind of conditional) and commands are an optional list of awk commands to be executed on lines satisfying the conditions specified by the pattern provided. If a pattern is not provided, the commands act upon all lines in the target file and if commands are not provided, awk lists all lines satisfying the pattern provided (like grep).

awk separates records defined by a record separator (newline by default) into fields defined by a field separator (space by default). Thus by default it separates input into lines and separates lines into words. Field and record separators can be reassigned either with command-line options or by altering the RS or FS built-in variables (see info awk for more details).

awk stores the fields in the current record in several variables $<n> where n is the nth field (starting from 1) so $1 is the first field, $2 is the second etc.

e.g. Given a PDB file with lines such as:

...
ATOM 49 CE1 PHE 4 ...
ATOM 50 HE1 PHE 4 ...
ATOM 51 CZ PHE 4 ...
ATOM 52 HZ PHE 4 ...
...

If you want the atom number corresponding to atoms with a certain type, you would search conditionally on the third field, $3 and print the second field, $2: awk '$3 == HE1 {print $2}' <protein>.pdb

awk also supports conditional patterns, for example for HE1 ($3) in residue number 4 ($5), we would use: awk '$3 == HE1 && $5 == 4 {print $2}' <protein>.pdb

There are many more commands in awk, all of which are listed in info awk, along with useful built-in variables (like the field separator).

== Useful built-in variables ==

NF - number of fields in the current input record
NR - current record number (all files)
FNR - current record number (in the current file)
FILENAME - current file name
RS - input record separator (default newline)
FS - input field separator (default space)

== Useful functions ==

gsub(r, s, [t]) - for pattern r in string t (if t is not specified, use the whole of the current record) replace with s and return the number of substitutions
sub(r, s, [t]) - same as gsub, but only performs one substitution and then stops
length(s) - returns the length of string s
index(s, t) - returns the index of string t in string s (starting at 1), or 0 if string t is not found
print{expressions} - prints the expressions specified separated by the OFS (output field separator)
tolower(s) - converts string s to lowercase
toupper(s) - converts string s to uppercase
strtonum(s) - converts string s to a number (assumes octal if it begins with 0, assumes hexadecimal if it begins with 0x or 0X)
substr(s, i, [n]) - returns the substring in string s beginning at position i which is at most n characters long (if n isn't specified, continues until the end of s)

There are also numerous mathematical and I/O functions which are documented in info awk.

== BEGIN and END ==

There are two special patterns which aren't compared against the input, their commands are executed (sensibly enough) before and after all of the other awk commands. These are BEGIN and END. Simply place these in place of the pattern and their commands will operate at the appropriate time.

← Older revision		Revision as of 15:44, 30 July 2008
Line 5:		Line 5:
	The general syntax for invoking awk is:		The general syntax for invoking awk is:

	awk -f <script-file> <target-file> - where script-file contains awk instructions		awk -f <script-file> <target-file> - where script-file contains awk instructions
	awk 'pattern {commands}' <target-file> - for command line scripting		awk 'pattern {commands}' <target-file> - for command line scripting

	e.g. awk '/alias/' ~/.bashrc - prints lines containing alias in your .bashrc file		e.g. awk '/alias/' ~/.bashrc - prints lines containing alias in your .bashrc file
	e.g. awk '/export PATH/ { print $2 }' ~/.bashrc - prints your path as defined in your .bashrc file		e.g. awk '/export PATH/ { print $2 }' ~/.bashrc - prints your path as defined in your .bashrc file

	Taking the command line example, pattern is an optional pattern (typically a regular expression or some kind of conditional) and commands are an optional list of awk commands to be executed on lines satisfying the conditions specified by the pattern provided. If a pattern is not provided, the commands act upon all lines in the target file and if commands are not provided, awk lists all lines satisfying the pattern provided (like grep).		Taking the command line example, pattern is an optional pattern (typically a regular expression or some kind of conditional) and commands are an optional list of awk commands to be executed on lines satisfying the conditions specified by the pattern provided. If a pattern is not provided, the commands act upon all lines in the target file and if commands are not provided, awk lists all lines satisfying the pattern provided (like grep).
Line 19:		Line 19:
	e.g. Given a PDB file with lines such as:		e.g. Given a PDB file with lines such as:

	...		...
	ATOM 49 CE1 PHE 4 ...		ATOM 49 CE1 PHE 4 ...
	ATOM 50 HE1 PHE 4 ...		ATOM 50 HE1 PHE 4 ...
	ATOM 51 CZ PHE 4 ...		ATOM 51 CZ PHE 4 ...
	ATOM 52 HZ PHE 4 ...		ATOM 52 HZ PHE 4 ...
	...		...

	If you want the atom number corresponding to atoms with a certain type, you would search conditionally on the third field, $3 and print the second field, $2: ~~awk '$3 == HE1 {print $2}' <protein>.pdb~~		If you want the atom number corresponding to atoms with a certain type, you would search conditionally on the third field, $3 and print the second field, $2:
			awk '$3 == HE1 {print $2}' <protein>.pdb

	awk also supports conditional patterns, for example for HE1 ($3) in residue number 4 ($5), we would use: ~~awk '$3 == HE1 && $5 == 4 {print $2}' <protein>.pdb~~		awk also supports conditional patterns, for example for HE1 ($3) in residue number 4 ($5), we would use:
			awk '$3 == HE1 && $5 == 4 {print $2}' <protein>.pdb

	There are many more commands in awk, all of which are listed in info awk, along with useful built-in variables (like the field separator).		There are many more commands in awk, all of which are listed in info awk, along with useful built-in variables (like the field separator).
Line 34:		Line 36:
	== Useful built-in variables ==		== Useful built-in variables ==

	NF - number of fields in the current input record		NF - number of fields in the current input record
	NR - current record number (all files)		NR - current record number (all files)
	FNR - current record number (in the current file)		FNR - current record number (in the current file)
	FILENAME - current file name		FILENAME - current file name
	RS - input record separator (default newline)		RS - input record separator (default newline)
	FS - input field separator (default space)		FS - input field separator (default space)

	== Useful functions ==		== Useful functions ==

	gsub(r, s, [t]) - for pattern r in string t (if t is not specified, use the whole of the current record) replace with s and return the number of substitutions		gsub(r, s, [t]) - for pattern r in string t (if t is not specified, use the whole of the current record) replace with s and return the number of substitutions
	sub(r, s, [t]) - same as gsub, but only performs one substitution and then stops		sub(r, s, [t]) - same as gsub, but only performs one substitution and then stops
	length(s) - returns the length of string s		length(s) - returns the length of string s
	index(s, t) - returns the index of string t in string s (starting at 1), or 0 if string t is not found		index(s, t) - returns the index of string t in string s (starting at 1), or 0 if string t is not found
	print{expressions} - prints the expressions specified separated by the OFS (output field separator)		print{expressions} - prints the expressions specified separated by the OFS (output field separator)
	tolower(s) - converts string s to lowercase		tolower(s) - converts string s to lowercase
	toupper(s) - converts string s to uppercase		toupper(s) - converts string s to uppercase
	strtonum(s) - converts string s to a number (assumes octal if it begins with 0, assumes hexadecimal if it begins with 0x or 0X)		strtonum(s) - converts string s to a number (assumes octal if it begins with 0, assumes hexadecimal if it begins with 0x or 0X)
	substr(s, i, [n]) - returns the substring in string s beginning at position i which is at most n characters long (if n isn't specified, continues until the end of s)		substr(s, i, [n]) - returns the substring in string s beginning at position i which is at most n characters long (if n isn't specified, continues until the end of s)

	There are also numerous mathematical and I/O functions which are documented in info awk.		There are also numerous mathematical and I/O functions which are documented in info awk.
Line 58:		Line 60:

	There are two special patterns which aren't compared against the input, their commands are executed (sensibly enough) before and after all of the other awk commands. These are BEGIN and END. Simply place these in place of the pattern and their commands will operate at the appropriate time.		There are two special patterns which aren't compared against the input, their commands are executed (sensibly enough) before and after all of the other awk commands. These are BEGIN and END. Simply place these in place of the pattern and their commands will operate at the appropriate time.

			== info awk ==

			awk has plenty more features, all of which can be found in the info file or with Googling, so the above is very much a brief introduction, have a look at info awk.

Quick guide to awk - Revision history

import>Khs26: Formatting and info awk

import>Khs26: Created page