Tuesday, 27 November 2012

SQL Introduction

DATABASE :
                        Database is a collection of coherent and meaning full data.
DBMS:
            DBMS (Database Management System)  is a system  which is allows you to INSERTING,UPDATING,DELETING and PROCESSING of data in database.

RDBMS:
                    Which is a DBMS  , It implements the relational data model
ORACLE :
                    Oracle is a DBMS which is implimented by the oracle corporation.

 SQL(Structured Query Language ):
                SQL is a command language.  It is the standard language for relational database management systems. This can be used to create the database structure. SQL statements are used to perform tasks such as update data on a database, or retrieve data from a database.

SQL is divided into the following

                  Ø  Data Definition Language (DDL)
Ø  Data Manipulation Language (DML)
Ø  Data Retrieval Language (DRL)
Ø  Transaction Control Language (TCL)
Ø  Data Control Language (DCL)

DDL -- create, alter, drop, truncate, rename
DML -- insert, update, delete
DRL -- select
TCL -- commit, rollback, savepoint
DCL -- grant, revoke

CREATE TABLE SYNTAX

Create table <table_name> (col1 datatype1, col2 datatype2 …coln datatypen);
Ex:
     SQL> create table student (no number (2), name varchar (10), marks number (3));

INSERT

This will be used to insert the records into table.
We have two methods to insert.
Ø  By value method
Ø  By address method

a) USING VALUE METHOD
    
     Syntax:
          insert into <table_name) values (value1, value2, value3 …. Valuen);

    
     Ex:
            SQL> insert into student values (1, ’sudha’, 100);
            SQL> insert into student values (2, ’saketh’, 200);
          
     To insert a new record again you have to type entire insert command, if there are lot of 
     records this will be difficult.
     This will be avoided by using address method.

b) USING ADDRESS METHOD
    
      Syntax:
          insert into <table_name) values (&col1, &col2, &col3 …. &coln);
     This will prompt you for the values but for every insert you have to use forward slash.
     
      Ex:
            SQL> insert into student values (&no, '&name', &marks);

Enter value for no: 1
Enter value for name: Jagan
Enter value for marks: 300
old   1: insert into student values(&no, '&name', &marks)
new   1: insert into student values(1, 'Jagan', 300)

SQL> /
Enter value for no: 2
Enter value for name: Naren
Enter value for marks: 400
old   1: insert into student values(&no, '&name', &marks)
new   1: insert into student values(2, 'Naren', 400)

c) INSERTING DATA INTO SPECIFIED COLUMNS USING VALUE METHOD
    
     Syntax:
           insert into <table_name)(col1, col2, col3 … Coln) values (value1, value2, value3 ….
                                                              Valuen);
     Ex:
            SQL> insert into student (no, name) values (3, ’Ramesh’);
            SQL> insert into student (no, name) values (4, ’Madhu’);

d) INSERTING DATA INTO SPECIFIED COLUMNS USING ADDRESS METHOD
    
     Syntax:
          insert into <table_name)(col1, col2, col3 … coln) values (&col1, &col2, &col3 …. &coln);
     This will prompt you for the values but for every insert you have to use forward slash.
     
     Ex:
            SQL> insert into student (no, name) values (&no, '&name');
Enter value for no: 5
Enter value for name: Visu
old   1:  insert into student (no, name) values(&no, '&name')
new   1:  insert into student (no, name) values(5, 'Visu')

SQL> /
Enter value for no: 6
Enter value for name: Rattu
old   1:  insert into student (no, name) values(&no, '&name')
new   1:  insert into student (no, name) values(6, 'Rattu')

SELECTING DATA

Syntax:
    Select * from <table_name>;              -- here * indicates all columns
or
    Select col1, col2, … coln from <table_name>;

Ex:
    SQL> select * from student;
   
        NO NAME            MARKS
        ---  ------             --------
         1   Sudha             100
         2   Saketh            200
         1   Jagan             300
         2   Naren             400
         3   Ramesh
         4   Madhu
         5   Visu
         6   Rattu

    SQL> select no, name, marks from student;

        NO NAME            MARKS
        ---  ------             --------
         1   Sudha             100
         2   Saketh            200
         1   Jagan             300
         2   Naren             400
         3   Ramesh
         4   Madhu
         5   Visu
         6   Rattu

    SQL> select no, name from student;

        NO NAME
        ---  -------
         1   Sudha
         2   Saketh
         1   Jagan
         2   Naren
         3   Ramesh
         4   Madhu
         5   Visu
         6   Rattu
                CONDITIONAL SELECTIONS AND OPERATORS

We have two clauses used in this
Ø  Where
Ø  Order by

USING WHERE

Syntax:
     select * from <table_name> where <condition>;
     the following are the different types of operators used in where clause.

v  Arithmetic operators         
v  Comparison operators
v  Logical operators

v  Arithmetic operators          -- highest precedence
+, -, *, /
v  Comparison operators
Ø  =, !=, >, <, >=, <=, <>
Ø  between, not between
Ø  in, not in
Ø  null, not null
Ø  like
v       Logical operators
Ø  And
Ø  Or                                -- lowest precedence
Ø  not

a) USING =, >, <, >=, <=, !=, <>
   
     Ex:
        SQL> select * from student where no = 2;


        NO NAME            MARKS
        ---  -------           ---------
         2   Saketh            200
         2   Naren             400
        
        SQL> select * from student where no < 2;

        NO NAME            MARKS
        ---  -------           ----------
         1   Sudha             100
         1   Jagan             300

        SQL> select * from student where no > 2;

         NO NAME            MARKS
         ---  -------           ----------
         3   Ramesh
         4   Madhu
         5   Visu
         6   Rattu

         SQL> select * from student where no <= 2;

         NO NAME            MARKS
         ---  -------           ----------
         1   Sudha             100
         2   Saketh            200
         1   Jagan             300
         2   Naren             400
      
         SQL> select * from student where no >= 2;

         NO NAME            MARKS
         ---  -------           ---------
         2   Saketh            200
         2   Naren             400
         3   Ramesh
         4   Madhu
         5   Visu
         6   Rattu

         SQL> select * from student where no != 2;

         NO NAME            MARKS
         ---  -------           ----------
         1   Sudha             100
         1   Jagan             300
         3   Ramesh
         4   Madhu
         5   Visu
         6   Rattu

         SQL> select * from student where no <> 2;

         NO NAME            MARKS
         ---  -------           ----------
         1   Sudha             100
         1   Jagan             300
         3   Ramesh
         4   Madhu
         5   Visu
         6   Rattu

b) USING AND
    
     This will gives the output when all the conditions become true.
    
     Syntax:
          select * from <table_name> where <condition1> and <condition2> and .. <conditionn>;
    
     Ex:

         SQL> select * from student where no = 2 and marks >= 200;


                              NO NAME            MARKS
         ---  -------           --------
         2   Saketh            200
         2   Naren             400

c) USING OR

     This will gives the output when either of the conditions become true.

     Syntax:
         select * from <table_name> where <condition1> and <condition2> or .. <conditionn>;

     Ex:
         SQL> select * from student where no = 2 or marks >= 200;

         NO NAME            MARKS
         ---  -------           ---------
         2   Saketh            200
         1   Jagan             300
         2   Naren             400

d) USING BETWEEN

     This will gives the output based on the column and its lower bound, upperbound.

     Syntax:
         select * from <table_name> where <col> between <lower bound> and <upper bound>;

     Ex:
         SQL> select * from student where marks between 200 and 400;

         NO NAME            MARKS
         ---  -------           ---------
         2   Saketh            200
         1   Jagan              300
         2   Naren              400

e) USING NOT BETWEEN

     This will gives the output based on the column which values are not in its lower bound,
     upperbound.

     Syntax:
     select * from <table_name> where <col> not between <lower bound> and <upper bound>;

     Ex:
         SQL> select * from student where marks not between 200 and 400;

         NO NAME            MARKS
         ---  -------           ---------
         1   Sudha             100

f) USING IN

    This will gives the output based on the column and its list of values specified.

    Syntax:
         select * from <table_name> where <col> in ( value1, value2, value3 … valuen);

     Ex:
         SQL> select * from student where no in (1, 2, 3);

         NO NAME            MARKS
         --- -------            ---------
         1   Sudha             100
         2   Saketh            200
         1   Jagan             300
         2   Naren             400
         3   Ramesh

g) USING NOT IN

     This will gives the output  based on the column which values are not in the list of values 
     specified.

     Syntax:
         select * from <table_name> where <col> not in ( value1, value2, value3 … valuen);

     Ex:
         SQL> select * from student where no not in (1, 2, 3);

         NO NAME            MARKS
         ---  -------           ---------
         4   Madhu
         5   Visu
         6   Rattu

h) USING NULL

     This will gives the output based on the null values in the specified column.

     Syntax:
         select * from <table_name> where <col> is null;

     Ex:
         SQL> select * from student where marks is null;

         NO NAME            MARKS
         ---  -------           ---------
         3   Ramesh
         4   Madhu
         5   Visu
         6   Rattu

i) USING NOT NULL

    This will gives the output based on the not null values in the specified column.

     Syntax:
         select * from <table_name> where <col> is not null;

     Ex:         
         SQL> select * from student where marks is not null;
         NO NAME            MARKS
         ---  -------           ---------
         1   Sudha             100
         2   Saketh            200
         1   Jagan             300
         2   Naren             400

j) USING LIKE

    This will be used to search through the rows of database column based on the pattern you
     specify.

     Syntax:
        select * from <table_name> where <col> like <pattern>;
    
     Ex:         
        i) This will give the rows whose marks are 100.

            SQL> select * from student where marks like 100;

         NO NAME            MARKS
         ---  -------           ---------
         1   Sudha             100
       
        ii) This will give the rows whose name start with ‘S’.

             SQL> select * from student where name like 'S%';

         NO NAME            MARKS
         ---  -------           ---------
         1   Sudha             100
         2   Saketh            200

        iii) This will give the rows whose name ends with ‘h’.

              SQL> select * from student where name like '%h';
        
         NO NAME            MARKS
         ---  -------           ---------
         2   Saketh            200
         3   Ramesh

        iV) This will give the rows whose name’s second letter start with ‘a’.

               SQL> select * from student where name like '_a%';

          NO NAME            MARKS
          ---  -------            --------
          2   Saketh            200
          1   Jagan             300
          2   Naren             400
          3   Ramesh
          4   Madhu
          6   Rattu
 
        V) This will give the rows whose name’s third letter start with ‘d’.

              SQL> select * from student where name like '__d%';

         NO NAME            MARKS
         ---  -------           ---------
         1   Sudha             100
         4   Madhu

        Vi) This will give the rows whose name’s second letter start with ‘t’ from ending.

               SQL> select * from student where name like '%_t%';

         NO NAME            MARKS
         ---  -------           ---------
         2   Saketh            200
         6   Rattu
        
        Vii) This will give the rows whose name’s third letter start with ‘e’ from ending.

                SQL> select * from student where name like '%e__%';

         NO NAME            MARKS
         ---  -------           ---------
         2   Saketh            200
         3   Ramesh

        Viii) This will give the rows whose name  cotains 2 a’s.

                    SQL> select * from student where name like '%a% a %';

         NO NAME            MARKS
          --- -------           ----------
         1   Jagan             300

* You have to specify the patterns in like using underscore ( _ ).

USING ORDER BY

This will be used to ordering the columns data (ascending or descending).

Syntax:
        Select * from <table_name> order by <col> desc;
By default oracle will use ascending order.
If you want output in descending order you have to use desc keyword after the column.

Ex:
        SQL> select * from student order by no;

        NO NAME            MARKS
        ---  -------           ---------
         1   Sudha             100
         1   Jagan              300
         2   Saketh            200
         2   Naren             400
         3   Ramesh
         4   Madhu
         5   Visu
         6   Rattu

        SQL> select * from student order by no desc;

        NO NAME            MARKS
        ---  -------           ---------
         6 Rattu
         5 Visu
         4 Madhu
         3 Ramesh
         2 Saketh            200
         2 Naren             400
         1 Sudha             100
         1 Jagan             300



Sunday, 25 November 2012

Sed Command in Unix and Linux


        Sed is a Stream Editor used for modifying the files in unix (or linux). Whenever you want to make changes to the file automatically, sed comes in handy to do this. Most people never learn its power; they just simply use sed to replace text. You can do many things apart from replacing text with sed. Here I will describe the features of sed with examples.

Consider the below text file as an input.

>cat file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
unixlinux which one you choose.

Sed Command Examples


1. Replacing or substituting string

Sed command is mostly used to replace the text in a file. The below simple sed command replaces the word "unix" with "linux" in the file.
>sed 's/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.
Here the "s" specifies the substitution operation. The "/" are delimiters. The "unix" is the search pattern and the "linux" is the replacement string.

By default, the sed command replaces the first occurrence of the pattern in each line and it won't replace the second, third...occurrence in the line.

2. Replacing the nth occurrence of a pattern in a line.

Use the /1, /2 etc flags to replace the first, second occurrence of a pattern in a line. The below command replaces the second occurrence of the word "unix" with "linux" in a line.
>sed 's/unix/linux/2' file.txt
unix is great os. linux is opensource. unix is free os.
learn operating system.
unixlinux which one you choose.
3. Replacing all the occurrence of the pattern in a line.

The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.
>sed 's/unix/linux/g' file.txt
linux is great os. linux is opensource. linux is free os.
learn operating system.
linuxlinux which one you choose.
4. Replacing from nth occurrence to all occurrences in a line.

Use the combination of /1, /2 etc and /g to replace all the patterns from the nth occurrence of a pattern in a line. The following sed command replaces the third, fourth, fifth... "unix" word with "linux" word in a line.
>sed 's/unix/linux/3g' file.txt
unix is great os. unix is opensource. linux is free os.
learn operating system.
unixlinux which one you choose.
5. Changing the slash (/) delimiter

You can use any delimiter other than the slash. As an example if you want to change the web url to another url as
>sed 's/http:\/\//www/' file.txt
In this case the url consists the delimiter character which we used. In that case you have to escape the slash with backslash character, otherwise the substitution won't work.

Using too many backslashes makes the sed command look awkward. In this case we can change the delimiter to another character as shown in the below example.
>sed 's_http://_www_' file.txt
>sed 's|http://|www|' file.txt
6. Using & as the matched string

There might be cases where you want to search for the pattern and replace that pattern by adding some extra characters to it. In such cases & comes in handy. The & represents the matched string.
>sed 's/unix/{&}/' file.txt
{unix} is great os. unix is opensource. unix is free os.
learn operating system.
{unix}linux which one you choose.

>sed 's/unix/{&&}/' file.txt
{unixunix} is great os. unix is opensource. unix is free os.
learn operating system.
{unixunix}linux which one you choose.
7. Using \1,\2 and so on to \9

The first pair of parenthesis specified in the pattern represents the \1, the second represents the \2 and so on. The \1,\2 can be used in the replacement string to make changes to the source string. As an example, if you want to replace the word "unix" in a line with twice as the word like "unixunix" use the sed command as below.
>sed 's/\(unix\)/\1\1/' file.txt
unixunix is great os. unix is opensource. unix is free os.
learn operating system.
unixunixlinux which one you choose.
The parenthesis needs to be escaped with the backslash character. Another example is if you want to switch the words "unixlinux" as "linuxunix", the sed command is
>sed 's/\(unix\)\(linux\)/\2\1/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
linuxunix which one you choose.
Another example is switching the first three characters in a line
>sed 's/^\(.\)\(.\)\(.\)/\3\2\1/' file.txt
inux is great os. unix is opensource. unix is free os.
aelrn operating system.
inuxlinux which one you choose.
8. Duplicating the replaced line with /p flag

The /p print flag prints the replaced line twice on the terminal. If a line does not have the search pattern and is not replaced, then the /p prints that line only once.
>sed 's/unix/linux/p' file.txt
linux is great os. unix is opensource. unix is free os.
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.
linuxlinux which one you choose.
9. Printing only the replaced lines

Use the -n option along with the /p print flag to display only the replaced lines. Here the -n option suppresses the duplicate rows generated by the /p flag and prints the replaced lines only one time.
>sed -n 's/unix/linux/p' file.txt
linux is great os. unix is opensource. unix is free os.
linuxlinux which one you choose.
If you use -n alone without /p, then the sed does not print anything.

10. Running multiple sed commands.

You can run multiple sed commands by piping the output of one sed command as input to another sed command.
>sed 's/unix/linux/' file.txt| sed 's/os/system/'
linux is great system. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you chosysteme.
Sed provides -e option to run multiple sed commands in a single sed command. The above output can be achieved in a single sed command as shown below.
>sed -e 's/unix/linux/' -e 's/os/system/' file.txt
linux is great system. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you chosysteme.
11. Replacing string on a specific line number.

You can restrict the sed command to replace the string on a specific line number. An example is
>sed '3 s/unix/linux/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.
The above sed command replaces the string only on the third line.

12. Replacing string on a range of lines.

You can specify a range of line numbers to the sed command for replacing a string.
>sed '1,3 s/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.
Here the sed command replaces the lines with range from 1 to 3. Another example is
>sed '2,$ s/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.
Here $ indicates the last line in the file. So the sed command replaces the text from second line to last line in the file.

13. Replace on a lines which matches a pattern.

You can specify a pattern to the sed command to match in a line. If the pattern match occurs, then only the sed command looks for the string to be replaced and if it finds, then the sed command replaces the string.
>sed '/linux/ s/unix/centos/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
centoslinux which one you choose.
Here the sed command first looks for the lines which has the pattern "linux" and then replaces the word "unix" with "centos".

14. Deleting lines.

You can delete the lines a file by specifying the line number or a range or numbers.
>sed '2 d' file.txt
>sed '5,$ d' file.txt
15. Duplicating lines

You can make the sed command to print each line of a file two times.
>sed 'p' file.txt
16. Sed as grep command

You can make sed command to work as similar to grep command.
>grep 'unix' file.txt
>sed -n '/unix/ p' file.txt
Here the sed command looks for the pattern "unix" in each line of a file and prints those lines that has the pattern.

You can also make the sed command to work as grep -v, just by using the reversing the sed with NOT (!).
>grep -v 'unix' file.txt
>sed -n '/unix/ !p' file.txt
The ! here inverts the pattern match.

17. Add a line after a match.

The sed command can add a new line after a pattern match is found. The "a" command to sed tells it to add a new line after a match is found.
>sed '/unix/ a "Add a new line"' file.txt
unix is great os. unix is opensource. unix is free os.
"Add a new line"
learn operating system.
unixlinux which one you choose.
"Add a new line"
18. Add a line before a match

The sed command can add a new line before a pattern match is found. The "i" command to sed tells it to add a new line before a match is found.
>sed '/unix/ i "Add a new line"' file.txt
"Add a new line"
unix is great os. unix is opensource. unix is free os.
learn operating system.
"Add a new line"
unixlinux which one you choose.
19. Change a line

The sed command can be used to replace an entire line with a new line. The "c" command to sed tells it to change the line.
>sed '/unix/ c "Change line"' file.txt
"Change line"
learn operating system.
"Change line"
20. Transform like tr command

The sed command can be used to convert the lower case letters to upper case letters by using the transform "y" option.
>sed 'y/ul/UL/' file.txt
Unix is great os. Unix is opensoUrce. Unix is free os.
Learn operating system.
UnixLinUx which one yoU choose.
Here the sed command transforms the alphabets "ul" into their uppercase format "UL"

Awk Command in Unix


        Awk is one of the most powerful tools in Unix used for processing the rows and columns in a file. Awk has built in string functions and associative arrays. Awk supports most of the operators, conditional blocks, and loops available in C language. 

One of the good things is that you can convert Awk scripts into Perl scripts using a2p utility. 

The basic syntax of AWK:

awk 'BEGIN {start_action} {action} END {stop_action}' filename

Here the actions in the begin block are performed before processing the file and the actions in the end block are performed after processing the file. The rest of the actions are performed while processing the file.

Examples:

Create a file input_file with the following data. This file can be easily created using the output of ls -l.

-rw-r--r-- 1 center center  0 Dec  8 21:39 p1
-rw-r--r-- 1 center center 17 Dec  8 21:15 t1
-rw-r--r-- 1 center center 26 Dec  8 21:38 t2
-rw-r--r-- 1 center center 25 Dec  8 21:38 t3
-rw-r--r-- 1 center center 43 Dec  8 21:39 t4
-rw-r--r-- 1 center center 48 Dec  8 21:39 t5
           From the data, you can observe that this file has rows and columns. The rows are separated by a new line character and the columns are separated by a space characters. We will use this file as the input for the examples discussed here. 

1. awk '{print $1}' input_file

Here $1 has a meaning. $1, $2, $3... represents the first, second, third columns... in a row respectively. This awk command will print the first column in each row as shown below.

-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
To print the 4th and 6th columns in a file use awk '{print $4,$5}' input_file

Here the Begin and End blocks are not used in awk. So, the print command will be executed for each row it reads from the file. In the next example we will see how to use the Begin and End blocks.

2. awk 'BEGIN {sum=0} {sum=sum+$5} END {print sum}' input_file

This will prints the sum of the value in the 5th column. In the Begin block the variable sum is assigned with value 0. In the next block the value of 5th column is added to the sum variable. This addition of the 5th column to the sum variable repeats for every row it processed. When all the rows are processed the sum variable will hold the sum of the values in the 5th column. This value is printed in the End block.


3. In this example we will see how to execute the awk script written in a file. Create a file sum_column and paste the below script in that file


#!/usr/bin/awk -f
BEGIN {sum=0} 
{sum=sum+$5} 
END {print sum}
Now execute the the script using awk command as 

awk -f sum_column input_file.

This will run the script in sum_column file and displays the sum of the 5th column in the input_file.

4. awk '{ if($9 == "t4") print $0;}' input_file

This awk command checks for the string "t4" in the 9th column and if it finds a match then it will print the entire line. The output of this awk command is

-rw-r--r-- 1 pcenter pcenter 43 Dec  8 21:39 t4
5. awk 'BEGIN { for(i=1;i<=5;i++) print "square of", i, "is",i*i; }'

This will print the squares of first numbers from 1 to 5. The output of the command is
square of 1 is 1
square of 2 is 4
square of 3 is 9
square of 4 is 16
square of 5 is 25
Notice that the syntax of “if” and “for” are similar to the C language.

Awk Built in Variables: 

You have already seen $0, $1, $2... which prints the entire line, first column, second column... respectively. Now we will see other built in variables with examples. 

FS - Input field separator variable:

So far, we have seen the fields separted by a space character. By default Awk assumes that fields in a file are separted by space characters. If the fields in the file are separted by any other character, we can use the FS variable to tell about the delimiter. 

6. awk 'BEGIN {FS=":"} {print $2}' input_file
OR
awk -F: '{print $2} input_file

This will print the result as 
39 p1
15 t1
38 t2
38 t3
39 t4
39 t5
OFS - Output field separator variable: 

By default whenever we printed the fields using the print statement the fields are displayed with space character as delimiter. For example

7. awk '{print $4,$5}' input_file

The output of this command will be
center 0
center 17
center 26
center 25
center 43
center 48
We can change this default behavior using the OFS variable as

awk 'BEGIN {OFS=":"} {print $4,$5}' input_file
center:0
center:17
center:26
center:25
center:43
center:48
Note: print $4,$5 and print $4$5 will not work the same way. The first one displays the output with space as delimiter. The second one displays the output without any delimiter. 

NF - Number of fileds variable:

The NF can be used to know the number of fields in line

8. awk '{print NF}' input_file
This will display the number of columns in each row.

NR - number of records variable: 
The NR can be used to know the line number or count of lines in a file.

9. awk '{print NR}' input_file
This will display the line numbers from 1.

10. awk 'END {print NR}' input_file
This will display the total number of lines in the file.

String functions in Awk:
Some of the string functions in awk are:

index(string,search)
length(string)
split(string,array,separator)
substr(string,position)
substr(string,position,max)
tolower(string)
toupper(string)

Advanced Examples:

1. Filtering lines using Awk split function

The awk split function splits a string into an array using the delimiter.

The syntax of split function is
split(string, array, delimiter)

Now we will see how to filter the lines using the split function with an example.

The input "file.txt" contains the data in the following format
1 U,N,UNIX,000
2 N,P,SHELL,111
3 I,M,UNIX,222
4 X,Y,BASH,333
5 P,R,SCRIPT,444
Required output: Now we have to print only the lines in which whose 2nd field has the string "UNIX" as the 3rd field( The 2nd filed in the line is separated by comma delimiter ).

The ouptut is:

1 U,N,UNIX,000
3 I,M,UNIX,222
The awk command for getting the output is:

awk '{ 
        split($2,arr,","); 
        if(arr[3] == "UNIX") 
        print $0 
} ' file.txt

Examples of Awk Command in Unix 

1. Inserting a new line after every 2 lines

We will see how to implement this using the awk command with an example.
The input "file.txt" contains the below data:

1 A
2 B
3 C
4 D
5 E
6 F
Let say, we want to insert the new line "9 Z" after every two lines in the input file. The required output data after inserting a new line looks as

1 A
2 B
9 Z
3 C
4 D
9 Z
5 E
6 F
9 Z
The awk command for getting this output is

awk '{ 
 if(NR%2 == 0) 
 {
  print $0"\n9 Z";
 } 
 else 
 {
  print $0
 }
     }' file.txt

2. Replace the Nth occurrence of a pattern

The input file contains the data.

AAA 1
BBB 2
CCC 3
AAA 4
AAA 5
BBB 6
CCC 7
AAA 8
BBB 9
AAA 0
Now we want to replace the fourth occurrence of the first filed "AAA" with "ZZZ" in the file.
The required output is:

AAA 1
BBB 2
CCC 3
AAA 4
AAA 5
BBB 6
CCC 7
ZZZ 8
BBB 9
AAA 0
The awk command for getting this output is

awk 'BEGIN {count=0} 
 { 
  if($1 == "AAA") 
  { 
   count++
  } 
  if(count == 4) 
  { 
   sub("AAA","ZZZ",$1) 
  } 
 } 
 {
  print $0
 }' file.txt
3. Find the sum of even and odd lines separately
The input file data:

A 10
B 39
C 22
D 44
E 75
F 89
G 67
You have to get the second field and then find the sum the even and odd lines.
The required output is

174, 172

The awk command for producing this output is

awk '{ 
 if(NR%2 == 1) 
 {
  sum_e = sum_e + $2
 } 
 else 
 {
  sum_o = sum_o + $2
 }
      } 
      END { print sum_e,sum_o }' file.txt
4. Fibonacci series using awk command
Now we will produce the Fibonacci series using the awk command.

awk ' BEGIN{ 
 for(i=0;i<=10;i++) 
 { 
  if (i <=1 )
  {
   x=0;
   y=1;
   print i;
  }
  else
  {
   z=x+y;
   print z; 
   x=y;
   y=z;
  }
 } 
    }'
The output is

0
1
1
2
3
5
8
13
21
34
55
5. Remove leading zeros from a file using the awk command. The input file contains the below data.

0012345
05678
01010
00001
After removing the leading zeros, the output should contain the below data.
12345
5678
1010
1
The awk command for this is.
awk '{print $1 + 0}' file.txt
awk '{printf "%d\n",$0}' file.txt