1

UNIX Basics

Contents
1.Introduction to Unix 2. Logging In and Basic Commands 3File system 4.managing files 5.File Access Permissions 6.vi Editor 7.Shell Basics 8. Standard I/O Redirection 9. Filters & some utilities 10. Controlling Processes 11. Shell Programming concepts 12. Labs 03 - 11 12 - 26 27 - 45 46 - 55 56 - 68 69 - 78 79 94 95 103 104 123 124 132 133 - 158 159 - 161

Module 1
Introduction to Unix

Objectives
4

Upon completing this module you should be able to understand the following:

what is an Operating System History of unix Operating system Unix Architecture More features of unix Unix Flavors Linux Flavors

What Is an Operating System?

5

Interface between Users and the Hardware

take care of Storage Management

take care of I/O device management

History of the UNIX Operating System

6

1969

AT&T Bell Labs UNICS system -designed by Ken Thompson & Dennis Ritchie UNICS finally became UNIX UNIX rewritten in C, making it portable to different hardware University of California at Berkeley (BSD) contributed many important features like vi, C shell etc. AT & T came back and started commercial productionEditions ->Systems

1970 1973 Mid 1970s

1982

Late 1980s AT & T released SVR4 unification of SV3.2,BSD,SunOs & XENIX

1991 Linux from Linus Torvalds

1990s

POSIX Standard, MIT introduced X-Windows

UNIX Architecture Kernel & Shell

7

User

Shell

Kernel

Hardware

UNIX Architecture System Calls

8

Though there are over a thousand commands on the system, they all use a handful of functions, called system calls, to communicate with the kernel.
All unix flavors use the same system calls and are described in the POSIX specification.

More Features of UNIX

9

Hierarchical file system Multi tasking Multi user The building block approach Pattern matching (wildcard characters) Toolkit(Applications, RDBMSs, Languages etc..) Programming facility Documentation

Flavors of UNIX
10

Digital Unix IRIX

SCO Open Server

SCO UnixWare IBM -AIX HP - UX Sun - Solaris

Clone of UNIX (LINUX)

11

Red Hat
Calders SuSE Mandrake Debian

12

Module 2

Logging In and Basic Commands

Objectives
13

Upon completing this module you should be able to understand the following:

Logging In and Out Command Line Format The Secondary Prompt Online Manual Pages id Command who Command date Command cal, clear Commands passwd Command finger command

A Typical Terminal Session

14

Log in to identify yourself and gain access.

Execute commands to do work.

Log off to terminate your connection.

Logging In and Out

15

telnet 192.168.10.1 User: Password:

exit or <ctrl>+<d> will terminate the login session

Command Line Format

16

Syntax: $ command [-options] [arguments]

1. 2. 3.

Separation :

mail f newmail

not mail f newmail

Order: mail f newmail not mal newmail f Multiple options: who m u or who mu not who m u

4.

Multiple arguments: mail team1 team2 not mail team1team2

The Secondary Prompt

17

$ echo 'hi >Good Morning

> is the default secondary prompt

Content of the Manual Pages

18

NAME SYNOPSIS DESCRIPTION RETURN VALUE ERRORS BUGS etc.

WARNINGS EXAMPLES AUTHOR FILES SEE ALSO

The Online Manual

19

Syntax: man [k|X] keyword in which X is the number of one of the manual sections Examples: $ man ls Display the ls" man page. $ man -k cat Display entries with keyword "cat". $ man passwd Display the "passwd" man page-Section 1. $ man 4 passwd Display the "passwd" man page-Section 4.

The id Command
20

Syntax: id Displays effective user and group identification for session

Example: $id uid =303 (user3) gid=300 (class)

The who Command

21

Syntax: who

Reports information about users who are currently logged on to a system

Examples: $ who root tty1p5 Jul 01 08:01 user11 tty1p4 Jul 01 09:59 user12 tty0p3 Jul 01 10:01

$ who am i user12 tty0p3 Jul 01 10:01

$ whoami user12

The date Command

22

Syntax: date Reports the date and time

Example: $ date Fri Jul 1 11:15:55 EDT 2005

The cal Command

23

Syntax: cal Reports the calendar of 2011 September(Current month)

Example: $ cal 8 2011 for Aug 2011 $ cal 2011 for the full calendar of year 2011

The passwd Command

24

Syntax: passwd Assigns a login password

Example: $ passwd Changing password for user1 Old password: New password: Re-enter new password:

The clear Command

25

Syntax:
clear Clears terminal screen

finger
26

Finger: Displays information about the users currently logged on

Eg:

$ finger user1 Login name: user1 Directory: /export/home/user1 shell:/usr/bin/sh On since Sep 05 09:10:12 on tty1 No plan

27

Module 3 File System

Objectives
28

Upon completing this module you should be able to understand the following:

What Is a File System? File System structure Hard Link File Types Ordinary Files Directory Files Symbolic Links Device files The File System Hierarchy Basic Commands pwd, cd, ls, mkdir, rmdir, touch

What Is a File System?

29

collection of control structures and Data blocks that occupy the space defined by a partition and allow for the storage and management of data.

Data + Meta data

File System structure

Primary Superblock
Backup Superblock Cylinder Group Block

Inode Table
Data Blocks
30

Hard Link
31

Allows a file to have more than one name and yet maintain a single copy on the disk. All the links have the same inode number Eg: $ ls i f1 1113 f1

$ ln f1 f1.lnk
$ ls i f1.lnk 1113 f1.lnk

File Types
32

Common File Types:

Ordinary files: regular files Directory files: table of contents, that stores a list of files/directories within that directory Device files: For every device there is a device file used by kernel to interact with the device.

Symbolic Link: Its link to other files

Ordinary Files
33

A regular file simply holds data. Regular files can hold ASCII text, binary data, image data, databases, application-related data, and more.

Directory Files
34

Directory name inum d1 4 f1 10

Inode Table # type mode links user group 4 dir 755 2 user1 group1 10 file 644 1 user1 group1 date Sep 5 9:30 Sep 5 9:45 size 512 12 loc

Symbolic Links
35

contains the path of the file to which it links

Exists even after the source file is removed and is exactly similar to Windows shortcut
Syntax: ln s soucefile linkfile Eg: ln -s /f1 f1.lnk

Overcomes 2 limitations of Hard Link: 1. possible across filesystems 2. can link to a directory

Device files
36

A device file provides access to a device.

# cd /devices/pci@1f,0/pci@1,1/ide@3 # ls -l brw------- 1 root sys 136, 0 Apr 3 11:11 dad@0,0:a Two types of device files: block device files character device file

The File System Hierarchy

37

usr

etc

devices

dev

var

tmp

bin

sbin

dsk

rdsk

adm

sadm

ls

who

system directories
38

/ :root directory, top level directory

/bin : user commands like cp, cron, cmp etc..

/usr/sbin: admin commands /dev : logical device files of all hardware devices /devices: physical device files /etc : System configuration files and user database

/tmp : to store temporary files

/usr : the binaries, shared libraries, shared documentation etc. /var: stores the log files and dynamic files

File Path name

39

A sequence of file names, separated by slashes (/), that describes the path, the system must follow to locate a file in the file system

Absolute pathname (start from the /-directory):

Eg: /export/home/user1/file1 Relative pathname (start from the current directory) ./test1 (. = current directory) ../team03/.profile (.. = parent directory)

pwd - Present Working Directory

40

pwd prints the Current Directory

ls - List Contents of a Directory

41

Syntax: ls [-adlFR] [pathname(s)]

Example: $ ls f1 f2 memo $ ls -F f1 f2* memo/ $ ls -aF profile f1 f2* memo/ $ ls memo f1 f2

cd Change Directory
42

syntax:

cd [dir_name]
Example: $ pwd /home/user3 $ cd memo; pwd /home/user3/memo $ cd ../..; pwd /home $ cd /tmp; pwd /tmp

mkdir, rmdir create & remove Directories

43

Syntax: mkdir directory eg:

$ mkdir d1

syntax: rmdir dir eg: $ rmdir /export/home/user1

working with Multiple Directories

44

Create multiple directories simultaneously:

$ mkdir -p dir1/dir2/dir3

Remove a directory and all its subdirectories $ rmdir -p dir1/dir2/dir3

The touch Command

45

Updates the atime and mtime of a file if it exists. Else create an empty file

Example: $ ls l -rw-rr-- 1 user1 group1 100 Sep 5 09:30 test $date Mon, Sep 5 10:00 2005 $ touch test file1

$ ls -l
-rw-rr-- 1 user1 group1 100 Sep 5 10:00 test -rw-rr-- 1 user1 group1 100 Sep 5 10:00 file1

46

Module 4
Managing Files

Objectives
47

Upon completing this module you should be able to understand the following:

File Characteristics cat more tail wc cp mv rm

File Characteristics
48

$ ls -l -rw-r--r-- 1 user3 class 37 Jul 24 11:06 f1 -rwxr-xr-x 1 user3 class 52 Jul 24 11:08 f2 drwxr-xr-x 2 user3 class 1024 Jul 24 12:03 memo

File Type

Links

Group

Permissions

Owner

Size

Timestamp

Name

cat - Display the Contents of a File

49

Syntax: cat [file...] Concatenate and display the contents of file(s) Examples: $ cat remind Your mother's birthday is November 29. $ cat note remind The meeting is scheduled for July 29. Your mother's birthday is November 29. $ cat abc abc 1234 CTRL 1234

more - Display the Contents of a File

50

Syntax: more [filename]... Display files one screen at a time Example: $ more funfile . . . --funfile (20%)-Q or q Quit more Return One more line Space bar One more page

tail - Display the End of a File

51

Syntax: tail [-n] [filename]... Display the end of file(s)

Example: $ tail -1 test soon as it is available.

wc command
52

The wc command counts the number of lines, words and bytes in a named file:

syntax:
wc [-c] [-l] [-w] filename Options: -c -l -w counts the number of bytes counts lines counts words

Example: $wc testfile 10 98 1000 testfile

cp - Copy Files
53

Syntax: cp [-i] file1 new_file Copy a file cp [-i] file [file...] dest_dir Copy files to a directory cp -r [-i] dir [dir...] dest_dir Copy directories
Example: cp file1 d1 copies file1 to d1 directory cp file2 file3 create a copy of file2 as file3 in the same directory

mv - Move or Rename Files

54

Syntax: mv [-i] file new_file Rename a file mv [-i] file [file...] dest_dir Move files to a directory mv [-i] dir [dir...] dest_dir Rename or move directories Example:

$ ls -F f1 f2* memo/ note remind $ mv f1 file1 $ ls -F file1 f2* memo/ note remind $ mv f2 memo/file2 $ ls -F file1 memo/ note remind $ ls -F memo file2*

$ mv note remind memo $ ls -F file1 memo/ $ls -F memo file2* note remind $ mv memo letters $ ls -F file1 letters/

rm - Remove Files
55

Syntax: rm [-if] filename [filename...] Remove files rm -r[if] dirname [filename...] Remove directories

Examples: rm f1 removes the file f1

rm r d1 remove the directory.

56

Module 5

File Access Permissions

Objectives
57

Access to files is dependent on a user's identification and the permissions associated with a file. This module will show how to Understand the read, write, and execute access to a file Permissions

ls (11, ls -l) chmod umask chown chgrp su setuid, setgid, sticky bit

File Permissions and Access

58

Access to files is dependent on a user's identification and the permissions associated with a file. This module will show how to Understand the read, write, and execute access to a file Permissions

ls (11, ls -l) chmod umask chown chgrp su

Determine what access is granted on a file Change the file access Change default file access Change the owner of a file Change the group of a file Switch your user identifier

Who can access a File?

59

The UNIX system incorporates a three-tier structure to define who has access to each file and directory: user The owner of the file group A group that may have access to the file other Everyone else The ls -l command displays the owner and group who has access to the file. $ ls -l -rw-r--r-- 1 user3 class 37 Jul 24 11:06 f1 -rwxr-xr-x 1 user3 class 37 Jul 24 11:08 f2 drwxr-xr-x 2 user3 class 1024 Jul 24 12:03 memo | | owner group

Types of Access
60

There are three types of access for each file and directory: Read files: directories: Write files: directories: Execute files: directories: contents can be examined. contents can be examined. contents can be changed. contents can be changed. file can be used as a command. can become current working directory.

Change Permissions (Symbolic Notation)

61

Syntax: chmod mode_list file... Change permissions of file(s)

mode_list [who[operator]permission] [ ,... ]

who user, group, other or all(u/g/a) operator + (add), - (subtract), = (set equal to) permission read, write, execute
Example: Original permissions: mode user group other rw-r--r-- rw- r-- r-$ chmod u+x,g+x,o+x file or $ chmod +x file Final permissions: mode user group other rwxr-xr-x rwx r-x r-x

Change Permissions(Octal Notation)

62

File and directory permissions can also be specified as an octal number:

Read Permission :4 Write Permission :2 Execute Permission:1

We can just add the numbers to specify the permission for each category Example: 6 means read and write, 7 means read, write and execute

eg: $ chmod 664 f1 will give read and write permissions for owner and group while only read for others

Default Permissions
63

The default protections for newly created files and directories are: -rw-rr-drwxr-xr-x

File Directory

644 755

These default settings may be modified by changing the umask value.

umask - Permission Mask

64

Umask specifies what permission bits will be set on a new file or directory when created. 777 022 666 022 rwxr-xr-x rw-rr

New Directory: New File :

755 644

The default value of umask is set in /etc/profile. This can be changed for all the users or a particular user

chown - Change File Ownership

65

Syntax: chown owner [:group] filename ...

Changes owner of a file(s) and, optionally, the group ID

Example: $ id uid=101 (user1), gid=101 (group1) $ cp f1 /tmp/user2/f1 $ ls -l /tmp/user2/f1 -rw-r----- 1 user1 group1 3967 Jan 24 13:13 f1 $ chown user2 /tmp/user2/f1 $ ls -l /tmp/user2/f1 -rw-r----- 1 user2 class 3967 Jan 24 13:13 f1 Only the owner of a file (or root) can change the ownership of the file.

The chgrp Command

66

Syntax: chgrp newgroup filename ... Example: $ id uid=303 (user3), gid=300 (class) $ ls -l f3 -rw-r----- 1 user3 class 3967 Jan 24 13:13 f3 $ chgrp class2 f3 $ ls -l f3 -rw-r----- 1 user3 class2 3967 Jan 24 13:13 f3 $ chown user2 f3 $ ls -l f3 -rw-r----- 1 user2 class2 3967 Jan 24 13:13 f3

su - Switch User Id
67

Syntax: su [user_name] Change your effective user ID and group ID Example: $ ls -l f1 -rwxr-x--- 1 user1 group1 3967 Jan 24 23:13 class_setup $ id uid=303 (user1), gid=300 (group1) $ su user2 Password: $ id uid=400 (user2), gid=300 (group1)

Special File Permissions setuid, setgid, sticky bit

68

setuid changes the effective user id of the user to the owner of the program chmod u+s f1 or chmod 4744 f1 setgid changes the effective group id of the user to the group of the program chmod g+s f1 or chmod 2744 f1

sticky bit ensures the deletion of files by only file owner in a public writable directory chmod +t f1 or Chmod 1744 f1

69

Module 6

vi Editor

Objectives
70

Upon completing this module you should be able to understand the following:

What Is vi? Starting and Ending a vi Session Cursor Control Commands Input Mode: i, a, O, o Deleting Text: x, dw, dd, dG Copying, moving and changing Text Searching for Text: /, n, N Find and Replace

What Is vi?
71

A screen-oriented text editor Included with most UNIX system distributions Command driven Categories of commands include General administration Cursor movement Insert text Delete text Paste text Modify text

Starting and Ending a vi Session

72

vi [filename] Start a vi edit session of file Example: $ vi testfile - If the file doesnt exist, it will be created - Otherwise vi will open the existing file

All modifications are made to the copy of the file brought into memory. :wq or :x or <shift-zz> :w :q :q! write and quit write quit Quit without saving

Cursor Control Commands

73

1G <ctrl-b> H

k
<up-arrow>

0 b,B h <left-arrow>

$ w,W l <right-arrow>

<down-arrow>
j L <ctrl-f> G

Input Mode: i, a, O, o
74

i will insert character at the present cursor position I will insert character at the beginning of the line a will append character at the present cursor position A will append character at the end of the line o will insert a blank line below O will insert a blank line above

Deleting Text: x, dw, dd, dG

75

deletes the current character

dw
dd dG

deletes the current word

deletes the current line delete all lines to end of file, including current line.

With any of these you can prefix a number Example: 3dd will delete 3 lines

d$ d0

to delete to the end of the line to delete to the start of the line

Copying, moving and changing Text

76

yy yw dw

copy the line copy the word cut the word p will paste the yanked lines below P will paste the yanked lines above

dd

cut the line

will overwrite the current character

R
cw

will replace all text on the right of the cursor position

will replace only the current word

Searching for Text: /, n, N

77

/director will locate for the first occurrence of the pattern director

n to locate the next occurrence N to locate the previous occurence

Find and Replace

78

To find and replace the word director with member :

:1,$s/director/member/g
1,$ represents all lines in the file

g makes it truly global. g will ensure that all occuences in each line is replaced. Without g only the first occurrence of each line will be replaced.

79

Module 7

Shell Features

Objectives
80

Upon completing this module you should be able to understand the following:

Shell functionalities Commonly Used Shells BASH Shell Features Aliasing, Command History Re-entering Commands The User Environment Setting Shell Variables Variable Substitution Command Substitution Transferring Local Variables to the Environment Functions of a Shell before Command execution.

Shell functionalities
81

command execution environment settings variable assignment variable substitution command substitution filename generation I/O redirection Interpretive programming language

Commonly Used Shells

82

/bin/bash /bin/ksh /bin/sh /bin/csh /bin/rksh /bin/rsh /bin/tcsh /bin/zsh

Bash shell Korn shell Bourne shell C Shell Restricted Korn shell Restricted Bourne shell An amended version of the C shell The Z shell

BASH Shell Features

83

A shell user interface with some advanced features:

Command aliasing File name completion Command history mechanism Command line recall and editing Job control Enhanced cd capabilities Advanced programming capabilities

Aliasing
84

Syntax: alias [name[=string]] Examples: $ alias dir=ls $ alias mroe=more $ alias mstat=/home/tricia/projects/micron/status $ alias laser="lp -dlaser" $ laser fileX request id is laser-234 (1 file) $ alias displays aliases currently defined $ alias mroe displays value of alias mroe mroe=more

Command History
85

The shell keeps a history file of commands that you enter.

The history command displays the last 16 commands. You can recall, edit, and re-enter commands previously entered.

Syntax: history [-n| a z] Display the command history. Example: $ history -2 list the last two commands 15 cd 16 more .profile $ history 3 5 3 date 4 pwd 5 ls list command numbers 3 through 5

Re-entering Commands
86

You type r c to re-enter command number c.

Example: $ history 3 5 list command numbers 3 through 5 3 date 4 pwd 5 ls

$r4 run command number 4 pwd /home/kelley

The User Environment

87

Your environment describes your session to the programs

you run. Syntax: env Example: $ env HOME=/home/gerry PWD=/home/gerry/develop/basics ... PATH=/usr/bin:/usr/contrib/bin:/usr/local/bin:\ /home/gerry/bin

Two Important Variables

88

The PATH variable

A list of directories where the shell will search for the commands you type The TERM variable Describes your terminal type and screen size to the programs you run

$ env ... PATH=/usr/bin:/usr/contrib/bin:/usr/local/bin $ TERM=70092

Setting Shell Variables

89

Syntax: name=value Example: color=blue PATH=$PATH:/usr/ucb

. . .

Variable Substitution
90

$ echo $PATH /usr/bin:/usr/contrib/bin:/usr/local/bin $ PATH=$PATH:$HOME:. $ echo $PATH /usr/bin:/usr/contrib/bin:/usr/local/bin:/home/user3:. $ echo $HOME /home/user3 $ file_name=$HOME/file1 $ more $file_name <contents of /home/user3/file1>

Command Substitution
91

Syntax: $(command) Example:

$ dt=date

This will store the string date in the variable dt not the value
This will do command substitution and store the result of date command in the variable dt

$ dt=$(date)

note: instead of $(date) we can also use `date`

Displaying Variable Values

92

$ echo $HOME /home/user3 $ env HOME=/home/user3 . SHELL=/usr/bin/sh $ set HOME=/home/user3 PATH=/usr/bin:/usr/contrib/bin:/usr/local/bin . color=lavender count=3 dir_name=/home/user3/tree $ unset dir_name

Transferring Local Variables to the Environment

93

Syntax: export variable

Transfer variable to environment

Functions of a Shell before Command execution.

94

Searches for a command Substitutes Shell variable values. Command substitution, I/O redirection Interpreted programming interface

95

Module 8

Standard I/O Redirection

Objectives
96

Upon completing this module you should be able to understand the following:

The Standard Files Input Redirection Output Redirection Creating a file with cat Error Redirection Combined Redirection Split Outputs

The Standard Files

97

Standard Input File (0)

Keyboard

Standard Output File (1)

Standard Error File (2)

Monitor
Monitor

operators: standard input redirection standard output redirection standard Error redirection 0< 1> 2> or < or >

Input Redirection
98

Default standard input $ mail user1 subject: Hi Test mail

<ctrl-d>

Redirect Input from a file: < $ mail user1 < letter

Output Redirection
99

Default standard output: $ ls f1 f2 f3

Redirect output from a file: > $ ls > ls.out

Redirecting and appending output to a file: >> $ who >> who.out

Creating a file with cat

100

While normally used to list the contents of files, using cat with redirection can be used to create a file $ ls file1

file2

file3

Using redirection
$ cat > newfile file created by using the output redirection <ctrl-d> $ ls file1

file2

file3

newfile

Error Redirection
101

Default standard error: $ cat file1 file3 This is file1 cat: cannot open file2

Redirecting error output to a file: 2> (To append: 2>>)

$ cat file1 file3 2>err This is file1

$ cat err
cat: cannot open file2 $ cat file1 file3 2> /dev/null

Combined Redirection
102

Combined redirects: $ command > outfile 2>errfile <infile $ command >> appendfile 2 >> errorfile <infile Association Examples: $ command > outfile 2 >&1
note: This is NOT the same as above $ command 2>&1 >outfile

Split Outputs
103

The tee command reads standard input and sends the data to both standard output and a file.

Example: $ ls | tee ls.save | wc l

104

Module 9 Filters & other commands

Objectives
105

Upon completing this module you should be able to understand the following:

grep, sort, find ,cut , tr ftp paste command split command uniq command diff(Differential file comparator) Command cmp command file command tar (tape archive program) Restoring Files

grep
106

Search for lines matching specified pattern syntax: grep [options] pattern [file1 file2 ..] Example: emp.dat Robin Bangalore John Rina Chennai Bangalore

$ grep Bangalore emp.dat Robin Bangalore Rina Bangalore

grep with Regular Expressions

107

grep regular_expression file

Valid metacharacters . Any single character

*
[aA] [a-f] ^a z$

Zero or more occurences of the preceding character

Enumeration: a or A Any one of the characters in the range of a through f Any lines that start with a Any lines that end with z

grep options
108

-v -c -l -n -i

print lines that do not match print only a count of matching lines print only the names of the files with matching lines number the matching lines ignore the case of letters when making comparisons

-w

do a whole word search

sort command
109

The sort command sorts lines and writes the result to standard output:

$ sort [ -t delimiter] [+field[.column]] [options]

Options: -d -r -n sorts in dictionary order. Only letters, digits and spaces are considered in comarisons reverses the order of the specified sort sorts numeric fields in arithmetic value

sort examples
110

$ cat animals dog.2 cat.4 elephant.10 rabbit.7 $ sort animals cat.4 dog.2 elephant.10 rabbit.7 $ cat animals | sort +0.1 rabbit.7 cat.4 elephant.10 dog.2 $ cat animals | sort -t. -n +1 dog.2 cat.4 rabbit.7 elephant.10

find command
111

Search one or more directory structures for files that meet certain specified criteria Display the names of matching files or execute commands against those files find path expression Examples: $ find / -name file1 ---search for files in whole system with name file1 $ find / -user user1 exec rm { } \; ---search for the files owned by user1 and delete them $ find / -user user1 ok rm { } \; ---search for the files owned by user1 and delete them interactively

find options
112

-type

f d +n -n n +x -x onum mode

ordinary file directory larger than "n" blocks smaller than "n" blocks equal to "n" blocks modified more than "x" days ago modified less than "x" days ago access permissions match "onum" access permissin match "mode"

-size

-mtime

-perm

-user
-o -newer

user1

finds files owned by "user1"

logical "or"

ref

searches for files that are newer than the reference file.

cut command
113

Used to cut fields from each line of a file or columns of a table

cut d: -f1,5 /etc/passwd cut c2 test second character of each line cut c-2 test first 2 characters of each line

tr command
114

Used to substitute the values

tr [a-z] [A-Z] will convert each small letter to caps

cat test | tr [a-z] [A-Z] will convert the small letters to caps in display
tr s will squeeze multiple space occurences of space into one

ftp
115

The ftp Command Syntax : $ ftp hostname get Gets a file from the remote computer. put Sends a local file to the remote system mget get multiple files from the remote system mput Sends multiple files to the remote system cd changes the directory in the remote system lcd changes the directory in the local system ls Lists files on the remote computer. ? Lists all ftp commands help command Display a very brief help message for command. bye Exits ftp.

The paste command

116

Used to paste files vertically eg. file1 name ram raghu address mvm rjnagar file2 age 29 25 paste file1 file2 name ram raghu address age mvm rjnagar 29 25

(To rearrange f1 f2 f3 to f3 f2 f1, cut each field,put into file then paste)
By default paste uses the tab character for pasting files, but d to specify one eg. paste d \| file1 file2 name ram raghu address|age mvm|29 rjnagar|25

The split command

117

Used to split up the files:

This command breaks up the input into several equi line segments. by defalult, split breaks a file into 1000 line pieces(or whatever is left) split creates files like xaa,xab,xac . xaz then xba, xbb . xbz and xzz . So there can be 676(26*26) files split 5 chap (here the chap file is broken into files of 5 lines size)

split chap small (the file names will be smallaa,smallab . smallzz)

The uniq command

118

used in EDP environment to clear the duplicate entries by mistakes 01|accounts|6213 01|accounts|6213 02|admin|5423 03|marketing|6521

eg. dept.list

03|marketing|6521
uniq dept.list 01|accounts|6213 02|admin|5423

03|marketing|6521
The input to uniq must be ordered data. So sort and send the data sort dept.list | uniq -uniqlist (uniqlist is the output file)

The diff(Differential file comparator) Command

119

Analyzes text files Reports the differences between files

diff [-options] file1 file2

The cmp command

120

$ cmp names names.old names names.old differ: byte 6, line 1 $cmp -l names names.old 6 12 151 7 102 156 8 157 145 ... ... ... cmp: EOF on names

The file command

121

The file command tells the type of file

Example: $ file /usr/bun/vi /usr/bin/vi:executable (RISC system/6000) or object module $ file c1 c1: ascii text $ file /usr/bin /usr/bin: directory

tar (tape archive program)

122

create a disk archive that contains a group of files or an entire directory structure -c -x -t -f copy extract list [only one can be present at a time] to specify the device name

tar []cvf etc.tar /etc/*

Restoring Files
123

tar xvf tarfile

Example: tar xvf etc.tar

selective extraction tar xvf etc.tar /etc/passwd tar tvf etc.tar will display the archive

124

Module 10 Controlling Processes

Objectives
125

Upon completing this module you should be able to understand the following:

Monitoring Process Controlling Processes Terminating Processes Kill Signals Running Long Processes Job Control

Monitoring Process
126

The ps command dislays process status information

$ ps -f UID john john john john PID 202 204 210 212 PPID 1 202 204 204 ... ... ... ... ... TTY tty0 tty0 yyu0 tty0 ... ... ... ... ... COMMAND -ksh ksh ls -R / ps -f

Controlling Processes
127

Foreground Processes:
$ ls -lR / >lsout

Background Processes: $ ls -lR / >lsout &

Terminating Processes
128

Foreground Processes: ctrl-c Interrupt key, cancels a forground process kill Sometimes the kill command is used to terminate foreground processes

Background Processes:

kill

kill is the only way to terminate processes

background

Kill Signals
129

Sig 01 02 03 09

Meaning hangup- you logged out while the process was still running interrupt- you pressed the interrupt(break) key sequence ctrl-c quit -you pressed the quit key sequence ctrl-\ Kill signal: The most powerful and risky signal that can be Signal cannot be avoided or ignored! Termination signal(Default): Stop a process Signal can be handled by programs

sent

15

Running Long Processes

130

The nohup command will prevent a process from being killed if you log off the system before it completes: $ nohup ls -lR / >lsout & [1] 100 If you do not redirect output, nohup will redirect output to a file nohup.out

$ nohup ls -lR / & [1] 102 Sending output to nohup.out

Job Control
131

jobs <ctrl-z> fg %jobnumber bg %jobnumber stop %jobnumber

Lists all jobs Suspends foreground task Execute job in foreground Execute job in background Suspends background task

Job Control Examples

132

$ ls -R / > out 2> errfile & [1] 273 $ jobs [1] + Running ls -R / > out 2>errfile & $ fg %1 ls -R / > out 2>errfile <ctrl-z> [1] + Stopped (SIGTSTP) ls -R / >out 2>errfile & $ bg %1 % jobs [1] + Running $ kill %1 [1] + Terminating

ls -R / >out 2>errfile &

ls -R / >out 2>errfile &

133

Module 11
Shell Programming Concepts

Objectives
134

Upon completing this module you should be able to understand the following:

echo & read commands Command Line Arguments The logical operators && and || The if Conditional Numeric comparison with test numeric, string and file comparison The case CONDITIONAL expr: Computation Command Substitution sleep & wait while loop, until and for loops

echo & read commands

135

echo used to display messages on the screen

read used to accept values from the users, make programming interactive

eg. echo Enter ur name

read name
echo Good Morning $name

Command Line Arguments

136

Shell programs can accept values through 1. read [Interactive used when there are more inputs] 2. From the command Line when u execute it[Non interactive- used when only a few inputs are there]

For eg. sh1 20 30

Here 20 & 30 are the command Line arguments.

Command Line args are stored in Positional parameters $1 contains the first argument, $2 the second, $3 the third etc. $0 contains the name of the file, $# stores the count of args

$* displays all the args

An example of Command Line args.

137

#! /bin/bash echo Program: $0 echo The number of args specified is $# echo The args are $*

sh sh1 prog1 prog2

The parameter $?
138

$? can be used to know the exit value of the previous command . This can be used in the shell scripts:

The logical operators && and ||

139

These operators can be used to control the execution of command depending on the success/failure of the previous command

eg.
grep director emp1.lst && echo Pattern found in file

grep manager emp2.lst || echo pattern not found

or grep director emp.lst && echo Pattern found || echo Pattern not found

exit : Script Termination

140

exit command used to prematurely terminate a program. It can even take args. grep $1 $2 | exit 2 echo Pattern found Job over

eg.

The if Conditional
141

if condition is true

then
execute commands else execute commands fi else is not a must : if condition is true then execute commands

fi

An example of if
142

eg1:

if grep director emp.lst then echo Pattern found else echo Pattern not found fi

Every if must have an accompanying then and fi, and optionally else

Numeric comparison with test

143

test is used to check a condition and return true or false

Relational operators used by if Operator Meaning

-eq
-ne -gt

Equal to
Not equal to Greater than

-ge
-lt -le

Gfreater than or equal to

Less than Less than or equal to

An example of test with numeric

144

echo Enter age read age

if test $age ge 18
then echo Major else echo Minor fi

[ ] short hand for test

145

echo Enter age read age

if [ $age ge 18 ]
then echo Major else echo Minor fi

if elif :Multi-way Branching

146

if condition then execute commands elif condition then execute commands elif condition then execute commands else execute conditions fi

echo Enter marks read marks if [ $marks ge 80 ] then echoDistinction elif [ $marks ge 60 ] then echo First Class else echo Pass fi

can have as many elif , else is optional

test String comparison

147

test with Strings uses = and !=

String Tests used by test Test -n str1 -z str1 Exit Status true if str1 is not a null string true if str1 is a null string

s1 = s2 true if s1 = s2
s1 != s2 str1 true if s1 is not equal to s2 true if str1 is assigned and not null

file tests
148

File related Tests with test

Test
-e file -f file

Exit Status
True if file exists True if fie exists and is a regular file

-r file
-w file -x file -d file -s file

True if file exists and is readable

True if file exists and is writable True if file exists and is executable True if file exists and is a directory True if the file exists and has a size >0

The case CONDITIONAL

149

case used to match an expression for more than one alternatives and uses a compact construct to permit multi way branching

case expression in pattern1) execute commands;; pattern2) execute commands esac

An example for case

150

echo 1. List of files 2. Processes of user 2. Quit

echo Enter choice

read choice case $choice in

1) ls ;;
2) ps f ;; 3) exit;; *) echo Wrong Choice esac

case Matching Multiple patterns

151

echo Do u want to continue (y/n) read answer case $answer in y|Y) echo Good ;; n|N) exit ;;

esac
case can also use wildcards like [yY][eE]* or [nN][oO]

expr: Computation
152

Shell doesnt have any compute features-depend on

expr

expr 3 + 5 expr $x + $y expr $x - $y expr $x \* $y expr &x / $y expr $x % $y division gives only integers as expr
can handle only integers

Command Substitution
153

x=5 x=`expr $x +1`

echo $x

it will give 6

sleep & wait

154

sleep 100

the program sleeps for 100 seconds

wait

wait for completion of all background processes

wait for completion of the process 138

wait 138

while loop
155

while condition is true do execute commands done eg. x=1 while test $x do -le 10

echo $x
x=`expr $x + 1` done (while true will set infinite condition)

until : whiles complement

156

x=1 until test $x gt 10

do
echo $x x=`expr $x + 1` done

For loop
157

for variable in list do execute commands done eg. for x in 1 2 3 4 5 do echo Value of x is $x

done

The list in for loop

158

1.

List from variables for var in $PATH $HOME $MAIL do echo $var done

2. List from Wild cards for file in *.c do

cc $file o $file{x}
done 3. List from positional parameters for pattern in $* do grep$pattern emp.lst || echo pattern $pattern done 4. List from command substitution for file in `cat clist` not found

Review
159 Who Who am I shows all users on the unix machine. shows the complete information about you.

Date
Id Cpio

shows the current date and time of the machine.

Shows the details of the user along with gid and groups. it is the command which saves and restores file

Cpio icvdum `usr2/*` < /dev/rmt0 How many shells? Two types of shells are there Bourne shell & C shell Bourne shell {sh, Ksh(korn), bash} C shell {csh & tcsh(tenex/tops)} History of shells First introduced sh shell used for programming and csh shell for interactive use, then the ksh introduced having all the C shells interactive features into the bourne shell syntaxs and finally bash introduced with all C shell and Korn shell features. Details in /etc/passwd Guest:X:501:513:U-ENZEN\Guest,S-1-5-21-117609710-1677128483-725345543-501:/home/Guest:/bin/bash Username:Password:UID:GID:UserID info:home directory:command shell

Review Cont.
160 Details in /etc/Shadow Your password is stored in /etc/shadow file.

Your encrpted password is not stored in /etc/passwd file. It is stored in /etc/shadow file.
Details in /etc/group wheel:*:0:root,rache group name: group password: group GID: list of users in the group Difference in /etc/profile, .profile & .bashrc /etc/profile is managed by system administrator contains shell initialization information required by all users. .profile is managed by you. You can add as much shell customization as you want. .profile is the users own login initialization .It is present in the root home directory of every user. .bashrc file is an initialization file run by each interactive invocation of the bash shell Viewing the file (ls, cat & wc) Use ls, ls a (for hidden files), ls lrt, ls F(for listing the items, whether it is file or directory, it show / for the directory), ls d(to list only directories), cat, cat n(for numbering the lines), cat b(it wont number the empty lines), wc , wc l(count the number of lines), wc w(count the number of words),wc m or c(Count the number of characters).

Review Cont.
161 Copying, moving & Renaming (cp, mv & rm) Cp(for files), cp r(for directories) source destination

Mv source destination
Rm, rm rf, Absolute path & Relative path There are two path define: Absolute path & Relative path Absolute path always start from root directory whereas relative path can be described from the current directory also. User permissions Chmod rwx Chmod go= *(block all other users to view ur files) or we can use chmod go-rwx. Chmod go+rwx(allow users the full permission) Chmod go-w <filename> (block other users writable permission). Letters used permissions U = owner, g = group, o = other, a = all, + = adding permissions to file, - = removing permissions to file. Difference between Process and a JOB Every command we give on unix starts a new process. Job can consist of multiple processes running in series or at the same time in parallel.

Review Cont.
162 Why exec command is used? To change your shell interpreter completely.

Variables
Variables are the word that holds the value. It is Fruit =Apple Fruit is a variable and Apple is the value of the variable Variable can be (a-z A-Z _ 0-9), but starts with (a-z A-Z _) Array variables Name[index]=value Eg. rohit[0-9]=(sweet, innocent,smart,intelligent) Accessing all items in an array variables: ${name[*]} & ${name[@]} Read only Variables To mark variable readonly by giving the readonly command. Types of Variables: Local, environment & Shell variables. Local variables are present in the current instance of a shell, whereas environment variables are available to any child process of the shell. Both the variables are created by user. Shell variables are special variables that are set by shell and is required by shell in order to function correctly.

Review Cont.
163 Substitution Filename Substitution

Using the wild cards (*) & (?).

Also use [a-z][A-Z][0-9] To negate any character, for eg. -[!a] Variable Substitution ${parameter:-word}(substituting the default value) If the parameter is null or unset,word is substituted to parameter. ${parameter:=word}(Assigning the default value) If the parameter is null or unset, parameter is set as value of word. ${parameter:?message}(Aborting due to variable errors) - If the parameter is null or unset, message is printed to standard error. This checks that variables are set correctly. ${parameter:+word} if the parameter is set, word is substituted for parameter. The value of parameter does not change. Command Substitution & Arithmatic substitution For ex. Date=`date` , Users=`who | wc l`

Quoting
Can be done in three ways (\), () & () i.e. backslash, single quote and double quote. Backslash Any character following by backslash lose its special meaning Single quote All special character between the single quotes loses its special meaning. Double quotes it allows its special meaning.

Review Cont.
164 $? = saves the exit status of the last executed command File Test

Options for files are

-b = if a block special file -c = if a character special file -d = for a directory -e = if file exist -f = for a regular file -g = if has its sgid bit set -h = for a symbolic link -k = for sticky bit -p = for a named pipe -r = for readable file -s = file size greater then zero -u = if has its suid bit set -w = for a writable file -x = for executable file -o = if owned by effective user ID.

Review Cont.
165 String Comparison -n = string has nonzero length; -z = string has zero length

String1=string2; string1!=string2
Numerical Comparision -eq, -ne, -lt, -le, -gt , -ge Compound Expression ! expr = true if expression is false. The expr can be any valid test command. Expr1 a expr2 = true if both expr are true (Logical AND &&) Expr1 o expr2 = true if either expr is true (Logical OR || ) CASE Statement Basic syntax, Case <word> in Pattern1) List1;; Pattern2) List2;; Esac Here word is compared against each pattern. Pattern is a string of regular characters and wildcard characters

Review Cont.
166 Loops Two main types of loops

While loop, until loop & for loop. Also one more type is there present in ksh and bash is select loop
While Loop Basic syntax While command Do List Done Here command is single command to execute, where list is a set of one or more commands to execute. Nested while loop, validating users input. Until Loop Similar to while loop, it is used until a condition is true. Basic syntax until command Do List Done

Review Cont.
167 For loop & Select loop Basic syntax for for

For name in word1, word2 . Wordn

Do List Done Basic syntax for select Select name in word1, word2 . Wordn Do List Done Loop controls Break & Continue

Review Cont.
168 Parameters Special Variables

$0 it defines the script name. the name of the command being executed.
$n these variable corresponding to the arguments with which script was invoked. $# number of arguments supplied to script. $* All the arguments are double quoted. $@ All the arguments are individually double quoted. $? the exit status of last command. $$ the process number of the current shell. $! the process number of last background command. Usage Statement Usage : $0 [arguments] [file|directory] Arguments & Options Ls aF fruit -aF is option and (-aF and fruit is an argument) Using basename Syntax basename filename Basename cmd takes the absolute or relative path and returns the file or directory.

Review Cont.
169 Option parsing using getopts Syntax getopts option-string variable

Getsopts f:o:v option

F is infile, o is outfile, v is verbose.

Input/Output Output to a terminal Echo & printf ECHO Embedding punctuation marks {*, (,),?} Embedding variable substitution {$Home, $Fruit} Embedding escape sequences {\n for new line, \t print a tab character, \c print a string without a default trailing newline} Printf it is same as echo, it wont produce a newline

Syntax printf %-mn<string>

(-) is for left aligned, by default it is right aligned.

Review Cont.
170 Output Redirection Command > file to redirect the complete logs.

Command >> file used for appending the logs in the file,previous data is not deleted.
Command | tee file used to redirect o/p to a file and the screen Input Redirection Command < file , for ex. Mail rohit@yahoo.co < Final_sheet Here document, syntax command << delimeter {document}delimeter For ex. Lpr << MYURLS {enter URLs}MYURLS Can be used with output redirection also, command > file << delimeter document delimeter Reading User Input Use the read command for taking the input from user. Syntax read name Exec Command Exec n>file & exec n>>file. N is an interger Redirecting STDOUT & STDERR in two separate files Command 1> file1 2> file2 Command > file1 2> file2.

Review Cont.
171 Functions It is created for reusing the commands.

Syntax name( ) { list all the commands}

Call the function using its name.

Text Editors Head it used to list the top 10 files which are recently used Ls lrt |head <number of files> Tail it is also same as head. But is can be used to view the logs of the file Tail f <filename> Grep It is used for search. -n for line number where the word is found, for ex. Grep n ls /home/rohit -I for case sensitive -l to search a word in number of files. -v to show the data except for that which is specified with v option. -c for counting the lines.

Review Cont.
172 Counting Words tr, sort & uniq

TR It is used for replacing the strings. For ex cat file | tr a-z A-Z
Tr s, use to replaces consecutive occurrence of same with only one occurrence of the character Sort it sort the data Sort rn is used to sort the output by numbers. Sort k is constructs the key in each line Uniq it sorts only unique lines For ex. # tr '!,"{}/=:$;()' ' ' <r4 | tr 'a-z' 'A-Z' | tr -s ' '| tr ' ' '\n' | sort | uniq -c To replace the r4 file and then sort o tans then fibe unique files. Uniq -c option is used for counting.

Review Cont.
173 File string Text using Regular Expression Sed & awk

If the search is made for any word that is in the beginning use ^<word>
If the search id made for any word that is in the end use <word>$ SED p option for print, d for delete & s for substitute For ex. sed /pattern1/pattern2/p but for s { sed s/pattern1/pattern2/} For globally changing the setting use /g in the last. Using multiple sed use the option {e}

Miscellaneous Commands Eval, : , sleep, find, exec, xargs, expr, bc

Labs
174

Lab:1 Write Script to see current date, time, username, and current directory Lab:2 Write script to print nos as 5,4,3,2,1 using while loop. Lab:3 How to write shell script that will add two nos, which are supplied as command line argument, and if this two nos are not given show error and its usage. Lab: 4 Write script to print given numbers sum of all digit, For eg. If no is 123 it's sum of all digit will be 1+2+3 = 6. Lab: 5.Write script to print given number in reverse order, for eg. If no is 123 it must print as 321.

Labs
175

contd.

LAB 4
# 1) Input number n

#
# # # # # #

2) Set sum=0, sd=0

3) Find single digit in sd as n % 10 it will give (left most digit) 4) Construct sum no as sum=sum+sd 5) Decrment n by 1 6) Is n is greater than zero, if yes goto step 3, otherwise next step 7) Print sum

if [ $# -ne 1 ]

LAB 5
# Script to reverse given no # # # # # # # 1) Input number n 2) Set rev=0, sd=0 3) Find single digit in sd as n % 10 it will give (left most digit) 4) Construct revrse no as rev * 10 + sd 5) Decrment n by 1 6) Is n is greater than zero, if yes goto step 3, otherwise next step 7) Print rev

Labs
176

contd.

LAB:1
echo "Hello, $LOGNAME" echo "Current date is `date`" echo "User is `who i am`" echo "Current direcotry `pwd`

LAB:2
i=5 while test $i != 0 do echo "$i " i=`expr $i - 1` done

Labs
177

contd.

LAB:3
if [ $# -ne 2 ] then echo "Usage - $0 x y" echo " exit 1 fi Where x and y are two nos for which I will print sum"

echo "Sum of $1 and $2 is `expr $1 + $2`"

Labs
178

contd.

LAB:4
i=5 if [ $# -ne 1 ] then echo "Usage: $0 number" echo " echo " I will find sum of all digit for given number" For eg. $0 123, I will print 6 as sum of all digit (1+2+3)"

exit 1
fi n=$1 sum=0 sd=0 while [ $n -gt 0 ] do sd=`expr $n % 10` sum=`expr $sum + $sd` n=`expr $n / 10` done echo "Sum of digit for number is $sum"

Labs
179

contd.

LAB:5
if [ $# -ne 1 ] then echo "Usage: $0 number" echo " echo " exit 1 I will find reverse of given number" For eg. $0 123, I will print 321"

fi
n=$1 rev=0 sd=0

while [ $n -gt 0 ] do sd=`expr $n % 10` rev=`expr $rev \* 10 + $sd` n=`expr $n / 10` done echo "Reverse number is $rev"