Advanced Bash-Scripting Guide

An in-depth exploration of the art of shell scripting

Version 5.4.03

10 August 2008

Mendel Cooper


thegrendel@theriver.com

This tutorial assumes no previous knowledge of scripting or programming, but progresses rapidly toward an intermediate/advanced level of instruction . . . all the while sneaking in little nuggets of UNIX® wisdom and lore. It serves as a textbook, a manual for self-study, and a reference and source of knowledge on shell scripting techniques. The exercises and heavily-commented examples invite active reader participation, under the premise that the only way to really learn scripting is to write scripts.

This book is suitable for classroom use as a general introduction to programming concepts.

The latest update of this document, as an archived, bzip2-ed "tarball" including both the SGML source and rendered HTML, may be downloaded from the author's home site. A pdf version is also available ( pdf mirror site). See the change log for a revision history.

Dedication

For Anita, the source of all the magic

Table of Contents
Part 1. Introduction
1. Why Shell Programming?
2. Starting Off With a Sha-Bang
2.1. Invoking the script
2.2. Preliminary Exercises
Part 2. Basics
3. Special Characters
4. Introduction to Variables and Parameters
4.1. Variable Substitution
4.2. Variable Assignment
4.3. Bash Variables Are Untyped
4.4. Special Variable Types
5. Quoting
5.1. Quoting Variables
5.2. Escaping
6. Exit and Exit Status
7. Tests
7.1. Test Constructs
7.2. File test operators
7.3. Other Comparison Operators
7.4. Nested if/then Condition Tests
7.5. Testing Your Knowledge of Tests
8. Operations and Related Topics
8.1. Operators
8.2. Numerical Constants
Part 3. Beyond the Basics
9. Variables Revisited
9.1. Internal Variables
9.2. Manipulating Strings
9.3. Parameter Substitution
9.4. Typing variables: declare or typeset
9.5. Indirect References
9.6. $RANDOM: generate random integer
9.7. The Double-Parentheses Construct
10. Loops and Branches
10.1. Loops
10.2. Nested Loops
10.3. Loop Control
10.4. Testing and Branching
11. Command Substitution
12. Arithmetic Expansion
13. Recess Time
Part 4. Commands
14. Internal Commands and Builtins
14.1. Job Control Commands
15. External Filters, Programs and Commands
15.1. Basic Commands
15.2. Complex Commands
15.3. Time / Date Commands
15.4. Text Processing Commands
15.5. File and Archiving Commands
15.6. Communications Commands
15.7. Terminal Control Commands
15.8. Math Commands
15.9. Miscellaneous Commands
16. System and Administrative Commands
16.1. Analyzing a System Script
Part 5. Advanced Topics
17. Regular Expressions
17.1. A Brief Introduction to Regular Expressions
17.2. Globbing
18. Here Documents
18.1. Here Strings
19. I/O Redirection
19.1. Using exec
19.2. Redirecting Code Blocks
19.3. Applications
20. Subshells
21. Restricted Shells
22. Process Substitution
23. Functions
23.1. Complex Functions and Function Complexities
23.2. Local Variables
23.3. Recursion Without Local Variables
24. Aliases
25. List Constructs
26. Arrays
27. /dev and /proc
27.1. /dev
27.2. /proc
28. Of Zeros and Nulls
29. Debugging
30. Options
31. Gotchas
32. Scripting With Style
32.1. Unofficial Shell Scripting Stylesheet
33. Miscellany
33.1. Interactive and non-interactive shells and scripts
33.2. Operator Precedence
33.3. Shell Wrappers
33.4. Tests and Comparisons: Alternatives
33.5. A script calling itself (recursion)
33.6. "Colorizing" Scripts
33.7. Optimizations
33.8. Assorted Tips
33.9. Security Issues
33.10. Portability Issues
33.11. Shell Scripting Under Windows
34. Bash, versions 2 and 3
34.1. Bash, version 2
34.2. Bash, version 3
35. Endnotes
35.1. Author's Note
35.2. About the Author
35.3. Where to Go For Help
35.4. Tools Used to Produce This Book
35.4.1. Hardware
35.4.2. Software and Printware
35.5. Credits
35.6. Disclaimer
Bibliography
A. Contributed Scripts
B. Reference Cards
C. A Sed and Awk Micro-Primer
C.1. Sed
C.2. Awk
D. Exit Codes With Special Meanings
E. A Detailed Introduction to I/O and I/O Redirection
F. Command-Line Options
F.1. Standard Command-Line Options
F.2. Bash Command-Line Options
G. Important Files
H. Important System Directories
I. Localization
J. History Commands
K. A Sample .bashrc File
L. Converting DOS Batch Files to Shell Scripts
M. Exercises
M.1. Analyzing Scripts
M.2. Writing Scripts
N. Revision History
O. Mirror Sites
P. To Do List
Q. Copyright
R. ASCII Table
Index
List of Tables
14-1. Job identifiers
30-1. Bash options
33-1. Operator Precedence
33-2. Numbers representing colors in Escape Sequences
B-1. Special Shell Variables
B-2. TEST Operators: Binary Comparison
B-3. TEST Operators: Files
B-4. Parameter Substitution and Expansion
B-5. String Operations
B-6. Miscellaneous Constructs
C-1. Basic sed operators
C-2. Examples of sed operators
D-1. Reserved Exit Codes
L-1. Batch file keywords / variables / operators, and their shell equivalents
L-2. DOS commands and their UNIX equivalents
N-1. Revision History
List of Examples
2-1. cleanup: A script to clean up the log files in /var/log
2-2. cleanup: An improved clean-up script
2-3. cleanup: An enhanced and generalized version of above scripts.
3-1. Code blocks and I/O redirection
3-2. Saving the output of a code block to a file
3-3. Running a loop in the background
3-4. Backup of all files changed in last day
4-1. Variable assignment and substitution
4-2. Plain Variable Assignment
4-3. Variable Assignment, plain and fancy
4-4. Integer or string?
4-5. Positional Parameters
4-6. wh, whois domain name lookup
4-7. Using shift
5-1. Echoing Weird Variables
5-2. Escaped Characters
6-1. exit / exit status
6-2. Negating a condition using !
7-1. What is truth?
7-2. Equivalence of test, /usr/bin/test, [ ], and /usr/bin/[
7-3. Arithmetic Tests using (( ))
7-4. Testing for broken links
7-5. Arithmetic and string comparisons
7-6. Testing whether a string is null
7-7. zmore
8-1. Greatest common divisor
8-2. Using Arithmetic Operations
8-3. Compound Condition Tests Using && and ||
8-4. Representation of numerical constants
9-1. $IFS and whitespace
9-2. Timed Input
9-3. Once more, timed input
9-4. Timed read
9-5. Am I root?
9-6. arglist: Listing arguments with $* and $@
9-7. Inconsistent $* and $@ behavior
9-8. $* and $@ when $IFS is empty
9-9. Underscore variable
9-10. Inserting a blank line between paragraphs in a text file
9-11. Generating an 8-character "random" string
9-12. Converting graphic file formats, with filename change
9-13. Converting streaming audio files to ogg
9-14. Emulating getopt
9-15. Alternate ways of extracting and locating substrings
9-16. Using parameter substitution and error messages
9-17. Parameter substitution and "usage" messages
9-18. Length of a variable
9-19. Pattern matching in parameter substitution
9-20. Renaming file extensions:
9-21. Using pattern matching to parse arbitrary strings
9-22. Matching patterns at prefix or suffix of string
9-23. Using declare to type variables
9-24. Indirect Variable References
9-25. Passing an indirect reference to awk
9-26. Generating random numbers
9-27. Picking a random card from a deck
9-28. Brownian Motion Simulation
9-29. Random between values
9-30. Rolling a single die with RANDOM
9-31. Reseeding RANDOM
9-32. Pseudorandom numbers, using awk
9-33. C-style manipulation of variables
10-1. Simple for loops
10-2. for loop with two parameters in each [list] element
10-3. Fileinfo: operating on a file list contained in a variable
10-4. Operating on files with a for loop
10-5. Missing in [list] in a for loop
10-6. Generating the [list] in a for loop with command substitution
10-7. A grep replacement for binary files
10-8. Listing all users on the system
10-9. Checking all the binaries in a directory for authorship
10-10. Listing the symbolic links in a directory
10-11. Symbolic links in a directory, saved to a file
10-12. A C-style for loop
10-13. Using efax in batch mode
10-14. Simple while loop
10-15. Another while loop
10-16. while loop with multiple conditions
10-17. C-style syntax in a while loop
10-18. until loop
10-19. Nested Loop
10-20. Effects of break and continue in a loop
10-21. Breaking out of multiple loop levels
10-22. Continuing at a higher loop level
10-23. Using continue N in an actual task
10-24. Using case
10-25. Creating menus using case
10-26. Using command substitution to generate the case variable
10-27. Simple string matching
10-28. Checking for alphabetic input
10-29. Creating menus using select
10-30. Creating menus using select in a function
11-1. Stupid script tricks
11-2. Generating a variable from a loop
11-3. Finding anagrams
14-1. A script that forks off multiple instances of itself
14-2. printf in action
14-3. Variable assignment, using read
14-4. What happens when read has no variable
14-5. Multi-line input to read
14-6. Detecting the arrow keys
14-7. Using read with file redirection
14-8. Problems reading from a pipe
14-9. Changing the current working directory
14-10. Letting let do arithmetic.
14-11. Showing the effect of eval
14-12. Echoing the command-line parameters
14-13. Forcing a log-off
14-14. A version of rot13
14-15. Using eval to force variable substitution in a Perl script
14-16. Using set with positional parameters
14-17. Reversing the positional parameters
14-18. Reassigning the positional parameters
14-19. "Unsetting" a variable
14-20. Using export to pass a variable to an embedded awk script
14-21. Using getopts to read the options/arguments passed to a script
14-22. "Including" a data file
14-23. A (useless) script that sources itself
14-24. Effects of exec
14-25. A script that exec's itself
14-26. Waiting for a process to finish before proceeding
14-27. A script that kills itself
15-1. Using ls to create a table of contents for burning a CDR disk
15-2. Hello or Good-bye
15-3. Badname, eliminate file names in current directory containing bad characters and whitespace.
15-4. Deleting a file by its inode number
15-5. Logfile: Using xargs to monitor system log
15-6. Copying files in current directory to another
15-7. Killing processes by name
15-8. Word frequency analysis using xargs
15-9. Using expr
15-10. Using date
15-11. Date calculations
15-12. Word Frequency Analysis
15-13. Which files are scripts?
15-14. Generating 10-digit random numbers
15-15. Using tail to monitor the system log
15-16. Printing out the From lines in stored e-mail messages
15-17. Emulating grep in a script
15-18. Crossword puzzle solver
15-19. Looking up definitions in Webster's 1913 Dictionary
15-20. Checking words in a list for validity
15-21. toupper: Transforms a file to all uppercase.
15-22. lowercase: Changes all filenames in working directory to lowercase.
15-23. du: DOS to UNIX text file conversion.
15-24. rot13: ultra-weak encryption.
15-25. Generating "Crypto-Quote" Puzzles
15-26. Formatted file listing.
15-27. Using column to format a directory listing
15-28. nl: A self-numbering script.
15-29. manview: Viewing formatted manpages
15-30. Using cpio to move a directory tree
15-31. Unpacking an rpm archive
15-32. Stripping comments from C program files
15-33. Exploring /usr/X11R6/bin
15-34. An "improved" strings command
15-35. Using cmp to compare two files within a script.
15-36. basename and dirname
15-37. A script that copies itself in sections
15-38. Checking file integrity
15-39. Uudecoding encoded files
15-40. Finding out where to report a spammer
15-41. Analyzing a spam domain
15-42. Getting a stock quote
15-43. Updating FC4
15-44. Using ssh
15-45. A script that mails itself
15-46. Generating prime numbers
15-47. Monthly Payment on a Mortgage
15-48. Base Conversion
15-49. Invoking bc using a here document
15-50. Calculating PI
15-51. Converting a decimal number to hexadecimal
15-52. Factoring
15-53. Calculating the hypotenuse of a triangle
15-54. Using seq to generate loop arguments
15-55. Letter Count"
15-56. Using getopt to parse command-line options
15-57. A script that copies itself
15-58. Exercising dd
15-59. Capturing Keystrokes
15-60. Securely deleting a file
15-61. Filename generator
15-62. Converting meters to miles
15-63. Using m4
16-1. Setting a new password
16-2. Setting an erase character
16-3. secret password: Turning off terminal echoing
16-4. Keypress detection
16-5. Checking a remote server for identd
16-6. pidof helps kill a process
16-7. Checking a CD image
16-8. Creating a filesystem in a file
16-9. Adding a new hard drive
16-10. Using umask to hide an output file from prying eyes
16-11. killall, from /etc/rc.d/init.d
18-1. broadcast: Sends message to everyone logged in
18-2. dummyfile: Creates a 2-line dummy file
18-3. Multi-line message using cat
18-4. Multi-line message, with tabs suppressed
18-5. Here document with parameter substitution
18-6. Upload a file pair to Sunsite incoming directory
18-7. Parameter substitution turned off
18-8. A script that generates another script
18-9. Here documents and functions
18-10. "Anonymous" Here Document
18-11. Commenting out a block of code
18-12. A self-documenting script
18-13. Prepending a line to a file
18-14. Parsing a mailbox
19-1. Redirecting stdin using exec
19-2. Redirecting stdout using exec
19-3. Redirecting both stdin and stdout in the same script with exec
19-4. Avoiding a subshell
19-5. Redirected while loop
19-6. Alternate form of redirected while loop
19-7. Redirected until loop
19-8. Redirected for loop
19-9. Redirected for loop (both stdin and stdout redirected)
19-10. Redirected if/then test
19-11. Data file names.data for above examples
19-12. Logging events
20-1. Variable scope in a subshell
20-2. List User Profiles
20-3. Running parallel processes in subshells
21-1. Running a script in restricted mode
23-1. Simple functions
23-2. Function Taking Parameters
23-3. Functions and command-line args passed to the script
23-4. Passing an indirect reference to a function
23-5. Dereferencing a parameter passed to a function
23-6. Again, dereferencing a parameter passed to a function
23-7. Maximum of two numbers
23-8. Converting numbers to Roman numerals
23-9. Testing large return values in a function
23-10. Comparing two large integers
23-11. Real name from username
23-12. Local variable visibility
23-13. Demonstration of a simple recursive function
23-14. Another simple demonstration
23-15. Recursion, using a local variable
23-16. The Fibonacci Sequence
23-17. The Towers of Hanoi
24-1. Aliases within a script
24-2. unalias: Setting and unsetting an alias
25-1. Using an and list to test for command-line arguments
25-2. Another command-line arg test using an and list
25-3. Using or lists in combination with an and list
26-1. Simple array usage
26-2. Formatting a poem
26-3. Various array operations
26-4. String operations on arrays
26-5. Loading the contents of a script into an array
26-6. Some special properties of arrays
26-7. Of empty arrays and empty elements
26-8. Initializing arrays
26-9. Copying and concatenating arrays
26-10. More on concatenating arrays
26-11. The Bubble Sort
26-12. Embedded arrays and indirect references
26-13. The Sieve of Eratosthenes
26-14. The Sieve of Eratosthenes, Optimized
26-15. Emulating a push-down stack
26-16. Complex array application: Exploring a weird mathematical series
26-17. Simulating a two-dimensional array, then tilting it
27-1. Using /dev/tcp for troubleshooting
27-2. Playing music
27-3. Finding the process associated with a PID
27-4. On-line connect status
28-1. Hiding the cookie jar
28-2. Setting up a swapfile using /dev/zero
28-3. Creating a ramdisk
29-1. A buggy script
29-2. Missing keyword
29-3. test24: another buggy script
29-4. Testing a condition with an assert
29-5. Trapping at exit
29-6. Cleaning up after Control-C
29-7. Tracing a variable
29-8. Running multiple processes (on an SMP box)
31-1. Numerical and string comparison are not equivalent
31-2. Subshell Pitfalls
31-3. Piping the output of echo to a read
33-1. shell wrapper
33-2. A slightly more complex shell wrapper
33-3. A generic shell wrapper that writes to a logfile
33-4. A shell wrapper around an awk script
33-5. A shell wrapper around another awk script
33-6. Perl embedded in a Bash script
33-7. Bash and Perl scripts combined
33-8. A (useless) script that recursively calls itself
33-9. A (useful) script that recursively calls itself
33-10. Another (useful) script that recursively calls itself
33-11. A "colorized" address database
33-12. Drawing a box
33-13. Echoing colored text
33-14. A "horserace" game
33-15. A Progress Bar
33-16. Return value trickery
33-17. Even more return value trickery
33-18. Passing and returning arrays
33-19. Fun with anagrams
33-20. Widgets invoked from a shell script
34-1. String expansion
34-2. Indirect variable references - the new way
34-3. Simple database application, using indirect variable referencing
34-4. Using arrays and other miscellaneous trickery to deal four random hands from a deck of cards
A-1. mailformat: Formatting an e-mail message
A-2. rn: A simple-minded file renaming utility
A-3. blank-rename: Renames filenames containing blanks
A-4. encryptedpw: Uploading to an ftp site, using a locally encrypted password
A-5. copy-cd: Copying a data CD
A-6. Collatz series
A-7. days-between: Days between two dates
A-8. Making a dictionary
A-9. Soundex conversion
A-10. Game of Life
A-11. Data file for Game of Life
A-12. behead: Removing mail and news message headers
A-13. ftpget: Downloading files via ftp
A-14. password: Generating random 8-character passwords
A-15. fifo: Making daily backups, using named pipes
A-16. Generating prime numbers using the modulo operator
A-17. tree: Displaying a directory tree
A-18. tree2: Alternate directory tree script
A-19. string functions: C-style string functions
A-20. Directory information
A-21. Object-oriented database
A-22. Library of hash functions
A-23. Colorizing text using hash functions
A-24. More on hash functions
A-25. Mounting USB keychain storage devices
A-26. Converting to HTML
A-27. Preserving weblogs
A-28. Protecting literal strings
A-29. Unprotecting literal strings
A-30. Spammer Identification
A-31. Spammer Hunt
A-32. Making wget easier to use
A-33. A podcasting script
A-34. Nightly backup to a firewire HD
A-35. An expanded cd command
A-36. A soundcard setup script
A-37. Locating split paragraphs in a text file
A-38. Insertion sort
A-39. Standard Deviation
A-40. A pad file generator for shareware authors
A-41. A man page editor
A-42. Petals Around the Rose
A-43. Quacky: a Perquackey-type word game
A-44. Nim
A-45. An all-purpose shell scripting homework assignment solution
A-46. An alternate version of the getopt-simple.sh script
A-47. Basics Reviewed
C-1. Counting Letter Occurrences
K-1. Sample .bashrc file
L-1. VIEWDATA.BAT: DOS Batch File
L-2. viewdata.sh: Shell Script Conversion of VIEWDATA.BAT
P-1. Print the server environment
R-1. A script that generates an ASCII table
  Next
  Introduction