The Shell Command Language

In The Interactive Shell, we learned several important concepts for users interfacing with the shell through interactive terminals. In this section, we will build on what we have learned and study the programmatic aspects of the shell command language. Before we get started, let’s review some of the key concepts of the interactive shell.

Basic Shell Operation

The shell operates a seven step process:

  1. Read a line of input

  2. Break the line into word and operator tokens

  3. Parse tokens to form a command[1]

  4. Expand parts of the command

  5. Redirect files and perform variable assignment

    • If command name is present, redirection affects the command, and variables are assigned in the command’s environment;

    • Otherwise, redirection is performed in a subshell, and variables are assigned into the shell environment.

  6. Execute the command if a command name is present

  7. If the command is a foreground command, wait for it to complete and collect its exit status

Quoting Rules

Backslashes (\) quote the next character after the backslash. Single quotes (’) quote all enclosed characters between the single quotes. Double quotes (”) also quote enclosed characters, but retain the special meaning of the dollar sign ($) and backquote (`). Within double quotes, the special meaning of the backslash (\) is retained only when used to escape $, `, “, \, or <newline>.

Expansion

There are four types of expansions which are performed on command words, the part of an assignment word following the ‘=’, and redirection filenames:

Tilde Expansion

Tilde (~) as the first character of a word, followed by an optional username ending at the first slash (/) or end of word, is expanded to that user’s home directory; an omitted username (“~” or “~/…”) expands to the contents of the HOME environment variable.

Parameter Expansion

${parameter} expands to the value of the given parameter, or an empty string if parameter is unset. Several modifiers may be used to alter the result of parameter expansion.

Command Substitution

`command` or $(command) executes command in a subshell environment. The output of command, with any trailing newlines removed, is substituted.

Arithmetic Expansion

$((expression)) expands to the result of expression. Variables may be referenced by name, and can also be assigned using assignment operators.

Command words also undergo field splitting and pathname expansion; assignment words and redirection filenames do not:

Field Splitting

The characters in the IFS variable are used to split words into field. This occurs only within substituted text from an expansion. The <space>, <tab>, and <newline> characters, if present in IFS, act as field separators, while any other characters act as field delimiters.

Pathname Expansion

Pattern matching sequences are expanded to separate fields for each matching filename.

Redirection

The redirection operators open and close files in different modes depending on the operator used. An io number immediately preceding the operator causes the file to be opened on that file descriptor number; otherwise, the default file descriptor number is 0 (stdin) for operators beginning with <, and 1 (stdout) for operators beginning with >. If there is no command name, redirection is performed in a subshell environment and does not affect the current shell or subsequent commands–but files are still opened and/or created as directed.

Variable Assignment

Assignment words at the beginning of a command of the form name=value cause that variable to be set; if there is no command name, the variable is set in the current shell environment, otherwise it is added to the command’s environment variables.

Execution

If a command name is present and contains a slash (i.e. is a pathname) the associated file is executed. Otherwise, the shell searches for and selects the first match from the following categories of commands, in the order listed:

  • Shell special built-in utilities

  • Shell functions

  • Regular built-in utilities

  • Executable files within each directory listed in the PATH environment variable.

Waiting

The shell waits on a foreground command and collects its exit status. This step is skipped for background commands–those ending with a & control operator.

Pipelines

A pipeline is a series of simple commands separated by the pipeline (|) operator. The stdout of each command is redirected to the stdin of the next so that multiple commands can be chained together to process a stream of input. Each command runs in a separate subshell environment and the shell waits only on and collects the exit status of the last command in the pipeline–unless the & operator is present, in which case the entire pipeline runs in the background as a single shell job.

Background Commands and Jobs

The shell is either processing input and executing commands (steps 1-6), or it is suspended and waiting for the current foreground command to finish (step 7). The current foreground command can be interrupted by pressing ctrl-z, causing the terminal to send a SIGTSTP signal to the running process. The command is placed in the shell’s list of background jobs in a stopped state, and the shell resumes processing input and executing commands.

Each background job is assigned a job id, and the list of jobs can be viewed with the jobs utility. A background job that is stopped may be resumed using the bg utility. A stopped or running background job can also be made to resume in the foreground with the fg utility.

If a background job attempts to read from the terminal, it will be stopped by a SIGTTIN signal and must be moved to the foreground with the fg utility in order to proceed with reading input; it can then be moved back to the background with the bg utility, when it is ready to continue running in the background without needing more input.

Commands terminated with the background control operator (&) are never initially waited for as described above; instead, they are assigned a job id and run in the background immediately.