Every skilled trade has its secrets -- those little tricks, techniques, and tools that make light of even the most complex task. For instance, my neighbor is a master carpenter. His naked eye can measure and transfer angles with great precision, miters join seamlessly, and his finishing work has earned him acclaim in local newspapers.
But what's more remarkable (at least to me -- a lay person and an accident waiting to happen) is the relative ease with which he works. After some 20 years in the trade, there isn't a shortcut he hasn't mastered. The shortcuts shave a smidgen of time here, some labor there, yet with repetitive tasks such as making cuts, driving nails, and assembling framing, the savings really add up.
Programmers, system administrators, and other UNIX® computer professionals have their own kind of specialized tools:
- Operating systems
- The shell
Give your fingers a break, don't break your fingers
As Part 1 showed, the power of the UNIX command line is unmatched. With just a few keystrokes and a bit of syntactic glue, including pipes (|),
tee, and redirection, you can assemble your own impromptu data transforms at each shell prompt.
For example, this command finds all of the text documents in your home directory that contain the words Monthly Report:
$ find /home/joe -type f -name '*.txt' -print | xargs grep -l "Monthly Report"
The command searches your entire home directory (
find /home/joe) to find all regular files (
-type f) with the suffix
.txt, and then runs the
grepcommand to search for the string
-loption prints the file's name if a match was found. Hence, the output of the command is a list of files that match.
While the command above is useful, it's onerous to remember and retype, especially if you use the command regularly. Moreover, when the command line is your primary interface to e-mail, files, tools (such as editors, compilers, monitors), and remote systems, any time and effort you can save at the command line can be better spent on the task at hand. After all, a thousand few fractions of a second really add up.
To make light of repetitive tasks, UNIX shells provide a variety of helpful shortcuts, including:
- A command history
- Environment variables
- Startup files
~(tilde). You can also refer to your home directory using the
$HOMEenvironment variable, as shown in Listing 1.
Listing 1. UNIX shell shortcuts
$ whoami strike $ echo ~ /Users/strike $ echo $HOME /Users/strike $ !! echo $HOME /Users/strike
That last command,
!!(two exclamation marks), might look a little strange, but it's a command history sigil that repeats the previous command verbatim. (Many shells also allow you to browse the list of previous commands using the up arrow key, or by pressing Control+P.)
Let's look at each kind of shell shortcut in more detail. This article is based on the Z shell (
zsh), which is typically installed in /bin/zsh. (If your system doesn't have the Z shell, ask your system administrator to install it.) The Z shell has a few special features; otherwise, all the examples shown here work in all modern UNIX shells.
Many command-line arguments are used so frequently that shells provide sigils, or symbols, as shorthand. You simply type the sigil in place of the argument.
As mentioned above,
~refers to your home directory. A similar shorthand,
~username, refers to username's home directory. For example,
~joerefers to joe's home directory. So, to copy a file from joe's doc directory to your info directory, you could type:
$ cp ~joe/doc/report.txt ~/info
Assuming that joe's home directory is in /guests and your home directory is /staff/bobr,
~joeis replaced with /guests/joe and
~becomes /staff/bobr, finally yielding the command
cp /guests/joe/doc/report.txt /staff/bobr/info. (See the sidebar, "Proofing your work" to learn how to preview your command line.)
Another valuable sigil is
..(two periods), shorthand for the directory immediately above the current directory. With
., the sigil for the current working directory, you can refer to files and directories in the file system relative to your current working directory.
For instance, if your current working directory is ~/jane/projects/lambda, the shorthand
../..refers to the directory two directories above, or ~/jane.
To refer to the directory that contains ~/jane, you can use
../../../("three directories above") or the path
The latter path says start at ~jane, and then go up one directory.
To copy a file to your current directory, you need not name it; simply refer to it as . ("dot"):
$ cp -pr /path/to/lots/of/stuff .
The former command recursively copies the /path/to/lots/of/stuff directory to your current directory, preserving the original time and date stamps. Path names that refer to
.are called relative path names. Path names that begin with a / (forward slash) or a ~ (tilde) are called absolute path names because you're referring to the file from the top of the file system, or from the top of a directory hierarchy.
Wildcards and patterns
With sigils, you reduce your typing time and can refer to a specific directory quickly and concisely. Wildcards are another form of shorthand to refer to the contents of a directory.
For example, assume that you have a directory containing 100 files. Some are C source code files that end with the suffix .c, others are object files with suffix .o, and still others are text documents (.txt), scripts (.sh), and executables (files with execute permission). To list only the
Cfiles, simply type:
$ ls *.c
*(typically called star rather than asterisk) means match any sequence of characters. The
.cfile name extension is a literal pattern that matches only a period followed by a lowercase c. So,
*.cmeans any sequence of characters followed by a period and a lowercase c. Given
*.c, the shell looks in the current directory (unless you provide a leading absolute or relative path name), finds every file name that matches the pattern, expands
*.cto that list of names, and passes the list as arguments to the
Listing 2 demonstrates the use of
*.cbased on the source code to
wget, the command-line download utility.
Listing 2. Use wildcards to find C source code files in a directory
$ ls *.c alloca.c ansi2knr.c cmpt.c connect.c convert.c ...
The process of expanding a wildcard to the list of matching file names is called globbing, and UNIX shells have a variety of globbing operators (so-called globs) to help you express what you're looking for:
- The glob
*(star) matches any character or sequence of characters, including an empty sequence.
- The glob
?(question mark) matches any single character.
- The glob
[ ](square brackets) matches any of the enclosed characters. Within the brackets, you can refer to a range of characters by using
-(hyphen), as in
[a-z]or all lowercase letters.
(The Z shell has many unique glob operators.
You can also repeat glob operators as necessary. Listing 3 provides additional examples.
1 $ ls -1 -a -F ./libs ChangeLog ChangeLog-branches/ Makefile Makefile.in alloca.c ansi2knr.c cmpt.c cmpt.o config.h config.h.in connect.c connect.h connect.o convert.c convert.h convert.o ... wget* 2 $ ls -a -F .* ./lib 3 $ ls -1 *.? alloca.c ansi2knr.c cmpt.c cmpt.o config.h connect.c connect.h connect.o convert.c convert.h convert.o ... 4 $ ls -1 ????.? cmpt.c cmpt.o 5 $ ls [a-c]?*.* alloca.c ansi2knr.c cmpt.c cmpt.o config.h config.h.in connect.c connect.h connect.o convert.c convert.h convert.o cookies.c cookies.h cookies.o
In Listing 3, Command 1 shows all the entries in the directory, including those entries that begin with . (dot) in a long list. (The
-aoption shows the so-called dot files; the
-1option lists everything in one column; and the
-Foption highlights directories with a / (forward slash) and executables with a * (star).)
Command 2 finds each entry whose name begins with a dot (hence .*). The third command finds only those items that have a one-letter suffix.
The fourth command finds only those items that have four characters followed by a dot and one character. Finally, Command 5 finds items that begin with lowercase a, lowercase b, or lowercase c and are followed by at least one letter, then anything, then a period, and then any suffix. As you can see, you can repeat the glob operators as needed.
So, what would
ls *.zyield (assuming no such files exist)? It yields a helpful error message:
$ ls *.z zsh: no matches found: *.z
A bit of (command) history
So far, you've seen how to specify paths and pick and choose files. You can express yourself at the command line. However, even if all command lines were short and sweet, chances are you would still get tired of typing the same thing over and over again. In particular, you would probably get weary of typing long, complex command lines with loads of options, or where the order of the arguments has to be just so. Luckily, most shells maintain a history of previous commands. To rerun a command, you simply find its entry in the history list and rerun it. And like other parts of the shell, shortcuts make references quick and easy.
To enable command history in Z shell, type:
$ HISTSIZE=500 $ SAVEHIST=500
Here, the commands specify that both the shell and the persisted history file should retain the last 500 commands. (By default, Z shell saves only the last 30 commands.) Check your shell's documentation for information on how to capture and persist command histories.
After working in the shell a while, you can view your command history by simply typing
$ history ... 781 /bin/ls -d */ 782 /bin/ls -F *(/) 783 /bin/ls -d -F *(/) 784 /bin/ls -d -F */ 785 /bin/ls -d */
Each command you run is assigned a sequential, numerical identifier. You use that identifier, such as 782, to refer to an entire command and to parts of each command. To rerun a command verbatim, type
!(exclamation mark) followed by the command's number:
$ !785 ChangeLog-branches/ doc/ po/ src/ util/ windows/
If you want a specific argument from a historical command, refer to the command with a ! (exclamation mark) and provide :N,where 0 refers to the command name, 1 refers to the first argument, and so on. For example, to extract the second argument of command 782 in the history log, type the code shown in Listing 4.
Listing 4. Extract the second argument from command 782
$ echo !782:2 echo *(/) ChangeLog-branches doc po src util windows $ ls AUTHORS COPYING INSTALL MACHINES AUTHORS COPYING INSTALL MACHINES $ echo !!:3 echo INSTALL $ history -2 788 ls AUTHORS COPYING INSTALL MACHINES 789 echo INSTALL $ echo !788^ echo AUTHORS AUTHORS $ echo !788$ echo MACHINES MACHINES
history -2prints the previous two commands. As shortcuts, you can refer to the first argument of a command (not the command name itself) using
^(carat), and you can refer to the last argument of a historical command with the shortcut
$(dollar sign). You can also refer to a range of arguments using a range notation, as shown in Listing 5.
Listing 5. A range notation
$ echo AUTHORS COPYING INSTALL MACHINES AUTHORS COPYING INSTALL MACHINES $ echo !!:1-2 echo AUTHORS COPYING AUTHORS COPYING
There are also other, more direct ways to recall historical commands. One way is to search for it:
$ ls I* $ ls M* $ echo !?M ls INSTALL
!?Masks for the most recent historical command line that contains an uppercase letter M.
Speaking fluent command line is an essential UNIX skill. But speaking UNIX doesn't stop at the shell prompt -- you must also communicate with the myriad of UNIX utilities. In UNIX, environment variables retain settings in your shell and allow you to propagate your preferences to each and every utility you launch from the command line.
Some environment variables -- called shell variables -- are used only by your shell to control its behavior. For instance, only the Z shell uses
$SAVEHIST, shown above, to manage command histories. Think of shell variables as settings.
Other environment variables are exported, or made globally available, and are copied into the process space (the environment) of every command you launch from the command line. For example,
$HOMEis a special environment variable that retains the location of your home directory. The UNIX login sequence sets
$HOME(and other environment variables), and then starts your shell, which in turn uses
$HOMEto find all your shell startup files. Other applications that you launch, such as SSH and FTP, refer to
$HOMEto find your .netrc file (used to store confidential, remote access passwords). Some environment variables -- such as
$SHELL-- every application uses. Other environment variables might be unique to an application.
To see all your current environment variables, type
printenv, as shown in Listing 6. (Depending on how your system administrator configured your system, you might have many more, or far fewer, environment variables than are shown here.)
Listing 6. View environment variables
$ printenv PATH=/Users/strike/bin:/Applications/xampp/xamppfiles/bin:/Users/strike/bin:/usr/bin:/ bin:/usr/sbin:/sbin HOME=/Users/strike SHELL=/bin/zsh USER=strike TERM=xterm-color LOGNAME=strike SHLVL=1 PWD=/Local/src/versions/wget/wget-1.9 OLDPWD=/Local/src/versions/wget/wget-1.9/src PERL5LIB=/Applications/xampp/xamppfiles/lib/perl5/site_perl/5.8.7:/Projects/IGSP/src CLICOLOR=true MANPATH=/Local/root/share/man:/usr/share/man:/opt/local/share/man INFOPATH=/opt/local/share/info LESS=-n
You likely recognize many of these variables; others might be new. The shell level (
$SHLVL) shows how many shells deep you are. A 1 indicates a login shell; a 2 means that you launched another shell from your login shell, and so on. You can use the value of
$SHLVLto change your prompt for each subsequent, nested shell.
$TERMreflects your terminal (probably terminal emulator) settings -- important for ensuring proper rendering of text, colors, as well as proper interpretation of keystrokes.
$PWDis your current working directory, while
$OLDPWDis your previous working directory. You can use both variables to quickly go back and forth between two directories, as shown in Listing 7.
Listing 7. Toggle between directories
$ echo $PWD /Users/strike $ echo $OLDPWD /Local/src/versions/wget/wget-1.9 $ cd $OLDPWD $ echo $PWD /Local/src/versions/wget/wget-1.9 $ echo $OLDPWD /Users/strike
The remaining environment variables in the list above are application-specific. Each retains preferences that control how each associated application works when you launch it.
$PERL5LIBis a search path for Perl to find custom libraries. The
$CLICOLORto render file types in color (directories in blue, executables in green, and so on). Custom application environment variables are typically documented in the program's man pages.
Setting an environment variable is identical to setting a shell variable. However, you must export the variable to make it globally available:
$ MYVARIABLE=$HOME/projectX $ export TMPDIR=/tmp/projectX
The former command sets a shell variable named
$MYVARIABLE. (The leading dollar sign is the shell prompt. When you set a variable, you don't provide the $. However, you do need the dollar sign, as in
$MYVARIABLE, whenever you use the variable.)
$MYVARIABLEis visible only to the shell, because it wasn't exported. To see a list of all shell variables, type
set. The output of
setincludes the environment variables, because those are available to the shell as well.
In the latter command,
$TMPDIRis set, exported, and available to all applications launched from the shell. One application that uses
$TMPDIRis the GNU Compiler Collection (GCC) compiler. The value you store in
$TMPDIRis where GCC generates its temporary files.
If you want to remove an environment variable, simply type
unsetand the name of the variables, as shown in Listing 8.
Listing 8. Remove an environment variable
$ set HOME=/Users/strike MYVARIABLE=/Users/strike/projectX TMPDIR=/tmp/projectX ... $ unset MYVARIABLE TMPDIR $ set HOME=/Users/strike ....
Aliases and startup files
The previous sections might have you concerned about just how much you have to type at the command line. Yes, there's a lot to learn -- this is because the shell environment is so rich. Remember, though, that with great power comes great productivity (many apologies to Spider-man).
To conserve those precious keystrokes and retain all the settings you've made, UNIX shells offer aliases and startup files, respectively. Aliases are shortcuts that you create. Startup files are read each time your shell starts and are the ideal place to store (and share) all your shell settings, such as shell variables (options), environment variables, and aliases.
An alias is a short sequence that you use instead of a longer command. You can think of an alias as a nickname for a command line. Instead of typing:
$ find /home/joe -type f -name '*.txt' -print | xargs grep -l "Monthly Report"
at the command prompt, you might type a nickname that you created:
The shell does the heavy lifting, replacing
findreportswith its expansion. To create the
alias findreports='find $HOME -type f -name "*.txt" -print | xargs grep -l "Monthly Report"'
Single quotation marks must delimit each alias. If you need quotation marks inside the alias, use double quotation marks. Z shell aliases can contain many shell primitives, including variables, pipes, redirection, other aliases, and other shell operands, as shown in Listing 9.
Listing 9. Z shell primitives
$ alias ll='/bin/ls -l' $ ll -d 2002* drwxrwxr-x 2 www-data www-data 4096 Jan 16 2002 2002-02 drwxrwxr-x 2 www-data www-data 4096 Jan 22 2002 2002-03 drwxrwxr-x 2 www-data www-data 4096 Apr 15 2002 2002-04 drwxrwxr-x 2 www-data www-data 4096 Apr 19 2002 2002-05 ... $ alias lt='ll -t' $ lt -d 2002* drwxrwxr-x 2 www-data www-data 4096 Apr 19 2002 2002-05 drwxrwxr-x 2 www-data www-data 4096 Apr 15 2002 2002-04 drwxrwxr-x 2 www-data www-data 4096 Jan 22 2002 2002-03 drwxrwxr-x 2 www-data www-data 4096 Jan 16 2002 2002-02 $ alias m='pinky | grep mstreicher' $ m mstreicher Martin Streicher ... $ alias snap='pinky >> ~/.pinky' $ snap $ snap $ cat ~/.pinky Login Name TTY Idle When Where mstreicher Martin Streicher pts/0 Jun 18 16:40 cpe-071-065-224-025.nc.res.rr.com Login Name TTY Idle When Where mstreicher Martin Streicher pts/0 Jun 18 16:40 cpe-071-065-224-025.nc.res.rr.com
llrefers to /bin/ls -- absolute paths are never replaced by alias substitution.
When you type
ll, it's replaced by its alias, and any remaining command-line arguments are appended. Hence,
ll -d 2002*is really the command
/bin/ls -l -d 2002*. The alias
lland adds the
-tflag to sort by creation time. The
ltalias expands to
/bin/ls -l -t -d 2002*. The
malias includes a pipe. The
snapalias uses redirection to append the output of a command to a file.
To see all the aliases set in your shell, just type
alias(with no arguments), as shown in Listing 10.
Listing 10. View all the aliases in your shell
$ alias alias findreports='find $HOME -type f -name "*.txt" -print | xargs grep -l "Monthly Report"' alias ll='/bin/ls -l' alias lt='ll -t' alias m='pinky | grep mstreicher' alias snap='pinky >> ~/.pinky' ...
If you want to remove an alias, just type
unaliasand the alias's name. You can also list multiple aliases at a time, as shown inListing 11.
Listing 11. View multiple aliases simultaneously
$ unalias m snap $ alias alias findreports='find $HOME -type f -name "*.txt" -print | xargs grep -l "Monthly Report"' alias ll='/bin/ls -l' alias lt='ll -t'
Finally, after you've worked hard to set up your environment just so, you'll want to keep your settings for next time. Indeed, you want your shell to be consistent from session to session and from instance to instance -- say, when multiple terminal windows are open on your workstation.
Shells include startup files to (re)initialize your environment when your shell starts. Startup files can be simple -- just a list of variables and values -- or quite complex, including customization logic and elaborate functions. Some users keep many sets of startup files, one set per project.
Z shell uses the startup files .zshrc and .zprofile, both of which reside in your home directory. (Other shells have similar files with similar names, and you can read your shell documentation for specifics. Some shells also provide for shutdown files, or files to run when your shell is exiting.) The .zshrc file is sourced, or read, and processed whenever you start a new shell; the .zprofile file is sourced only when you start a login shell.
After you've configured your shell, take a snapshot of your settings and save them in one of the shell startup files:
$ set >> $HOME/.zshrc $ alias >> $HOME/.zshrc
Note: You might want to edit the resulting .zshrc file and remove variables that are session-specific.
Whew! This installment of Speaking UNIX covered a lot of ground, but your diligence should yield vast rewards. Work smarter, not harder, and save the extra time to do really important things, like play slashem.
Next time, UNIX Tutorial goes positively old school. I'll forgot those trendy browsers and examine how to connect, download, upload, transfer, and communicate entirely from the command line.