Shell Basics

Navigating the File System

• Easiest way to learn basic Linux commands is to see them in action
• Type pwd (short for "print working directory") to find out where you are
• (Many Unix commands have cryptic names like this)

   $ pwd
   /home/hpotter/swc
				

• Then type ls (for "listing") to see what's in the current directory

   $ ls
   LICENSE.txt     conf        data        docs    index.swc   license.swc     print.css   swc.css     tests
   Makefile        config.mk   depend.mk   img     lec         press           sites       swc.dtd     util
				

• To see what's in the data directory, type ls data

   $ ls data
   bio   elements   haiku.txt   morse.txt   pdb   solarsystem.txt
				

(The “$” in the examples above is the shell "prompt" character. It means the shell is waiting for keyboard input.)

• Alternatively:
• Type cd data to 'go into' directory data
  • I.e., change the current working directory to data
• Type ls on its own
• Type cd .. to go up one level in the directory hierarchy
  • I.e., back to the directory you started from in this example

   $ cd data
   $ pwd
   /home/hpotter/swc/data
   $ ls
   bio   elements   haiku.txt   morse.txt   pdb   solarsystem.txt
   $ cd ..
   $ pwd
   /home/hpotter/swc
				

Creating Files and Directories

• Rather than messing with the course files, let's create a temporary directory and play around in there:

  • 
       $ mkdir tmp
       $ cd tmp
       $ ls
    			

  • Note that there's normally no output from mkdir, but mkdir -v (verbose) would print a confirmation message
• Use a plain text editor of your choice to create a file called earth.txt with the following contents:

  • 
       Name: Earth
       Period: 365.26 days
       Inclination: 0.00
       Eccentricity: 0.02
    			

  • TextEdit (on OS X) also runs in a window of its own
  • Pico or Vim (on Unix) takes over the shell window temporarily

• The easiest way to create a similar file venus.txt is to copy earth.txt and edit it...

  
     $ cp earth.txt venus.txt
     $ edit venus.txt
     $ ls -t
     venus.txt   earth.txt
  		

  • “ -t ” tells ls to list by modification time, instead of alphabetically

Part I: Exercises

Part I: Exercises (continued)

Redirecting Input and Output

• A running program is called a process
• Every process automatically has three connections to the outside world:
  • Standard input (stdin): connected to the keyboard
  • Standard output (stdout): connected to the screen
  • Standard error (stderr): also connected to the screen and used for error messages

 



Redirecting stdin and stdout

• You can tell the shell to connect standard input and standard output to files instead:
  

  command < input_file

  ...reads from input_file instead of from the keyboard. You don't need to use this very often, because most Unix commands let you specify the input file (or files) as command-line arguments.
  Similarily,
  

  command > output_file

  ...writes to output_file instead of to the screen. Only "normal" output goes to the file, not error messages.
  Finally,
  

  command < input_file > output_file

  ...does both.
  Note that redirection takes effect command-by-command, rather than permanently.

Redirection Examples

• Save number of words in all text files to words.len:

  • 
       $ cd bio
       $ wc *.txt > words.len
    				

  • Nothing appears on the screen because all output is being sent to the file words.len
  • Check contents using cat

    
       $ cat words.len
          7   66  468 albus.txt
          5   46  311 ginny.txt
          5   49  342 harry.txt
          5   49  331 hermione.txt
          6   54  364 ron.txt
         28  264 1816 total
    				

Potential Pitfalls

• Try typing cat > junk.txt
  • No input file specified, so cat reads from the keyboard with output sent to a file
  • The world's dumbest text editor
• When you're done, use rm junk.txt to get rid of the file
  • NEVER type rm * unless you're really, really, REALLY sure that's what you want to do.
• Don't redirect out to the same file, e.g. "sort words >words"
  • The shell sets up redirection before running the command
  • Redirecting output to an existing file truncates it, making it empty
  • sort then reads the empty file
  • Original contents of words is permantly lost

Pipes

• Suppose you want to use the output of one program as the input to another. E.g., use wc -w *.txt to count the words in some files, then sort -n to sort numerically.
• One obvious solution is to send output of first command to a temporary file, then read from that file:

  
     $ wc -w *.txt > words.tmp
     $ sort -n words.tmp
       46 ginny.txt
       49 harry.txt
       49 hermione.txt
       54 ron.txt
       66 albus.txt
      264 total
     $ rm words.tmp
  			

Pipes (continued)

• But the right answer is to use a pipe
  • Written as “ | ”
  • Tells the OS to connect the standard output of the first program to the standard input of the second:

    
       $ wc -w *.txt | sort -n
         46 ginny.txt
         49 harry.txt
         49 hermione.txt
         54 ron.txt
         66 albus.txt
        264 total
    				
    				
    				

  • More convenient and less error prone than temporary files

Pipeline and Filters

• Can chain any number of commands together
  • and combine with input and output redirection

    
       $ grep 'Title' spells.txt | sort | uniq -c | sort -n -r | head -10 > popular_spells.txt
    				

• Any program that reads from standard input and writes to standard output can use redirection and pipes
  • Such programs are often called filters
  • If your programs work like filters, you (and other people) can combine them with other standard tools
  • A combinatorial explosion of goodness

Environment Variables

• The OS stores some environment variables for every process
  • Like variables in a program, each has a name and a value
  • By convention, names are all upper case
  • Values are always strings
• Type printenv at the command prompt to get a listing:

  
     $ printenv
     MANPATH=:/usr/local/sge/man
     XDG_SESSION_ID=2215
     HOSTNAME=watson.cgl.ucsf.edu
     TERM=xterm-256color
     SHELL=/usr/local/bin/bash
     HISTSIZE=1000
     SSH_CLIENT=169.230.11.112 63047 22
     OLDPWD=/home/socr/a/tef
     SSH_TTY=/dev/pts/2
     SVN_EDITOR=vi
     USER=tef
     SSH_AUTH_SOCK=/tmp/ssh-OLrXGv99V3/agent.33503
     MAIL=/var/spool/mail/tef
     PATH=/home/socr/a/tef/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/sge/bin/linux-x64:.
     LC_COLLATE=C
     PWD=/usr/local/www/rbvi/preview/htdocs/Outreach/bmi219/slides
     LANG=en_US.UTF-8
     SGE_ROOT=/usr/local/sge
     HISTCONTROL=ignoredups
     SHLVL=1
     HOME=/home/socr/a/tef
     LOGNAME=tef
     PRINTER=lys
     SSH_CONNECTION=169.230.11.112 63047 169.230.27.29 22
     LESSOPEN=||/usr/bin/lesspipe.sh %s
     XDG_RUNTIME_DIR=/run/user/20021
     DISPLAY=localhost:10.0
  			

Environment Variables (continued)

• Get a variable's value by putting “$” in front of its name:
  • So ls $HOME is the same as ls /home/rweasley (if you're Ron Weasley)
  • Use the echo command to print out a variable's value

    
       $ echo $HOME
       /home/socr/a/tef
    				

    (Don't confuse the shell prompt character (also “$”) with the “$” used to identify a variable.)

  • Question: why must you type echo $HOME, and not just “$HOME”?
  • If a variable hasn't been defined, its value is the empty string (rather than an error)

Important Environment Variables

Setting Environment Variables

Environment Configuration

• To set a variable's value automatically when you log in, edit ~/.bashrc
  “~” is a shortcut meaning "your home directory"

  
     # Add personal tools directory to PATH.
     PATH=$HOME/bin:$PATH
  
     # Personal settings
     export EDITOR=/local/bin/emacs
     export PRINTER=gryffindor-laserwriter
  
     # Change default behavior of commands.
     alias ls="ls -F"
  			

  Note: .bashrc files can become very complex...
• Many applications look for personal configuration files in the user's home directory
  • By convention, their names begin with "." so that a normal ls won't show them
  • Once upon a time, the "rc" at the end meant "run commands"

How the Shell Finds Programs

• The PATH environment variables defines the shell's search path
• When you run a command like broom, the shell:
  • Splits $PATH into components to get a list of directories
  • Unix uses “ : ” as a separator
  • Looks for the program in each directory in left-to-right order
  • Runs the first one that it finds
• Example:
  • PATH is /home/rweasley/bin:/usr/local/bin:/usr/bin:/bin:/python3
  • Both /usr/local/bin/broom and /home/rweasley/bin/broom exist
  • /home/rweasley/bin/broom will be run when you type broom at the command prompt
  • Can run the other one by specifying the path, instead of just the command name

Common Search Path Entries

• /bin, /usr/bin: core tools like ls
  • Note: the word "bin" comes from "binary", which is geekspeak for "a compiled program"
• /usr/local/bin: optional (but common) tools, like the gcc C compiler
• $HOME/bin: tools you have built for yourself
  • Remember, $HOME is your home directory
• It is also common to include “ . ” (the current working directory) in your path
  • Allows you to run a program in the current directory using whatever, instead of ./whatever

File Ownership and Permissions

• On Unix, every user belongs to one or more groups
  • The groups command will show you which ones you are in
• Every file is owned by a particular user and a particular group
  • Can assign read (r), write (w), and execute (x) permissions independently to user, group, and others
  • Read: can look at contents, but not modify them
  • Write: can modify contents
  • Execute: can run the file (e.g., it's a program)
• ls -l shows this information
  • Along with the file's size and a few other things
• Permissions displayed as three “rwx” triples
  • "Missing" permissions shown by “ - ”
  • So “rw-rw-r--” means:
    1. User and group can read and write
    2. Everyone else can read, but not write
    3. No one can execute
    

Directory Permissions

• Execute permission means something different for directories
  • Allows you to "go into" a directory, but does not mean you can read its contents
• If directory tools has permission “rwx--x--x”, then:
  • If someone other than the owner does ls tools permission is denied
  • But anyone who wants to can run the command tools/pfold

Changing Permissions

• Change permissions using chmod
  • chmod u+x broom allows broom's owner to run it
  • chmod o-r notes.txt takes away the world's read permission for notes.txt
• Any set of shell commands can be turned into a program!
  • If it's worth doing again, it's worth automating

Changing Permissions (continued)

• Create a file called nojunk that contains the following two lines:

  
     #!/usr/bin/bash
     rm -f *.junk
  			

  • Use man rm to find out what the "-f" flag does
  • “#!/usr/bin/bash” means "interpret the contents of this file using the Bash shell"
    • Any program name can follow the "#!"
    • We'll see some possibilities later
• Change permissions of nojunk to “rwxr-xr-x”
• Run it with ./nojunk
  • Or if “$HOME/bin” is in your search path, move it there

Avoid this pitfall!

• Don't call your temporary test programs “test”
  • There's already a /usr/bin/test program
  • Your PATH may cause that program to be run instead of yours
  • Confusion will result, so use something else, e.g. “tryit”

Part II: Exercises

Part II: Exercises (continued)

Part II: Exercises (continued)

Part II: Exercises (continued)

Part II: Exercises (continued)