How can I change to the previous directory instead of going up?

The command

cd -

will perform the swap you need on most of the mainstream shells, the older longer variant is

cd "$OLDPWD"

which will use the environment variable that contains the previous working directory.

The POSIX man page for cd mentions:

DESCRIPTION

If, during the execution of the above steps, the PWD environment variable is changed, the OLDPWD environment variable shall also be changed to the value of the old working directory (that is the current working directory immediately prior to the call to cd).

OPERANDS

  -  When a hyphen is used as the operand, this shall be equivalent to the command:

cd "$OLDPWD" && pwd 

which changes to the previous working directory and then writes its name.

In addition to bryan's answer, it's worth mentioning there's also pushd and popd, which build up directories like a stack. This is also available on Windows NT; however, it is not available in all shells.

For example, we can go to three different directories, and you'll always see your stack when you call pushd:

charon:~ werner$ pushd Documents/
~/Documents ~

charon:Documents werner$ pushd ../Movies/
~/Movies ~/Documents ~

charon:Movies werner$ pushd ../Downloads/
~/Downloads ~/Movies ~/Documents ~

And when you call popd three times in a row, you get to those directories in the stack in reverse order. At the same time, the stack will be emptied again.

charon:Downloads werner$ popd
~/Movies ~/Documents ~

charon:Movies werner$ popd
~/Documents ~

charon:Documents werner$ popd
~

charon:~ werner$ popd
-bash: popd: directory stack empty

If you are using Zsh; it has an AUTO_PUSHD option, which will automatically push cd's onto the stack.