How to detect that emacs is in terminal-mode?

The window-system variable tells Lisp programs what window system Emacs is running under. The possible values are

x

Emacs is displaying the frame using X.

w32

Emacs is displaying the frame using native MS-Windows GUI.

ns

Emacs is displaying the frame using the Nextstep interface (used on GNUstep and Mac OS X).

pc

Emacs is displaying the frame using MS-DOS direct screen writes.

nil

Emacs is displaying the frame on a character-based terminal.

From the doc.

Edit: it seems that window-system is deprecated in favor of display-graphic-p (source: C-h f window-system RET on emacs 23.3.1).

(display-graphic-p &optional DISPLAY)

Return non-nil if DISPLAY is a graphic display.
Graphical displays are those which are capable of displaying several
frames and several different fonts at once.  This is true for displays
that use a window system such as X, and false for text-only terminals.
DISPLAY can be a display name, a frame, or nil (meaning the selected
frame's display).

So what you want to do is :

(if (display-graphic-p)
    (progn
    ;; if graphic
      (your)
      (code))
    ;; else (optional)
    (your)
    (code))

And if you don't have an else clause, you can just:

;; more readable :)
(when (display-graphic-p)
    (your)
    (code))

The answers mentioning window-system and display-graphic-p aren't wrong, but they don't tell the complete picture.

In reality, a single Emacs instance can have multiple frames, some of which might be on a terminal, and others of which might be on a window system. That is to say, you can get different values of window-system even within a single Emacs instance.

For example, you can start a window-system Emacs and then connect to it via emacsclient -t in a terminal; the resulting terminal frame will see a value of nil for window-system. Similarly, you can start emacs in daemon mode, then later tell it to create a graphical frame.

As a result of this, avoid putting code in your .emacs that depends on window-system. Instead, put code like your set-frame-size example in a hook function which runs after a frame is created:

(add-hook 'after-make-frame-functions
  (lambda ()
    (if window-system
      (set-frame-size (selected-frame) 166 100)))))

Note that the 'after-make-frame-functions hook isn't run for the initial frame, so it's often necessary to also add frame-related hook functions like that above to 'after-init-hook.