Is there any way to invert file selections on Mac OS X like we do it on Windows?

The key you're looking for is Cmd.

Select the files – any view should work – then move with the mouse cursor to the empty space next to the file list. Hold Cmd and drag to invert the selection:

If you don't want to use the mouse, an alternative way would be to use AppleScript instead. Open up Automator.app, create a new Service. From the left pane, drag Run AppleScript to the right, and paste the following:*

on run {input, parameters}

  tell application "Finder"
    set inverted to {}
    set fitems to items of window 1 as alias list
    set selectedItems to the selection as alias list
    repeat with i in fitems
      if i is not in selectedItems then
        set end of inverted to i
      end if
    end repeat
    select inverted
  end tell

  return input
end run

Set this service to receive No Input from Finder.app. Like this:

Save this service as Invert Selection. Then, head to System Preferences » Keyboard » Keyboard Shortcuts, and add your shortcut to the service, for example ShiftCmdI:

Now, select any files in Finder and press the shortcut to invert the selection.

_{* I found this on the Apple Mailing Lists, no idea who wrote it though.}