How to detect if OS X is in dark mode?

Don't think there's a cocoa way of detecting it yet, however you can use defaults read to check whether or not OSX is in dark mode.

defaults read -g AppleInterfaceStyle

Either returns Dark (dark mode) or returns domain pair does not exist.

EDIT:

As Ken Thomases said you can access .GlobalPreferences via NSUserDefaults, so

NSString *osxMode = [[NSUserDefaults standardUserDefaults] stringForKey:@"AppleInterfaceStyle"];

If osxMode is nil then it isn't in dark mode, but if osxMode is @"Dark" then it is in dark mode.

Swift 2 -> String ("Dark", "Light")

let appearance = NSUserDefaults.standardUserDefaults().stringForKey("AppleInterfaceStyle") ?? "Light"

Swift 3 -> Enum (Dark, Light)

enum InterfaceStyle : String {
   case Dark, Light

   init() {
      let type = UserDefaults.standard.string(forKey: "AppleInterfaceStyle") ?? "Light"
      self = InterfaceStyle(rawValue: type)!
    }
}

let currentStyle = InterfaceStyle()

You can detect this using NSAppearanceCustomization method effectiveAppearance, by checking for darkAqua.

Swift 4 example:

extension NSView {
    var isDarkMode: Bool {
        if #available(OSX 10.14, *) {
            if effectiveAppearance.name == .darkAqua {
                return true
            }
        }
        return false
    }
}