Solution 1:

I think you are looking for the LSUIElement in the Info.plist

LSUIElement (String). If this key is set to “1”, Launch Services runs the application as an agent application. Agent applications do not appear in the Dock or in the Force Quit window. Although they typically run as background applications, they can come to the foreground to present a user interface if desired.

See a short discussion here about turning it on/off

Solution 2:

You can use what is called Activation Policy:

Objective-C

// The application is an ordinary app that appears in the Dock and may
// have a user interface.
[NSApp setActivationPolicy: NSApplicationActivationPolicyRegular];

// The application does not appear in the Dock and does not have a menu
// bar, but it may be activated programmatically or by clicking on one
// of its windows.
[NSApp setActivationPolicy: NSApplicationActivationPolicyAccessory];

// The application does not appear in the Dock and may not create
// windows or be activated.
[NSApp setActivationPolicy: NSApplicationActivationPolicyProhibited];

Swift 4

// The application is an ordinary app that appears in the Dock and may
// have a user interface.
NSApp.setActivationPolicy(.regular)

// The application does not appear in the Dock and does not have a menu
// bar, but it may be activated programmatically or by clicking on one
// of its windows.
NSApp.setActivationPolicy(.accessory)

// The application does not appear in the Dock and may not create
// windows or be activated.
NSApp.setActivationPolicy(.prohibited)

This should hide the dock icon.

See also

Solution 3:

To do it while abiding to the Apple guidelines of not modifying application bundles and to guarantee that Mac App Store apps/(Lion apps ?) will not have their signature broken by info.plist modification you can set LSUIElement to 1 by default then when the application launches do :

ProcessSerialNumber psn = { 0, kCurrentProcess };
TransformProcessType(&psn, kProcessTransformToForegroundApplication);

to show it's dock icon, or bypass this if the user chose not to want the icon.

There is but one side effect, the application's menu is not shown until it losses and regains focus.

Source: Making a Checkbox Toggle The Dock Icon On and Off

Personally i prefer not setting any Info.plist option and use TransformProcessType(&psn, kProcessTransformToForegroundApplication) or TransformProcessType(&psn, kProcessTransformToUIElementApplication) based on what is the user setting.

Solution 4:

In Xcode it is shown as "Application is agent (UIElement)" and it is Boolean.

In your Info.plist control-click to an empty space and select "Add Row" from the menu Type "Application is agent (UIElement)" Set it YES.

TO make it optional I added the following line to my code (thanks Valexa!)

 // hide/display dock icon
if (![[NSUserDefaults  standardUserDefaults] boolForKey:@"hideDockIcon"]) {
    //hide icon on Dock
    ProcessSerialNumber psn = { 0, kCurrentProcess };
    TransformProcessType(&psn, kProcessTransformToForegroundApplication);
}