Why does NSColor.controlTextColor change according to background color?

If you want to pass in any color and then determine which text color would be more ideal - black or white - you first need to determine the luminance of that color (in sRGB). We can do that by converting to grayscale, and then checking the contrast with black vs white.

Check out this neat extension that does so:

extension NSColor {

    /// Determine the sRGB luminance value by converting to grayscale. Returns a floating point value between 0 (black) and 1 (white).
    func luminance() -> CGFloat {
        var colors: [CGFloat] = [redComponent, greenComponent, blueComponent].map({ value in
            if value <= 0.03928 {
                return value / 12.92
            } else {
                return pow((value + 0.055) / 1.055, 2.4)
            }
        })
        let red = colors[0] * 0.2126
        let green = colors[1] * 0.7152
        let blue = colors[2] * 0.0722
        return red + green + blue
    }

    func contrast(with color: NSColor) -> CGFloat {
        return (self.luminance() + 0.05) / (color.luminance() + 0.05)
    }

}

Now we can determine whether we should use black or white as our text by checking the contrast between our background color with black and comparing it to the contrast with white.

// Background color for whatever UI component you want.
let backgroundColor = NSColor(red: 0.5, green: 0.8, blue: 0.2, alpha: 1.0)
// Contrast of that color w/ black.
let blackContrast = backgroundColor.contrast(with: NSColor.black.usingColorSpace(NSColorSpace.sRGB)!)
// Contrast of that color with white.
let whiteContrast = backgroundColor.contrast(with: NSColor.white.usingColorSpace(NSColorSpace.sRGB)!)
// Ideal color of the text, based on which has the greater contrast.
let textColor: NSColor = blackContrast > whiteContrast ? .black : .white

In this case above, the backgroundColor produces a contrast of 10.595052467245562 with black and 0.5045263079640744 with white. So clearly, we should use black as our font color!

The value for black can be corroborated here.

EDIT: The logic for the .controlTextColor is going to be beneath the surface of the API that Apple provides and beyond me. It has to do with the user's preferences, etc. and may operate on views during runtime (i.e. by setting .controlTextColor, you might be flagging a view to check for which textColor is more legible during runtime and applying it).

TL;DR: I don't think you have the ability to achieve the same effect as .controlTextColor with an NSColor subclass.

Here's an example of a subclassed element that uses its backgroundColor to determine the textColor, however, to achieve that same effect. Depending on what backgroundColor you apply to the class, the textColor will be determined by it.

class ContrastTextField: NSTextField {

    override var textColor: NSColor? {
        set {}
        get {
            if let background = self.layer?.backgroundColor {
                let color = NSColor(cgColor: background)!.usingColorSpace(NSColorSpace.sRGB)!
                let blackContrast = color.contrast(with: NSColor.black.usingColorSpace(NSColorSpace.sRGB)!)
                let whiteContrast = color.contrast(with: NSColor.white.usingColorSpace(NSColorSpace.sRGB)!)
                return blackContrast > whiteContrast ? .black : .white
            }
            return NSColor.black
        }
    }

}

Then you can implement with:

let textField = ContrastTextField()
textField.wantsLayer = true
textField.layer?.backgroundColor = NSColor.red.cgColor
textField.stringValue = "test"

Will set your textColor depending on the layer's background.