How to make a smooth, rounded, volume-like OS X window with NSVisualEffectView?

Update for OS X El Capitan

The hack I described in my original answer below is not needed on OS X El Capitan anymore. The NSVisualEffectView’s maskImage should work correctly there, if the NSWindow’s contentView is set to be the NSVisualEffectView (it’s not enough if it is a subview of the contentView).

Here’s a sample project: https://github.com/marcomasser/OverlayTest

Original Answer – Only Relevant for OS X Yosemite

I found a way to do this by overriding a private NSWindow method: - (NSImage *)_cornerMask. Simply return an image created by drawing an NSBezierPath with a rounded rect in it to get a look similar to OS X’s volume window.

In my testing I found that you need to use a mask image for the NSVisualEffectView and the NSWindow. In your code, you’re using the view’s layer’s cornerRadius property to get the rounded corners, but you can achieve the same by using a mask image. In my code, I generate an NSImage that is used by both the NSVisualEffectView and the NSWindow:

func maskImage(#cornerRadius: CGFloat) -> NSImage {
    let edgeLength = 2.0 * cornerRadius + 1.0
    let maskImage = NSImage(size: NSSize(width: edgeLength, height: edgeLength), flipped: false) { rect in
        let bezierPath = NSBezierPath(roundedRect: rect, xRadius: cornerRadius, yRadius: cornerRadius)
        NSColor.blackColor().set()
        bezierPath.fill()
        return true
    }
    maskImage.capInsets = NSEdgeInsets(top: cornerRadius, left: cornerRadius, bottom: cornerRadius, right: cornerRadius)
    maskImage.resizingMode = .Stretch
    return maskImage
}

I then created an NSWindow subclass that has a setter for the mask image:

class MaskedWindow : NSWindow {

    /// Just in case Apple decides to make `_cornerMask` public and remove the underscore prefix,
    /// we name the property `cornerMask`.
    @objc dynamic var cornerMask: NSImage?

    /// This private method is called by AppKit and should return a mask image that is used to 
    /// specify which parts of the window are transparent. This works much better than letting 
    /// the window figure it out by itself using the content view's shape because the latter
    /// method makes rounded corners appear jagged while using `_cornerMask` respects any
    /// anti-aliasing in the mask image.
    @objc dynamic func _cornerMask() -> NSImage? {
        return cornerMask
    }

}

Then, in my NSWindowController subclass I set up the mask image for the view and the window:

class OverlayWindowController : NSWindowController {

    @IBOutlet weak var visualEffectView: NSVisualEffectView!

    override func windowDidLoad() {
        super.windowDidLoad()

        let maskImage = maskImage(cornerRadius: 18.0)
        visualEffectView.maskImage = maskImage
        if let window = window as? MaskedWindow {
            window.cornerMask = maskImage
        }
    }
}

I don’t know what Apple will do if you submit an app with that code to the App Store. You’re not actually calling any private API, you’re just overriding a method that happens to have the same name as a private method in AppKit. How should you know that there’s a naming conflict? 😉

Besides, this fails gracefully without you having to do anything. If Apple changes the way this works internally and the method just won’t get called, your window does not get the nice rounded corners, but everything still works and looks almost the same.

If you’re curious about how I found out about this method:

I knew that the OS X volume indication did what I want to do and I hoped that changing the volume like a madman resulted in noticeable CPU usage by the process that puts that volume indication on screen. I therefore opened Activity Monitor, sorted by CPU usage, activated the filter to only show “My Processes” and hammered my volume up/down keys.

It became clear that coreaudiod and something called BezelUIServer in /System/Library/LoginPlugins/BezelServices.loginPlugin/Contents/Resources/BezelUI/BezelUIServer did something. From looking at the bundle resources for the latter, it was evident that it is responsible for drawing the volume indication. (Note: that process only runs for a short time after it displays something.)

I then used Xcode to attach to that process as soon as it launched (Debug > Attach to Process > By Process Identifier (PID) or Name…, then enter “BezelUIServer”) and changed the volume again. After the debugger was attached, the view debugger let me take a look at the view hierarchy and see that the window was an instance of a NSWindow subclass called BSUIRoundWindow.

Using class-dump on the binary showed that this class is a direct descendant of NSWindow and only implements three methods, whereas one is - (id)_cornerMask, which sounded promising.

Back in Xcode, I used the Object Inspector (right hand side, third tab) to get the address for the window object. Using that pointer I checked what this _cornerMask actually returns by printing its description in lldb:

(lldb) po [0x108500110 _cornerMask]
<NSImage 0x608000070300 Size={37, 37} Reps=(
    "NSCustomImageRep 0x608000082d50 Size={37, 37} ColorSpace=NSCalibratedRGBColorSpace BPS=0 Pixels=0x0 Alpha=NO"
)>

This shows that the return value actually is an NSImage, which is the information I needed to implement _cornerMask.

If you want to take a look at that image, you can write it to a file:

(lldb) e (BOOL)[[[0x108500110 _cornerMask] TIFFRepresentation] writeToFile:(id)[@"~/Desktop/maskImage.tiff" stringByExpandingTildeInPath] atomically:YES]

To dig a bit deeper, you can use Hopper Disassembler to disassemble BezelUIServer and AppKit and generate pseudo code to see how the _cornerMask is implemented and used to get a clearer picture of how the internals work. Unfortunately, everything in regard to this mechanism is private API.

I remember doing this sort of thing long before CALayer was around. You use NSBezierPath to make the path.

I don't believe you actually need to subclass NSWindow. The important bit about the window is to initialize the window with NSBorderlessWindowMask and apply the following settings:

[window setAlphaValue:0.5]; // whatever your desired opacity is
[window setOpaque:NO];
[window setHasShadow:NO];

Then you set the contentView of your window to a custom NSView subclass with the drawRect: method overridden similar to this:

// "erase" the window background
[[NSColor clearColor] set];
NSRectFill(self.frame);

// make a rounded rect and fill it with whatever color you like
NSBezierPath* clipPath = [NSBezierPath bezierPathWithRoundedRect:self.frame xRadius:14.0 yRadius:14.0];
[[NSColor blackColor] set]; // your bg color
[clipPath fill];

result (ignore the slider):

Edit: If this method is for whatever reason undesirable, can you not simply assign a CAShapeLayer as your contentView's layer then either convert the above NSBezierPath to CGPath or just construct as a CGPath and assign the path to the layers path?

The "smooth effect" you are referring to is called "Antialiasing". I did a bit of googling and I think you might be the first person who has tried to round the corners of an NSVisualEffectView. You told the CALayer to have a border radius, which will round the corners, but you didn't set any other options. I would try this:

layer.shouldRasterize = YES;
layer.edgeAntialiasingMask = kCALayerLeftEdge | kCALayerRightEdge | kCALayerBottomEdge | kCALayerTopEdge;

Anti-alias diagonal edges of CALayer

https://developer.apple.com/library/mac/documentation/GraphicsImaging/Reference/CALayer_class/index.html#//apple_ref/occ/instp/CALayer/edgeAntialiasingMask

Despite the limitations of NSVisualEffectView not antialiasing edges, here's a kludgey workaround for now that should work for this application of a floating title-less unresizeable window with no shadow - have a child window underneath that draws out just the edges.

I was able to get mine to look like this:

by doing the following:

In a controller holding everything:

- (void) create {

NSRect windowRect = NSMakeRect(100.0, 100.0, 200.0, 200.0);
NSRect behindWindowRect = NSMakeRect(99.0, 99.0, 202.0, 202.0);
NSRect behindViewRect = NSMakeRect(0.0, 0.0, 202.0, 202.0);

NSRect viewRect = NSMakeRect(0.0, 0.0, 200.0, 200.0);

window = [FloatingWindow createWindow:windowRect];

behindAntialiasWindow = [FloatingWindow createWindow:behindWindowRect];
roundedHollowView = [[RoundedHollowView alloc] initWithFrame:behindViewRect];

[behindAntialiasWindow setContentView:roundedHollowView];
[window addChildWindow:behindAntialiasWindow ordered:NSWindowBelow];

backingView = [[NSView alloc] initWithFrame:viewRect];

contentView = [[NSVisualEffectView alloc] initWithFrame:viewRect];
[contentView setWantsLayer:NO];
[contentView setState:NSVisualEffectStateActive];
[contentView setAppearance:
 [NSAppearance appearanceNamed:NSAppearanceNameVibrantLight]];
[contentView setMaskImage:[AppDelegate maskImageWithBounds:contentView.bounds]];

[backingView addSubview:contentView];

[window setContentView:backingView];
[window setLevel:NSFloatingWindowLevel];
[window orderFront:self];


}

+ (NSImage *) maskImageWithBounds: (NSRect) bounds
{
return [NSImage imageWithSize:bounds.size flipped:YES drawingHandler:^BOOL(NSRect dstRect) {

    NSBezierPath *path = [NSBezierPath bezierPathWithRoundedRect:bounds xRadius:20.0 yRadius:20.0];

    [path setLineJoinStyle:NSRoundLineJoinStyle];
    [path fill];

    return YES;
}];
}

RoundedHollowView's drawrect looks like this:

- (void)drawRect:(NSRect)dirtyRect {
[super drawRect:dirtyRect];
// "erase" the window background
[[NSColor clearColor] set];
NSRectFill(self.frame);

[[NSColor colorWithDeviceWhite:1.0 alpha:0.7] set];

NSBezierPath *path = [NSBezierPath bezierPathWithRoundedRect:self.bounds xRadius:20.0 yRadius:20.0];
path.lineWidth = 2.0;
[path stroke];

}

Again, this is a hack and you may need to play with the lineWidth / alpha values depending on the base color you use - in my example if you look really closely or under lighter backgrounds you'll make out the border a bit, but for my own use it feels less jarring than not having any antialiasing.

Keep in mind that the blending mode won't be the same as the native osx yosemite pop-ups like the volume control - those appear to use a different undocumented behindwindow appearance that shows more of a color burn effect.