If OS X doesn't use the X Server, then what does it use?

It uses Quartz. But that's a bit of a marketing term in so much as there is not one single thing called Quartz, but a group of things that work together to make the system called Quartz. Specifially, Quartz encompasses the set of systems needed to do rendering (Quartz 2D), font management, graphics card interaction (Quartz Compositor), OpenGL support, 3D acceleration and what not.

The Quartz Compositor is the piece of the Quartz bundle that provides the server and window compositing and management features normally performed by the XServer and XClient pair.

An addendum to this question might be: why didn't Apple just use X11 and the X Server?

And there's a great answer to that addendum question on Slashdot from one of the primary Quartz developers:

What Apple is providing is an Apple-original window system that is graphics model agnostic, as well as a vector drawing system that maps very well to PDF, which is a sort of PostScript without the non-graphical operators. This is packaged under the name 'Quartz' for easy reference by Marketing types.

The window system is designed to support both buffered (like an offscreen PixMap) and unbuffered windows, and is graphics model agnostic, working equally well with QuickDraw, OpenGL, the Quartz drawing engine, X11, and third party solutions, and managing window geometry for the Classic, Carbon, and Cocoa environments. The server portion is a hybridization of screen arbiter and compositor models (and if that's all Geek to you, don't worry about it).

The Quartz drawing engine supports drawing primitives similar to the graphics primitives that might be found in the DPSClient single-operator primitives library for X and NeXTSTEP. There are no math and flow control primitives, as these can be done more efficiently in the native compiled code. There are no DPS or PS wrappers, as this optimization for server-side graphics is not needed in the Quartz client-side graphics model.

The operations provide imaging and path construction and filling operations as well as some interesting other bits that map well into the direction that 2D drawing is headed. (See Longhorn, or the X raster projects.) The drawing engine can output to rasters (like a window!), as well as PS and PDF streams to feed printers. The Mac OS X printing system takes advantage of the capabilities of Quartz to support all sorts of printers, and make the life of printer driver developers much, much easier.

Things we'd need to add/extend in X Window software (protocol+server+manager+fonts+...):

1. Extend font server and services to vend outlines and antialiased masks, support more font types, handle font subsetting.
2. Extend drawing primitives to include PS-like path operations.
3. Add dithering and phase controls.
4. Add ColorSync support for drawing and imaging operations, display calibration
5. Add broad alpha channel support and Porter-Duff compositing, both for drawing in a window and for interactions between windows.
6. Add support for general affine transforms of windows
7. Add support for mesh-warps of windows
8. Make sure that OpenGL and special video playback hardware support is integrated, and behaves well with all above changes.
9. We find that we typically stream 200 Mb/sec of commands and textures for interactive OpenGL use, so transport efficiency could be an issue.

So, yes, it looks like we can use X for Quartz. All we need do is define extensions for and upgrade the font server, add dithering with phase controls to the X marking engine, add a transparency model to X imaging with Porter-Duff compositing support, make sure GLX gets in, upgrade the window buffering to include transparency, mesh warps, and really good resampling, and maybe augment the transport layer a bit.

Ummm... There doesn't appear to be much code left from the original X server in the drawing path or windowing machinery, and it doesn't appear that apps relying on these extensions can work with any other X server. Just what did we gain from this?

Oh, yeah. My mom can run an xterm session on her desktop now without downloading the Apple X11 package, a shareware X server or buying a software package.

Been there, evaluated that.