Solution 1:

The software you are looking for is called "Digital Rights Management" software, often abbreviated to "DRM".

Searching for this term will find a wide range of third party tools and solutions for protecting your videos. The offerings will range from massive companies like Adobe with their Access solution to numerous smaller companies.

Protecting Streamed/Online Content

Controlling access to a video file to be played on viewer's computer is a difficult problem. Your best option is to provide online only access to your video content. By serving the video content yourself, you can more easily control access.

Have you considered using YouTube and their paid channel service?

Protecting Local Content

There is currently no built-in DRM mechanism included with Mac OS X. Requiring a user to install a third party video codec is likely to limit your audience.

However, Flash remains a popular plug-in and can be used to provide digital rights management. Flash works both in the web browser and for stand alone applications.

Searching for flash protect local video returns software claiming to support multiple platforms, including Apple:

Protect Software

We protect your software, video and content effectively against illegal copies and sharing. On CD/DVD/BD, USB-Stick, in local networks or via the Internet - We have the best solution for copy protection.

Brightcove

Robust content protection across every screen is an essential part of any video monetization strategy. Brightcove helps you protect your content and your business model with the most advanced encryption and DRM technologies, as well as geographic, time-based and user-based restrictions, to prevent unauthorized access, downloads, and copying of your valuable content.

Multimedia OwnerGuard

Multimedia OwnerGuard is designed to support a wide range of Multimedia Files including (swf, flv, mp4, f4v, 3gp, avi, rm, mkv, 3gg, 3g2, mov, m4v, m4a, f4p, f4a, f4b, mp3, mpg, asf, wma, wmv, aac, ogg, aif, flac). Now you can use full features of OwnerGuard DRM Technology to protect and distribute your Flash SWF, FLV and many other media files while OwnerGuard protects your digital ownership rights.

SWF Protection

Encrypt and protect Flash SWF files from decompilers. Protect not only Actionscripts, but also all images, sounds, buttons and sprites resources in your SWF files.

Try a Flash Based Solution

Try one of the Flash based video protection programs returned by the search above; I have no experience with any, so will not directly recommend one. The list above is extracted from the first page of Google results.

Solution 2:

For what is worth, this is more of a opinion (experience) then a clear answer to your specific question.

It has become a "sport" to some to crack anything password protected.

So no matter what you use it will be cracked if the content is of high interest.

The best protection I found is to use your Ownership information embedded in the video it self.

Modify the video and add logo or subtitle or Icon or what you like that is displayed constantly or frequently, your choice.

It is almost impossible to remove it, and it wont badder regular user since they know it is from you anyway.

This method also has a positive side effect. Anyone can copy and redistribute to they liking, while doing so promoting your video.

There is no point for me listing the crack methods just to prove the point but easy to Google them.