How to make Time Machine backups over network without OS X server?

The officially sanctioned answer: buy an Apple Time Capsule device.

Unsupported but well-known alternative: enable a hidden option that allows you to use any network drive. Open the Terminal application and run this one command:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

It will allow you to select any network drive from System Preferences: Time Machine.

The network drive should be a HFS+ file system over AFP (standard file sharing from another Mac). I would strongly discourage you from using any other setup if you are not prepared to manually restore one file at a time from the backup.

Some full and detailed instructions are available here, but it's pretty much just a matter of sharing the folder you want with AFP only, mounting it on the client for the first time, and selecting it in Time Machine.

If that doesn't work for you, or you need more configuration options, install OS X Server. It's an officially supported feature, and Server only costs $20 from the Mac App Store. Unlike older versions, Server for 10.8 is just an MAS install and doesn't require reinstalling the OS.

Side note: do not ever use the TMShowUnsupportedNetworkVolumes option if you value the integrity of your backups. Unsupported volumes are unsupported for a reason: Time Machine requires certain specific file-sharing features to ensure that the backups aren't corrupted if, for example, the network connection drops in the middle of a backup. Enabling this option disables checking for those features and I have seen it result in corrupted backups quite a few times.

I was successful at setting up an Ubuntu 12.04 computer running netatalk (AFP) as a file server and backing up two Macs on 10.8 and restoring one after a crash.