Can I stream any video played with VLC player to Apple TV?

It's not using VLC (although it uses some of its components and should be able to open anything VLC can open), but it looks like there's a way to do this. It requires on-the-fly transcoding for formats the AppleTV doesn't support (essentially anything not .mp4 or .m4v), which means it may take a fair bit of CPU power on your Mac, especially for HD stuff. It also means that aside from files the AppleTV natively supports, you're not getting bit-perfect renditions of the files, it's a lossy translation, but if you have a decently fast Mac, it should be pretty good.

The main tool you'll want is AirFlick. It's a pretty basic program that sends a URL to the AppleTV that tells it to open a stream from your computer. It also handles transcoding the non-native files. It's not very well documented; it looks like the latest versions use a built-in copy of ffmpeg for transcoding, but some of the resources around the web suggest that it may need VLC installed as well.

Usage is pretty straight forward.

1. Open up AirFlick
2. Select your AppleTV from the drop-down in the upper-left
3. Drag the file you want to play onto the droplet, and hit play.

It should take around 30 seconds to buffer the transcoding, then start playing. It's definitely beta software, so YMMV.

From what I can tell, the transcoding support is pretty simplistic. If you want to configure the transcoding process, you need to use custom VLC or ffmpeg command-line options and pass the output to AirFlick.

This guide from TUAW should give you a starting point for how to do that, but you'll want to consult other VLC or ffmpeg resources for specific configurations - it's beyond my expertise. The basic idea seems to be:

1. Use VLC or ffmpeg to transcode the source (in the example, that's dvdread:///dev/rdisk5@1:3-, replace that with the path to your file of choice) on-the-fly to an AppleTV compatible format
2. Pipe that output to mediastreamsegmenter which breaks the video into a streaming playlist the AppleTV can read
3. Host the streaming playlist on a webserver on your Mac
4. Use AirFlick to pass the URL of the playlist to your AppleTV