Scale-out distributed storage with snapshots
Solution 1:
Have you looked into CephFS (rather than running a filesystem on a Ceph block device)? I haven't used it in production, but it claims to have the features you want (including snapshots of the filesystem or arbitrary subtrees).
Solution 2:
Fast forward to 2019. Ceph is now around for quite some time and with the jewel Release Version 10.2.0 CephFS is declared production ready.
The Ceph setup has also been greatly reduced in complexity with the new BlueStore storage backend that was introduced in Ceph 12 Luminous release.
Give it a try.