Ceph: Why is a greater number of "placement groups" a "bad thing"?

Solution 1:

I am currently investigating ceph as an alternative to our data storage as well. I found your question and did some reading and am hoping this idea makes sense. The way they do dynamic distribution of data would suggest that if you have a high number of OSDs (significantly more than the replication level). Then it seems like it would be possible (and likely) that the distribution algorithm would put some parts of files on a huge number of OSDs, such that if you lost N nodes (where N is greater than your replication level) its highly likely you would lose your data (or at least have a significant amount of corruption). Which isn't really a surprise. I would expect to have data loss if you lost 3 nodes in your cluster (like their example) unless your replication level was 4 or higher.