Ceph with mainstream (desktop) SSDs?

I am searching for an answer about using SSDs drivers, which are build for the mainstream. Afaik ceph uses for its marketing to buy cheap disks for entprise ready storages. But I cannot find any details or tests about this.

E.g. if you take a look at the 1TB SATA 2.5" e.g. SanDisk Ultra 3d or Samsung 960evo both are pretty cheap right now to build a fast storage array with ceph

But if I read on, there are a bunch of limitations mentioned for storage arrays (for SANs or RAID-arrays) to avoid cache on the drives, to ensure a high DWPD. So does this also match do ceph? If considering this facts the price for the same sized SSD is about factor 3 to 5 more expensive between mainstream and enterprise SSDs.

So considering from economical perspective, I could buy at least 3 times more drives with mainstream SSDs compared to the enterprise grade SSDs for RAIDs or SAN storages.

Can someone tell me, if the mainstream SSDs will work well with ceph on production environments?

What endurance per named SSD drive can be expected?

Solution 1:

I can only highly recommend to use SSDs with capacitors that protect against power interruption. For Ceph this not only protects data integrity but far more important it will result in much less latency. SSDs without capacitors are blocking when they write out their cache. SSDs with capacitors just ignore cache flushes (which ceph is sending out) because they are always able to write out their cache due to the capacitors help during a power loss situation.

We started out with 860 EVOs. It was a nightmare of throughput. 860 PROs were better, but the PM883 (with power-loss protection, also Samsung) really made a huge difference.

Be careful with Samsung though: They don't work well with AMD SATA Controllers. At least the SB7x0/SB8x0/SB9x0 SATA Controller. AMD and Samsung are pointing at each other why NCQ support is broken. Which sucks. This is 2021.