Btrfs over mdadm RAID6?
Solution 1:
In 2016, Btrfs RAID-6 should not be used.
You can see on the Btrfs status page that RAID56 is considered unstable. The write hole still exists, and the parity is not checksummed. Scrubbing will verify data but not repair any data degradation.
To answer your questions:
is btrfs over mdadm raid6 reliable ?
You want one Btrfs volume to sit on top of md RAID-6, meaning that Btrfs will be unaware of the RAID. This is as reliable as having one filesystem formatted as Btrfs, which is by default a duplicated copy of the metadata and only a single copy of your data.
will the bitrot protection and snapshots keep working ?
Using your proposed setup, Btrfs will detect rotted bits, but it can't fix them because there's only one copy of your data. If something were to happen to your md RAID-6 array, you'd be looking at data loss.
Snapshots would still work, though, but also without repairs during scrubbing.
are there some drawbacks to this setup ?
You'd be trusting md to keep your data intact, but md doesn't know about what your data is―that's Btrfs's job. Btrfs can't repair an inconsistency happening at md's level.
Here's my personal example of what would happen if something were to go wrong with md RAID-6.
are there better options ?
If you're looking for an alternative to BTRFS RAID-6, consider ZFS RAID-Z2, which offers a reliable implementation of RAID-6 that checks the integrity and repairs your data as well as snapshots.
As far as drawbacks for ZFS RAID-Z2, it can't be shrunk or reshaped, and resizing should only be done one disk at a time, which can take a very long time.
See also: Overview of ZFS
Solution 2:
well an raid1 btrfs on top of raid6 mdadm splited in two can be a solution. You can split every hdd in two equal partitions, group one partition per hdd in 6 hdd sets in two raid6 mdadm volumes and then make a raid1 btrfs from this two raid6 mdadm volumes