ZFS: will RAM shortage cause trouble?

Don't do it. Dedupe on ZFS is bad news if you haven't planned well for it...

What can happen?

• Your system can slow down considerably in certain operations. Holding the dedplication table in RAM isn't the best use of resources.
• Your system can stall for DAYS if you delete data or filesystems the wrong way.

If there's any doubt, you shouldn't be using dedupe. Buy more disks. Design for more space. Use COMPRESSION instead... it's a much smarter choice for most data sets. I have yet to find a good application for ZFS deduplication that was worth the planning headache.

Also see:
Backup storage server with ZFS
and
ZFS - destroying deduplicated zvol or data set stalls the server. How to recover?