What is the impact of increasing "nofile" limits in /etc/security/limits.conf?

I've found a number of articles describing how to increase the limits for the number of open files through /etc/security/limits.conf, but I don't understand the impact of doing so. Many times I see people updating 1024 to 2048. Ok, those file handles must cost RAM or something. Why not increase it to 100000? What resource am I eating up with open files?

A question about how to increase the limits: https://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux

This is the limit on the number of files that a single process can have open at a time. Sockets, pipes, and terminals count too. There is almost no software in existence that can handle more than about 20,000 files open at a time, so there's no point in setting the limit higher than that.

Just like to add that value of nofiles depend on "/proc/sys/fs/nr_open" as mentioned above and ulimit uses setrlimit() to set resource limit.