Slower/cached Linux file system required
I know it sounds odd but I need a slower or cached filesystem.
I have a lot of firewalls that are syslog'ing their data to a pair of Linux VMs which write these files to their 'local' (actually FC SAN attached) ext3-formatted disks and also forward the messages to our Splunk servers.
The problem is that the syslog server is writing these syslog messages as hundreds, sometimes thousands, of tiny ~4k writes per second back to our FC SAN - which can handle this workload right now but our FW traffic's going to be growing by at least a factor of 5000% (really) in coming months and that'll be a pain for the SAN, I want to fix the root cause before it's a problem.
So I need some help figuring out a way of getting these writes cached or held-off in some way from the 'physical' disks so that the VMs fire off larger, but less frequent, writes - there's no way of avoiding these writes but there's no need for it to do so many tiny ones.
I've looked at the various ext3 options, setting noatime and nodiratime but that's not made much of a dent in the problem. Obviously I'm investigating other file systems but thought I'd throw this out in case others have the same problem in the future.
Oh and I can't just forward these messages to Splunk, our firewall team insist they're in their original format for diag purposes.
Perhaps commit ext3 mount option would help you? For example, commit=60 would flush all the data and the meta-data only once per minute.
Obligatory warning: this might lead to data loss of up to one minute of data (if you pass the commit=60 value).
Filesystem: Disable write barriers if your devices are using them, and disable atime updates across the board.
But chances are you can also tune your syslog at the expense of greater loss of data in the case of a failure event (power, etc).
Example directives from syslog-ng (which may not be what you are using):
flush_lines()Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch.flush_timeout()Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.
The destination is in this case is a disk file.