NFS Caching Issue

We are having an issue that happens intermittently during a code deployment. NFS caches the files as they are read, but if a file is read during a code deploy it stays in a dirty state as if the file wasn't changed during the deploy. The only way we can alleviate this issue is by clearing the NFS cache after the deploy.

Our webserver returns blank pages for all requests until the NFS cache is cleared.

Is there a setting or does anyone have a suggestion on how to fix this?

We are running NFS v3 at the moment.

Exists different options to cache file attribute on a NFS mount (client side):

NOAC
Use the noac mount option to achieve attribute cache coherence among multiple clients. Almost every file system operation checks file attribute information. The client keeps this information cached for a period of time to reduce network and server load. When noac is in effect, a client’s file attribute cache is disabled, so each operation that needs to check a file’s attributes is forced to go back to the server. This permits a client to see changes to a file very quickly, at the cost of many extra network operations.

lookupcache=none
If the client ignores its cache and validates every application lookup request with the server, that client can immediately detect when a new directory entry has been either created or removed by another client. You can specify this behavior using lookupcache=none. The extra NFS requests needed if the client does not cache directory entries can exact a performance penalty. Disabling lookup caching should result in less of a performance penalty than using noac, and has no effect on how the NFS client caches the attributes of files.

actimeo=n
Using actimeo sets all of acregmin, acregmax, acdirmin, and acdirmax to the same value.

• acregmin=n, The minimum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 3-second minimum.
• acregmax=n, The maximum time (in seconds) that the NFS client caches attributes of a regular file before it requests fresh attribute information from a server. If this option is not specified, the NFS client uses a 60-second maximum.
• acdirmin=n, The minimum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. If this option is not specified, the NFS
  client uses a 30-second minimum.
• acdirmax=n, The maximum time (in seconds) that the NFS client caches attributes of a directory before it requests fresh attribute information from a server. If this option is not specified, the NFS
  client uses a 60-second maximum.

Taken from the NFS man page.

I hope this help.

Your specifying a lot of NFS options, most of which (rsize/wsize/soft) are not needed because they are already (usually) the defaults (in Linux);

NFS Mount Options rsize=32768,wsize=32768,timeo=30,retrans=10,intr,noatime,soft,async,nodev

However, if you are running Linux, you should probably look into setting the following NFS options;

actimeo=3 (sets acreg[max|min],acdir[max|min], can be lowered to 1 if you're still seeing speed problems)

lookupcache=none (ignores cache completely, default is "all")

noac (Disables attribute caching)

This blog post seems to have more information

you only need to set acdirmax=1. By default the NFS mount uses Close-To-Open cache consistency. This means it will check the modification time and permissions of the file every time it opens a file, regardless of cached data. Issues occur if you add new files to a directory or overwrite files with new versions.

When files are overwritten on server A, just after server B read that same directory, the lookup cache on server B will hold a stale copy of these files. So when we try to access attributes of a file, it will send a request to the fileserver, but it will use the filehandle that is being stored in the cache, eventhough it was replaced by a new file with a different handle. Since the file with the old handle is gone it will report “File not Found”

So the solution to this problem would be to invalidate the lookup cache for a specific directory. Disabling the lookup cache completely is not an option, as it will overload the file server with NFS requests. The lookup cache checks if the cache is still valid by checking the timestamp of the directory that contains the file. If this is in the attribute cache, it will use it from cache. So to resolve your issues you should make sure that the attribute cache, specific for directories, has the lowest cache time possible.

This can be done using mountoptions : acdirmax=1