Optimization of Soft Page Faults

Based on your provided facts, I think the mentioned application was doing rapid memory allocation (e.g malloc() ). It seems that it allocated a block, possibly releasing it and then allocate again. Usually memory allocator will keep freed memory block(s) in cache, but it could be that in your case, it was forced to be released.

I think there's not much you can do here, since we're dealing with application behaviour, not kernel or other aspect. However, I think the situation could be altered by using different memory allocator. Try to google "memory allocator". For example, in Linux, the default memory allocator is ptmalloc. Example of alternative is : http://goog-perftools.sourceforge.net/doc/tcmalloc.html