Solution 1:

The strace command might be of some use. It will show you in what system calls the time is being taken up:

$ strace -cTv zip /tmp/test.zip /usr/share/dict/words
  adding: usr/share/dict/words (deflated 73%)
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.000055           2        34           read
  0.00    0.000000           0        21           write
  0.00    0.000000           0        12         3 open
  0.00    0.000000           0         9           close
  0.00    0.000000           0         3         3 access
  0.00    0.000000           0         1           rename
  0.00    0.000000           0         3           brk
  0.00    0.000000           0         1           gettimeofday
  0.00    0.000000           0         4           munmap
  0.00    0.000000           0         3           mprotect
  0.00    0.000000           0         6           _llseek
  0.00    0.000000           0         6           rt_sigaction
  0.00    0.000000           0        12           mmap2
  0.00    0.000000           0         8         1 stat64
  0.00    0.000000           0         1         1 lstat64
  0.00    0.000000           0         9           fstat64
  0.00    0.000000           0         1           fcntl64
  0.00    0.000000           0         1           set_thread_area
------ ----------- ----------- --------- --------- ----------------
100.00    0.000055                   141         8 total

Solution 2:

On Linux you can read /proc/[pid]/status; the VmPeak field is the maximum virtual memory size, VmHWM is the maximum resident set size.

The getrusage() syscall might, or might not, help. The struct rusage contains e.g. a maxrss field, but at least on Linux this is never filled in.

Solution 3:

If you don't mind repeating the execution a few times you can use ulimit -Hv to set the memory limit for the shell (in Bash) and then binary search the minimum when the application successfully exits.