How to measure average execution time of a script?

Solution 1:

No, your idea of averaging is correct.

Script execution depends on lots of factors, and however it is to be split up between setup time (loading interpreter in memory, setting up, and possibly compiling code to bytecode or machine code) and true execution time.

To better focus on inner execution time, you do the loop in the script itself (i.e. instead of calculating one factorial, you calculate it 100 times within one execution of the script. The script will be setup once, and the inner routine will execute 100 times).

To focus on total time, you execute the script one hundred times and average the results. Ideally, you should separate those executions enough that the system returns in a "reference state" (or a script-unrelated state) every time. For example, the interpreter itself will be cached in memory, so that the very first execution of the script will be appreciably slower than the subsequent ones.

To get a better insight on the algorithm, I think the best way is something like this (on an otherwise idle machine):

wrap the algorithm in one single function.
in the controlling application:
- call the function once
- get the system ("wall clock") time and add 10 (or a reasonable N) seconds
- enter the loop and start counting iterations
- after each call to the function, increment the counter
- if the system time is below the saved time, do another loop
- get exact N, possibly floating point, from current wall clock time
- display the counter divided by N: that's the number of iterations/second.

The application only runs once, all setup and priming is done by the first untimed iteration, so this should minimize the overheads (except maybe for the time call).

If the function receives an input, you'd do well to supply it a random sequence of inputs using a PRNG seeded with a fixed value, to ensure both versions of the function being tested receive the same values. This avoids one function performing apparently better due to "lucky numbers" (e.g. I remember a variation of the Hillsort algorithm that performed measurably better if the number of items to be sorted was in the form 2^k-1 with small ks).

Solution 2:

You can run iterations of the program in a loop; and divide the total time by the number of iterations:

time for i in {1..10}; do sleep 1; done
real    0m10.052s
user    0m0.005s
sys 0m0.018s

Solution 3:

there is a tool called multitime that does exactly this: running a command several times, measuring how long it takes (real/user/system with mean, min/max and median time automatically computed)

For instance, for measuring a similar script 100 times:

multitime -q -n 100 "fact1.sh"
===> multitime results
1: -q fact1.sh
            Mean        Std.Dev.    Min         Median      Max
real        0.122       0.032       0.086       0.116       0.171       
user        0.148       0.044       0.096       0.137       0.223       
sys         0.023       0.019       0.000       0.014       0.061

Solution 4:

This is old but it came up so high on google when I was looking for a command I used previously but couldn't find. Anyways, my preferred way of doing this is:

perf stat -r 10 -B sleep 1

This gives quite a bit of details including average execution time right at the end:

1.002248382 seconds time elapsed                   ( +-  0.01% )

Solution 5:

python -m timeit -n 1 -r 100 -s 'import os' 'os.system("ruby fac1.rb")'

For -n, -r and other options see https://docs.python.org/2/library/timeit.html#command-line-interface