"Measure" vs. "measurement"

Solution 1:

Inasmuch as it's relevant to this context, a measure is the standard, system, or unit by which something is measured, e.g.,

Philosophers have long pondered the true measure of a man.

The standard measure of distance is the meter.

or an unstated or undefined amount, e.g.,

This recipe calls for a measure of egg for texture.

She failed to show any measure of sympathy.

A measurement is the datum or numerical value obtained by measuring, e.g.,

His measurements are off by an order of magnitude.

Either word can serve as the act of measuring.

His measure(ment) of the frequency failed to account for the red-shift in the observations.

For your example,

The measure of this algorithm's performance is its execution time. Therefore, our testing suite gathers measurements of the execution under different runtime conditions. Our testing suite itself adds overhead that will not be present in a deployment environment, however, and this must not be forgotten when we analyze its measure(s/ment/ments) of the underlying algorithm's performance.

Solution 2:

In normal usage, "measure" is a verb referring to the action of laying down a ruler or clocking a speed or whatever. "Measurement" is a noun that can be, (a) the act of measuring, or (b) the quantity to be measured or the result of measuring.

For example, "I measured the length of this piece of wood." (measure - verb) "I used a ruler to perform this measurement." (measurement - the action) "Please find the measurements of this room." "The measurement was 14 inches." (measurement - result)

English-speakers occasionally use "measure" as a noun for the result of measurement. Like, "The resulting measure was three liters." But this is a relatively rare. Most people say "measurement".

Solution 3:

Numerical values that are measured are measurements, where I would define measure as "to express an observed magnitude in terms of defined units", and measurements are the numbers produced by the act of measuring (measurement). This is exactly what you are doing with your execution times.

The noun form of measure can refer to a value that indicates a characteristic, but is not strictly measured. For example, the Gini coefficient is a measure of a country' economic inequality. The value was not obtained by comparing some observable magnitude against a "Gini scale", it was calculated from multiple possibly subjective statistical samples.