Does the word 'incrementation' exist?
Solution 1:
Whether a word can be said to "exist" or not is a very difficult question whose answer depends on many things. Clearly, for example, the sequence of characters that "incrementation" comprises exists, or otherwise you couldn't have written it down in the first place!
So, instead, I will answer an easier and less philosophical question: are there a number of uses of "incrementation" in the wild with the meaning you suggested? The answer is decidedly yes.
Now, whether that's a good choice, of course, is something else entirely. Some people might say "incrementation" is too technical or constructivist of a word for regular, everyday usage. In that case, you should stick with just "increments" or "incrementing" instead of using the less common word.
Solution 2:
There are about 358,000 Google hits for "incrementation" (with upright quotes, to limit searches to pages with this exact word), so it surely exists as a word in actual usage—even though Google results are not an exact measure of anything and even though there are probably occurrences of the word in languages other than English, too.
The word “incrementation” appears to be used mostly in programming, mathematics, and natural sciences. For example, in some programming languages, according to established terminology for them, “the incrementation operator” means the “++” operator, which increments the value of its argument by one. There is apparent need for a word like “incrementation”. Even though we could say “the incrementing operator”, it’s not really a term-like expression, and when we need to discuss how the process of incrementing something actually happens, “incrementation” is more convenient and readable than “process of incrementing”.
Dictionaries like Merriam-Webster and Oxford dictionaries do not contain it, but they often lack new words and words with limited scope of use. This particular word might be common enough to make its way to dictionaries, but this is not automatic; their editors need to do things, and they also exercise their judgment. Dictionaries also have normative aspects: words may be excluded since they are considered as “not correct” in some sense. Spelling dictionaries like those used by Microsoft Word tend to be more limited than large general dictionaries, since when a word is rare, its occurrence in text may well be unintentional, a “typo” or other mistake, so it is useful to inform the user about a potential problem.
Solution 3:
It's common in computer science to turn verbs into nouns simply because labelling processes is at the heart of what we do.
The verb for this is to reify (or to objectify), which itself has the noun form, reification.
Incrementation isn't a pretty word. Stylistically, I'd prefer to use increment as an adjective, 'the increment operator', or using 'incrementing' as a noun, where possible, but I certainly agree there are places where such a word could be valuable.
Is it a word or isn't it? There's no formal way to say, because there's no definitive source in English for what is a word and what isn't. Consider that many programming terms don't mean what their general English equivalents mean. Many terms don't appear in dictionaries at all.
It's understandable and it communicates a familiar idea. If you ask me, yes, it's a word.