Single word which means 'suffix' or 'prefix'

Affix

an additional element placed at the beginning or end of a root, stem, or word, or in the body of a word, to modify its meaning.

Affix is appropriate. Per wiktionary, it means:

(linguistics) A bound morpheme added to a word’s stem; formerly applied only to suffixes (also called postfixes), the term as now used comprises prefixes, suffixes, infixes, circumfixes, and suprafixes.

Per wikipedia,

An affix is a morpheme that is attached to a word stem to form a new word. Affixes may be derivational, like English -ness and pre-, or inflectional, like English plural -s and past tense -ed. They are bound morphemes by definition; prefixes and suffixes may be separable affixes. Affixation is, thus, the linguistic process speakers use to form different words by adding morphemes (affixes) at the beginning (prefixation), the middle (infixation) or the end (suffixation) of words.

If you expressly exclude infixes, such that the set contains only prefixes and suffixes, then the group would be adfix, a subset of affix.

adfix (plural adfixes)
A type of affix, which is attached to the outside of a stem (an existing word), to form a new word. The adfix category branches into prefixes (ones which appear before the stem) and suffixes (ones which appear after the stem). [emphasis added]