Full-stack vs Full Stack, Back-end vs Back end, Front-end vs Front end
The usual principle is that an attributive adjective gets hyphenated (it was a hard-fought victory), but a predicative adjective does not (the victory was hard fought).
The punctuation usually follows the syllabic stress. Note that in hard-fought victory, there is only one stressed syllable in hard-fought, whereas in the victory was hard fought, there are two stressed syllables in hard fought.
So you'd have front-end system.
The noun phrase front end shouldn't be hyphenated: both words are stressed.