a perfection of/perfect

If you use the word perfection in your sentence, it can be interpreted as there having been some kind of process, perhaps a lengthy one, that took place.

This can be seen from one of the senses of the definition of perfection:

[Merriam-Webster]
3 : the act or process of perfecting

Because your sentence starts with the novel achieves, stress is already put on there being an action. Using perfection emphasizes a time-based period during which the quality of the form was improved to the point of it being perfect.

But if you just use the word perfect, that subtle interpretation isn't as obvious.

Put more simply, if I'm being forced to explain a difference between the two sentences, I might think of it in this way:

The novel achieves a perfection of form that is quite new.

I could associate this with somebody spending months, if not years, working arduously to produce the final result, and finally achieving it. The emphasis is more on the action than on the result.

The novel achieves a perfect form that is quite new.

Here, I might simply think that the author managed to get lucky, or was simply brilliant, and managed to produce something special. The emphasis is more on the result than on the action that led up to it.

Generally speaking, either form of the word can be used, and most likely understood in the same way. But, if you're looking for a subtle distinction, that would be it.