Is it incorrect to write "I rate this book a 3 out of 4 stars?"

Both "I rate this book three out of four stars" and "I rate this book a three out of four stars" sound acceptable to me.

"I rate this book a three" seems grammatical to me (although maybe "I give this book a three" would be a more common verb in that kind of sentence). So I think the use of the indefinite article would remain grammatical when the additional clarifying information "...out of four stars" is added to the end of the sentence. "Three" would function in that context as a noun, I guess.

If there were some reason to consider it incorrect to use "a" here, it would not be because of the presence of this earlier in the sentence. This acts as the determiner for the noun book; the article a, which comes after book, is clearly not meant to be a determiner for book.

Based on a comment by tchrist, my current hypothesis is that speakers who object to your sentence don't find it natural to interpret "a three out of four stars" as "a three (out of four stars)", the way I did in the second paragraph of this post. I think that most people would accept "rate/give it a three out of four", with no following noun. For some reason, adding a noun after the second numeral seems to potentially change the interpretation of the first numeral's grammatical role.

When writing about points or scores, I prefer digits to words.

If we accept that there is no problem with "a score of 3"

a score of 90%
a score of 3–2
He had an IQ score of 120

Source: Longman Dictionary

Then something that scores 3 stars, is perfectly fine

• a score of 3 stars

When the context is clear, such as a restaurant, book or movie review, the grade refers to a score, which can be anything: badges, stars, chef hats etc.

• I rate it a 3 out of 4 stars

In other words
   I rate this book a 3 [out of 4 stars]
   This book has a [score of] 3 out of 4 stars

If the determiner e.g. this is used with one noun, another determiner can be used with a different noun or plural noun phrase in the same sentence. For example,

• The match ended in a draw
• My house had a market value of $700,000
• That table costs a whopping 4,000 euros

Your editor does not sound like a native English speaker. They may understand the grammatical rules of English, but they do not have an ear for the language.

A good editor who grew up with English would never say "'A' is wrongly inserted; you have already used 'this' as a determiner, and there is no need for another one."

That's just not how English works, and that one sentence is a terrible piece of writing.

I would not trust writing advice from anyone who writes like that.

Your original sentence does sound better without the "a", but it's still not quite right.

A great editor would have an easy answer for you: "Why don't you try this: 'I rate this book 3 stars out of 4.'"

That is pretty close to the ideal way of writing what you want to say here.