What does Théoden’s quote “Oft evil will shall evil mar” in Tolkien’s Lord of the Rings mean?

That's exactly what it means.

The desires and plans of evil people ("evil will;" "will" in this case being the noun relating to intent and desire) often ("oft") ruin ("mar") the cause of evil.

That is, the phrase says that evil people are selfish, petty, and short-sighted, and that this quality in evil individuals often impairs the grander world-embracing schemes of capital-E-Evil. It's ironic, and the context of Wormtongue and the Palantir is a great example of that irony.

This is very similar in meaning to a more usual phrase:

come back to bite you

If something will come back to bite you, it will cause problems for you in the future. Her unpleasant remarks may well come back to bite her later.

http://idioms.thefreedictionary.com/come+back+to+bite+you

In this case it may also mean that one evil person is likely to spoil the plans of another evil person. Such people want everything their own way and don't like to co-operate. If they have opposing ideas then conflict will follow. This is often seen in gang warfare.