How do I say 'I don't want' without phrasing it in terms of wanting?

The underlying difference between wanting and your other possibilities is that the others, describing active operations, are clearly 2-valued. "Wanting", on the other hand, describes a mental state which can inhabit a continuum, ranging from desiring a thing or consequence, to rejecting a thing or consequence, with a neutral state being part of the continuum. The neutral state, or "don't-care" is an important possibility in any discussion of desire, so it should not be overlooked.

As a result, there is no particular reason to expect that a single word will encompass both the neutral and the negative. The simple negation ("I don't want") is generally adequate to express the lack of wanting, and if the exact form of not wanting is important, then the choices such as "I don't care" or "I really don't want" can be used.

I suggest indifference: lack of interest in or concern about something; an indifferent attitude or feeling.

In your example:

"I am indifferent about going there."