Adjust the other end of a visual selection

I'm not sure I'm actually understanding what you want, but perhaps it is this:

:help v_o

Basically, while text is visually highlighted, you can press o and it will jump to the other "end" of the visual region without modifying it.