What is the meaning of "ought not"?
Consider this example:
A few strong branches over water reach for what they ought not reach.
Which of the meanings comes closest to “ought not” in this sentence?
Is it “doesn't have to”, “should not” or “must not”?
Some might prefer the sentence to have to before reach, but otherwise it is well-formed. It’s impossible to be sure of the meaning out of context. It could be should not or it could be must not, or possibly something else, but not doesn't have to (which would in any case have to be don't have to in order to agree in number with the subject).
There appear to be several issues entangled here.
One issue, the answer to the OP's original question, is that they're all different modal constructions, each with different sources, different syntax, different idioms, and different possible senses, which nevertheless often overlap, in the right contexts. And they're all irregular as hell.
Another issue is the metaphoric source of ought, which is, as many of us know, the old past participle of the verb to owe, and is caught up with the concepts of credit and debit, authority and duty. One ought to do something if one owes one's action (to some person, or some abstract principle) to pay a debt, to pay one's due(s), to do one's due-ty, etc, etc.
Ought (or more commonly oughta) is principally a weak Necessity deontic modal, like should, rooted in a Face and Credit social system (rather than a Strict Authority system, like the strong Necessity modals must and hafta) but there is an epistemic sense as well, usually functioning wherever emotions can be invoked or inferred, as in anger over a misplaced item. Compare
- Dammit! It oughta be here.
with
- Dammit! It should be here.
which sounds a little weaker to my ear by comparison. But that could be personal usage bias. There's a lot of that, with all modals.
Still another issue is the use of ought as a true modal auxiliary verb (which means no to on infinitive complements), or as a semi-modal auxiliary, (like need or dare, which allow to before infinitive complements, under certain circumstances).
Ought is a paraphrase of a true modal auxiliary (in this case should) that hasn't quite assimilated yet, at least not enough to lose the to everywhere. Like the true modal auxiliary should, it precedes not, but there are variant past constructions -- hadn't oughta, shouldn't oughta. Also like should (and would, could, might and must), it's formed from a preterite stem but can refer to the future naturally.
But should never takes to. However, ought, if it's separated from its infinitive by a negative, can suffer stack overflow, lose its modal binding field, revert to regular verb status, and allow automatic generation of an infinitive complementizer to. This alternation of ought not go vs ought not to go resembles the behavior of need and dare, which are also Negative Polarity Items as modals, though with a different syntax.
Outside negative polarity environments, though, the to in ought to seems likely to remain for a while. It is reinforced by the bisyllabic nature of all the periphrastic modal proto-auxiliaries oughta, hafta, gonna, wanna, etc, which are all in the same kind of boat, but whose individual grammars all leak in different places.
In general usage, should not. "Doesn't Have to" does not really make sense to me in that context, whilst "must not" appears too strong.
A few strong branches over water reach for what they should not reach
However looking at that sentence it does seem a little odd. I am still confident that that is the best fit.
We normally say A few strong branches over water reach for what they ought not to reach.
I'd rephrase it as "... reach for what they aren't supposed to reach". The closest meaning to this is should not.