In sentences with "too" and "enough" what word does the to-infinitive modify?
As pointed out in this answer,
Too is a Negative -- too
AdjtoVPmeans 'soAdjthatNot VP'
A bag that is too heavy (for anyone) to lift means a bag that can't be lifted (by anyone).
As for what modifies what, the infinitive is the [to VP] part of the [too Adj [to VP]] construction.
That infinitive doesn't modify any single word; rather, it says there is Adj enough to negate VP.