Meaning and usage of "to no end"

As I learned the phrase, it's simply he annoys me no end, meaning (literally) without limit and (actually) a great deal. He annoys me to no end would mean 'without any purpose', but since most people aren't annoying deliberately, I can't see how the phrase would be useful.

The OED discusses no end in the context of end, which is noted as of Old English/Saxon origins to mean 1a. "the extremity or outermost part (in any direction) of a portion of space, or of anything extended in space; utmost limit," and 1b. "a limit of magnitude or multitude." No end is defined as a colloquialism that means "a vast quantity or number (of)," and in modern slang as "‘immensely’, ‘to any extent’; and (with of) qualifying a predicate." The provided example quotes are below.

• 1623 J. Bingham tr. Xenophon Hist. 143 You‥made no end of promises.
• 1856 C. Reade Never too Late x, Box at the opera costs no end.
• 1859 F. W. Farrar Eric 55 You are no end cleverer and stronger.
• 1866 Trollope Belton Estate III. v. 121 What comfort have I in a big house, and no end of gardens?
• 1871 F. C. Burnand More Happy Thoughts (ed. 2) xix. 143 He ‘makes no end of stuff’, or ‘loses no end of stuff’‥on the Derby.
• 1872 E. Peacock Mabel Heron I. i. 9 You will have no end of trouble.
• 1883 W. Besant All in Garden Fair viii, Keats was no end of a fellow.
• 1909 R. E. Knowles Attic Guest 54 You'll have no end of fun with him.
• 1912 Chambers's Jrnl. Dec. 769/1 ‘I really must show this to Champneys,’ thought Michael; ‘it will please him no end.’
• 1955 Ess. & Stud. 8 5 A few clean strokes of Occam's razor would have helped Mr. Jackson no end.
• 1958 H. Babcock I don't want to shoot Elephant 8, I often walk fifteen miles a day while hunting.‥ This puzzles my wife no end.
• 1970 New Yorker 3 Oct. 90/2 Thomas had been impressed no end by the sight of Klüver‥fixing an art-and-technology malfunction with a pair of pliers.