Choose a note in Preview PDF
This is indeed an unintuitive situation. To delete the note:
Right-click the note. This can be done with the mouse or by tapping with two fingers on a trackpad.
The note will highlight and a menu will show up. Click anywhere outside the menu to get rid of the menu. Now the note has a gray outline.
Click Edit > Delete at the menu bar.
Here is a GIF of the process. Note the first click is a right-click. 