Are you looking for something like this perhaps?

To show popup over selected row(s) only

  private void maybeShowPopup(MouseEvent e) {
     if (e.isPopupTrigger()) {

        // get row that pointer is over
        int row = table.rowAtPoint(e.getPoint());

        // if pointer is over a selected row, show popup
        if (table.isRowSelected(row)) {
           popup.show(e.getComponent(), e.getX(), e.getY());
        }
     }
  }

Or the converse, to prevent popup from showing over selected rows only:

  private void maybeShowPopup(MouseEvent e) {
     if (e.isPopupTrigger()) {
        int row = table.rowAtPoint(e.getPoint());
        int[] selectedRows = table.getSelectedRows();

        if (!table.isRowSelected(row)) {
           popup.show(e.getComponent(), e.getX(), e.getY());
        }
     }

It's an interesting question, because it highlights missing api on JComponent :-)

As we all know, the recommended way to register popupMenus is to use the componentPopupMenu property. Related api is

 void setComponentPopupMenu(JPopupMenu);
 JPopupMenu getComponentPopupMenu();
 Point getPopupLocation(MouseEvent);

what is missing (and actually needed for this requirement) is

JPopupMenu getComponentPopupMenu(MouseEvent);

this lack is all the more annoying, as the getPopupLocation is called (by AWTEventHelper deep in the LAF) after getComponentPopup(). So there's no leeway for a hack like storing the last mouse event which might have triggered the popup and then decide which/if to return popup. And returning null for the location will only result in showing it at the mouse location

The only (dirty) hack (around my utter reluctance to get my hands dirty with a MouseListener ;-) is to override getComponentPopup and decide there whether or not to return it based on current mouse position

    table = new JTable(model) {

        /** 
         * @inherited <p>
         */
        @Override
        public JPopupMenu getComponentPopupMenu() {
            Point p = getMousePosition();
            // mouse over table and valid row
            if (p != null && rowAtPoint(p) >= 0) {
                // condition for showing popup triggered by mouse
                if (isRowSelected(rowAtPoint(p))) {
                    return super.getComponentPopupMenu();
                } else {
                    return null;
                }
            }
            return super.getComponentPopupMenu();
        }

    };

the side-effect is that popup showing isn't triggered by keyboard as long as the mouse is anywhere above the table, which might or not be a problem.