Filter based on another table

You can use a NOT EXISTS to filter out what shouldn't be there.

WITH img AS (
  SELECT id as image_id, label, mu
  FROM images
  WHERE id = 1
)
SELECT 
  images.file_path
, images.id as image_id
, ratings.mu as rating_mu
, ratings.label as rating_label
FROM images 
CROSS JOIN img
INNER JOIN ratings
  ON images.id = ratings.image 
 AND ratings.label = img.label 
WHERE NOT EXISTS (
  SELECT 1
  FROM comparison c
  WHERE images.id in (c.img1, c. img2) 
    AND img.image_id in (c.img1, c. img2)
)
ORDER BY abs(img.mu - ratings.mu) ASC
LIMIT 1;

file_path    | image_id | rating_mu | rating_label
:----------- | -------: | --------: | -----------:
/img/foo.png |        3 |       2.5 |            1

Test on db<>fiddle here