How I can select rows from a dataframe that do not match?

If I understand correctly, you need the negation of the %in% operator. Something like this should work:

subset(b, !(y %in% a$x))

> subset(b, !(y %in% a$x))
  y
5 5
6 6

Try the set difference function setdiff. So you would have

results1 = setdiff(a$x, b$y)   # elements in a$x NOT in b$y
results2 = setdiff(b$y, a$x)   # elements in b$y NOT in a$x

You could also use dplyr for this task. To find what is in b but not a:

library(dplyr)    
anti_join(b, a, by = c("y" = "x"))

#   y
# 1 5
# 2 6