Are "skill set" and "skill sets" both acceptable?

Skill is a noun, not an adjective. However in that phrase it is used as a noun adjunct, so it serves as an adjective.

Aside from that, I agree entirely. "Well, that matches my skill set" and "The two jobs need completely different skill sets" being valid singular and plural uses, respectively.

In reference to all of one's skills, the term 'skill set' includes all of a person's skills with none left out of the fold or on the sidelines. But if we wanted to convey a breakdown of skills into different areas and components, the term 'multiple' can be used to more clearly convey a plurality of sets, as in 'multiple skill sets'. In this instance, 'sets' is plural.