“if they were” is real condition or hypothetical?

Solution 1:

"If" is the thing that is making it hypothetical, so both your examples are hypothetical. However, you would say "going camping", not "going on the camping"... or you could say "going on the camping trip".

The first one is how Americans would say it: "If he wasn't going on the camping trip, he would stay home." The second is how UK-influenced countries use it: "If he weren't going camping, he would stay (at) home."

The real condition would be "He is not going camping, so he will stay home."

Solution 2:

I don't think your distinction between 'real' and 'hypothetical' is a valid one, and your examples are not idiomatic English.

If the person did go camping, you would say "If he had not gone camping, he would have stayed at home." A hypothetical condition would be "If he were not going camping, he could have accepted the invitation to his friend's party."