What triggers the ability to swap opponents?

Occasionally I have seen a button presented to me next to the Scout and Attack buttons to Swap Enemy, which will let me get a different opponent if the current one is too hard. I know this button isn't always available - what is the criteria for making it appear? (Number of defeats, time, combination?)

It is automatically triggered by time.

When after one day, that opponent will be able to be changed. Other things like defeats won't count. Time is the main reason for switching opponents.

Once again, no other factor will be used, which means defeats, victory points, and amount of free villages. For more information about the game, check out the Boom Beach Wiki.

It's just a matter of time - If you wait long enough, you can skip any matchmade opponents. I think the timer is a few days, but I haven't sat down and timed it.