Only allow certain blocks to be broken with mining fatigue

I want to make sure players cant break any blocks but wheat, while they have the mining fatigue effect. Is there a way to detect if they are trying to break the wheat or is there a way to only allow wheat?

Detecting if a player is breaking wheat is (as far as i know) not possible.
However if you only want people to be able to break wheat, you first have to put everyone into adventure mode with:

/gamemode 2 @a

Now, no one can break blocks anymore. Give the people you want to be able to break wheat a tool you can obtain by typing:

/give @p minecraft:diamond_hoe 1 0 {CanDestroy:["minecraft:wheat"]}

That will give you a diamond hoe that is able to break wheat. If you want a different tool just replace diamond_hoe with a different item name. You can find those item names in the minecraft wiki by looking into the box on the right.