How do you remove a SPECIFIC level of potion effect from a player?

When you apply the effect to the player, set some scoreboard value to 1. Set it back to 0 when clearing the effect from the player. This way you can keep track of what effects the player might or might not have.

To create a scoreboard do scoreboard objectives create speed1 dummy. Then set the player's score to 1 using scoreboard players set <player> speed1 1. After that give all players with a score of 1 speed using effect @a[score_speed1_min=1] speed 1000000 <whatever value> true. When clearing the effect do the exact opposite, set the score back to 0 and clear the effects of all players with a score of 1.

You should have the command blocks applying and clearing the effects on a clock. If you set score speed2 for player1 to 0, while a player2 has speed1 set to 1, this will happen:

  1. Speed (all levels) will get cleared from player1 and player2
  2. Speed 1 will be given to player2

So: Before - Player1 has speed 2, player2 has speed 1. After - Player1 has no speed, player2 still has speed 1

Please excuse me for my possibly not very good English, I hope this helped you :)