Do EVs take immediate effect?

Notice: This post only applies to games up to HeartGold/SoulSilver. The EV system has changed since Black/White. For more details, see DragonLord's answer.

Technically, every 4 EVs gained will result in 1 extra stat at level 100. For example, if you took two level 100 Snorlax, with identical natures and IVs, but one had 4 EVs in Attack and the other had 0 EVs in Attack, the former would have 1 more point in its Attack stat than the latter.

The EV-effected stats gained during level-up don't all happen all at once. DisgruntledGoat described what occurs in a good way:

On level up, the game will calculate how much of a stat point increase you would have at that level and adds it on. For example, if you have a level 50 Gyarados and you earn 200 EVs (50 stat points), you won't get +1 extra point per level up. Instead, you'll get around +25 on the first level up (because at lv50 you would have earned about half the points), then +1 for roughly every two levels after that.

Of course, if your Gyarados was level 99, had cleared EVs, and then you battled all those low level Pokémon, you would probably see a very large stat boost from 99 to 100.

As of Generation V (Black and White), yes. EVs are now applied to stats after every battle. This means that player-visible stat increases may occur when they are not expected, such as an increase in maximum HP after a battle without leveling up—and I've observed this playing Pokémon Black.

In Generations III and IV, EVs are generally applied only on leveling up, and Pokémon do not gain EVs when at level 100 or otherwise from a battle that does not yield experience points, except by stat-boosting items. EVs are applied when the Pokémon is deposited into and withdrawn from the Pokémon Storage System (box) as well, which can be done to force any pending EVs to be applied between level-ups or at level 100.

Since Generation III, every 4 EVs adds one point to a particular stat at level 100. For lower-level Pokémon, this stat gain is prorated based on the Pokémon's current level (for example, a level 50 Pokémon with 120 EVs for HP gains 15 HP from EVs, rather than 30 at level 100). Generations I and II use a system called Stat Exp that differs significantly from the current EV system.

See this Bulbapedia article for more information on effort values.

As of Generation VI (X and Y), all EV changes are applied immediately, regardless of whether they are the result of battle, Super Training or otherwise.