Red Mushroom House in World 4

Super Mario Bros 3, World 4 has a Red Mushroom House with a rock blocking its path. The Hammer Bros have never left me with an actual hammer, and I never seem to have one when I get to World 4.

Is there a hammer I missed, or is there another way to get to the Red Mushroom House?

There's a hammer located in World 3 (take the canoe and beat the mini-boss on the other side, the mushroom house there is guaranteed to give you one) but there's also a couple places to use it in World 3, if you like. If you hold on to it through World 3 and you'll be able to use it to access that red mushroom house in World 4.

Here's a picture of the world map where the canoe is. The second island with only one house is where you want to be.