Where is the castle?

I was told by a man to go to the castle and search it for the Golden Leaves that he lost there.

Problem is, I cannot find the castle. I have been wandering around in circles for a while now, and it is getting a little ridiculous.

Where is the castle? What is it near?

Directions

Here is the path you need to take:

(There will be a gate blocking your path directly above here for now)

(there will be a monkey here, you will need to give him bananas. This is part of the trade quest which I detail below)

(there will be a bush over the steps)

Trade Quest: Getting the Bananas

As stated above, you need to give the monkey some bananas in order for him to build a bridge so you can proceed. To do this you need to have completed part of the games trade sequence. The sequence you need to complete is as follows:

1.Obtain the Yoshi doll from the crane game.

2.Trade the Yoshi Doll for the Ribbon with the married couple.

3.Trade the Ribbon to the bowwow (located inside a shed) for some dog food.

4.Trade the Dog Food for the Bananas to the crocodile named sale.

5.Trade the Bananas to the Monkey for the stick. He will then build the bridge for you to cross.

Map

Here is the complete map with the directions, and with the location of the trade items marked (view it in another page to see it in full size):

(images taken from wikicheats) (zeldadungeon.net used as a reference for the trade quest sequence)

The castle is NNW of Richard's Villa, Richard being the person who sent you on this fetch quest.

Follow the red line from Richard's Villa to reach it. The first time you visit, the castle gate is up; a button inside will let the gate down, so you can take the blue path to enter/exit instead.
_{Note for Link's Awakening DX players: you can get a photo with Richard by standing in front of the gate. If you lower it before you do this, you can't get it later!}

"Woah," you say to yourself -- "there's no bridge there!" That is correct! You also need to have progressed the main trading quest far enough to have dropped by Sales' House o' Bananas on the beach. _{^{(For those who haven't, yet: Yoshi Doll (Trendy Game), Bow (house north of Rooster statue), Dog Food (small kennel next to Madam MeowMeow's house))}}. You'll recognize Sales' house because the trees out front actually have bananas on them.

After you've done this, you can give the bananas to Kiki the monkey, which will cause him to effectively build a bridge for you, letting you cross. (You'll also get the next trade quest item.)

This can only be done after completing Level 2 and returning Bow-Wow to Madam Meowmeow, though you can't talk to Richard in the first place unless you've ditched the dog. If you try to do it before, Kiki and Bow-Wow will get in a fight and Kiki will run off until you return. You can't otherwise enter Kanalet Castle, because you need the flippers to enter water, which you don't get until later.

Richard lives in the southern part of Ukuku Prairie. Talk to him and he'll ask you to find his five Golden Leaves at Kanalet Castle. Go north along the trail to Kanalet Castle, and you'll find there's no way through the gate. Go east and you'll find a monkey. Assuming you've been following the Trading Game, talk to him and give him your bananas. He and his friends will construct a bridge leading into Kanalet Castle. Pick up the stick they leave behind and cross the bridge. Go north to the Phone Booth and turn west. Slash the grass to reveal a tunnel under the wall. This leads to a rare 2-D scene outside of a true dungeon. Go west through it and climb the ladder. Now that you're inside the castle walls, you can start looking for leaves.

Source