How to create custom map in AOE3?

That's rather complex.

You can create a scenario ingame, but in order to create a custom map, you must obtain a certain skill: RMS. It's the abbreviation for Random Map Scripting. There's no editor out there to create a custom map in AOE3, you must do it all by yourself, you must write it yourself.

If you really are eager to learn that, you can visit those sources:

• An overview about the process of creating a custom map in AOE3.
• Library about RMS.
• Forum about RMS concerning AOE3.

If you follow everything written down there, you are ready to go and you can start off by creating a cool map. Wish you good luck!