Fastest way to get to Hyrule castle town from Laynaru tower?

I have been trying to get to Hyrule castle town (for unexplainable reasons), but every time I get killed by guardians. They seem to spawn more nearby, and I can't get the map from the tower! What is the fastest way to get to Hyrule castle town? If possible, I would like to not deal with touching any guardians. What I need:

• A path that won't take me by guardians but is semi-optimized
• Horse-friendly (optional)

To better illustrate what I said in my comment to the question, I decided to write an answer. Ideally, your route would have you avoid Hyrule Fields altogether because that's where a lot of guardians like to roam. So thinking about that, what I would propose is a route similar to this.

Basically I would head out of the tower and get myself to be on the mountain range directly east of Castle Town. The yellow line means that I would then attempt to fly in to the north-east corner of the town. I don't remember if there are guardians there or not so you would have to watch and see. I also don't know what kind of stamina you have but I imagine with enough food/potions it could be done. If the flying is not possible, an alternative suggestion would be to get into the water north of the north-eastern part of town and try to climb up the cliff to it. Neither of these ideas have been tested or optimized and there is a probably a better solution, but one thing I loved about this game is you don't have to take the obvious route to get somewhere. James Bond that ish and get creative with it.