How are Arbitrary Code Execution (ACE) exploits discovered (such as in OoT)?
Solution 1:
From my current understanding, code-savvy speedrunners read through the game's code (even as low down as assembly language)
This is basically the only way to do it. Arbitrary code execution bugs involve injecting machine code, often at the byte-level, so they fundamentally require a deep understanding of the game and architecture internals.
To find them, bugs where the game crashes or memory is corrupted are usually a great kicking-off point. Usually these are found by random community members, posted to some forum, and then analyzed by other community members who have the expert knowledge required to exploit the bug. In other cases, the expert will spend many many long hours stepping through the assembly code, looking for exploits.
The specifics of how buffer overflow and other exploits work is well beyond the scope of this site. We have an entire other Stackexchange for that sort of thing: ReverseEngineering.SE (RetroComputing.SE is another great resource, if you're asking about older consoles)