What is the equivalent of chmod 777?

Using cacls you can do this same type thing, example.

cacls myfile.txt /g everyone:f

Roughly:

icacls notepad.exe /grant Everyone:F

Note that this isn't exactly the same as deny ACL entries override allow entries, so if there are any deny ones you may need to remove them.

Generally, I honestly have to question the motives of what you are attempting to do. There should never be a reason to replace core operating system files with other ones.

You can easily associate text files with Notepad2 without replacing notepad.exe which should be the preferred way of dealing with this.