How do I link an installed app in iBooks Author?
It is possible in WebKit so it is probably possible in iBooks too. The URL depends on the application, as it's the developer who chooses it. More info here, with the example of launch center, an app which is basically a list of URLs to apps.
Try it with the free app Twitterrific (make your link to twitterrific:// – no http://!).