Possible to integrate Keynote in custom UI?
Solution 1:
Yes, on OS X - the technology used to make most programs is Objective C where messages get passed from the UI to the logic and a model-view-controller idiom is used.
If you are skilled, you can do things like patch the binary to swizzle a method or use SIMBL to modify apps that do not have a plug in system like Keynote.
These reverse-engineering feats require some very detailed skill and can break each time the app is updated, so something that is theoretically possible becomes unlikely to happen as something you can download for free with the work already being done for you.
Your commend about the quicklook ability to preview keynotes is where I would start if you wanted to develop a true application to read keynote files. From there, you would need to reverse engineer the storage format by discovering that a .key file is really just a fancy zip file and go from there.
Solution 2:
Don't forget that a Keynote can be exported to the QuickTime .mov
format. This preserves transitions and builds, and even click-to-advance. I'm certain this could be embedded into a custom UI.
Solution 3:
As you said, Quick Look is the way to go. But the support for eg. advanced animations isn't perfect I think. You could also try to export your .key to HTML if you have direct access, and via injected js take the control over the playback (The html is actually images that are animated through js/css). Put it into a NSWebView or something.