How big is this app update?

The size is normally the size of the full application. What I do is I open the application info screen and look at the file size. Remember, App Store updates are full versions of the file being downloaded, and the existing one replaced.

There is unfortunately no way to do this quickly, you have to check each application individually from the update screen by tapping on its icon.

As pointed out by @cksum, when finding the app on the iPhone, scroll to the bottom and it will show the size.

If got exactly what you need! If you jailbreak your device, there is an app called AppStore Update Size from the BigBoss repository, it's free and it looks like this: