Ubuntu live CD and installing new applications onto a USB drive
Solution 1:
It is absolutely possible.
The mechanism built into Ubuntu is available. Basically, use the usb-creator tool or LiveUSB Creator (available once you've booted the LiveCD) to create a LiveUSB system on the USB stick.
Here's an install-it-from-Windows tutorial.
There are lots of posts on the Ubuntu forums about persistence; you'll want to check there in your research. These posts seem particularly useful with 9.04:
installing on a 16gb
This post suggests (while using the LiveUSB Creator) just "created one without any persistence space, then manually created an EXT2 partition labelled 'casper-rw' and on boot-up I had a system with persistence"
If the system you're on can boot from USB, that's all you need. If not, then use the LiveCD or build yourself a boot floppy (ha! -- tho you can probably build a boot CD instead of a floppy...).
Hope that helps!