How to save settings on bootable USB drive?

If you use the USB Disk Creator (available from the Live CD I think, see here amongst many other places for more detail) you have the option of reserving space on the USB stick for holding updates. If you create a USB stick this way then changes, including updated and added software, should be preserved between boots.

Make sure you reserve plenty of space for such updates: you will no doubt need more than you think eventually as over time more and more packages need to be updated/added and any log files that are updated over time will get stored there to. If you only have a 1Gb USB stick, I suggest reserving all that is left after the system block is allocated for updates - I would recommend a 2Gb stick or larger though to allow more space.

It is possible to install a "proper" system to a USB drive too, though using the tool mentioned above is probably much simpler. The tool creates a live-CD like system plus a shadow file for updates, which is less efficient in some ways, but should work perfectly well for the use you describe.