What's the optimal Handbrake setting to convert AVI for editing in iMovie/Final Cut?

Don't forget the built-in Automator actions for encoding. The video plugin is very nice and has pre-sets starting with 480p (VGA size) and up.

The Encode for: Greater compatibility toggle keeps the audio and video at a lower bit rate. Depending on what your tolerance for "less degradation" is, you might experiment with Higher quality if you see too many H.264 artifacts on the default setting. It's unlikely as the encoder has a lot of room with 1.5 Mbps as the ceiling for the video data rate.

You could set up a few services to test the various settings and then make a folder action to apply your favorite encode setting to anything you drop in that folder.

Automation is fun—you might even be able to set up a folder action to do your uploading as part of this process.

I would suggest you convert it to MP4 with the H.264 Codec. Handbreak is a pretty straight-forward applicaiton in terms of video conversion; in the video tab you can see the input and output size in pixels (so your video getting reduced in size should be avoided in a single glance), the framerate, which is best to keep on "same as source" and your target quality. Here is where the quality/size question arises.

As explained in the Handbrake wiki:

With the average bitrate or target size methods, you control the size of the output file but give up control over the video's quality.

Constant quality mode does the opposite; you specify a quality level and HandBrake adjusts the bitrate (that is, the size) to meet it.

https://trac.handbrake.fr/wiki/ConstantQuality

So basically you can select a target size, and your video will be squeezed to that, at the expense of quality, you can choose an average bit rate, and all your video will be set those many kilobytes per second, or you can select a Constant Quality and handbreak will make sure your video looks as you want it, regardless of size.

Its a tough decision, and depends greatly on your source video. If youre uploading to the internet I's suggest you select "2 pass encoding" and set the bit-rate to arround 1MB. Because this depends a lot on your video (eg. vids with lots of grain and movement usually need a higher bit rate) this might be enough or might not (I'm pretty sure it will though :))

Good luck!