Handbrake: How to retain sharpness when down-sampling video during an encode?

You've provided no details of your Handbrake version or options, nor the codecs and bitrates of your sources or transcode results, but I'm going to take a stab at it. What you refer to as a loss of 'sharpness' is the result of three generations of lossy video encoding. In short, things got blurry because you decimated the quality of the original source.

First, let's manage expectations. Going from 1280x720 to 852x480 is a >50% reduction in number of pixels, but it will most likely not result in a 50% smaller file AND one which meets your quality standards. Maybe if you were encoding the two files from the same original source, but you shouldn't be surprised that a higher resolution/bitrate file can yield a better picture (projected into any sized window) than a reduced bitrate/scaled/recompressed version.

My guess is your video went from a high quality 720p/1080p source (high bitrate h264 or VC-1, likely BluRay) to your file (720p mid-range bitrate h264) to this attempted transcode (480p lower bitrate h264). This file has already been through put through a lossy compression algorithm twice and you're surprised when a the third round of compression combined with a bitrate reduction and scaling made the file less sharp? That's just common sense.

If you have access to the original source ripping from there will definitely get you the smallest file that meets your quality standards. Removing just one lossy generation will yield significantly sharper results. If a much smaller file is really important (say cause you bought a 16GB iPad and want to maximize the quantity of videos rather than quality of each) feel free to arbitrarily set an average video bitrate and waste time with 2-pass encoding trying to get the file close to your target size.

But if you're actually concerned about quality you should set the x264 settings to 'Constant Quality' and render a 30 second clip multiple times using various CRF values (Constant Rate Factor - an arbitrary unit) to empirically find which CRF value meets your needs. Although HandBrake gives you a huge slider from 0 to 51 for CRF, sane values are in the 19-24 range (default is 20). After selecting your source you can change the toggle from Chapters to seconds and choose any 20-30 second range you'd like to compare. Rename the destination file (e.g. "test-crf20.m4v" or whatever) click 'Add to Queue', adjust CRF a little rename and repeat. Batch render half a dozen and compare each of them to the resized VLC window. Pick the smallest file which doesn't look degraded and try encoding the entire video with that CRF and see if you get the file size savings your looking for with a video that's still 'sharp-enough'.