Only use XHDPI drawables in Android app?
I guess that's a good way to go. The only downside I can think of is the resource overhead on small scale devices and possible artifacts because of the downscaling. Actually at this year's Google IO Chris Pruett recommended embedding only high resolution assets and let opengl handle the scaling.
As of Android 1.6, different densities are handled, including XHDPI (which wasn't officially added until 2.2). Your app will first look for an image matching its density, but it can look into larger "buckets" like XHDPI and then perform the scaling for you.
It's best to include specific assets for the densities you want to support. An image that's 100x100 takes 40kb; and image that's 200x200 takes 160k (uncompressed). So, any XHDPI assets used on MDPI devices have four times the amount of data that you need, which has to be handled when the app starts and your resources are prepared. Lower memory use means greater efficiency, less chance for an OutOfMemoryException.
Further, specific types of images will look bad when automatically scaled. In particular, images with fine lines or fine patterns will have their detail muddied. When you shrink the images by hand, you can choose the algorithm that best matches your needs (linear, bicubic, lanczos, etc.).
If you're worried about the time it takes to do the resizing yourself, you can incorporate a batch process or use tools such as Nine Patch Resizer: http://code.google.com/p/9patch-resizer/
I tested in a simple app (develop for Android 2.1) using only xhdpi images and it works fine in small, medium and high resolutions... even I tested in an Android 2.1 (small resolution) and it opens the imagen without problem.
Maybe the thing with the memory is true, so its necessary someone test this.