How do I create a nice-looking DMG for Mac OS X using command-line tools?

After lots of research, I've come up with this answer, and I'm hereby putting it here as an answer for my own question, for reference:

1. Make sure that "Enable access for assistive devices" is checked in System Preferences>>Universal Access. It is required for the AppleScript to work. You may have to reboot after this change (it doesn't work otherwise on Mac OS X Server 10.4).

2. Create a R/W DMG. It must be larger than the result will be. In this example, the bash variable "size" contains the size in Kb and the contents of the folder in the "source" bash variable will be copied into the DMG:

   hdiutil create -srcfolder "${source}" -volname "${title}" -fs HFS+ \
         -fsargs "-c c=64,a=16,e=16" -format UDRW -size ${size}k pack.temp.dmg
   

3. Mount the disk image, and store the device name (you might want to use sleep for a few seconds after this operation):

   device=$(hdiutil attach -readwrite -noverify -noautoopen "pack.temp.dmg" | \
            egrep '^/dev/' | sed 1q | awk '{print $1}')
   

4. Store the background picture (in PNG format) in a folder called ".background" in the DMG, and store its name in the "backgroundPictureName" variable.

5. Use AppleScript to set the visual styles (name of .app must be in bash variable "applicationName", use variables for the other properties as needed):

   echo '
      tell application "Finder"
        tell disk "'${title}'"
              open
              set current view of container window to icon view
              set toolbar visible of container window to false
              set statusbar visible of container window to false
              set the bounds of container window to {400, 100, 885, 430}
              set theViewOptions to the icon view options of container window
              set arrangement of theViewOptions to not arranged
              set icon size of theViewOptions to 72
              set background picture of theViewOptions to file ".background:'${backgroundPictureName}'"
              make new alias file at container window to POSIX file "/Applications" with properties {name:"Applications"}
              set position of item "'${applicationName}'" of container window to {100, 100}
              set position of item "Applications" of container window to {375, 100}
              update without registering applications
              delay 5
              close
        end tell
      end tell
   ' | osascript
   

6. Finialize the DMG by setting permissions properly, compressing and releasing it:

   chmod -Rf go-w /Volumes/"${title}"
   sync
   sync
   hdiutil detach ${device}
   hdiutil convert "/pack.temp.dmg" -format UDZO -imagekey zlib-level=9 -o "${finalDMGName}"
   rm -f /pack.temp.dmg 
   

On Snow Leopard, the above applescript will not set the icon position correctly - it seems to be a Snow Leopard bug. One workaround is to simply call close/open after setting the icons, i.e.:

..
set position of item "'${applicationName}'" of container window to {100, 100}
set position of item "Applications" of container window to {375, 100}
close
open

There's a little Bash script called create-dmg that builds fancy DMGs with custom backgrounds, custom icon positioning and volume name.

I've built it many years ago for the company that I ran at the time; it survives on other people's contribution since then, and reportedly works well.

There's also node-appdmg which looks like a more modern and active effort based on Node.js; check it out as well.