Encourage the JVM to GC rather than grow the heap?

You could try specifying -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio to control heap expansion and shrinking:

• -XX:MinHeapFreeRatio - when the percentage of free space in a generation falls below this value the generation will be expanded to meet this percentage. Default is 40.
• -XX:MaxHeapFreeRatio - when the percentage of free space in a generation exceeded this value the generation will shrink to meet this value. Default is 70.

You might also want to experiment with concurrent GC by specifying -XX:+UseConcMarkSweepGC. Depending on your application it could keep the heap size lower at the cost of additional CPU cycles.

The JVM is otherwise going to use the memory you specify as available when it's optimal do to so. You could specify a lower amount like -Xmx768m to keep it contained and it might run fine, although you would increase your risk of running out of memory in a heavy load scenario. Really the only way to use less memory overall is to write code that uses less memory :)

Update Java 15 added two more collectors. Some of the relevant options in ZGC are:

• -XX:+UseZGC enable ZGC
• -XX:+ZProactive and -XX:+ZUncommit Both are enabled by default. ZGC will proactively try to both free garbage objects and to release the freed memory to the OS. The rest of the options tweak how aggressively it's done.
• -XX:ZCollectionInterval=seconds How often the GC runs. Set to a few seconds to ensure that garbage is cleaned up as quickly as possible. (By default it's 0, ie no guarantee the GC will run.)
• -XX:ZUncommitDelay=seconds Sets the amount of time (in seconds) that heap memory must have been unused before being uncommitted. By default, this option is set to 300 (5 minutes). Set a lower value to get back the memory more quickly
• -XX:SoftMaxHeapSize The GC will fire more often whenever the application uses more memory than this limit. (This is probably the answer the OP is looking for.)
• -XX:SoftRefLRUPolicyMSPerMB Not sure how often this option is relevant, but it controls how long some cached objects remain in memory.

Original Answer There are four (actually, five) different garbage collectors in HotSpot, and your options are different for each one.

• The serial collector has -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio, which let you more or less directly control the behavior. I don't have much experience with the serial collector however.
• The parallel collector (-XX:+UseParallelOldGC) has -XX:MaxGCPauseMillis=<millis> and -XX:GCTimeRatio=<N>. Setting a lower maximum pause time generally causes the collector to make the heap smaller. However, if the pause time is already small the heap won't become smaller. OTOH, if the max pause time is set too low, the application may spend all of its time collecting. Setting a lower gc time ratio also generally makes the heap smaller. You are telling the gc that you're willing to dedicate more CPU time to collection in return for a smaller heap. However, this is just a hint and may have no effect. In my opinion, the parallel collector is close to untunable for the purpose of minimizing the heap's size. This collector works better (it tunes itself) if you let it run for a while and the application's behavior stays constant.
• The CMS collector (-XX:+UseConcMarkSweepGC) generally requires a larger heap to accomplish its main goal of low pause time, so I won't discuss it.
• The new G1 collector (-XX:+UseG1GC) isn't as brain-dead as the older collectors. I find that it chooses a smaller heap size on its own. It has a lot of tuning options, although I've only begun to study them. -XX:InitiatingHeapOccupancyPercent, -XX:G1HeapWastePercent, -XX:G1ReservePercent and -XX:G1MaxNewSizePercent may be of interest.

Take a look at the list of java flags.