Is there a mechanism in Unix (or Linux) to stop an in-progress core dump?

Generally: no, there is no way to Reliably kill a coredump.

That being said there is a possibility (at least in linux) for commercial *NIX probably no way

The possibility lies in the fact that the 3.x series of the kernel is able to interrupt file writing. One possibility is to find the thread that is doing the dumping and repeatedly send SIGKILL to it until it succeeds.

This patch series fixes the issue to some level.

Other possibility is to use the alternate syntax for the coredump_pattern. The manual says that since 2.6.19 instead of a pattern you can use a pipe and a program (with params) that will handle the dump. Ergo you will have control which dump will get written to where (/dev/null being the obvious candidate for your useless cores).

This patch also deserves a bit of attention: http://linux.derkeiler.com/Mailing-Lists/Kernel/2010-06/msg00918.html