Define a custom hash() method for use with ES6 maps

Is something like this possible?

No, this is a known flaw of ES6 Collections. All they do is check for reference identity, and there is no way to change that.

The best thing you can do (if hash consing the instances is not an option as you say) is not to use objects for the keys. Instead, use strings that encode the Key values, and convert back and forth between the two representations. Given that you consider your keys to be immutable, this should not pose a problem.

I've created a class called CanonMap in my library big-m to encapsulate mapping by hash instead of reference.

By default, it works with tuples, Dates, and simple objects:

const { CanonMap } = "big-m";

const myMap = new CanonMap();
myMap.set(
  ["Farooq", "867-5309"],
  36.59
);

myMap.get(
  ["Farooq", "867-5309"]
) === 36.59;

myMap.set(
  {name: "Farooq", number: "867-5309"},
  36.59
);

myMap.get(
  {number: "867-5309", name: "Farooq"} // Insensitive to key ordering
) === 36.59;

myMap.set(new Date(2012, 6, 5), "Tuesday");
myMap.get(new Date(2012, 6, 5)) === "Tuesday";

It can also be extended with a custom "canonizer" function that determines how to hash values:

import {naiveCanonize, jsonCanonize, JsonCanonMap, CanonMap} from "big-m";

// Same as default canonizer, but with greater recursion depth (default is 2)
new CanonMap([], 6);

// Canonize by ID with fallback to naive
const customCanonMap = new CanonMap([
  [{id: "TEST1", x: 7}, 77],
  [{ x: 7 }, 88]
], lookup => lookup.id || naiveCanonize(lookup));

customCanonMap.get({id: "TEST1", x: 8}) === 77; // Ignores other values, uses ID
customCanonMap.get({x: 8}) === undefined; // Uses all fields, so lookup fails

// Default canonizer with JSON.stringify
new CanonMap([], jsonCanonize);
// equivalent to
new CanonMap([], lookup => JSON.stringify(lookup));
// also equivalent to
new JsonCanonMap(); 

Finally, to implement a CanonMap that makes use of a prototype hash function on the object itself, as you described, you could do something like this:

const selfHashingCanonMap = new CanonMap([], lookup => {
  if ("hash" in lookup) {
    return lookup.hash();
  } else {
    return naiveCanonize(lookup);
  }
});