Is there a word for the object or destination of a pilgrimage?

Solution 1:

Besides being the literal destination for many such pilgrimages, the word Mecca (sometimes mecca) is often used in this way – i.e., metaphorically.

Solution 2:

As I said in my comment, finding the right word to fill in the blank is not easy.

If you go on a pilgrimage to X, X is the _ of the pilgrimage.

In fact, I think the right word in this context is either destination or site. However, one makes a devotional pilgrimages to a shrine, which may provide the right word for a slightly different sentence.

Note: as in JR's answer, shrine can be used in a non-devotional sense too: "His daughter's bedroom was a shrine to Abba."