What does “it” do in “the man whose unique right it was to assist the King to mount his horse”?

This 'it' is called an expletive (or dummy) pronoun, and fills the subject of the relative clause, and is related to the infinitival clause to assist the King to mount his horse.

This is part of a general pattern of predicates that can take clauses as their subjects. Since clauses tend to be long, we tend to like to put them towards the right edge of the sentence in English. This is commonly called extraposition. The it then fills the space where the clause would normally have been, because English finite clauses must have subjects.

Here's the pattern without the relative clause:

  1. To assist the King is his right.
  2. It is his right to assist the King.
  3. To assist the King makes him happy.
  4. It makes him happy to assist the King.
  5. To assist the king is important.
  6. It is important to assist the King.

As you can see in each of the pairs of examples, the infinitival clause can either be the subject of the sentence, or appear after the main predicate. It is in that latter case that the the expletive pronoun is used to take the place of the subject.

So in your specific example the structure is somewhat more complicated because the the clause it was to assist the king is a relative clause modifying man, but the basic structure is the same.

But your intuition about your two versions of the structure are completely correct, although that's an artefact of the main predicate of the sentence being a noun phrase (his right); with the other examples I gave of course, the inversion is not possible, i.e., you can't say *Important is to assist the King.

Some comments on relative clauses

The reason the structure appears to be different from the simpler examples above comes from the fact that it is part of a relative clause, and in particular, a relative clause formed with a possessive relative pronoun. This adds quite a bit of complexity to the structure, but it is still fundamentally the same. To see this it's necessary to explain a little about relative clause structure.

Semantically and syntactically, a relative clause is a clause which modifies a noun phrase (called the head of the relative clause). Because of this modification, the clause must have something "missing" that is related to the head, usually via a relative pronoun. We can think of the relative pronoun as having moved from the missing position much in the same way that an interrogative pronoun moves from some position in the sentence to form a question. We can think of this as involving the following steps, where the symbol t represents the empty position from which the relative pronoun has moved. Here's a derivation for the relative clause the man who I saw.

  • [I saw who] move relative pronoun->
  • [who [ I saw t ]] combine with 'man' ->
  • [[man] [who [ I saw t ]]] combine with 'the'->
  • [the [[man][who [ I saw t ]]]]

This is the simplest kind of relative clause. But we can also form a relative clause where the relative pronoun doesn't move by itself (because it can't) but instead takes along a larger phrase that contains it. The simplest version of this involves possessive relative pronouns, as in the man whose brother I saw. The derivation is basically the same:

  • [I saw whose brother] move relative pronoun->
  • [whose brother [ I saw t ] ] combine with 'man' ->
  • [[man] [whose brother [ I saw t ]]] combine with 'the'->
  • [the [[man][whose brother [ I saw t ]]]]

In these examples, I deliberately made phrase containing the relative pronoun move from the object position of the verb, but relative pronouns can move from most positions in the clause. In your example, the phrase containing the relative pronoun is the main predicate of the relative clause itself. So the derivation proceeds as follows:

  • [it is whose unique right to serve the King] move relative pronoun->
  • [whose unique right [ it is t to serve the King ]] combine with 'man' ->
  • [[man] [whose unique right [ it is t to serve the King ]]] combine with 'the'->
  • [the [[man] [whose unique right [ it is t to serve the King ]]]

In the structure without the it the base form is slightly different, because the phrase containing the relative pronoun has moved from the subject position of the clause (as you point out in your example which is like 7):

  1. His unique right is to assist the King.

But the derivation proceeds the same as above, except that the position that the phrase containing the relative pronoun moves from is different.

  • [ whose unique right is to serve the King] move relative pronoun->
  • [whose unique right [ t is to serve the King ]] combine with 'man' ->
  • [[man] [whose unique right [ t is to serve the King ]]] combine with 'the'->
  • [the [[man] [whose unique right [ t is to serve the King ]]]