Solution 1:

It is certainly obscure, and possibly a mistake - as Edwin Ashworth reasonably suggests. I am a little forgiving of a major author and offer a simpler but similar construction to illustrate a speculative meaning.

Please consider "I answer the question for readers, and straightway their comments."

This is a short (perhaps too short, and even ungrammatical) way of saying "I answer the question for readers, and straightway {come} their comments."

From the same perspective I suggest

"... and straightway their inoffensive, not to say deprecatory and humble air"

= "and straightway {comes} their inoffensive, not to say deprecatory and humble{,} air".

My own additions are in {}.

Solution 2:

No, "their" is used quite correctly here, as a possessive. It simply modifies "air". To simplify the phrasing by removing some words, it basically says:

Their inoffensive... air towards him... is marvellous.

The thrust of the passage seems to be that the attitude ("air") of naval officers towards their commander may be defiant in public view, and perhaps when that commander is not present to take offense. But those same officers are more conciliatory in a private setting when that commander is present.

Solution 3:

I agree broadly with Anton, but will suggest that the sentence in question is elliptical. Melville is saying that the officers' attitude on deck is certainly not the strangest thing - No! the strangest thing is their inoffensive, not to say deprecatory and humble air towards him at dinner. The quote says:

It is not the least among the strange things bred by the intense artificialness of sea-usages, that while in the open air of the deck some officers will, upon provocation, bear themselves boldly and defyingly enough towards their commander; yet, ten to one, let those very officers the next moment go down to their customary dinner in that same commander’s cabin, and straightway [the observer will see that] the most strange thing is their inoffensive, not to say deprecatory and humble air towards him, as he sits at the head of the table; this is marvellous, sometimes most comical.