Is there a word or phrase to say that someone's opinion carries more weight because they aren't biased to say it anyway?

The opposite of 'Empty vessels make the most sound.'

One answer is 'Still waters run deep.'

still waters run deep

A quiet person may be very profound, as in

• Susie rarely says much, but still waters run deep.

[The American Heritage® Dictionary of Idioms by Christine Ammer]

For a phrase, the boy who cried wolf is fitting (describing Larry Summers in this case):

Someone who claims that something is happening when it really isn't, which results in the rejection of subsequent valid claims. The expression comes from one of Aesop's fables, in which a young shepherd lies about a wolf threatening his flock so many times that people do not believe him when he and his flock are in legitimate danger. — Farlex

But it's not just an idiom, it's also a snowclone: the X who cried Y. For example:

The Boy Who Cried Inflation:
Persistent warnings about rising inflation have proved to be false alarms. Could this time be different?

I find the example of bias unhelpful at best, but in many circumstances the unbiased protagonist might be described as:

“A disinterested party”

i.e. someone with no material or reputational interest in a matter.

I provide this example, not merely to illustrate the reason for not impoverishing the language by using ‘disinterested’ to mean ‘uninterested’, but because it offers a different shade of meaning from ‘unbiased’.

A person of authority of extreme virtue can be unbiased yet have a personal stake in the outcome of a judgement, e.g. the semi-legendary Lucius Junius Brutus — unbiased but not disinterested. A prejudiced judge could, for example, be racial biased, but gain no advantage from condemning someone unjustly — disinterested, but biased.