Why Past Perfect In Specified Sentence

I am reading The Valley of Fear by Arthur Conan Doyle. And in the fourth chapter we have this part:

"It all could have been arranged. Mr Douglas was an American, or had lived in America for a long time. So had Mr Barker."
"Ames, the butler--"
"What about him? Is he reliable?"
"ten years with Sir Charles Chandos--as solid as a rock. He has been with Douglas ever since he took the Manor House five years ago. He has never seen a gun of this sort in the house"
"The gun could fit into any box. How could he swear there was no such gun in the house?"
"Well, anyhow, he had never seen one."

And according to Advanced Learners' Grammar by Mark Foley & Diane Hall - Past Perfect is usually used in: actions and states before a time in the past, a sequence in the past, or to describe unfulfilled intentions.

But in this sentence "he had never seen one" I do not really feel the usage of this tense. Is it implied that:- he had never seen one (before we spoke with him)"? Or are there any other explanation?

"How could he swear there was no such gun in the house?".

This is a rhetorical question which was posed "at some time in the past".

"He had never seen one" refers to a time before that time in the past.

Hence it is written in the past perfect.

Compare:

Q1: How can he swear there was no gun in the house? A1: He has never seen one [as of now].

Q2: How could he swear there was no gun in the house? A2: He had never seen one [as of the time he swore].

Even if it's still true that he has yet to see a gun, the tense in A2 is backshifted to suit the past sense in Q2.