Past tense of "must" when meaning logical probability
Solution 1:
Try the alternative:
He must have been very clever.
It has the meaning of past.
Solution 2:
"He must be very clever" is close in meaning to "He has to be very clever", so you can use "He had to be very clever" for the past.
Similarly "He must have been very clever" is close to "He has to have been very clever", with a past form "He had to have been very clever".