Odd British pluperfect subjunctive construction?
Solution 1:
As FumbleFingers noted, keep is acceptable. As to which is 'more right', if it is possible to have degrees of correctness, that is a question of style rather than grammar.
In the antediluvian days of my youth, "I insisted that he kept..." could mean only that he kept ... at some past time, and at some later time I insisted on the truth of this. "I insisted that he keep ..." was the required form if I was reporting my past insistence on a subsequent pattern of behaviour from him. This difference is perhaps clearer in these two sentences:
I insisted that my parents lived in London. My actual words wore "My parents live in London, I tell you!"
I insisted that my parents live in London. My actual words to my parents were "You must live in London. I insist on this".
Most speakers of BrE today would use the past-tense form in both situations.
Solution 2:
It can go either way in different contexts.
If the court had insisted that everyone keep a straight face, the meaning of it is what you're supposing: that the court is, in the time frame referenced, insisting that portrait subjects remain serious in their portraits.
If the court really had insisted that everyone kept a straight face, the object of the instance could be an action in the past and be correct. In this sense, kept could be used to explain that the court had insisted that everyone had remained serious during [during an implicitly referenced event], possibly because someone was questioning or criticizing the court on a point to the contrary. This isn't what's intended by use of the word in the article.
It's important to remember that the tense of the verb that is being performed by your subject does not have to agree in tense with the object of that verb (if the verb is transitive, like in this case). For example, if I'm talking about a baseball game I played last week, it would sound ridiculous to say something like, "In the second inning, coach insisted that I stole a base." The reason for this is simple: the verb "insisted" indicates that the insisting was done in the past, but the object of the verb is something that happened in the present of that context. "Coach insisted that I steal a base," is the correct way to say it because the object of my coach's insistence is a mandate, and we typically use the present tense of verbs for mandates. ("Eat all your veggies," "Don't run with scissors!", "Visit your grandparents on Saturday.")
For the second context I mention, the word "kept" as the object of the verb "insisted" isn't serving as a mandate--it's serving as a regular verb. If I insist that I did something, I'm insisting (in the present) that I did something (in the past, relative to the context of the insistence happening in the present). If I insisted that I will do something, I insisted (in the past) that I will do something (in the future, relative to the context of the insistence in the past). And so on. I would only use "kept" as the object of the verb like in the question if I were emphasizing that someone kept something by insisting on it.