How do you call it when someone makes faces while trying to remember something

You can use the expression "to strain one's facial muscles".

(ref. 1) The effort strained his facial muscles. She returned the smile. “Just fine. I have the latest data on compression strength and fracture resistance for the honeycomb structure.

(ref. 2) Kuster~Laap strained his facial muscles searching his memories. »I'm not very familiar with these convoys.

You can also use "screw one's face up". (OALD¹)

(ref.) Monk screwed his face in concentration . “ How about Dixie ? That's better yet . ” “ Dixie what ? ” Barney asked patiently . “ Dixie nuthin ' , ” Monk said . “ Just plain Dixie . ”

¹_{After correction; see user Mitch's comment}