What determines which challenges show on the custom class and lobby screens?

On the custom class and multiplayer lobby screens, if you hit X (on the Xbox 360), challenges are displayed.

What determines which challenges are shown using this interface, and is it different between the two screens?

The challenges on the custom class screen are generally related to some part of your custom classes. I believe the ones on the multiplayer lobby are similar. I believe they are selected based on what challenges you are close to completing - sort of a quick, heads up progress indication or hit list of challenges.

As you move through the custom class interface, it will show you challenges related to the slot you're selecting. For instance, it will show Slot 1 Perk challenges when you're highlighting the Slot 1 Perk. These change based on what you currently have in a weapon or perk slot.