What does "working a system of favors" mean?

In this context, a system of favours is a feeling of obligation to grant a favour in the present due to favours received in the past.

Let me illustrate with a fictional story.

Let’s say Captain Barbell saves Ferrous Man. Now Ferrous Man owes Captain Barbell a favour. Then Ferrous Man retires from hero work to set up a family. But Captain Barbell needs him to invent a time machine to save the world, with a risk on the side that FM’s family might disappear if some butterfly flapped a bit differently in the past. FM is reluctant and could have just said, “No”, except he owes CB a big favour.

Ferrous Man could politely decline by asking the rhetorical question of whether it is worth risking his family in order to help. The expected answer, of course, is “No”. But having entered this “system of favours” due to CB’s earlier favour, he can’t just brush the request aside with a polite rhetorical question. That makes the question “harder to ask”.

"A system of favours" means a system where you pay each other in favours, not in money.

To "work" a system of favours means to "make it work for you". There's an appropriate definition on dictionary.com, but you need to click "see more definitions" to see the verb (used with object) and then click "see more" to get the right definition:

verb (used with object) [...]

28 to use or manage (an apparatus, contrivance, etc.): It is easy to work the camera in this mobile device. She can work many power tools.

32 to operate (a mine, farm, etc.) for productive purposes: to work a coal mine.