why is it always "on foot" not "on feet"? [duplicate]

While we normally use both our feet to walk, why is it grammatically acceptable to say "on foot" not "on feet"?

The funny thing is in German this expression is also used in singular: on foot - zu Fuß. I've just had a look into a Latin and Greek dictionary though I have no big standard dictionaries. It seems in Latin pedibus, ablative plural, with feet, was used but also pedem ferre, accusative singular, an idiomatic expression for to go or to come. Maybe the singular was a kind of artistic literary device of variation by using the singular instead of the plural and this poetic use gained general acceptance.

Both are used:

He stands on feet of clay.

and

I will follow you on foot.