is there an upper bound for damage reduction with armor and resistances?

Solution 1:

There is an upper bound for damage reduction... it's 100%.

However:

1. It's not possible to reach 100% damage reduction.

2. Effective HP (as a percentage of 'base HP') is linear with respect to resistance and armor (they're calculated separately, then multiplied). At level 60, every 3000 armor (or 300 resistance) gives you +100% EHP.

3. It may be possible to reach 95% reduction just from armor, but it's not a good idea. You'd need 57000 armor (which probably isn't possible even with BiS +armor gear). However, to get 95% DR from resistance and armor combined, you need about 10400 armor and 1040 resist all, which is nearly 1/3rd the item budget.

See this D3 forum post for much more detail.