Do I need to add armv6 support when limiting apps to iOS 4.0+?

Sorry, but you aren't right. The iPhone 3G and iPod Touch 2G are able to run iOS 4 (barely) and they include a armv6 processor.

ARMv8 / ARM64 = iPhone 5s, iPad Air, Retina iPad Mini
ARMv7s = iPhone 5, iPhone 5c, iPad 4
ARMv7  = iPhone 3GS, iPhone 4, iPhone 4S, iPod 3G/4G/5G, iPad, iPad 2, iPad 3, iPad Mini  
ARMv6  = iPhone, iPhone 3G, iPod 1G/2G

iOS 4.3+ requires ARMv7. If your Deployment Target is 4.3, you can exclude armv6 support.

As of XCode 4.5 you cannot build for armv6 @Paul de Lange

You can safely drop armv6 support IF you set your deployment target to 4.3. This is because iOS 4 can be installed on armv6 devices only up to 4.2.