Is there a limit to how old a device/iOS version iTunes can backup?

Yes, the latest versions of iTunes (at least going back through Spring of 2016 & possibly winter 2015; I don't quite remember) no longer support the iOS 3 and lower backup protocol. I haven't found it documented anywhere, I just know when people started emailing me about the issue.

Unfortunately, any version of iTunes that speaks BackupAgent (the old protocol) won't support an iPhone running iOS 10. I actually don't know if BackupAgent2 will handle an iOS 3 backup. If it doesn't, you would need to make a backup of the iOS 3 device using an old iTunes, restore to an iOS 9 device using that same version of iTunes, make a new backup of the iOS 9 device, then update iTunes and restore to the iOS 10 device.

Depending on what you need to migrate, there may be easier ways :)