iSCSI separation from Ethernet via VLAN
What I'm worried about is whether or not I should better separate non-iSCSI traffic from the 172.16.200.X subnet with firewall rules so that port 22 (ssh) is blocked out on all servers.
If you use DNS names to connect to other servers and those resolve to the LAN addresses you should be fine. (Alternatively, you can use the LAN IP addresses directly, of course.)
If you really want to disable all non-iSCSI traffic on the SAN you'll need to either
- configure all services to bind to LAN IP addresses only
- use local firewalls on the servers to filter all unwanted traffic
- use ACLs on the iSCSI switch ports to filter all unwanted traffic
If you do filter, just permitting iSCSI and denying everything else is the correct approach.
Does having ethernet traffic on an iSCSI network create problems (either lag or errors?)
The main reason to separate LAN and SAN traffic is that you want to make sure your storage network cannot clog up at all events. If it did, it would rapidly cause I/O errors, in turn causing data loss and even corruption. A (very) low volume of stray traffic isn't anything to really worry about.
However, I'd use the ACL approach if service bindings (#1) aren't practical or if there are other server administrators taking things lightly. E.g. dynamic DNS updating very easily puts your iSCSI IPs in DNS and any inter-server traffic can quickly land in the SAN.