Local and public IPs on the same switch?

It's possible, but it's not advised, especially when there are better ways.

First, switches don't care about your IP addresses, they care about your MAC addresses. They're "layer 2" devices. IP addresses are layer 3, so they're pretty much irrelevant to the switching side of things.

To make sure that I've got your infrastructure correct, you have servers A, B, C, and D. Each one of them has 2 NICs. You want to take NIC#1 on each server, and configure them with external, internet facing IP addresses, then take NIC#2 on each server, and configure them with private IPs?

I have to ask why, at this point.

If it's for dedicated bandwidth, you would be better served to bond NIC#1 and NIC#2 into one logical interface, which can double the bandwidth.

If it's for security, then you'll have to give some more information, because there's no added security from using private IPs on a switch with public network connections. You aren't going to be broadcasting anything to the internet*, but at the same time, any network broadcasts from the network cards on the private IP block (things like ARP/RARP requests and the like) will get sent to your upstream router. It won't forward them or respond, but it certainly doesn't do anything for you.

(* - probably not, anyway)

Now, if you're still security conscious, why not use VLANs on the switch to segregate the external network from the internal network? The VLANs will create two logical switches*, which will prevent the leaking of your layer 2 broadcast info to the router, and in general, segregation of "private" networks into distinct logical layer 2 networks is preferable.

(* - I'm simplifying, but in essence, this is what it does)

Under some situations, I'd be okay with doing this if the switch were a relatively modern switch that has 802.1q VLAN tag support.

I'd create one vlan for the public traffic and one vlan for the private traffic, and use private addresses for the private traffic. Put one set of ports into one vlan (all untagged), and another set of ports into another vlan.

Put your public interfaces / addresses into one vlan, then put the private addresses into the other.

Beware that if any of the public facing computers gets broken into, you'll be exposing the private interfaces of the other hosts on the private network to that rooted computer. You should treat all of these interfaces as untrustworthy when you're setting up and vetting up your security model.

Thinking about this a little -- the vlan model is about the same as setting up ip address aliases and putting everything into the same broadcast domain / vlan. In some ways it is even better because you can do LACP between the computers and the switch and get better performance and link redundancy. But I still wouldn't do it because it is ugly to rely on your router / firewall to drop traffic to RFC1918 space.