Designing segmented LAN with fairly shared hi-speed internet access on a tight budget
- Does our idea pass your sanity check?
I would consider handling the VLAN interfaces in pfSense rather than mess up ESXi with 54 extra NIC's that it doesn't really need to know about (personal preference).
You didn't mention anything about your addressing, but assuming you're using RFC1918 private addresses (so you have full control), make sure the addressing makes sense. Personally, I like to make sure my VLAN tags match my subnets. In this instance, I would do something like this:
10.0.0.0/24 => Admin
10.1.101.0/24 => Apartment 1 (VLAN 101)
10.1.102.0/24 => Apartment 2 (VLAN 102)
...
10.1.154.0/24 => Apartment 54 (VLAN 154)
I added 100 to the VLAN's since you don't want to use VLAN 1 for Apartment 1. I also kept the admin and apartment subnets in a separate /16 so you can cleanly route the apartment VLAN's (in a single CIDR) separately in the future if need be.
How will you handle when tenants want to have inter-apartment traffic? (eg, I'm really good friends with my neighbour and I want to share the videos on my NAS with him/her). This isn't so much a technical question as a policy question.
- Any obvious flaws or pitfalls in the design?
You're going to end up with a lot of NIC's in pfSense that you're going to have to make sure can't be routed between. I haven't used pfSense for years, but in principal you need to setup a default DENY policy. This will prevent Apt X routing traffic to Apt Y, without having to manually ensure that policy is always in place. Then create the individual ALLOW policies to only let each apt out via the internet connection.
- Is it reasonable to expect 300+ Mbps routing from pfSense running virtualised?
I don't see why not when you're using gigabit NIC's. Virtualization doens't add that much overhead.
- What sort of link aggregation can/should we use for linking the two different brands of switches together - preferably providing both fault tolerance and n times 100 Mbps uplink
LACP.
- Can we expect VLAN to work as expected, combining tagged and untagged, across different brands of switches, through link aggregation and passed through to VMWare?
802.1q is a "standard"... Depending how old the switches are, if they were designed when 802.1q was still being finalized they might do something "weird", but you should be pretty safe.
- Given the proposed design, I'm not quite sure how we would go about handling routing and allowing access, to the administrative VLAN from selected apartments (of those who are managing the setup), but in the big picture this is a minor thing.
The default deny policy will prevent this, so you just need to add explicit ALLOW's to permit it.