Trying to understand the interactions between two different subnets on the same network

Solution 1:

The theory of the subnet mask is that it defines what part of the IP address is the network address and what part of the IP address is the host address:

10.100.0.1 - IP address;

255.0.0.0 - Subnet mask;

10 - network address, 100.0.1 - host address.

Hosts within same subnet can talk directly to each other. That means if host A and B are located within the same subnet and A wants to talk to B then A will send it's traffic directly to B. If host A wants to talk to host C which is located in a different subnet then A will have to route this traffic to the gateway which knows (hopefully) how to reach different network. So, it is up to the host to define where to send traffic:

1. Directly to the host (second host is within the same subnet)
2. To the gateway (second host belongs to a different subnet)

What happens in your case is that your "Authorized" clients have IP addresses 10.100.0.10 - 10.100.0.250 (I assume the subnet mask is 255.0.0.0). The server has IP address 10.100.0.1. To a host from the "Authorized" range this server is located in the same subnet.

If host 10.100.0.10 from the "Authorized" range wants to talk to the server - it first checks if this server is located within the same subnet or not. For the host 10.100.0.10 with subnet mask 255.0.0.0 same subnet would be all hosts within the range 10.0.0.1 - 10.255.255.254. Server's IP address happens to be in this range. For this reason a host from "Authorized" range makes an attempt to reach the server directly and (assuming they are located on the same Layer 2 network) this attempt succeeds.

In this case even though server has different subnet mask - it happens to be located in the bigger subnet (which is also a subnet for the "Authorized" clients). If your server will have different second byte in the IP address (10.150.0.1 for example) it will be unable to reply to the host from "Authorized" range, because from the server's perspective, the "Authorized" range would look like a different subnet and server would need to send traffic to a router. If there would be no router - then there would be no communication.

If you want to separate your network to the "Guests" and "Authorized" parts then you need to make them to be located in the different subnets that do not overlap.

For example:

1. "Guests" - 10.10.0.1, subnet mask 255.255.0.0
2. "Authorized" - 10.20.0.1, subnet mask 255.255.0.0

Server would be located within "Authorized" part of the network having IP address 10.20.0.100, subnet mask 255.255.0.0.

With this setup these subnets will be effectively separated from each other, since parts of IP addresses representing their subnet will differ:

1. 10.10 for Guests
2. 10.20 for Authorized

At this point communication between these subnets will be possible only via router that has interfaces in both subnets.

Also, it is worth mentioning, that while all your computers share same Layer 2 network nothing will prevent a Guests to manually assign themselves IP addresses from the "Authorized" range. This will effectively make them to be part of the Authorized network.

Solution 2:

All the "Authorized" and "Guest" machines are on the same subnet, so it's no surprise that they all can reach each other.

The server's restricted subnet mask makes it think that only the "Authorized" computers are on the same subnet, so it ARPs for them directly and can reach them.

The server thinks the "Guest" computers are on a different subnet, so it tries to send their packets to its default gateway (that is, at the Ethernet layer, it addresses them to the default gateway's MAC address; they're still addressed to the "Guest" computers at the IP layer). If the server has no default gateway defined, or if its default gateway is unreachable or misconfigured, these packets won't be able to reach the "Guest" computers.

Solution 3:

Since the packets are outside their LAN range, they send the packets to their default router. Their default router forwards them to their destination and sends an ICMP redirect to the source. Whether or not the ICMP redirect works, the traffic still gets there.

You definitely should not do things this way.