How to emulate an inline host in the middle of 2 other hosts in VMware Workstation?

In your schema we can see that you need two separate LANs. Let's call them LAN-Attacker and LAN-Victim. On the Attacker VM and Victim VM you will need a single virtual network adapter for each VM. On the IPS VM you will need two virtual network adapters. You can add and configure the adapters in the Virtual Machine Settings Window on the Hardware tab.

Do not be confused by the fact that there are two separate LANs. They can be in the same IP subnet if your IPS is going to perform as a bridge (layer 2 device). They can also be in two different IP subnets if the IPS is going to act as a router (layer 3). This depends just on your networking configuration inside the virtual machines.

Now there are two options how to configure and connect the two LANs.

LAN Segments

In VMware Workstation 8.0 and newer you can use LAN Segments for local virtual networks which need to communicate only with virtual machines. This configuration is considerably easier. In the Virtual Machine Settings Window on the Hardware tab select a Network Adapter and click on the LAN Segments buttons. Create the two LAN Segments LAN-Attacker and LAN-Victim. On each virtual network adapter select the corresponding LAN Segment.

Be aware of the fact that the machines which are connected only to LAN Segments will be able to communicate (over network) neither with the physical host nor with the external physical networks.

Virtual Networks vmnetx

In all versions of VMware Workstation you can use virtual networks. You can configure them using the Virtual Network Editor (in the Edit menu). The virtual networks are called vmnetx where x is the number of the virtual network. Either configure unused ones or create new ones. There are three types of virtual networks:

• Bridged - They are connected to a physical network your physical host has access to on the layer 2. The virtual machines connected to this vmnet then look like they are directly connected into the physical network.
• NAT - There is a virtual network but the physical host performs dynamic NAT so the machines connected to this vmnet can communicate with physical networks. (and each to other - see below)
• Host-only - This vmnet is like NAT but without the NAT and access to the outside physical networks. So the machines connected to this vmnet can communicate only each to others including the physical host if you select the option "Connect a host virtual adapter".

In your case you will use either Host-only or NAT (if the machines need to communicate with the outside world). In fresh VMware Workstation installation vmnet0 - vmnet2 are pre-defined so you can probably use vmnet3 as LAN-Attacker and vmnet4 as LAN-Victim.

In the virtual machines you then assign the corresponding vmnets to the virtual network adapters a similar way like LAN Segments above just select the option "Custom: Specific virtual network" instead of "LAN Segment".