PF OpenBSD states

There are a number of things you can do here.

To see which hosts are responsible for the large number of state table entries you can do pfctl -vs state.

To add more state table entries you can do what you suggested (set limit states to a bigger number), but if there is an underlying issue you probably don't want to do that.

You can also consider adjusting the state timeout values (set timeout), possibly using adaptive timeouts, in order to get rid of old/stale states more quickly.

See the manpage for pf.conf and the manpage for pfctl for more information

[Ref: Hitting the PF state table limit]

The PF State tables set the limit of connections that have been authorized, and thus limit the number new connections that the firewall will accept. You may have excess bandwidth available, but if there is no free capacity in the State Tables, then your firewall becomes a bottle-neck.

The configured limits for state information is accessible through pfctl:

# pfctl -sm
states        hard limit    10000   
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   200000

The above limits pre-sets the allocated memory of the defined structures, such that they are always available, and it also limits growth of the said data structures. If your firewall traffic exceeds the above settings, then performance will be affected.

It is now important to monitor the effects of your traffic on the counters for the above limits. The generic -si output gives us clues to where to further investigate potential bottle-necks in our firewall.

# pfctl -si
Status: Enabled for XXXXXXXXXXXXXXXX          Debug: Urgent

State Table                          Total             Rate
  current entries                       34
  searches                        96379206           15.2/s
  inserts                           726196            0.1/s
  removals                          726162            0.1/s

On the above gateway, connected to two infrequently used laptops, the current entries is very low relative to the hard limit 10000 above. Obviously, the current entries will fluctuate due to use, and on a busier gateway may fluctuate significantly.

[Ref: Hitting the PF state table limit, Open BSD state hard-limit reached]

An important counter to monitor from pfctl -si is the memory counter. The same details should be available through systat pf.

From an active gateway linking our 6 sites, we get the following from a standard install, no modification to state tables.

# pfctl -si | grep memory
  memory                          209230            0.1/s

The counter highlights how often PF has failed at least one of the pool(9). The higher the number, the higher the frequency of incidences where packets arriving at the firewall have most likely been dropped due to one of the hardware limits.

Our above example shows 209,230 times the memory limit was hit.

The next review is to check with Kernel memory allocations, using vmstat. To narrow our search down to the effects on the pf state table we check the entry for pfstatepl.

Below, we grab the lines with state or Fail (so we can get the column headers)

# vmstat -m | grep -E "state|Fail"
Name      Size  Requests  Fail   InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg  Idle
pfstatepl  296  213123877 209235  5075  1050     0  1050 1050     0   2308   526
pfstatekeypl 
pfstateitempl

pfstatepl is the label for memory allocated for the struct pf_state (/usr/src/sys/net/pf_ioctl.c) The failures do seem to be significant.

pfctl -vvsi | grep congestion