pfSense: Possible to traffic capture the actual WAN port?

Yes, you have to do it via SSH and not the GUI Diagnostics>Packet capture screen though, as it'll capture on the ng0 PPPoE interface. Just tcpdump as normal on the actual Ethernet interface being used for the PPPoE connection. Send it to file and pull it across to a machine with Wireshark for easier analysis. Something like:

tcpdump -i em0 -s 0 -w /tmp/pppoe.pcap

to grab everything on em0 to the file /tmp/pppoe.pcap


To expand upon Chris's answer, the actual command i used was:

>tcpdump -i xl0 -e 'not (pppoes and ip)'

Where

-i [interface]

specifies my WAN interface, which for me is xl0

-e 

include the source and destination MAC addresses, so i can see who is sending what packets to who

'not (pppoes and ip)'

is an expression that says i want to exclude ip packets inside of a PPPoE Session packet.

Note: tcpdump has options to monitor just the PPPoE "discovery" packets (PADI, PADO, PADR, PADS):

>tcpdump -i xl0 -e pppoed

This would show just the PPPoE discovery packets. i preferred the other syntax, because once the PPPoE session is established, i then get to see login/password authentication with my ISP, as well as IP/DNS/route/MTU information.


This led me to discover that pfSense is sending a PPPoE Active Discovery Terminate (PADT) packet out of the blue. That is why my link is going down - pfSense is hanging up randomly.

The PADT tag indicates the end of the session, and nobody is allowed to send anymore packets on that session. From RFC2516 - A Method for Transmitting PPP Over Ethernet (PPPoE):

5.5 The PPPoE Active Discovery Terminate (PADT) packet

This packet may be sent anytime after a session is established to
indicate that a PPPoE session has been terminated. It may be sent by
either the Host or the Access Concentrator. The DESTINATION_ADDR
field is a unicast Ethernet address, the CODE field is set to 0xa7 and the SESSION_ID MUST be set to indicate which session is to be terminated. No TAGs are required.

When a PADT is received, no further PPP traffic is allowed to be sent
using that session. Even normal PPP termination packets MUST NOT be
sent after sending or receiving a PADT. A PPP peer SHOULD use the
PPP protocol itself to bring down a PPPoE session, but the PADT MAY be used when PPP can not be used.

My next question will, of course, be

Why is pfSense hanging up randomly?


The answer is that an ATM switch inside Bell Canada's concentrator is faulty. The card would randomly reset, losing any memory of my PPPoE session.

Meanwhile, my router doesn't know its PPPoE session is no longer valid, and continues to try to communicate over that same SessionID. The concentrator, not recognizing that session, ignores any packets coming from me.

pfSense, after sensing 10 seconds of no traffic, begins sending LCP Echo-Requests, at 10s intervals, to the remote network. After 40 seconds of not getting any responses, pfSense brings down the PPPoE session and starts a new one:

PADI

The concentrator, seeing a broadcast requesting to start a session, responds; and shortly thereafter the new PPPoE session is established.

Current (faulty) ATM switch MAC:

00:90:1a:a0:a1:f4 Unisphere Solutions (was: redstone communications)

New ATM switch MAC:

to be installed 8/6/2010