【Troubleshooting and Questions】Kubernetes Pod-to-Pod Communication Cross Nodes

Finally, I found the root cause. I use VirtualBox to setup the VM, there is 2 network interface. One is NAT for the internet-facing, and another is Host-Only for k8s cluster communication. Flannel always uses the NAT. This is not correct. We need to config the correct network interface vaule in kube-flannel.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  ...
spec:
  ...
  template:
    ...
    spec:
      ...
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.13.1-rc1
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        # THIS LINE
        - --iface=enp0s8