Stopping IPTables is the only way to make Kubernetes cluster work
We have an inhouse 1.17.5 K8s cluster - 5 nodes. I cannot deploy, collect logs, anything on the cluster when IPTables is enabled.
[root@lpdkubpoc01a ~]# kubectl run --generator=run-pod/v1 --rm utils -it --image quaytest.phx.aexp.com/sanupin/utils:1.0 bash
[root@lpdkubpoc01a couchbase-autonomous-operator-kubernetes_2.0.1-linux-x86_64]# kubectl exec -it hello-0 bash
[root@lpdkubpoc01a ~]# kubectl logs kube-proxy-kqs7m --tail 10 -n kube-system
[root@lpdkubpoc01a ~]# kubectl logs couchbase-operator-d9696755c-tqx57
All the above operations just hang when IPTables is enabled.
The API server logs (I can get this since it's on the same VM as my control plane and I am logged on to) show that there's problems connecting to port 10250
Trace[1253082920]: [11.90845011s] [11.906664621s] Transformed response object
E0728 21:39:10.658466 1 status.go:71] apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get", URL:"https://10.22.77.12:10250/containerLogs/default/couchbase-operator-d9696755c-tqx57/couchbase-operator", Err:(*errors.errorString)(0xc000098260)}
I0728 21:39:10.658761 1 trace.go:116] Trace[128874851]: "Get" url:/api/v1/namespaces/default/pods/couchbase-operator-d9696755c-tqx57/log,user-agent:kubectl/v1.17.5 (linux/amd64) kubernetes/e0fccaf,client:10.22.76.244 (started: 2020-07-28 21:39:06.353221799 +0000 UTC m=+80919.504899548) (total time: 4.30550213s):
Trace[128874851]: [4.305499605s] [4.303636525s] Transformed response object
I have configured 10250 on ALL my nodes:
[root@lpdkubpoc01a ~]# iptables -L | grep 10250
ACCEPT tcp -- anywhere anywhere tcp dpt:10250
ACCEPT tcp -- anywhere anywhere tcp spt:10250
[root@lpdkubpoc01a ~]# iptables -L | grep 10250
ACCEPT tcp -- anywhere anywhere tcp dpt:10250
ACCEPT tcp -- anywhere anywhere tcp spt:10250
[root@lpdkubpoc01a ~]# iptables -L | grep 10250
ACCEPT tcp -- anywhere anywhere tcp dpt:10250
ACCEPT tcp -- anywhere anywhere tcp spt:10250
But no luck yet with accessing any logs.
I have calico pod network running:
[root@lpdkubpoc01a couchbase-autonomous-operator-kubernetes_2.0.1-linux-x86_64]# kubectl get pods -n kube-system -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-58c67bc699-g2dzw 1/1 Running 0 47h 192.168.76.195 lpdkubpoc01a.phx.aexp.com <none> <none>
calico-node-9khc6 1/1 Running 0 47h 10.22.77.15 lpdkubpoc01e.phx.aexp.com <none> <none>
calico-node-fc9kp 1/1 Running 0 47h 10.22.77.12 lpdkubpoc01c.phx.aexp.com <none> <none>
calico-node-htxbh 1/1 Running 0 47h 10.22.76.245 lpdkubpoc01b.phx.aexp.com <none> <none>
calico-node-q59vd 1/1 Running 0 47h 10.22.77.13 lpdkubpoc01d.phx.aexp.com <none> <none>
calico-node-zkwtr 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
coredns-598947db54-dtsjk 1/1 Running 0 47h 192.168.76.193 lpdkubpoc01a.phx.aexp.com <none> <none>
coredns-598947db54-mrjjl 1/1 Running 0 47h 192.168.76.194 lpdkubpoc01a.phx.aexp.com <none> <none>
etcd-lpdkubpoc01a.phx.aexp.com 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
kube-apiserver-lpdkubpoc01a.phx.aexp.com 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
kube-controller-manager-lpdkubpoc01a.phx.aexp.com 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
kube-proxy-2z5rx 1/1 Running 0 47h 10.22.76.245 lpdkubpoc01b.phx.aexp.com <none> <none>
kube-proxy-55jgf 1/1 Running 0 47h 10.22.77.15 lpdkubpoc01e.phx.aexp.com <none> <none>
kube-proxy-f5k5f 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
kube-proxy-gskwj 1/1 Running 0 47h 10.22.77.13 lpdkubpoc01d.phx.aexp.com <none> <none>
kube-proxy-kqs7m 1/1 Running 0 47h 10.22.77.12 lpdkubpoc01c.phx.aexp.com <none> <none>
kube-scheduler-lpdkubpoc01a.phx.aexp.com 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
Below is my IPTables configuration on the server which happens to host a hello world pod:
When IPTables is disabled, no problems doing any operation.
EDIT:
Below is the complete non-working firewall
[root@lpdkubpoc01b ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
1459K 268M cali-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Cz_u1IQiXIMmKD4c */
790K 62M KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
790K 62M KUBE-EXTERNAL-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */
2 144 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 /* 001 accept all icmp - Puppet Managed by fw_base */
221K 16M ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 /* 002 accept all to lo interface */
0 0 REJECT all -- !lo * 0.0.0.0/0 127.0.0.0/8 /* 003 reject local traffic not on loopback interface */ reject-with icmp-port-unreachable
474K 193M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* 004 accept related established rules */ state RELATED,ESTABLISHED
2 128 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 22 /* 005 ssh - port 22 */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 8089 /* 006 splunk client - port 8089 */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 9898 /* 007 tripwire client - port 9898 */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 5666 /* 009 nrpe/nagios client - port 5666 */
789 47340 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 17472 /* 110 allow taniumclient access - port 17472 */
10 600 LOGIT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 /* 990 forward new SYN input to LOGIT chain */
771K 61M DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* 999 drop everything else */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:10250
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 cali-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:wUHhoiAYhphO9Mso */
0 0 KUBE-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */
0 0 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
Chain OUTPUT (policy ACCEPT 10956 packets, 1349K bytes)
pkts bytes target prot opt in out source destination
783K 84M cali-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:tVnHkvAo15HuiPy0 */
129K 25M KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp spt:10250
Chain KUBE-EXTERNAL-SERVICES (1 references)
pkts bytes target prot opt in out source destination
Chain KUBE-FIREWALL (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-FORWARD (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ mark match 0x4000/0x4000
0 0 ACCEPT all -- * * 192.168.0.0/16 0.0.0.0/0 /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- * * 0.0.0.0/0 192.168.0.0/16 /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED
Chain KUBE-KUBELET-CANARY (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-PROXY-CANARY (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-SERVICES (3 references)
pkts bytes target prot opt in out source destination
Chain LOGIT (1 references)
pkts bytes target prot opt in out source destination
10 600 LOG tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* 991 configure LOGIT chain to log everything as DROP INBOUND TCP */ LOG flags 0 level 4 prefix "DROP INBOUND TCP "
Chain cali-FORWARD (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:vjrMJCRpqwy5oRoX */ MARK and 0xfff1ffff
0 0 cali-from-hep-forward all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:A_sPAO0mcxbT9mOV */ mark match 0x0/0x10000
0 0 cali-from-wl-dispatch all -- cali+ * 0.0.0.0/0 0.0.0.0/0 /* cali:8ZoYfO5HKXWbB3pk */
0 0 cali-to-wl-dispatch all -- * cali+ 0.0.0.0/0 0.0.0.0/0 /* cali:jdEuaPBe14V2hutn */
0 0 cali-to-hep-forward all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:12bc6HljsMKsmfr- */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:MH9kMp5aNICL-Olv */ /* Policy explicitly accepted packet. */ mark match 0x10000/0x10000
Chain cali-INPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 4 -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:PajejrV4aFdkZojI */ /* Allow IPIP packets from Calico hosts */ match-set cali40all-hosts-net src ADDRTYPE match dst-type LOCAL
0 0 DROP 4 -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:_wjq-Yrma8Ly1Svo */ /* Drop IPIP packets from non-Calico hosts */
0 0 cali-wl-to-host all -- cali+ * 0.0.0.0/0 0.0.0.0/0 [goto] /* cali:8TZGxLWh_Eiz66wc */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:6McIeIDvPdL6PE1T */ mark match 0x10000/0x10000
1466K 269M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:YGPbrUms7NId8xVa */ MARK and 0xfff0ffff
1466K 269M cali-from-host-endpoint all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:2gmY7Bg2i0i84Wk_ */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:q-Vz2ZT9iGE331LL */ /* Host endpoint policy accepted packet. */ mark match 0x10000/0x10000
Chain cali-OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Mq1_rAdXXH3YkrzW */ mark match 0x10000/0x10000
0 0 RETURN all -- * cali+ 0.0.0.0/0 0.0.0.0/0 /* cali:69FkRTJDvD5Vu6Vl */
0 0 ACCEPT 4 -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:AnEsmO6bDZbQntWW */ /* Allow IPIP packets to other Calico hosts */ match-set cali40all-hosts-net dst ADDRTYPE match src-type LOCAL
787K 85M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:9e9Uf3GU5tX--Lxy */ MARK and 0xfff0ffff
787K 85M cali-to-host-endpoint all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:OB2pzPrvQn6PC89t */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:tvSSMDBWrme3CUqM */ /* Host endpoint policy accepted packet. */ mark match 0x10000/0x10000
Chain cali-failsafe-in (0 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:wWFQM43tJU7wwnFZ */ multiport dports 22
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:LwNV--R8MjeUYacw */ multiport dports 68
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:QOO5NUOqOSS1_Iw0 */ multiport dports 179
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:cwZWoBSwVeIAZmVN */ multiport dports 2379
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:7FbNXT91kugE_upR */ multiport dports 2380
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:ywE9WYUBEpve70WT */ multiport dports 6666
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:l-WQSVBf_lygPR0J */ multiport dports 6667
Chain cali-failsafe-out (0 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:82hjfji-wChFhAqL */ multiport dports 53
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:TNM3RfEjbNr72hgH */ multiport dports 67
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:ycxKitIl4u3dK0HR */ multiport dports 179
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:hxjEWyxdkXXkdvut */ multiport dports 2379
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:cA_GLtruuvG88KiO */ multiport dports 2380
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Sb1hkLYFMrKS6r01 */ multiport dports 6666
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:UwLSebGONJUG4yG- */ multiport dports 6667
Chain cali-from-hep-forward (1 references)
pkts bytes target prot opt in out source destination
Chain cali-from-host-endpoint (1 references)
pkts bytes target prot opt in out source destination
Chain cali-from-wl-dispatch (2 references)
pkts bytes target prot opt in out source destination
0 0 cali-fw-cali5bdd8f7a3d4 all -- cali5bdd8f7a3d4 * 0.0.0.0/0 0.0.0.0/0 [goto] /* cali:Miz_dfm_OqFqStOj */
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:j9w09IE3nQzJkJrt */ /* Unknown interface */
Chain cali-fw-cali5bdd8f7a3d4 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:RQw05lu9TEo6E9J7 */ ctstate RELATED,ESTABLISHED
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:01JI5c18EIipS498 */ ctstate INVALID
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:VyKbvKvpg3t6bqdZ */ MARK and 0xfffeffff
0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:rxc0MECUB43_dUKZ */ /* Drop VXLAN encapped packets originating in pods */ multiport dports 4789
0 0 DROP 4 -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:IhtTke9qggxJCRvo */ /* Drop IPinIP encapped packets originating in pods */
0 0 cali-pro-kns.default all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:V5EIVhFIRYorU3ee */
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Gx-cKFSZy-GfPVMs */ /* Return if profile accepted */ mark match 0x10000/0x10000
0 0 cali-pro-ksa.default.default all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:FSUAC6Xrp8hOklGS */
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:80aV6ux2xgqiIrzU */ /* Return if profile accepted */ mark match 0x10000/0x10000
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:dSsEIrLZxSINZx51 */ /* Drop if no profiles matched */
Chain cali-pri-kns.default (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:7Fnh7Pv3_98FtLW7 */ MARK or 0x10000
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:ZbV6bJXWSRefjK0u */ mark match 0x10000/0x10000
Chain cali-pri-ksa.default.default (1 references)
pkts bytes target prot opt in out source destination
Chain cali-pro-kns.default (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:oLzzje5WExbgfib5 */ MARK or 0x10000
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:4goskqvxh5xcGw3s */ mark match 0x10000/0x10000
Chain cali-pro-ksa.default.default (1 references)
pkts bytes target prot opt in out source destination
Chain cali-to-hep-forward (1 references)
pkts bytes target prot opt in out source destination
Chain cali-to-host-endpoint (1 references)
pkts bytes target prot opt in out source destination
Chain cali-to-wl-dispatch (1 references)
pkts bytes target prot opt in out source destination
0 0 cali-tw-cali5bdd8f7a3d4 all -- * cali5bdd8f7a3d4 0.0.0.0/0 0.0.0.0/0 [goto] /* cali:SsXGJ85OfhKFm0ei */
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:pP7bpa3eH6NFcNsD */ /* Unknown interface */
Chain cali-tw-cali5bdd8f7a3d4 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:uNKFNt79CGfLzpK9 */ ctstate RELATED,ESTABLISHED
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:ObEnKmaWBF0EWLU3 */ ctstate INVALID
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:JMSA5XB5i9j9eeav */ MARK and 0xfffeffff
0 0 cali-pri-kns.default all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:ogNpbuRSm1qaUhka */
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:owLwcUjuD59m1Twf */ /* Return if profile accepted */ mark match 0x10000/0x10000
0 0 cali-pri-ksa.default.default all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:89--cr3F1NqYju12 */
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:KbMS7iLpFxJMiJWe */ /* Return if profile accepted */ mark match 0x10000/0x10000
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:u5nFxLY0XdpnCUkZ */ /* Drop if no profiles matched */
Chain cali-wl-to-host (1 references)
pkts bytes target prot opt in out source destination
0 0 cali-from-wl-dispatch all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Ee9Sbo10IpVujdIY */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:nSZbcOoG1xPONxb8 */ /* Configured DefaultEndpointToHostAction */
Also, the host network is 10.22.76.0/23
Pod network is 192.168.0.0/16
Please help!
Your rule to accept traffic to TCP port 10250 will never match, because it is at the end of the INPUT chain, and appears after the rule to DROP everything. It should be moved up, before the rules that log and drop traffic.
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD_direct 0 -s 10.43.0.0/16 -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD_direct 1 -d 10.43.0.0/16 -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD_direct 2 -m comment --comment "Policy explicitly accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
sudo firewall-cmd --reload
where 10.43.0.0/16 is my K8s cluster network. In my situation this is calico bug which will fixed in 3.18 version. Iptables overwrite rules created by calico, and you should again rewrite iptables rules for calico.