Kubernetes minikube fails to start with error: Invalid user docker from 127.0.0.1

My server is fedora 27 with SELinux enforced. I am setting Kubernetes. My setup fails early as I can't even create clusters. Below is the error I get:

% minikube start --vm-driver kvm2 --logtostderr --v=10 
I0214 12:44:01.346196   10856 notify.go:109] Checking for updates...
I0214 12:44:01.768416   10856 start.go:96] Viper configuration:
Aliases:
map[string]string{}
Override:
map[string]interface {}{"v":"10"}
PFlags:
.........
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
..............
Defaults:
map[string]interface {}{"wantreporterror":false, "wantreporterrorprompt":true, "showdriverdeprecationnotification":true, "alsologtostderr":"false", "log_dir":"", "wantupdatenotification":true, "reminderwaitperiodinhours":24, "wantkubectldownloadmsg":true, "wantnonedriverwarning":true, "v":"0"}
Starting local Kubernetes v1.9.0 cluster...
Starting VM...
I0214 12:44:01.769248   10856 cluster.go:74] Skipping create...Using existing machine configuration
........
Found binary path at /usr/bin/docker-machine-driver-kvm2
Launching plugin server for driver kvm2
Plugin server listening at address 127.0.0.1:44533
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
I0214 12:44:01.916793   10856 cluster.go:83] Machine state:  Running
(minikube) Calling .DriverName
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x834be0] 0x834b90  [] 0s}  42660 <nil> <nil>}
About to run SSH command:
exit 0
Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain

Here I need to enter to get the prompt. When listing running processes containing minikube, I find this:

188:qemu     10734     1  9 12:43 ?        00:00:37 /usr/bin/qemu-system-x86_64 -machine accel=kvm -name guest=minikube,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-minikube/master-key.aes -machine pc-i440fx-2.10,accel=kvm,usb=off,dump-guest-core=off -m 1954 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid fef86dfc-66d5-4033-87a2-106f6b470737 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-minikube/monitor.sock,server,nowait -mon` chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot menu=off,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x4 -drive file=/home/gabx/.minikube/machines/minikube/boot2docker.iso,format=raw,if=none,id=drive-scsi0-0-2,readonly=on -device scsi-cd,bus=scsi0.0,scsi-id=2,drive=drive-scsi0-0-2,id=scsi0-0-2,bootindex=1 -drive file=/home/gabx/.minikube/machines/minikube/minikube.rawdisk,format=raw,if=none,id=drive-virtio-disk0,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=a4:d6:75:fd:a0:af,bus=pci.0,addr=0x2 -netdev tap,fd=28,id=hostnet1,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=a4:d6:75:fd:a0:af,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x7 -msg timestamp=on

So a process has been started by qemu user, but I don't know if this is the way it shall be.

Some debugging help:

$ journalctl -u sshd.service
Invalid user docker from 127.0.0.1 port 34998

$ journalctl -u libvirtd.service
Starting Virtualization daemon...
Feb 14 12:29:12 dahlia systemd[1]: Started Virtualization daemon.
Feb 14 12:29:13 dahlia dnsmasq[16256]: read /etc/hosts - 3 addresses
Feb 14 12:29:13 dahlia dnsmasq[16256]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Feb 14 12:29:13 dahlia dnsmasq-dhcp[16256]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Feb 14 12:29:13 dahlia dnsmasq[9727]: started, version 2.78 cachesize 150
Feb 14 12:29:13 dahlia dnsmasq[9727]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify
Feb 14 12:29:13 dahlia dnsmasq[9727]: warning: no upstream servers configured
Feb 14 12:29:13 dahlia dnsmasq-dhcp[9727]: DHCP, IP range 192.168.39.2 -- 192.168.39.254, lease time 1h
Feb 14 12:29:13 dahlia dnsmasq-dhcp[9727]: DHCP, sockets bound exclusively to interface virbr0
Feb 14 12:29:13 dahlia dnsmasq[9727]: read /etc/hosts - 3 addresses
Feb 14 12:29:13 dahlia dnsmasq[9727]: read /var/lib/libvirt/dnsmasq/minikube-net.addnhosts - 0 addresses
Feb 14 12:29:13 dahlia dnsmasq-dhcp[9727]: read /var/lib/libvirt/dnsmasq/minikube-net.hostsfile
Feb 14 12:31:48 dahlia libvirtd[9650]: 2018-02-14 12:31:48.661+0000: 9650: info : libvirt version: 3.7.0, package: 3.fc27 (Fedora Project, 2017-12-04-17:14:09, buildhw-06.phx2.fedoraproj
Feb 14 12:31:48 dahlia libvirtd[9650]: 2018-02-14 12:31:48.661+0000: 9650: info : hostname: dahlia.thetradinghall.com
Feb 14 12:31:48 dahlia libvirtd[9650]: 2018-02-14 12:31:48.661+0000: 9650: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error

Last line is printed each time I start minikube.

NOTE:

  1. sshd is running on a custom port, users are login with no password but a key, and sshd config has this line:

    Allowuser User1 user2

  2. Iptables is active with among other lines:

    Chain FORWARD (policy DROP)

  3. two dnsmasq processes are running, one owned by root and other nobody

    154:nobody 1681 1 0 Feb13 ? 00:00:01 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/minikube-net.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

  4. There is no docker user nor group on my machine

  5. running kubectl print errors:

    % kubectl version

    Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"archive", BuildDate:"2018-01-15T15:56:33Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
    The connection to the server localhost:8080 was refused - did you specify the right host or port?

    % kubectl cluster-info

    Kubernetes master is running at http://localhost:8080 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. The connection to the server localhost:8080 was refused - did you specify the right host or port

Thank you for any help and hint about how minikube command is supposed to run and why this docker user which can't connect? Can it be at the root of my troubles? Why this message connection to the server localhost:8080 was refused when running kubectl ?


rm -rf ~/.minikube

and try again.

I have given up on kvm2 on fedora 27 and gone back to VirtualBox for vm-driver.

Regarding your other questions-

  • qemu is a virtual machine application. This is what runs the kubernetes bits for minikube, and what backs the kvm2 vm-driver.
  • there does not need to be a docker user on your machine. The docker user exists on the qemu machine.
  • minikube ssh is short for

    ssh -i ~/.minikube/machines/minikube/id_rsa docker@

This failure essentially means that the ssh key inside the qemu vm differs from what should be the same key on your machine for this vm. In other words, things are hosed, you should start over.