Did I just get hacked?
Solution 1:
EDIT 2:
there is one good reason why this post is attracting so much attention: you managed to record the whole, live session of an intruder on your PC. This is very different from our everyday experience, where we deal with the discovery of the consequences of his actions and try to redress them. Here we see him at work, see him having some problems with establishing the backdoor, retrace his steps, work feverishly (perhaps because he was sitting at your desk, as suggested above, or perhaps, and in my opinion more likely, because he was unable to make his malware run on the system, read below), and try to deploy fully self-contained instruments of control. This is what security researchers witness daily with their honey traps. For me, this is a very rare chance, and the source of some amusement.
You have definitely been hacked. The evidence for this does not come from the snippet of the auth.log file you displayed, because this reports an unsuccessful login attempt, occurring over a short time span (two secs). You will notice that the second line states Failed password, while the third one reports a pre-auth disconnect: the guy tried and failed.
The evidence comes instead from the content of the two files http://222.186.30.209:65534/yjz and http://222.186.30.209:65534/yjz1 which the attacker downloaded onto your system.
The site is currently open to anyone to download them, which I did. I first ran file on them, which showed:
$ file y*
yjz: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.2.5, not stripped
yjz1: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.9, not stripped
Then I brought them onto a 64-bit Debian VM I have; an examination of their content thru the strings command revealed much that was suspicious (reference to various well-known attacks, to commands to be substituted for, a script that was clearly used to set up a new service, and so on).
I then produced the MD5-hashes of both files, and fed them to Cymru's hash database to see whether they are known agents of malware. While yjz is not, yjz1 is, and Cymru reports a probability of detection by anti-virus software of 58%. It also states that this file was last seen some three days ago, so it is reasonably recent.
Running clamscan (part of the clamav package) on the two files I obtained:
$ clamscan y*
yjz: Linux.Backdoor.Gates FOUND
yjz1: Linux.Trojan.Xorddos FOUND
so we are now certain that standard Linux software can identify it.
What should you do?
Though rather new, neither system is very new, see this Jan. 2015 article on XorDdos, for instance. So most free packages should be able to remove it. You should try: clamav, rkhunter, chkrootkit. I have Googled around, and seen that they claim to be able to spot it. Use them to check on the predecessor's work, but after running these three programs you should be ready to go.
As for the larger question, what should you do to prevent future infections, Journeyman's answer is a good first step. Just keep in mind that it is an ongoing struggle, one that all of us (including me!) may very well have lost without even knowing it.
EDIT:
At Viktor Toth's (indirect) prompt, I would like to add a few comments. It is certainly true that the intruder encountered some difficulties: he downloads two distinct hacking tools, changes their permissions several times, restarts them several times, and tries many times to disable the firewall. It is easy to guess what is happening: he expects his hacking tools to open a communication channel toward one of his infected pcs (see later), and, when he does not see this new channel spring up on his control GUI, fears his hacking tool is being blocked by the firewall, so he repeats the installation procedure. I agree with Viktor Toth that this particular stage of his operation does not seem to bring the expected fruits, but I would like to encourage you very strongly not to underestimate the extent of the damage inflicted on your pc.
I provide here a partial output of strings yjz1:
etc/init.d/%s
/etc/rc%d.d/S90%s
--del
chkconfig
remove
update-rc.d
/etc/cron.hourly/gcc4.sh
/etc/rc.d/rc%d.d/S90%s
--add
defaults
/proc/%d/exe
/proc/self/exe
HOME=/
MYSQL_HISTFILE=/dev/null
#!/bin/sh
# chkconfig: 12345 90 90
# description: %s
### BEGIN INIT INFO
# Provides: %s
# Required-Start:
# Required-Stop:
# Default-Start: 1 2 3 4 5
# Default-Stop:
# Short-Description: %s
### END INIT INFO
case $1 in
start)
stop)
esac
sed -i '/\/etc\/cron.hourly\/gcc4.sh/d' /etc/crontab && echo '*/3 * * * * root /etc/cron.hourly/gcc4.sh' >> /etc/crontab
etc/init.d/%s
GET %s HTTP/1.1
%sHost: %s
POST %s HTTP/1.1
%sHost: %s
Content-Type: application/x-www-form-urlencoded
Content-Length: %d
%s%s
Accept: */*
Accept-Language: zh-cn
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; TencentTraveler ; .NET CLR 1.1.4322)
Connection: Keep-Alive
This provides evidence of tampering with the services (in /etc/init.d and in /etc/rc.d), with crontab, with the history file of mysql, and a couple of files in proc which are links to bash (which suggests a custom-made fraudulent version of your shell has been planted). Then the program generates an HTTP request (to a Chinese-speaking site,
Accept-Language: zh-cn
which gives substance to David Schwartz's comment above), which may create even more havoc. In the request, binaries (Content-Type: application/x-www-form-urlencoded) are to be downloaded to the attacked pc (GET) and uploaded to the controlling machine (POST). I could not establish what would be downloaded to the attacked pc, but, given the small size of both yjz and yjz1 (1.1MB and 600kB, repectively), I can venture to surmise that most of the files needed to cloak the rootkit, i.e. the altered versions of ls, netstat, ps, ifconfig,..., would be downloaded this way. And this would explain the attacker's feverish attempts to get this download going.
There is no certainty that the above exhausts all possibilities: we certainly lack part of the transcript (between lines 457 and 481) and we do not see a logout; furthermore, especially worrisome are lines 495-497,
cd /tmp; ./yd_cd/make
which refer to a file we did not see downloaded, and which might be a compilation: if so, it means the attacker has (finally?) understood what the problem with his executables was, and is trying to fix it, in which case the attacked pc has gone for good. [In fact, the two versions of the malware which the attacker downloaded onto the hacked machine (and I onto my 64bit Debian VM) are for an unsuitable architecture, x86, while the name alone of the hacked-into pc gives away the fact that he was dealing with an arm architecture].
The reason why I wrote this edit is to urge you as strongly as possible either to comb your system with a professional instrument, or to re-install from scratch.
And, by the way, should this prove useful to anyone, this is the list of of the 331 IP addresses to which yjz tries to connect. This list is so large (and probably destined to become larger still) that I believe this is the reason for tampering with mysql. The list provided by the other backdoor is identical, which, I presume, is the reason for leaving such an important piece of information out in the open (I think the attacker did not wish to make the effort to store them in kernel format, so he put the whole list in a clear-text file, which is probably read-in by all of his backdoors, for whichever OS):
61.132.163.68
202.102.192.68
202.102.213.68
202.102.200.101
58.242.2.2
202.38.64.1
211.91.88.129
211.138.180.2
218.104.78.2
202.102.199.68
202.175.3.3
202.175.3.8
202.112.144.30
61.233.9.9
61.233.9.61
124.207.160.110
202.97.7.6
202.97.7.17
202.106.0.20
202.106.46.151
202.106.195.68
202.106.196.115
202.106.196.212
202.106.196.228
202.106.196.230
202.106.196.232
202.106.196.237
202.112.112.10
211.136.17.107
211.136.28.231
211.136.28.234
211.136.28.237
211.147.6.3
219.141.136.10
219.141.140.10
219.141.148.37
219.141.148.39
219.239.26.42
221.130.32.100
221.130.32.103
221.130.32.106
221.130.32.109
221.130.33.52
221.130.33.60
221.176.3.70
221.176.3.73
221.176.3.76
221.176.3.79
221.176.3.83
221.176.3.85
221.176.4.6
221.176.4.9
221.176.4.12
221.176.4.15
221.176.4.18
221.176.4.21
58.22.96.66
218.104.128.106
202.101.98.55
211.138.145.194
211.138.151.161
211.138.156.66
218.85.152.99
218.85.157.99
222.47.29.93
202.101.107.85
119.233.255.228
222.47.62.142
122.72.33.240
211.98.121.27
218.203.160.194
221.7.34.10
61.235.70.98
113.111.211.22
202.96.128.68
202.96.128.86
202.96.128.166
210.21.3.140
210.21.4.130
211.95.193.97
211.98.2.4
211.98.4.1
211.162.61.225
211.162.61.235
211.162.61.255
211.162.62.1
211.162.62.60
221.4.66.66
202.103.176.22
202.96.144.47
210.38.192.33
202.96.134.33
202.96.134.133
202.96.154.15
210.21.196.6
221.5.88.88
202.103.243.112
202.193.64.33
61.235.164.13
61.235.164.18
202.103.225.68
221.7.136.68
202.103.224.68
211.97.64.129
211.138.240.100
211.138.242.18
211.138.245.180
221.7.128.68
222.52.118.162
202.98.192.67
202.98.198.167
211.92.136.81
211.139.1.3
211.139.2.18
202.100.192.68
211.97.96.65
211.138.164.6
221.11.132.2
202.100.199.8
202.99.160.68
202.99.166.4
202.99.168.8
222.222.222.222
202.102.224.68
202.102.227.68
222.85.85.85
222.88.88.88
210.42.241.1
202.196.64.1
112.100.100.100
202.97.224.68
219.235.127.1
61.236.93.33
211.93.24.129
211.137.241.34
219.147.198.230
202.103.0.68
202.103.0.117
202.103.24.68
202.103.44.150
202.114.0.242
202.114.240.6
211.161.158.11
211.161.159.3
218.104.111.114
218.104.111.122
218.106.127.114
218.106.127.122
221.232.129.30
59.51.78.210
61.234.254.5
202.103.96.112
219.72.225.253
222.243.129.81
222.246.129.80
211.142.210.98
211.142.210.100
220.168.208.3
220.168.208.6
220.170.64.68
218.76.192.100
61.187.98.3
61.187.98.6
202.98.0.68
211.93.64.129
211.141.16.99
202.98.5.68
219.149.194.55
211.138.200.69
202.102.3.141
202.102.3.144
58.240.57.33
112.4.0.55
114.114.114.114
114.114.115.115
202.102.24.34
218.2.135.1
221.6.4.66
221.131.143.69
202.102.8.141
222.45.0.110
61.177.7.1
218.104.32.106
211.103.13.101
221.228.255.1
61.147.37.1
222.45.1.40
58.241.208.46
202.102.9.141
202.102.7.90
202.101.224.68
202.101.226.68
211.141.90.68
211.137.32.178
202.96.69.38
211.140.197.58
219.149.6.99
202.96.86.18
101.47.189.10
101.47.189.18
118.29.249.50
118.29.249.54
202.96.64.68
202.96.75.68
202.118.1.29
202.118.1.53
219.148.204.66
202.99.224.8
202.99.224.67
211.90.72.65
211.138.91.1
218.203.101.3
202.100.96.68
211.93.0.81
222.75.152.129
211.138.75.123
202.102.154.3
202.102.152.3
219.146.1.66
219.147.1.66
202.102.128.68
202.102.134.68
211.138.106.19
211.90.80.65
202.99.192.66
202.99.192.68
61.134.1.4
202.117.96.5
202.117.96.10
218.30.19.40
218.30.19.50
116.228.111.118
180.168.255.18
202.96.209.5
202.96.209.133
202.101.6.2
211.95.1.97
211.95.72.1
211.136.112.50
211.136.150.66
119.6.6.6
124.161.97.234
124.161.97.238
124.161.97.242
61.139.2.69
202.98.96.68
202.115.32.36
202.115.32.39
218.6.200.139
218.89.0.124
61.139.54.66
61.139.39.73
139.175.10.20
139.175.55.244
139.175.150.20
139.175.252.16
168.95.1.1
210.200.211.193
210.200.211.225
211.78.130.1
61.31.1.1
61.31.233.1
168.95.192.1
168.95.192.174
61.60.224.3
61.60.224.5
202.113.16.10
202.113.16.11
202.99.96.68
202.99.104.68
211.137.160.5
211.137.160.185
219.150.32.132
202.98.224.68
211.139.73.34
61.10.0.130
61.10.1.130
202.14.67.4
202.14.67.14
202.45.84.58
202.45.84.67
202.60.252.8
202.85.128.32
203.80.96.9
203.142.100.18
203.142.100.21
203.186.94.20
203.186.94.241
221.7.1.20
61.128.114.133
61.128.114.166
218.202.152.130
61.166.150.123
202.203.128.33
211.98.72.7
211.139.29.68
211.139.29.150
211.139.29.170
221.3.131.11
222.172.200.68
61.166.150.101
61.166.150.139
202.203.144.33
202.203.160.33
202.203.192.33
202.203.208.33
202.203.224.33
211.92.144.161
222.221.5.240
61.166.25.129
202.96.103.36
221.12.1.227
221.130.252.200
222.46.120.5
202.96.96.68
218.108.248.219
218.108.248.245
61.130.254.34
60.191.244.5
202.96.104.15
202.96.104.26
221.12.33.227
202.96.107.27
61.128.128.68
61.128.192.68
218.201.17.2
221.5.203.86
221.5.203.90
221.5.203.98
221.7.92.86
221.7.92.98
The following code
#!/bin/bash
echo 0 > out
while read i; do
whois $i | grep -m 1 -i country >> out
done < filename
cat out | grep -i cn | wc -l
on the above list shows that 302 out of a total 331 addresses are in mainland China, the remaining ones are in Hong Kong, Mongolia, Taiwan. This adds further support to David Schwartz's contention that this is mostly a Chinese bot ring.
EDIT 3
At @vaid's request (the author of the OP, read his comment below), I will add a comment about how to strengthen security of a basic Linux system (for a system providing many services, this is a far more complex topic). vaid states he did the following:
Reinstall the system
changed root password to a 16 character long password with mixed lower- and uppercase letters and characters and digits.
Changed the username to a 6 mixed character long username and applied the same password as used for root
changed SSH port to something above 5000
turned off SSH root login.
This is fine (except I use a port above 10,000 since many useful programs use the ports below 10,000). But I cannot emphasize enough the need to use cryptographic keys for ssh login, instead of passwords. I will give you a personal example. On one of my VPSes, I was uncertain whether to change the ssh port; I left it at 22, but used crypto keys for authentication. I had hundreds of break-in attempts per day, none succeeded. When, tired to check daily that no one had succeeded, I eventually switched the port to something above 10,000, break-in attempts went to zero. Mind you, it is not that hackers are stupid (they are not!), they just hunt down easier prey.
It is easy to activate a crypto key with RSA as a signature algorithm, see comment below by Jan Hudec (thanks!):
cd; mkdir .ssh; chmod 700 .ssh; cd .ssh; ssh-keygen -t rsa (then hit <kbd>ENTER>/kbd> three times); cat id_rsa.pub >> authorized_keys; chmod 600 *
Now all you have to do is to copy the file id_rsa to the machine from which you want to connect (in a directory .ssh, also chmod'ed to 700), then issue the command
ssh -p YourChosenNonStandardPort -i ~/.ssh/id_rsa me@RemoteMachine
When you are sure that this works, edit on the server (=the machine you want to connect to) the file /etc/ssh/sshd_config, and change the line
#PasswordAuthentication yes
to
PasswordAuthentication no
and restart the ssh service (service ssh restart or systemctl restart ssh, or something like this, depending on distro).
This will withstand a lot. In fact, there are currently no known exploits against the current versions of openssh v2, and of RSA as employed by openssh v2.
Lastly, in order to really bolt down your machine, you will need to configure the firewall (netfilter/iptables) as follows:
iptables -A INPUT -p tcp --dport YourChosenNonStandardPort -j ACCEPT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -P INPUT DROP
iptables -P OUTPUT ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
This, 1) allows ssh connections from both LAN and WAN, 2) allows all input which was originated by your requests (for instance, when you load a Web page), 3) drops everything else on the input, 4) allows everything on the output, and 5-6) allows everything on the loopback interface.
As your needs grow, and more ports need to be opened, you may do so by adding, at the top of the list, rules like:
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
to allow for instance people to access your Web browser.
Solution 2:
Welcome to the Internet - where any open SSH server is likely going to get probed, brute-forced, and have various indignities inflicted upon it.
To start, you need to completely wipe the storage on the product. Image it if you want to pass it on for forensics, but the Linux install on it is now suspect.
Bit of guesswork but
You got brute-forced or use a common password. It's security by obscurity but you don't want a dictionary password or to use a root account open to SSH. Disable root SSH access if it's an option or at least change the name so they need to guess both. SSHing as root is terrible security practice anyhow. If you must use root, log in as another user and use su or sudo to switch.
Depending on the product, you might want to lock down SSH access in some way. A total lock-down sounds like a good idea, and allows users to open it up as needed. Depending on what resources you can spare, consider only allowing IP addresses in your own subnet, or some kind of login throttling system. If you don't need it on the final product make sure it's turned off.
Use a non standard port. Security by obscurity again, but it means an attacker needs to target your port.
Do not ever use a default password. The best approach I've seen is to randomly generate a password for a specific device and ship it with your product. Best practice is key based authentication, but I've no idea how you'd approach that on a mass market product.