How to close a port with no process attached?
This question is similar to Network port open, but no process attached? and netstat shows a listening port with no pid but lsof does not
I tried all i can do(as root: netstat, lsof, ls -al /proc/*/fd etc.), but i can't find the pid.
Anyway, i have to close or release the port without pid, because my process want to listen on it. Is anyway to do it?
I don't want to reboot the server. Because there is a process in my system, it will update bin files in all of my servers then deploy all servers automatically. The deployment will be failed when 7123 or some other ports were listened.
Thanks.
More Details
There are three servers in this situation, all of them are Rackspace's servers, and all 'bad port' is udp port 7123.
Reboot can solve this problem, i have tried it on one of those three servers. But i don't want to reboot the server.
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 213120 0 134.213.205.214:7123 0.0.0.0:* -
The result of nmap shows this port is not closed.(@Enzo)
root@auto:~# nmap 134.213.205.214 -p 7123 -sU
Starting Nmap 6.40 ( http://nmap.org ) at 2018-04-16 12:01 UTC
Nmap scan report for auto (134.213.205.214)
Host is up.
PORT STATE SERVICE
7123/udp open|filtered unknown
Nmap done: 1 IP address (1 host up) scanned in 2.12 seconds
Update
For now, reboot is the only way to close/release this port. But reboot is a bad idea, it makes service stop.
Solution 1:
There's the possibility your system is running in an inconsistent status. I would start by doing some basic diagnostics.
Check the UDP socket and its process.
Run sudo netstat -lunp or sudo ss -lunp to see whether that UDP socket (on port 7123) is busy. sudo is needed for a normal user to escalate privileges and see the PID and the process name who's "listening" on that port. Without sudo there will be a - instead of the PID and process name. Take note of the the PID. In case there's none listed, then you can start thinking your system has been compromised as the process is capable of hiding itslef by manipulating the internal kernel structures for the processes!
If you cannot see the PID with high privileges, than you'd better to insulate that server as it's probably been compromised and hacked.
Check whether the listening process is also "talking"
I suggest to use nmap or the more general netcat tool. With nmap you can try this command:
sudo nmap -sV -v -Pn -sU -p U:7123 134.213.205.214
It tries to understand the protocol spoken at port UDP:7123. sudo is required for the specific scan type (UDP service scan). If you see data that don't match what you are expecting, than probably either the process is gone wild or is in an inconsistent status. Or it's been replaced by some other (malicious?) code.
With netcat you need some manual intervention to generate traffic to the server:
netcat -u 134.213.205.214 7123
Whatever you'll type will be sent to your application on port UDP:7123. You can also pipe some random data (but don't expect any meaningful result) with:
cat /dev/urandom | netcat -u 134.213.205.214 7123
Check the system and application logs (if any)
The former are usually stored at /var/log/ directory. Application log could be anywhere else. The tool lsof can be of help if you check the PID on its 2nd column. I'd keep an eye on these logs during the subsequent actions.
Check the process table for that PID
The command ps can provide for a lot of details about processes. My personal favorite is:
PS_FORMAT="ruser,pid,ppid,s,%cpu,rss,cmd" ps ax --sort=pid
You can see on the second column the PID you are looking for, along with the real user ID (ruser, 1st column), the parent PID (ppid, 3rd column), the status (s, 4th), the % CPU usage (%cpu, 5th), the resident set size (rss, 6th) and the command line with arguments (cmd, 7th). In my opinion, for this very case, the process status (it's a single letter) and the percentage of used CPU are the key values, along with the command line.
Check on the man page for all the details for ps and to fine tune the output.
Check the binaries
If you have another machine with the same architecture and OS and the same process running in an expected way, you can check whether the binaries match byte-by-byte. In case they don't, you'd better reinstall those binaries from a known secure source.
Let's assume the program is /usr/local/bin/myserver.
Calculate and take note of its checksum. Something like this:
sha512sum /usr/local/bin/myserver
If the program has been compiled to use dynamic libraries, then you'd also check them. The list of used dynamic libraries is obtained with:
ldd /usr/local/bin/myserver
Be warned: the output can be quite lengthy, but for each line you'd calculate and take note of the checksum for comparison.
In case any discrepancy with the reference trusted system is found, I'd suggest for a re-installation of the whole system and applications. It's a radical approach but I think that the target system cannot be trusted any more.
Kill and restart the process
I would try to kill the rogue process (provided that you know its PID) and restart it to check whether the rogue behavior sticks. In order to kill a process given its PID you can run:
sudo kill -s SIGKILL <PID>
I need to warn you that there is a number of cases in which a process won't die or will die later than expected. It depends mostly upon the process status shown by the ps command run earlier. The referenced man page reports some information about the process status column at the "PROCESS STATE CODES" paragraph.
Use the command ps to check again the process status. It should normally be kicked out within a few seconds and disappear from the process list.
Once the process has been killed you can try to restart it and see whether the rogue behavior repeats.
As a general rule of thumb, if you don't change anything to the system, the behavior will hardly change as well. That is, just rebooting the application or the system will hardly solve the problem: it will just be pushed forward in time.