Suspicious users with numbers are devouring entire CPU [duplicate]

On my test server which I have docker-run gitlab-ce, redis server and some other important services, I noticed I have an uninvited guest, kdevtmpfsi. I tried everything proposed by the community but I see kind of intelligence in this one.

I runs some processes under nonexisting users, it started by gitlab-+, but I killed all the processed with this user. Now, I see a different behaviour. It runs some processes under some users with numbers, 998, 997, 996, etc.

All the commands they run are not existing on my machine. I don't have a local postgres, redis-server,gitlab-exporter etc.

28741 999       20   0 2873420 2.289g      0 S 331.8 29.4   1:31.19 kdevtmpfsi

Can anyone help?

There are two things happening here:

1. There is indeed a miner running. Googling for kdevtmpfsi gives a lot of results.
2. It is likely that this is happening inside a container, so the numerical UID and that the file doesn't exist on the host are both normal.

So, likely one of the containers got compromised. Whether they broke out of it is unknown.

I'd bet on "no", because it is extra effort and more chance to get caught (container hosts have a lot better security than containers) and doesn't gain them much -- this is a fire-and-forget miner that they will not contact again, when it is shut down, not much is lost.

Still, you can't be sure, so the proper and diligent thing to do would be to nuke the site from orbit.