Since I started playing with the ancient version of Linux Deepin in my junior year (2011), it’s been 14 years now. During this long period, I’ve used Debian/RedHat/Arch-based distros, but except for accidentally deleting my graduation thesis in my final year of grad school, I’ve never encountered any malware or viruses on Linux distributions.
However last week, our company server… actually got infected…
At first when the maintenance vendor told me our server might be infected, I was skeptical, because the technical support staff seemed quite unreliable.
I watched the whole remote operation process… their command familiarity… was even worse than mine… They just opened TOP and went through process names one by one, didn’t even know how to filter by user or sort by CPU usage. They even asked me about some obviously normal processes… After rebooting, they looked at TOP and said “seems fine now” (while CPU was still at 100%)… After messing around for a while, they just said “probably a mining virus, you need to reinstall the system” and stopped responding…
Although our data and system are separated, reinstalling would still cost us extra time reconfiguring everything. Moreover, prescribing medicine without diagnosing the problem, doesn’t satisfy my curiosity might not actually solve the problem…
So I decided to Google first…
It really was a mining virus
Initial Google searches didn’t yield much. Besides CPU being at 100% with no visible process responsible (which matched), netstat didn’t show any connections to mining pools, and systemctl mysteriously stopped working, preventing me from installing some tools mentioned in this blog.
At this point I still wasn’t thinking virus, until I found a blog suggesting using lsattr and chattr to check/modify systemctl permissions to fix software installation issues. I found that lsattr was missing from the server, and chattr gave a very strange error. I initially thought maybe these tools weren’t installed?
Then I remembered - we have another server! After logging into that one… wow, the other server did have lsattr and chattr… I went back to check what chattr was on the infected server… cat revealed it had clearly been replaced!
Then it hit me - top/htop/netstat etc. must have all been modified too!
Discovering malware modifications and countermeasures
- lsattr: Completely deleted, couldn’t check file attributes. Replaced with file from the other server.
- chattr: Replaced and set with ‘i’ and ‘a’ attributes so even root couldn’t delete it. Couldn’t reinstall chattr to fix. Replaced with file from the other server.
- top/htop: Must have been modified to hide malicious processes. Replaced top from the other server, reinstalled htop.
- systemctl/service management: Added a service to launch malware. Modified systemctl couldn’t show the malicious service. Removed immutable attributes and reinstalled systemd.
- ls: Replaced to hide malicious executables and service files. Replaced with file from the other server.
- reboot: Modified (SHA didn’t match the other server). Replaced with file from the other server, but still couldn’t reboot properly after fixing.
- netstat: Replaced to hide malicious network connections. Reinstalled net-tools.
Afterword
I guess you really do learn from experience. I never imagined malware would evolve to… just modify all the tools you’d use to detect it…
Also, I’ve… really become a sysadmin now…
