Series: Linux Mastery Road to Cloud | Domain 9
There’s a moment every Linux learner hits where SSH stops feeling like a magic portal and starts feeling like infrastructure. That moment happened for me when I exposed my home server to the open internet at limonlab.online — and within hours, I was watching hundreds of failed login attempts roll through my auth logs from IPs all over the world.
That’s when SSH stopped being “the thing I type to connect to a server” and became something I actually needed to understand. This post is everything I learned in Domain 9, written the way I wish someone had explained it to me.
First — What Actually Happens When You Type ssh user@host
Most tutorials skip this completely and jump straight to key generation. But understanding the connection lifecycle is what separates someone who uses SSH from someone who can troubleshoot it.
Here’s what happens between pressing Enter and seeing a shell prompt:
1. TCP connection — Your machine opens a TCP connection to port 22 (or whatever port sshd is listening on). If this fails, you get “Connection refused” immediately. If it hangs, something is blocking the packet at the network level — a firewall, a routing issue, or the host is down but the TCP stack on your router is still alive.
2. SSH protocol handshake — Both sides announce their SSH version. You’ll see something like SSH-2.0-OpenSSH_8.9p1 in verbose mode.
3. Algorithm negotiation — Client and server compare lists of supported KEX (key exchange) algorithms, ciphers, and MACs (message authentication codes). They pick the strongest one both support. If there’s zero overlap — which happens when you SSH into a very old server — you get the no matching key exchange method found error. More on this shortly.
4. Host key verification — The server sends its public host key. Your SSH client checks it against ~/.ssh/known_hosts. If it’s a new server, you get the fingerprint prompt (this is TOFU — Trust On First Use). If the key changed since your last connection, you get the scary WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED message.
5. User authentication — Now the server knows who it is to you. Now it needs to know who you are to it. This is where your key, your password, or other auth methods come in.
6. Session — Channel is opened, shell is spawned, you’re in.
Understanding this chain is the difference between staring at a hanging terminal and knowing exactly where to look.
Key Generation — ED25519, Not RSA
When I started, I generated RSA keys because every old tutorial said to. Then I learned why ED25519 is better:
- Shorter keys, same or better security
- Faster to generate and verify
- Resistant to certain attacks that affect RSA at lower bit sizes
Generating a proper ED25519 key pair:
ssh-keygen -t ed25519 -C "limon@limonlab" -f ~/.ssh/id_limonlab
The -C flag is just a comment — it helps you identify which key is which when you have several. The -f flag sets the filename so you’re not overwriting your default key.
This creates two files:
~/.ssh/id_limonlab— your private key. Never leaves your machine. Never.~/.ssh/id_limonlab.pub— your public key. This goes on the servers you want to access.
If someone asks you what the -b flag does — it sets the key length in bits. It’s meaningful for RSA (use 4096). For ED25519 it’s ignored entirely because the key size is fixed by the algorithm.
The Permission Chain — Where Most Beginners Get Stuck
This one bit me hard. My key was correct, the server had my public key in authorized_keys, and SSH was still asking for a password. The fix wasn’t the file — it was the directory above it.
SSH is paranoid by design. It checks permissions at every level before it trusts your authorized_keys file:
| Path | Required Permission | Why |
|---|---|---|
~ (home directory) | 755 or more restrictive | If your home dir is world-writable, anyone could plant files |
~/.ssh | 700 | Only you should read or write here |
~/.ssh/authorized_keys | 600 | Only you should read this |
The key insight: SSH will silently ignore your authorized_keys if any parent directory is too permissive. It won’t tell you why. It’ll just fall back to password auth. Check the full chain when key auth isn’t working:
ls -ld ~
ls -ld ~/.ssh
ls -l ~/.ssh/authorized_keys
Fix with:
chmod 755 ~
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
sshd_config — Hardening a Real Server
This is where running a public server changed my understanding completely. Within hours of going live, /var/log/auth.log looked like this:
Failed password for root from 185.234.x.x port 42891 ssh2
Failed password for root from 45.142.x.x port 33201 ssh2
Invalid user admin from 193.32.x.x port 55012 ssh2
Your sshd_config lives at /etc/ssh/sshd_config. Here’s what I actually set and why:
Disable root login entirely:
PermitRootLogin no
There’s also prohibit-password which allows root login but only with a key — not a password. no is stricter and what most production servers should use unless there’s a specific reason.
Disable password auth completely:
PasswordAuthentication no
Once your key is on the server, turn this off. The bots trying to brute force passwords can’t do anything if password auth is disabled. This is the single most impactful hardening step.
Restrict which users can log in:
AllowUsers limon deploy
Even if someone creates another account on the server, they can’t SSH in unless they’re in this list.
Limit auth attempts per connection:
MaxAuthTries 3
This limits authentication attempts per connection — not connections. It’s worth combining with fail2ban for real brute force protection.
Idle session timeout:
ClientAliveInterval 300
ClientAliveCountMax 2
SSH sends a keepalive packet every 300 seconds. If it gets no response twice in a row, it drops the connection. That’s 10 minutes of total idle time before disconnect. Useful for cutting dead sessions that are holding open connections.
Always test your config before reloading:
sshd -t
Run this every single time before you systemctl reload ssh. One syntax error in sshd_config and you’ve locked yourself out of a remote server.
~/.ssh/config — Stop Typing Long Commands
Once you’re managing more than one server, not having a config file is just pain. This file lets you alias connections and pre-configure options:
Host homelab
Hostname 192.168.1.50
User limon
Port 2222
IdentityFile ~/.ssh/id_limonlab
Host *
ServerAliveInterval 60
IdentitiesOnly yes
After this, ssh homelab replaces ssh -i ~/.ssh/id_limonlab -p 2222 limon@192.168.1.50. The Host * block applies to all connections — good for global defaults like keepalive settings.
ProxyJump is worth understanding here. Let’s say you have a bastion server that’s the only one publicly reachable, and your app servers are only reachable from the bastion. Instead of SSH-ing to the bastion and then SSH-ing again from there:
Host appserver
Hostname 10.0.0.5
User deploy
ProxyJump limon@bastion.limonlab.online
One command — ssh appserver — and SSH handles the jump transparently. This is exactly how many cloud environments are structured.
Port Forwarding — Tunneling Through SSH
This is the part that confused me the most at first. Once I built the mental model, it clicked.
Local forwarding (-L): Bring a remote port to your local machine.
ssh -L 3307:localhost:3306 limon@myserver.com
Read this as: “On my local machine, open port 3307. Any traffic I send to port 3307 should be forwarded through the SSH tunnel to localhost:3306 as seen from the remote server.”
Useful when: A database isn’t exposed publicly (it shouldn’t be), but you want to connect to it from your laptop using a GUI like DBeaver.
Remote forwarding (-R): Expose your local port on the remote machine.
ssh -R 2222:localhost:22 limon@myserver.com
Read this as: “On the remote server, open port 2222. Any traffic sent to that port should come back through the tunnel to my port 22.”
Useful when: You’re behind a firewall or NAT that won’t let inbound connections reach you, but you need someone to be able to SSH into your machine.
Dynamic forwarding (-D): Create a SOCKS proxy.
ssh -D 1080 limon@myserver.com
This creates a local SOCKS5 proxy on port 1080. Point your browser at it and all your traffic routes through the remote server. More flexible than -L because it handles any destination, not just one specific host:port.
SSH Agent — Stop Typing Your Passphrase
You should protect your private key with a passphrase. But typing it every single time you SSH somewhere is painful. That’s what ssh-agent is for.
The agent runs in memory and holds your unlocked private key. You unlock it once at the start of your session:
eval $(ssh-agent)
ssh-add ~/.ssh/id_limonlab
After that, any SSH connection that needs that key uses the agent — no passphrase prompt.
The SSH_AUTH_SOCK environment variable is how SSH clients find the running agent. If that variable isn’t set, they can’t reach it.
Agent forwarding (-A) lets the remote server use your local agent — so from serverA you can SSH to serverB using your local keys without copying them to serverA. Sounds convenient. It is. It’s also a security risk: if someone has root on serverA, they can use your agent socket to impersonate you while your connection is active. Don’t use -A on servers you don’t fully trust.
SCP vs Rsync
SCP is simple. It copies files over SSH. That’s it.
# Copy remote nginx logs to local
scp -r limon@myserver.com:/var/log/nginx/ ~/logs/
Rsync is smarter. It compares source and destination, only transfers what changed, and can compress in transit.
rsync -avz limon@myserver.com:/var/log/nginx/ ~/logs/
The -a flag preserves permissions, timestamps, symlinks. -v is verbose. -z compresses during transfer.
For a 2GB directory where 95% of files haven’t changed, rsync transfers maybe a few MB. SCP transfers all 2GB every time. For syncing directories regularly — backups, deployments — rsync is almost always what you want.
The --delete flag removes files from the destination that no longer exist at the source. This makes rsync behave like a true mirror. It’s useful but treat it with respect — a wrong path argument can delete things you wanted to keep.
Known Hosts and TOFU
The first time you SSH to a new server:
The authenticity of host 'myserver.com' can't be established.
ED25519 key fingerprint is SHA256:abc123...
Are you sure you want to continue connecting (yes/no)?
This is TOFU — Trust On First Use. You’re being asked to manually verify the server’s identity. Most people just type yes without checking the fingerprint. That’s a habit worth breaking if you’re connecting to production servers, because TOFU is vulnerable to MITM attacks on that first connection.
Once you accept, the server’s key is stored in ~/.ssh/known_hosts. Every future connection, SSH compares the key to what’s stored.
If you rebuild a server and its host key changes, SSH will refuse to connect and show the big warning. Fix it by removing the old entry:
ssh-keygen -R myserver.com
StrictHostKeyChecking no bypasses all of this — SSH will connect to any server without verifying its key. You’ll see this in automation scripts sometimes. It’s a bad idea in production because it defeats one of SSH’s core protections against impersonation.
Reading Auth Logs
Running a public server means reading logs regularly. Here are the patterns that actually matter:
Authentication refused: bad ownership or modes for directory /home/limon
Permission chain problem. Check ls -ld ~, ls -ld ~/.ssh, ls -l ~/.ssh/authorized_keys.
Received disconnect from x.x.x.x: Too many authentication failures
Your SSH client tried too many keys before finding the right one (or none worked). Hit by MaxAuthTries. Use IdentitiesOnly yes in your config so SSH doesn’t try every key it knows.
Connection closed by authenticating user limon x.x.x.x port 54321 [preauth]
The connection was dropped before authentication completed. Could be a client timeout, an abrupt disconnect, or a bot that gave up. The [preauth] tag means they never successfully authenticated.
Crypto Negotiation — The One Error That Surprises People
If you ever SSH into an older server (some network equipment, legacy systems) and get:
no matching key exchange method found. Their offer: diffie-hellman-group1-sha1
The server only supports an old, weak KEX algorithm that modern OpenSSH refuses to use by default. For a one-off connection you can override it:
ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 user@oldserver
The + means “add this to my list” rather than replacing it. Don’t put this in your default config — it weakens security globally. Use it only for the specific host that needs it.
SSH Multiplexing — For When You Connect to the Same Host Constantly
If you’re SSH-ing to the same server dozens of times a day (during deployment work, debugging sessions), each connection does the full TCP + crypto handshake. That overhead adds up.
Multiplexing lets multiple SSH sessions share one underlying TCP connection:
Host myserver
ControlMaster auto
ControlPath ~/.ssh/cm-%r@%h:%p
ControlPersist 10m
After your first connection, subsequent connections reuse the existing socket. ControlPersist 10m keeps the master connection alive for 10 minutes after you close the last session — so the next connection is instant.
What I Carried Away From This Domain
Running a public server turns SSH from a topic into a necessity. The bots that hit limonlab.online don’t care about my learning curve — they’re scanning every port, trying every username, hammering away constantly.
What actually protects the server isn’t complexity. It’s the basics done properly:
- ED25519 keys, not passwords
- PermitRootLogin no
- PasswordAuthentication no
- Correct permission chain on ~/.ssh
- Reading the logs regularly
The rest of it — forwarding, multiplexing, agent, config files — those make SSH comfortable to work with every day. But security comes from the fundamentals.
That’s Domain 9 done. Next up: Domain 10 — Disk & Storage Management.
This post is part of my Linux Mastery Road to Cloud series — documenting my path from zero Linux knowledge to a junior cloud/DevOps role. All commands tested on my live Ubuntu server at limonlab.online.
Leave a Reply