Brute-force.

I just realised that my CentOS hosts don’t have Denyhosts running on them. cPanel is supposed to include some sort of anti-brute-force protection, but I don’t think it’s terribly reliable, since after installing Denyhosts on my main cPanel VM, I received the following in my admin email:

Added the following hosts to /etc/hosts.deny:

222.66.76.146 (unknown)
59.60.6.202 (unknown)
85.186.25.39 (unknown)
134.34.33.30 (unknown)
59.106.17.100 (unknown)
217.39.150.194 (pbx.altmore.co.uk)
202.63.105.91 (unknown)
202.65.218.140 (static-ip-140-218-65-202.rev.dyxnet.com)
201.28.114.173 (201-28-114-173.customer.tdatabrasil.net.br)
211.112.95.3 (unknown)
220.225.147.101 (unknown)
218.248.1.163 (unknown)
58.53.194.72 (unknown)
72.233.71.101 (101.71.233.72.static.reverse.ltdomains.com)
121.151.178.9 (unknown)
194.244.37.210 (unknown)
220.225.40.205 (unknown)
83.15.231.245 (eot245.internetdsl.tpnet.pl)
200.91.217.75 (200-91-217-75-host.ifx.net.co)
67.110.178.197 (67.110.178.197.ptr.us.xo.net)
58.211.137.77 (unknown)
62.111.247.77 (host-ip77-247.crowley.pl)
89.106.12.190 (reverse-89-106-12-190.grid.com.tr)
148.245.173.248 (unknown)
82.127.123.140 (LSt-Amand-152-31-44-140.w82-127.abo.wanadoo.fr)
200.253.204.134 (seliga.maracanau.ce.gov.br)
85.125.68.108 (85-125-68-108.work.xdsl-line.inode.at)

This VM has been running for less than a month, too.

Random brute-force attacks on SSH are as common as the spam in your inbox – anything with SSHd running and a public IP address is vulnerable to this, and unless you make a habit of reading your security logs (and who really has time to do that?) you need something like Denyhosts.

Ubuntu Server ships with only a bare minimum of packages, so I can understand why I have to install it myself there – but I’m a bit disappointed that CentOS doesn’t come with anything. It’s not the fault of CentOS, since they just repackage Redhat Enterprise – does RHEL 4 ship with nothing watching over SSH? I’d be really unimpressed if that were the case and I’d bought an RHEL licence.

As an aside, while I know that my CentOS installs are not the same as a standard install-from-the-DVD install, due to the method in which Xen VMs are created, my day job involves installing CentOS 4 systems, and I know that it doesn’t ship with Denyhosts. X11, Gnome, and a ton of junk, but no Denyhosts.

The moral of this story is: no matter how good your passwords are, a brute-force attack will eventually compromise them, your machine – virtual or otherwise – will be pwnd and used to compromise other machines, perform DDoSes for the Russian Mafia, host kiddie porn, or one of a thousand other things that are Seriously Bad News. Unless you’ve disabled password authentication entirely and just use SSH keys, you need Denyhosts (or something like it).

Post-mortem

Somewhat late, here’s the run-down on what went wrong with our big upgrade.

The executive summary is: very little.

Saturday’s hardware upgrades went reasonably well, although we did discover that the hardware just doesn’t like 2GB DIMMs, and caused a minor panic by forgetting to activate the second disk in the BIOS.

The RAM issue isn’t major: in the short term we have more than enough RAM for the VMs we need now, and in the medium term four 1GB DIMMs won’t cost much and should see us fine for the lifetime of the hardware. In the long term, this server is going to be replaced or supplimented with something newer which will happily take whatever RAM I throw at it.

Sunday’s software work was mixed. A hardware oversight caused delays in getting started as we couldn’t boot from the Ubuntu Server CD. Once we resolved that it went fairly smoothly. We discovered that Ubuntu Server for x86_64/AMD64 didn’t have the oh-so-nice ‘ubuntu-xen-server’ package, so I had to manually type out a short list of package names. It wasn’t a huge hardship.

The LVM partitioning all worked out excellently and the revised disk plan I came up with on the train probably saved us hours.

It still involved about two hours being stuck behind the KVM cart in a corner of a datacentre, something I always find particularly unpleasant, and as a result we ran away pretty much as soon as we had Ubuntu Server installed.

Xen and LVM were actually installed sitting in a Starbucks under Canary Wharf DLR station, and after heading into town (via a detour back to the datacentre to correct a BIOS issue we’d discovered) I worked on getting the Gentoo install working as a VM in a pub. After giving up due to running out of battery power, and having tracked down several issues (mainly making sure that /proc and /sys weren’t automounted by Gentoo’s RC scripts), we went back to our hotel.

Rather than giving up entirely, I paid the hotel wifi tax and surprised myself by getting it working in about 10 minutes.

We brought the Gentoo system back up at the stroke of midnight, and everything seemed pretty good.

Monday was left for relaxing and enjoying a nice meal in a pub in the docklands, doing some shopping, and watching a DVD.

At this point, the work was done and the trip was over and, in retrospect, it all seemed to have worked. But there were horrors to come when I tried to create some CentOS VMs on the train back to Edinburgh…