Security Through Isolation

They say no good deed goes unpunished.  In internet hosting, that’s almost always the case.  For the last fifteen years I’ve had servers that I’ve given friends accounts on.  At first they were co-located machines I built by hand, then leased servers, and now cloud VMs.  I hosted friends and family’s blogs, sites for activism causes that I or friends believed in.  I’ve even had a web site of a well known Silicon Valley venture capitalist on there.

Unfortunately whenever you do that, especially whenever you hand out accounts or host web applications that people were once enthusiastic about but then moved on from, security is going to become a problem.  A few months ago while doing maintenance on the machine I noticed that an account for someone had been logging in, except it shouldn’t have been, since I created the account while trying to troubleshoot a problem that we solved another way.  But there it was, in the last login log.  Digging into that directory I realized that the password had been simple (in the heat of troubleshooting you don’t always make the best decisions for security), and someone had brute forced the SSH login.  The machine had been compromised and used as an IRC bot host.  How very 1997.

Today I decided to migrate between cloud providers.  While I could host all this stuff for free at HP Cloud, but I do some dangerous stuff in the context of my account, and it’s nice to have a stable VM elsewhere.  It’s also nice to see what the competition are up to.  I’d been hosting this VM at Rackspace ever since they bought my preferred VM provider, Slicehost, but while poking around pricing I realized I could cut my monthly bill in half if I migrated to Linode.  I’ve admired Linode’s geek-friendly control panel, ever since I tried it while designing the beta versions of HP Cloud’s.  I was also on an ancient Ubuntu 9 version at Rackspace, and this would be a good opportunity to upgrade the OS and software.

After copying all the home directories and web sites over, I did a last pass to pick up any straggler processes.  These usually live in cron jobs, so that’s where I went looking.  Lo and behold, an account I’d setup for a friends mom to host her business web site had been compromised at some point.  Another bot.  Joy.

Fortunately her web sites had long since been migrated off the server, so I was able to disable her account (and remove the stashed authorized_keys file with a bot installer in them) and a bunch of others I knew wouldn’t be used, but it really goes to show how vulnerable these machines can be.  Who knows how they got her password.  It might have been an easily crackable password, it might have been a web script compromise, it might have been an email exploit.  More than a few of these usernames and passwords are sitting in mailboxes or in saved FTP connection files on easily crackable machines.

Two weeks ago I got an email from a former client.  We built a pretty complex web site for her in 2003, lots of bells and whistles.  It’s held up pretty well, but it hasn’t had any serious maintenance work in a lot of years.  She’d gotten a call from the FBI, saying that data from her web site was circulating in Russia.  Fortunately it was just an exported mailing list, not encrypted passwords or other secure data.  In her case I think one of her employees had either an easily guessable password, or a trojan was installed on her computer that logged her keystrokes.  How do you guard against the guardians?  Nobody was thinking of two-factor authentication for small business web sites in 2003, but the next time I build one, I sure will be.

Sharing accounts on a machine, or having admin accounts into a web based system is an inherently insecure thing.  The more keys there are to a lock, the more likely someone you don’t want to have one will get one.  I created user accounts on our shared server because that’s how you did it, back in the day.  Create a user account, setup a directory for the web site, add a database for them, and let them go.  Now we have linux kernel exploits that let anyone with user level privileges become superusers.  Adding random accounts to systems and handing out the passwords is an insane thing to do.

So the only hope we can have of having any kind of security is by shrinking the permissions scope down.  When everyone has user accounts on a machine, that entire machine is vulnerable.  When everybody has a small VM, only that VM is vulnerable (usually).  Even better, give them a single-process Linux Container, like those managed by docker.io, and suddenly they don’t even necessarily need to manage dependencies anymore.

I’m sure docker has its own set of security issues, but hopefully we’re more cognizant of them now.  Don’t create unnecessary user accounts.  Use password protected SSH keys.  Don’t re-use ssh keys.  Keep your dependencies up to date.  Watch the security mailing lists.  It really starts to sound like something the hosting provider should be doing…

So I think there’s a real opportunity here for a trusted IaaS operator to create a generic Linux Containers As A Service offering.  Push down one level from VM into Process.  Bring your own docker image, buy a set amount of RAM (say, 128 meg for a big PHP or Python process) and bill by the minute. Route them inside of the machine through some kind of nginx or go based proxy, like CloudFoundry does, but a little less specialized.  Something between CloudFoundry and an IaaS VM.  Upgrades in CloudFoundry are a pain.  If I could just shuffle a docker.io image around, that’d be way easier.  Oh, and don’t sign up for too many hosted services.  Each one of those you use is like another shared account, and the more you share your data with, the more likely it is it’ll be exposed.  Build small, build focused.

So back to security.  This is a plea to all those who have friends who’ve given them accounts on servers, or people who run servers and create accounts for friends.  I know the complex password requirements are a pain at work or on bank web sites, but they’re really even more important in less maintained environments.  Nobody’s watching that shared server, keeping it water tight is a shared responsibility.  If someone creates an account for you, change the password immediately.  They won’t remember to go delete it if you end up not using it, and if they set it to something simple, there’s a good chance someone will be able to brute force it.  Don’t store passwords in plain text anywhere.  Don’t email them to people, or have people email them to you.  You’ll be happier in the long run if you don’t.  Use best practices, and save us all some heartache.