security | Jennifer Kramer

Security Through Isolation

They say no good deed goes unpunished. In internet hosting, that’s almost always the case. For the last fifteen years I’ve had servers that I’ve given friends accounts on. At first they were co-located machines I built by hand, then leased servers, and now cloud VMs. I hosted friends and family’s blogs, sites for activism causes that I or friends believed in. I’ve even had a web site of a well known Silicon Valley venture capitalist on there.

Unfortunately whenever you do that, especially whenever you hand out accounts or host web applications that people were once enthusiastic about but then moved on from, security is going to become a problem. A few months ago while doing maintenance on the machine I noticed that an account for someone had been logging in, except it shouldn’t have been, since I created the account while trying to troubleshoot a problem that we solved another way. But there it was, in the last login log. Digging into that directory I realized that the password had been simple (in the heat of troubleshooting you don’t always make the best decisions for security), and someone had brute forced the SSH login. The machine had been compromised and used as an IRC bot host. How very 1997.

Today I decided to migrate between cloud providers. While I could host all this stuff for free at HP Cloud, but I do some dangerous stuff in the context of my account, and it’s nice to have a stable VM elsewhere. It’s also nice to see what the competition are up to. I’d been hosting this VM at Rackspace ever since they bought my preferred VM provider, Slicehost, but while poking around pricing I realized I could cut my monthly bill in half if I migrated to Linode. I’ve admired Linode’s geek-friendly control panel, ever since I tried it while designing the beta versions of HP Cloud’s. I was also on an ancient Ubuntu 9 version at Rackspace, and this would be a good opportunity to upgrade the OS and software.

After copying all the home directories and web sites over, I did a last pass to pick up any straggler processes. These usually live in cron jobs, so that’s where I went looking. Lo and behold, an account I’d setup for a friends mom to host her business web site had been compromised at some point. Another bot. Joy.

Fortunately her web sites had long since been migrated off the server, so I was able to disable her account (and remove the stashed authorized_keys file with a bot installer in them) and a bunch of others I knew wouldn’t be used, but it really goes to show how vulnerable these machines can be. Who knows how they got her password. It might have been an easily crackable password, it might have been a web script compromise, it might have been an email exploit. More than a few of these usernames and passwords are sitting in mailboxes or in saved FTP connection files on easily crackable machines.

Two weeks ago I got an email from a former client. We built a pretty complex web site for her in 2003, lots of bells and whistles. It’s held up pretty well, but it hasn’t had any serious maintenance work in a lot of years. She’d gotten a call from the FBI, saying that data from her web site was circulating in Russia. Fortunately it was just an exported mailing list, not encrypted passwords or other secure data. In her case I think one of her employees had either an easily guessable password, or a trojan was installed on her computer that logged her keystrokes. How do you guard against the guardians? Nobody was thinking of two-factor authentication for small business web sites in 2003, but the next time I build one, I sure will be.

Sharing accounts on a machine, or having admin accounts into a web based system is an inherently insecure thing. The more keys there are to a lock, the more likely someone you don’t want to have one will get one. I created user accounts on our shared server because that’s how you did it, back in the day. Create a user account, setup a directory for the web site, add a database for them, and let them go. Now we have linux kernel exploits that let anyone with user level privileges become superusers. Adding random accounts to systems and handing out the passwords is an insane thing to do.

So the only hope we can have of having any kind of security is by shrinking the permissions scope down. When everyone has user accounts on a machine, that entire machine is vulnerable. When everybody has a small VM, only that VM is vulnerable (usually). Even better, give them a single-process Linux Container, like those managed by docker.io, and suddenly they don’t even necessarily need to manage dependencies anymore.

I’m sure docker has its own set of security issues, but hopefully we’re more cognizant of them now. Don’t create unnecessary user accounts. Use password protected SSH keys. Don’t re-use ssh keys. Keep your dependencies up to date. Watch the security mailing lists. It really starts to sound like something the hosting provider should be doing…

So I think there’s a real opportunity here for a trusted IaaS operator to create a generic Linux Containers As A Service offering. Push down one level from VM into Process. Bring your own docker image, buy a set amount of RAM (say, 128 meg for a big PHP or Python process) and bill by the minute. Route them inside of the machine through some kind of nginx or go based proxy, like CloudFoundry does, but a little less specialized. Something between CloudFoundry and an IaaS VM. Upgrades in CloudFoundry are a pain. If I could just shuffle a docker.io image around, that’d be way easier. Oh, and don’t sign up for too many hosted services. Each one of those you use is like another shared account, and the more you share your data with, the more likely it is it’ll be exposed. Build small, build focused.

So back to security. This is a plea to all those who have friends who’ve given them accounts on servers, or people who run servers and create accounts for friends. I know the complex password requirements are a pain at work or on bank web sites, but they’re really even more important in less maintained environments. Nobody’s watching that shared server, keeping it water tight is a shared responsibility. If someone creates an account for you, change the password immediately. They won’t remember to go delete it if you end up not using it, and if they set it to something simple, there’s a good chance someone will be able to brute force it. Don’t store passwords in plain text anywhere. Don’t email them to people, or have people email them to you. You’ll be happier in the long run if you don’t. Use best practices, and save us all some heartache.

Being Burgled

I don’t ask for audience participation very often, but today I’d like you to do me a favor. The next time you’re home, walk around and take pictures of every room in your house with your cell phone. Pretend you’re documenting the place for when they make a movie of your life. Feel free to cast your favorite Hollywood stars as the main characters.

These photos will be really valuable if (or perhaps just when) someone kicks open your front door and steals your stuff. It doesn’t happen very often, but in our zip code it happened 541 times in 2011. There are about 35,000 residences in our zip code, which means about a 1 in 65 chance a given house will be broken into. Last Tuesday it happened to us, while I was presenting about Software Bot Platforms at SXSW Interactive.

While you’re taking pictures of everything, make sure you have photos of everything that’s worth over a hundred bucks or so, especially those TVs, PlayStations, Xboxes, and the like. I’d suggest you flip those things over and take pictures of the serial numbers, too. That information’s really hard to dig up once they’re gone.

Lets say someone does decide to break into your house and steal your stuff. Most burglaries happen through the front or back door, just kicking the thing in (or finding the key you left under the mat or rock). If you’re like most people, when you installed your front door you might have used the cheap screws and shallow deadbolt. If you don’t know, unscrew the screws. If they’re shorter than 3 inches, replace them with nice strong 3 or 3 1/2 inch screws from the hardware store. If you’re leaving the house, always lock the deadbolt. The handle latch can be pried open with a screwdriver, or kicked open with one swift kick. Most burglaries happen between 10am and 3pm, prime “I’m just going to leave for a few minutes” time. If you have a really cheap deadbolt, think about upgrading to a grade 1 or grade 2 deadbolt. If you’d really like to secure your front door, consider a metal reinforcing strip. They make them for french doors, too. If it takes more than a few minutes to get your door open, they’re probably going to leave. You can drive yourself crazy researching bump key resistant locks, but if you really want the best, you can spend quite a bit.

I’d guess the people who broke into our house were inside less than five minutes. They look for houses that are easy to get away from. We live on a corner two blocks from major north/south and east/west arteries. They’re probably going to hit your bedroom first, they’ll pull out the drawers in your night stands (looking for jewelry or guns or small electronics). Maybe like our break in, they’ll grab a random bag to store their loot. Maybe a bag with a lot of memories to you. After the break in you’ll marvel at the things they didn’t take, the jewelry box they didn’t find, the cameras or hard drives, but some things will still be gone, and inevitably they will be things you care about.

Then they’ll hit the living room. They’ll grab things that are easy to sell, like your TV, your Xbox and Playstation. They’ll clear out your collection of Xbox games. Later you’ll realize that while the TV and devices are replaceable, the save games sitting on that Xbox’s hard drive are not. The three playthroughs of Borderlands 1 and 2 you did with your wife, with all the awesome characters and loot? All gone. The hours you spent with your team in Mass Effect, and how you were a couple hours from the end, eagerly awaiting the last DLC? Gone. You’re probably not going to finish Assassin’s Creed 3 now, and thank goodness you never even started Skyrim.

Your daughter may point at the place the TV was and say “uh oh”. The first couple of times it’s cute, but it’s also painful. Just be glad she wasn’t there when they broke in. You’ll probably also spend some time wondering why they would bother to take her new pair of red sneakers. But then you’ll realize that the people who broke into your house might have kids too, and then you’ll just be sad for the world.

They’ll do really strange things, like take the Xbox from your living room but leave the power supply, and take the power supply for your other Xbox in your daughters room but leave the console. Of course, the power supplies are different, so you don’t even have one working Xbox anymore. It’s kind of senseless.

They’ll grab the work laptop off your desk in the office, making a giant mess in the process (as if your office wasn’t a mess enough already). But they’ll graciously pull your USB VPN key out of it, and drop it on the floor before they leave, which almost makes you think that this was a Jason Bourne-esque black bag job and they’ve installed keyloggers and microscopic cameras everywhere to infiltrate your work accounts. That would explain why they didn’t touch your wife’s laptop on the desk opposite. But now you’re just talking crazy.

When you get the call that someone broke into your house and took all your computers, you immediately think the worst, that everything is gone and in the hands of mafioso or triad hackers who intend to destroy you digitally as well. This is a good way to think, because it probably isn’t far from the mark, and certainly won’t be over the next decade.

In our case we were lucky. Irma’s personal laptop was covered by papers, so they didn’t grab it. My main work laptop was entirely encrypted, and my other work laptop didn’t have anything loaded on it. They didn’t get my Time Machine backup drive, from which they could have reconstructed my entire life. My laptop was with me, as were our iPads. They got an old phone we were using for audio streaming, and the old iPad we used as a white noise generator for our daughter, but those didn’t have anything special on them. They didn’t steal our network attached storage device, which would probably be a treasure trove to the wrong people. In the end, we were very lucky.

But this could just as easily happen to any of you, so please, do me a big favor…

Your To-Do List

1. Reinforce your doors. Always use your deadbolt when you leave the house. Both front and back doors. If you have a sliding glass door in the back, figure out a way to secure that thing. Don’t leave any of your windows unlocked. Don’t leave a spare key out in anything like an obvious place. Lock the door into your garage when you leave, garage door openers are easy to fake out and even manual ones are really easy to open.

2. Catalog all your stuff, it’ll make the insurance process easier, and you can do it in a half an hour with your iPhone. Serial numbers for anything that has them. Entire room shots. You never know what they’ll take. If you can, set a reminder to do it again in 6 months. Google Calendar is great for that. Back up those pictures on a PC or in the cloud. Make sure you have a passcode lock on your phone, those things are slipperier than a bucket full of eels. Same with your iPads or other personal electronic devices.

3. Make sure you back up your machines. We use CrashPlan. It’s money well spent. If you have a Mac and have a Time Machine backup (and you should), be sure to encrypt it (if it’s local) or hide the drive in an inconspicuous place (if it’s over a network). If you have a Mac, also turn on FileVault, so even if they get your computer, they can’t read the contents of your hard drive. Always require a password to wake from sleep or screen saver or login on boot. Pick a good password, something you don’t use anywhere else. Make sure someone else knows it, in case you get hit by a truck. If you have a PC, here’s an article that may help. If you have an Xbox and an Live Gold account, turn on Cloud Saves and use them.

4. Make sure Find My Device is turned on for all your iPhones, iPods, iPads, and Macs. You can wipe a machine remotely if it gets out of your hands and is still connected to wifi. Make sure you can login to iCloud and all your stuff is listed. One of my friends recommends Cerberus for Android.

5. You can secure your stuff a bit. If I’d had a cable lock on my TV, and they hadn’t been able to just lift it off the wall mount, it would probably still be here. There are also locks for your laptop, but that’s a pain if you like to be mobile. If your laptop lives on your desk, it might be worth it. Some larger TV wall mounts have holes for locks. If yours doesn’t, you might be able to thread a cable lock through it and make it harder to pull off the wall.

6. If you live near Austin and would like to fix up your security but don’t know how to do any of this stuff, are scared of screwdrivers, have a phobia of the hardware store, etc, let me know and we’ll get it done.

Ergo

In the next day or two we’ll pick up a new TV to replace the one from the living room. My daughter will be able to watch Sesame Street again. We’ll probably get another Xbox, and maybe another copy of Borderlands 2. I don’t think I’ll ever go back to Mass Effect. That save game was their world, I can’t re-create that. We’re probably going to install a camera in our entry way that watches the entry way and street, something that even keeps running when the power’s cut. Of course then there’s the back door, or a window. Your home isn’t a castle, it’s a barely held together shed with a bunch of memories and possessions inside that anyone with a reciprocating saw and 2 minutes of time could compromise. If someone wants your stuff, there isn’t much you can do to stop them, which in the end is the terrifying thing, because all we’d really like back is our peace of mind.

Remembering Aaron Swartz (1986 – 2013)

Aaron Swartz took his own life yesterday. Today, the Internet mourns, or at least, the parts of the Internet who were aware of him. Nearly everyone online is touched by his work, but most will be oblivious to his passing. It’s starkest on Twitter, where half of the tweets I read are about Aaron, and half are from people who haven’t a clue.

I met Aaron in 2003, at the SXSW EFF party Polycot co-sponsored and organized. The idea that a non-profit and a 3 person web development company could book a club a block away from the Austin Convention Center for a SXSW party shows you how long ago that was. Aaron was speaking about Creative Commons at SXSW that year. I forget how, but we somehow ended up running around together, trying to get the DSL working at the club (we ended up driving to another Polycot’s apartment and snarfing a router, because Texture’s was locked down).

Aaron would have been 16 or 17 at the time, and I remember him hauling around a backpack with his laptop in it that was nearly bigger than him. Aaron was a prodigy, you could tell by being around him that he lived on finding solutions to problems. He was the kind of person you sometimes wish you were, motivated, energetic, brilliant, but also wish you weren’t, because the prospect of it can be terrifying. I wasn’t surprised when he went on to contribute to reddit, and start his data freedom and political justice efforts. He was that kind of guy.

Aaron ran into trouble with the law a few years ago, after dropping a laptop into a data closet at MIT and snarfing down a couple million documents from the pay-per-access Scientific and Academic Journal Archive JSTOR, with the intent of uploading them freely on the internet. JSTOR declined to prosecute, but Carmen Ortiz, the US Attorney for Massachusetts decided to push ahead, charging Aaron with a felony which held a maximum penalty of 35 years in prison and a million dollar fine. The expert witness in the case has some notes. Aaron fell into some pretty deep depression, as freedom loving, introspective intellectuals are prone to, and in the end, took his own life.

This is where Aaron’s story and mine start to mirror each other. Before I got out of my teens, I had my own run in with our nation’s legal system, though mine was more tech business related than internet freedom related. I did something I felt at the time was just, and then faced the possibility of consequences. I can certainly sympathize with the feeling of helplessness you get. Introspective nerds aren’t used to the criminal justice system, and we aren’t used to systems where we don’t understand anything and are unable to make any change. In a computer system or a network you can learn, fix, and modify. The justice system, likely for most of us, just exists as a giant monolithic machine that chews people up. The prospect of getting caught up in a machine like that is terrifying, not to mention just losing a giant chunk of your life and becoming a societal outcast. This can weigh heavily on a person, especially one who thrives on solving new technical problems and feels themselves on the side of freedom and justice. With technology Aaron had agency, he had some power, but even with high profile friends, facing the machinations of the state, he felt he had no recourse.

We all also like to think we’re good people, we hold doors for people, we make room for people in traffic, we pay our taxes, we vote. When we get a chance, we strive to do the right thing. When you’re accused of a crime, especially when you’re doing something you feel is morally right, that can be crushing. Suddenly society looks different. You are, at least in some way, a bad person. You’ve been separated from society, pushed out of the public body like a virus or thorn. It isn’t implicit in every interaction, but you feel it, and it lingers there, at the back of your mind. It takes a long time for that to go away, and in the mean time, if you’re prone to depression, things can get very dark.

In the end, Aaron’s storyline and mine diverge. The charges against me were dropped, and after a few years of legal wrangling where everyone’s lawyers made some good money but the participants just had sleepless nights, the entire thing was settled out of court. In the end, life goes on. Lesson learned. No black marks, no permanent damage, no ticking the ‘convicted of a felony’ box on forms.

Aaron was facing more than that for a more righteous cause, and it got the better of him. In the end we all lose, even the state attempting to impose justice. People like to think that they want freedom and justice, that they’d strive for it and fight for it if they had reason and opportunity, but the price is high, and we are all too comfortable.

Tim Berners-Lee posted this:

Aaron dead.
World wanderers, we have lost a wise elder.
Hackers for right, we are one down.
Parents all, we have lost a child. Let us weep.