Dwarf Fortress, Facebook, Big Data and the Search for Story

Last night after driving home from the Austin PyLadies meetup, my wife sat in our driveway for 20 minutes listening to the end of an episode of WNYC’s Radiolab.  Later, after we’d headed to bed, she spent another 20 minutes retelling the story to me, minus Radiolab’s flourish and production.  The story was still interesting second hand, and comes down to this (I’ll wait if you’d like to go listen to the episode of Radiolab, I’m sure it’s excellent):

Two people discover hundreds of letters from WWII on the side of Route 101.  They’re from soldiers replying to a woman on the homefront.  The soldiers call her mom, but she isn’t their mother.  The two ask around, no one knows anything about them.  One of them, a creative writing professor, ends up using the letters as projects for his students.  He gives them a letter, and their task is to create a story around it.  A soldier, a woman stateside, an unlikely connection.  The other discoverer wants to track down relatives, she wants to uncover the truth.  She ends up discovering it, but he’d rather not know.  He wants the possibilities.

Even told second hand, the story stuck with me on a meta-level.  There aren’t a lot of things that would make my wife sit in the car in the driveway for 20 minutes listening to the radio, but a good story is one.  We love stories, we love it when they’re well crafted and well told.  But we also love the possibilities of them.  Sometimes we don’t want the truth, we want magic, we want to dream the dream of what could be.  Sometimes the truth can’t exist, and the closest we can get is a dim outline of it.  Sometimes the dream is better.

The Promise: Stories that Tell Themselves

A few days ago I ran across a blog post by Tynan Sylvester, a designer on the game Bioshock Infinite.  It’s all about the dream of simulations for game designers, how we think that by creating more and more complex systems, we might eventually build a system that is complex enough to manifest stories.  Austin Grossman’s latest novel, YOU, is about that, in a way.  The protagonist is a game designer and the antagonist is just a manifestation of some long-running game rules.  As game designers, we want to design games that surprise us.  That’s the ultimate payoff, to build a game that entertains you, and not just a twitch game that is enjoyable for its mechanics, but a game with stories compelling enough to sit in the car in the driveway for 20 minutes at 9 o’clock at night.

Lots of game designers have tried to do this. Tynan talks specifically about systems in early versions of Bioshock where the player would have to play autonomous bots (splicers, gatherers and protectors) off each other to progress.  They hoped that amazing, emergent gameplay would be the result.  In the end it didn’t work, and the game moments that they’d hoped would happen spontaneously ended up being heavily scripted.  Players crave story, but that story can’t be left up to their persistence and chance, especially when creating a commercial title.  In that environment, a great story has to be guaranteed.

Dwarf Fortress: Madness in Text Mode

There are a few notable exceptions to this principle, and they’re mainly smaller games driven by singular minded creators.  The best example of this is Dwarf Fortress, a massive and inscrutable simulation game where the the player takes on the role of an overseer, and the titular dwarves are simulated autonomous entities inhabiting the world.  Dwarves have names and hair colors, what Tynan calls Hair Complexity, things that add perceived simulation depth without effecting anything else.  (When was the last time you played an RPG where a plot point hinged on your hair style?)  They also have more integrated systems like hunger and social needs.  They have personalities, they get sad, and sometimes they go crazy.  The dwarves live in a randomly generated world, so your game isn’t like my game, and even my second game won’t be like my first.

Dwarf Fortress has a very dedicated core following, and one of the reasons is that it really lives at the edge of apophneia, the experience of seeing meaningful patterns emerge from random data.  At the core of Dwarf Fortress is a collection of rules governing behavior.  A dwarf without food will eventually starve.  A dwarf without personal interaction may eventually go crazy.  Dwarves are scared of wolves.  Dwarves exist in a world generated fractally, a world that feels real because it mirrors patterns in nature.  Therefor, as more and more rules get layered on, and more and more people play more and more games and get better and better at creating experimental mazes for these digital rats to play in, stories begin to appear, or so we perceive.

Two of the most famous stories to come out of Dwarf Fortress games are Boatmurdered, the tale of an epic game played out by members of the Something Awful forums in 2007, and Bronzemurder, a beautiful infographic-style tale of a dwarf fortress and a terrible monster.  Go read it, it’s great.

Dwarf Fortress didn’t generate these stories, though.  People played the game, sometimes hundreds or thousands of times, and while gazing into the mandala of the game, they nudged and pulled the threads of the world and created stories based on the events that occurred there.  Dwarf Fortress isn’t a windup toy, it’s a god-game, and the players impact on the game world is more than negligible.  The stories generated there are as much created by the players as by the game.

I Fight For the Users

While my wife was out at PyLadies last night, I coincidentally watched TRON: Legacy.  It occurred to me as I was thinking about writing this post, that it’s a movie about this possibility: The dream of a world inside a computer, a world created by a brilliant programmer, a world that once set in motion can create stories, unexpected events and enthralling narrative.  The creator steps aside, and no longer controls the game from the top-down.  The creator becomes a god among men, watching things unfold from their level.

In TRON: Legacy, the magic of digital life comes in the form of Quorra, the last of the ISOs, Isometric entities that appear spontaneously from the wasteland of the computer.  Digital DNA, digital life.  Enough rules, enough circuitry, enough care and magic happens.  That premise is exciting, and to programmers it’s intoxicating.  For those of us in the digital generation, that’s the dream we live with.  That’s what we keep trying to make happen wherever we go and whatever project we work on, be it big data or software bots.

But the lone programmer, no matter how brilliant, and working for no matter how long, can only produce so much code.  Stories from one person only grow so far, only change so much, and rarely surprise and enthrall.  Dwarf Fortress as a dwarf isn’t a game most people would play.  It’s hard to see the overall story, and the game isn’t good at presenting it.  But if there were more players…

EVE Online: More Interesting to Read About Than to Play

If it’s possible (albeit insanely difficult) to have stories appear in a single player game, it must be easier for stories to manifest in a multi-player game, right?  Games like World of Warcraft have largely fixed, planned out stories.  It comes back to the challenge that Bioshock had, complex systems are exciting to designers, but players want immediate story gratification.  Complex systems take dedication to understand, dedication most players don’t have.  When new multiplayer games are announced they sometimes hint at players making a real impact on the world, but those systems usually fail to live up to the hype.  The latest game to promise this is The Elder Scrolls Online.  We’ll see if they can do it.

One game that does this and thrives is EVE Online.  EVE is a massively multiplayer online space combat simulation, one that spans an entire universe.  It’s possible to play EVE as a loner, but it’s also possible to align yourself with a faction, and have your small efforts merge with hundreds or even thousands of others to build armadas and giant dreadnaught ships, to control entire solar systems and even galaxies.  The designers and administrators of EVE take a largely hands-off approach.  They don’t want to kill the golden goose, so they design the game for balanced conflict, and let the players sort it out.

Every once in a while something epic happens in EVE, either a massive fraud, an invasion a faction planned for months, or a random accident that led to a game-rebalancing war.  There are battlefield reports, and once the space dust settles, people start to put together a history, and an accessible storyline appears.  Here are a few great EVE stories.  More people probably enjoy the reports of epic battles in EVE through these stories than actually play the game.  To quote a MetaFilter comment thread: “This game sounds stressing as hell if you really play it and not just dither around. Fascinating to read about, however, almost like news from a parallel universe.”

You could say that EVE is a computer program for generating stories, and in fact the’ve even made a deal to do a TV show based on player stories from the EVE universe.  Except again we find that that EVE isn’t the thing generating the stories, EVE is just a place where the stories happen.  To a player only experiencing the events inside the game it may seem mysterious and amazing, and it certainly is to those of us who read about the events afterwards, but it’s really just a sandbox.  People play pretend with enforceable rules, but you can’t separate a story that happens inside of EVE with the real life stories that happen outside of it: The scheming that happens on IRC or in forums, the personal vendettas, the flexible allegiances  and the real-world money that flows through the system.  There’s no way to watch something occur inside of EVE, and even if you had perfect clarity on everything that happened inside, have any way of knowing for sure what really caused it.  If you take away the players, the legions of dedicated fans scheming and plotting, you just have an empty universe.

Facebook and the Timeline of Truth

I think a lot of web developers secretly wanted to be game designers.  Becoming a game designer is difficult, there aren’t as many jobs and the hours are terrible.  Instead we build web sites, but we’re building systems too, and we want to tell stories.

I joined Facebook back in April of 2006.  I had a @swt.edu address from Southwest Texas State (now Texas State University) from an extremely brief stint (sub 1 day) as an IT staff-member, so I got in a few months before they opened it for everyone.  Getting into a new, exclusive social network is a bit like finding a new simulation.  We hope the software can tell us new stories, that it can make some sense of the data it has.  With Facebook the promise was that if it collected enough information about us, it could tell us that magical story.  That’s what Timeline was supposed to do.  Give Facebook enough photos, enough checkins, enough friend connections, enough tagged posts and it would be able to tell the story of our lives.

In the end, though, Timeline doesn’t tell you a real story.  It reminds you of stories you’ve heard and experienced, but Facebook is only a dumb algorithm working with imperfect data.  It’s smart enough to target ads, but it can’t understand the meaning, and it can’t remix the data in really compelling ways.  It can’t be Radiolab.  Most of the time the prioritization it comes up with I just want to turn off.  Its attempts at story are so bad I’d rather use my own organic cognitive story filters.

With every new Facebook feature announcement, with Google+ or the next thing that processes all your activity, the promise is that the system can get better at telling those stories.  We want to believe it will happen.  We want to believe that a couple thousand web developers and a couple billion dollars could create a story machine, but I’m not sure it can.  I was reading an article about HP’s R&D budget the other day that said Facebook invests 27.5% of revenue in R&D, a larger percentage than any other company they tracked.  You can bet a good chunk of that is going towards the search for story, in some form or another.

Weaving a Web

I’d be remiss if I didn’t mention Weavrs at this point, since they are essentially digital actors that derive stories from the mess of social media.  Weavrs are designed specifically for apophneia, they produce content one step up from random, and rely on our desire for patterns to throw away the things that don’t fit.  We project stories on to them, and for a project with the limited resources that it had, it’s exceedingly good at it.

My weavr twin is posting about HP Moonshot servers.  That’s almost eerie, but it’s also posting about hockey tickets.  The story makes sense if I’m picky about the things I include, but it isn’t an internally consistent narrative.  The narrative is impressed on it by the people who see it, like reading digital tea leaves.  Your story of my weavr is different than mine.

With enough resources and time, weavrs might become a real story machine.  That’s a moonshot program, though, and I don’t know who’s going to step forward and make that happen.  Investment follows money, and right now the money is racing towards big data.

Autonomy: Billions and Billions

The lure of story, the promise of meaning from the chaos of data isn’t limited to games or the social web.  It’s the romantic beating heart of big data.  It’s the stories about Target knowing you’re pregnant before you do.  It’s what lured HP to spend $8.8 billion dollars more than it was worth to acquire Autonomy.

Autonomy’s main product is called the Intelligent Data Operating Layer, or IDOL (symbology, ahoy!).  They call the processing of information with it Meaning-Based Computing.  From what I’ve heard it’s certainly good at what it does, but while it promises Meaning from Data, and that promise separated HP from 9 Instagrams or 2,500 Flickrs, there has to be some apophenia at work here.  Just like watching solar system battles inside of EVE gives you a piece of the story and playing hundreds of games of Dwarf Fortress will result in games worth telling stories about, the system data is never the entire picture.

I really like Stephen Wolfram.  Stephen believes in the fundamental computability of everything.  While I love reading his blog posts, and I am interested in and admire his idea, I have to wonder how far the hyperbole is from actual execution.  Given enough computable facts and enough understanding about the structure of narrative, a perfect Wolfram|Alpha should be able to tell me stories about the real world.  But it can’t.  They aren’t even trying to approach that.  Wolfram|Alpha isn’t creating Radiolab.  They want answers, not stories.  You know what tells stories? Dirty, messy, all-too-human Wikipedia.

A Different Kind of Magic

My friend Matt Sanders works for a bay area company called Librato.  Librato is a big data startup, having pivoted from some other work to running a service that collects vast amounts of metrics and provides dashboards on top of it.  With Librato Metrics you can feed data points, set alert triggers, create graphs, and watch activity.  It’s big data without the prediction.  It promises no magic, but relies on our own.  It optimizes data for processing by human eyeballs.

The 3 pounds of grey matter between your ears is still the best computer we have, running the best software for deriving stories and making sense of data.  Librato works because it doesn’t try to be what it can’t.  Google Analytics tries to offer Intelligence Events, but more often than not, it can’t offer anything more helpful than that visits are up from Germany 34%.  You would think that by combining traffic source analysis with content changes and deep data understanding Google would be able to tell you why visits are up from Germany, but most of the time that basic percentage is the best it can offer.  It still takes that 3 pounds of meat to pull together the data and interpret it into a story.  While computers may be generating articles on company reports or sports games, they’re not creating Radiolab.

Wrapping Up

I think there’s still a lot of room for innovation here.  The Archive Project I dreamed of long ago is essentially a system for telling stories and discovering meta-stories.  Maybe someone will finally build it.  Maybe the next Dwarf Fortress will be a world that runs persistently in the cloud, a world where our games interact with other people’s games, where crowdsourced Hair Complexity snowballs until you can get lost in the story if you want to. A game where if you want to turn off a random path and follow it down to the river you’ll find a fisherman who will tell you a tale interesting enough to make you sit in your car for 20 minutes, enthralled by a narrative.

Maybe the framing of a story is what big data needs to become personally relevant.  Maybe that’s its magic trick.  Maybe narrative is the next great big data frontier.

Future Past

I sometimes wonder about the generation of kids growing up today, in this big data, analytic-driven, always-on world.  I wonder how they will embrace it, like we embraced computers and connectivity.  I wonder if they’ll have the ability to hear the prognostications of the computer, to listen to the story from the machine, and consider it a kind of truth.  To internalize it, but also keep it separate.  To know the machine knows a truth, but not necessarily the absolute truth.  Maybe that will be their power, the thing they can do that those of us from the generation before can’t. Maybe that is where the dream finally comes true.