The Secret Life of a Dev Team

This isn’t an MMO news site — we try to focus on practical advice, not ongoing drama — but recently Blizzard made a particularly interesting decision that I want to use to illuminate what goes on inside a dev team. And believe me, unless you’ve seen it first-hand it can be difficult to believe what actually goes on inside a dev team.

A Decision Badly Communicated

In WoW Patch 2.3, Blizzard inadvertently introduced a new creature (a ghostly wolf) that was obviously not intended to be tamable by hunters, but which could be tamed through very clever use of game mechanics (including putting together a small group and dropping a couple hundred gold in specialized equipment). This wolf had no special qualities other than a transparent appearance and no gameplay benefits over other wolves in the game. Players loved it because it looked cool, but it was obviously a mistake so they asked for clarification from Blizzard … and got it: a Blizzard CS Rep posted in the European forums that Blizzard had “no plans to address this issue – it will still be possible in future for everyone who wishes to tame this NPC to do so.” Unfortunately, six weeks later Blizzard apparently changed its mind; the ghostly wolves were removed in a stealth hotfix early one morning. Frantic inquiries resulted in a post by a Community Manager on the US forums, that said: “Through a hotfix we’ve recently removed the ability to tame a Grimtotem Spirit Guide [ed. -- that is, the ghostly wolf].” Although no direct reason was given for the change, they were apparently concerned about the precedent set by “(t)he unintended nature of the taming, the undead status of the guide, appearance of the wolf in relation to the feel of the hunter class, and the complex processes of taming”. Players who had already tamed a ghostly wolf, by the way, got to keep their pet.

To sum up, Blizzard removed content that did not affect game balance after stating that they would not remove the content, and they did so in a stealth hotfix with no prior notice despite the fact that doing so caused a number of players to waste their hard-earned gold.

It’s Not That Simple

Now, I’m not here to rant about this as a player. (Plenty of people are doing that elsewhere!) But let’s look at this situation as a developer: Obviously this isn’t great for Blizzard — it will blow over (as these things always do), but in the meantime it’s doubtless causing some headaches for their CS and Community teams, especially increased CS tickets (which means increased wait times for everyone). Things like this are almost never the core reason that players quit the game, but as we’ve seen from exit surveys they are often the straw that breaks the camel’s back — the trigger that prompts a fence-sitting player to cancel their subscription. A situation like this is not the end of the world, but it’s definitely to be avoided.

So why did Blizzard handle this situation the way that they did?

Sometimes an unpopular, unannounced change is well worth the cost, if there is some good reason to make the change — for example, if a bug threatens game balance and announcing the fix beforehand increases the chance that a lot of players will exploit the bug. And it is even possible to come up with a situation in which a detailed explanation cum apology afterwards is a bad idea because other similar bugs may still remain. But was that the situation here? Since the bug was a misplaced content flag and they let hunters who had already tamed the beast keep it, that seems unlikely.

Sometimes the best reasons for a change aren’t visible to players — and aren’t intended to ever be known by players. For example, as producer of AC1 I once decided to delay a fix for an unfortunate economy-wrecking crafting bug for several weeks. Why in the world would I do that? Because our publisher was in charge of hotfixes; because I wasn’t satisfied that the personnel they had available was sufficient to perform a hotfix safely at that time; because the political situation between us and the publisher was such that attempting to negotiate a safe hotfix would have severely hampered other, more important development priorities; because I knew that in the end, the game would recover from the bug, but I couldn’t guarantee that the team would recover from pissing off the publisher. I’m certain that from the players’ point of view, my communications about that situation were appallingly obtuse. But again, that hardly seems likely to be the situation here.

What Really Happened?

I can’t actually tell you why Blizzard decided to handle the ghostly wolf matter the way they did. I don’t know anything about the internals of WoW’s development process or team. But I’ve seen similar situations develop so many times that I can make a pretty good guess as to what happened. The following fictitious story could easily happen at any of a half-dozen MMO companies:

So. It all started when CS was bombarded with questions about the ghostly wolf. The lead on duty first tried asking the Community team if they knew anything about it. CS and Community don’t always get along — Community is too close with the Marketing team, and everyone hates and fears Marketing — but at least they answer e-mails, unlike the damned Dev team. But unfortunately Community didn’t have a clue either, although they had been noticing similar questions.

So straws were drawn and the loser (the Community lead in this case) was dispatched to go ask his contact on the Dev team. Unfortunately there had been another flare up between the producer and the head of Operations (who ran the CS and Community teams) just last week, so the frequently-ignored edict that all contact must be through official channels was back in force for the moment. The Community lead eventually found his contact, a production under-flunky, racing up and down the halls with a bug list in hand. He tried to ask about the ghostly wolf and got a shouted “E-mail me!” over the shoulder as the contact ran off. The Community lead went back to his desk to e-mail the contact, as requested, but he knew he wouldn’t get a response for a couple of days at least.

In the meantime, a diligent CS rep who was desperately trying to be helpful managed to sneak into the Dev team’s area without being caught. She found the designer in charge of the Dustwallow Marsh revamp (the new content that inadvertently introduced the ghostly wolf) and asked him if it was going to be fixed. He thought for a second and then said, “No, I don’t see why it would be. It doesn’t do any harm and it’s kind of cool.” Triumphant, she returned to her desk with an actual answer to reassure the players.

Two days later, the community contact on the Dev team brought up the question in a weekly Dev team leads meeting. The Dustwallow Marsh designer wasn’t present as he wasn’t a lead. After some discussion, the producer decided that he wanted to leave the option open for adding some sort of undead-pet hero class to hunters at some later time, maybe in the expansion after next. Of course it did mean nerfing the existing pets, but since everyone in the room knew that it was easier to just go along with the producer on these things, and since no one in the room would actually have to deal with the backlash, this seemed like the easiest solution. So after the meeting, the programming AP entered a bug into the bug tracking system and made sure it got assigned to a programmer with some free time for an immediate fix.

The next day, however, the lead designer found out that CS had told players that the ghostly wolf would stay. She tracked this information back to the Dustwallow Marsh designer, who convinced her that it was really quite a cool use of game mechanics and anyway, everyone knew that the producer was smoking crack and there were no plans for a undead pets in the next two years. And moreover, a statement had already been made. So the lead designer told the programming AP to back out the bug and then went to see the producer to let him know.

The programming AP didn’t have the right flags on his account to remove a bug so he went to the QA lead for help. The QA lead became rather irate that no one had mentioned this priority bug to her yet, but since that happened several times a week let it pass with nothing more than a terse e-mail to the programmer who was working on the bug. The programmer, however, was friends with a QA flunky who was convinced that the QA lead hated her (the flunky) and therefore took the terseness personally and decided to fix the damned bug anyway, because who was QA to say what was a bug or not? Only the Dev team can decide that something shouldn’t be fixed! So motivated, the fix was completed and checked in in record time. However, since the bug had been retracted, there was unfortunately no record of the fix.

In the meantime, the lead designer was meeting with the producer, who seized on the fact that the carefully laid plans of the Dev team were being dismantled because of a lowly CS rep who wouldn’t follow procedure. He dashed off a rude note to the head of Operations (just another salvo in their ongoing feud) and then ranted for a bit about his plans for undead pets. The lead designer did eventually talk him around into leaving the ghostly wolf for now, though, largely by exaggerating how much time a proper fix would take away from development on the current expansion.

And so matters rested for many weeks, until the build with the fix went live. Suddenly CS and Community were being deluged with questions about the ghostly wolf again. The CS lead immediately called the Dev team contact, who quickly discovered that the programmer really had fixed the bug even though he shouldn’t have. The contact called the producer who, remembering only about half of how this all came about, called the Community lead to bitch him out and incidentally get a statement up at once. The Community lead was fairly annoyed at being yelled at by the producer over the situation when he had played very explicitly and carefully by the rules, posted the most obtuse statement he thought he could get away with and then pretended to be in meetings the rest of the day.

And CS was left to pick up the pieces.

The Big Secret

The big secret is that the typical team of 40+ individuals is completely incapable of working as a single entity. There are cliques, there are squabbles, there are stupid dramas getting in the way of everything. Most dev teams suffer from incredible levels of internal miscommunication and politics.

Whenever I interview with an MMO company, I talk about the need for strong internal communication. Most of the time my interviewer just smiles and nods. They usually believe that they already have great internal communication. They are always wrong. Strong communication doesn’t just come from conveying information — it also requires trust, understanding of intent, passion for the game, and the ability to put aside personal disputes for the good of the team.

A live team can become one coherent whole and when that happens everybody is so much more productive, it’s incredible. A gelled team is a beautiful thing, and once you’ve been part of one you will never want to work any other way. But bringing strong organization to a team of 40+ requires extremely masterful leadership as well as hard work, plus some good old-fashioned luck.

[By the way, if the concept of "gelled teams" is new to you, I recommend the classic book Peopleware: Productive Projects and Teams, by Tom DeMarco and Timothy Lister.]

This entry was posted in Production. Bookmark the permalink.

5 Responses to The Secret Life of a Dev Team

  1. Nikos Beck says:

    It doesn’t have to be a 40+ person team to have poor communication. I was on a team of six, fixing bugs on the bug list that the lead designer was never informed of. It was flagged “approved” by a lowly designer after discussing it with QA. It took a day for the lead designer to audit every single bug pending and fixed within the previous two weeks, huddle with QA about the procedure, huddle with programming about the procedure, add fixes to reverse or modify the fixes that were already in the game. It was a mess.

  2. Sandra says:

    That is very true. We had our own miscommunications on AC1 even when it was a team of four-ish. But I think that people assume the big ‘professional’ teams are better somehow, when in fact they are usually worse by orders of magnitude.

  3. Cathy Nicoloff says:

    I must have been too insulated to see that level of infighting and politics from QA (or else I am oblivious to such, hmm). However, things that got put out as sudden “stealth nerfs” often fell into two categories.

    1. Exploits. Like you mentioned above.
    2. A fix that was scheduled for a later patch, but was already coded up, so it got thrown in a patch early. When this happened, the start of testing in QA would have been delayed for some reason (maybe the release candidate build was broken for several days, or the release candidate was missing stuff that should have been delivered a week ago). A dev lead or the producer might look at a list of bugs that were already marked “RESOLVED FIXED” for the next patch and say, “When you get a sec, check these half-dozen changes into the release candidate tree before QA starts testing,” because everyone likes it when more bugs than expected get fixed ahead of schedule. The community people who are responsible for posting a weekly list of upcoming changes might not be told, or might only find out when the official patch notes are generated the day before the patch, so the players don’t find out until the last minute. Cue player uproar and conspiracy theories about a “stealth nerf”.

    The second option leans heavily on “lack of internal communication” like you mentioned, but is minus the internal warfare in your hypothetical scenario.

  4. Dkhjfnop says:

    cool post dude