What’s a QA team without a spec? A goddamned nuisance and a waste of time, that’s what.
Man I hate when QA people do their jobs without specs! It’s so irritating. When Asheron’s Call 2 launched, there were thousands of outstanding bugs in the QA database that we opted not to fix before launch. Sounds bad, doesn’t it? But if you looked at them, you’d understand. Hundreds of them were bugs about how buildings were floating 2 virtual inches off the ground; if you zoomed the camera down to the floor and looked up, you could see that these structures were very slightly hovering.
Hundreds more were about items that “popped in” too soon or too late; the “art degrades” for the items weren’t set up right, so they seemed jumpy. And so on… thousands of little tiny nits.
Why does that piss me off? Surely those bugs should have been in the bug database, right? Even if they’re not fixed immediately, they’ll get fixed at some point! Right, true. Except for this:
Asheron’s Call 2 launched with over 3000 severe yet unrecognized bugs. It is not unreasonable to argue that AC2’s early failings were due to the lack of quality in things like crafting, combat, and skills. Every time the QA people entered a bug about a floating object, that was time they weren’t spending finding more serious bugs. For instance:
- Many of the crafting recipes were about ten times harder to create than intended. The creatures that dropped the needed parts spawned incredibly rarely, due to an oversight.
- Many of the quests could easily become broken if the steps of the quest were done in the wrong order. Fixing them then required the assistance of a customer service representative.
- Almost every skill in the game was broken in some way. Some skills literally did nothing; others were too costly or too powerful; some started out super strong and then got weaker as you leveled up the skill.
And so on. Serious bugs, very much worth fixing. These didn’t make it into the database at all; the live team stumbled upon them when players started screaming about them. What the hell, QA? What the hell?
I’m just being a jerk to QA here. One of the reasons that QA did such a terrible job on Asheron’s Call 2 was because there were almost no design specs.
It was the extremely talented Jesse Kurlancheek (a.k.a. “Devilmouse”) who was responsible for the skills in AC2. Because of the agreement Turbine had made with their publisher Microsoft, he was obligated to create at least 600 skills for the game before it shipped, broken into 30 different skill trees. Problem was, he only had three months to design and implement them all. Jesse famously told his boss, “I can either implement the skills, or I can document the skills, but there’s no time to do both.” And he was right. But they chose to implement instead of documenting, and the result was tragedy.
Without any idea of how skills were supposed to work, QA never looked at any of them. To be fair, QA should have at least poked around with them some, but they just weren’t motivated to spend any time on them because the designer would almost always say “no, that’s not a bug, that’s how it’s supposed to work!” So the QA team spent their time wandering around looking for objects that were hovering a tiny bit off the ground, instead. At least those were irrefutable bugs!
The lack of specs was devastating. When the game launched, the live team needed to get the game into a maintainable state. Without specs, we had no idea what Jesse had intended. Worse yet, a few months later, not even Jesse remembered what the intention of each skill was. So we created elaborate analysis software in order to locate skill deviations, and slowly reverse-engineered the intentions, like archaeologists exploring an ancient culture. The lack of specs cost us thousands of man-hours. It made us look like total dicks, too, when we fixed the outrageous bugs we found.
The “reap” abilities in AC2 stole the health of an enemy and gave it to you. However, as is typical of this sort of power, you couldn’t steal more life than you were missing. So if you were only missing 100 health, you could never steal more than 100 from the enemy. Tragically, there was an accidental minus-sign in the implementation. If you reaped somebody and you were fully healthy, instead of doing no damage, you did double damage. Reaps were the most effective attacks in the game, provided you hadn’t been hurt too much. It made PvP in particular a nightmare, and players were howling for us to fix it. When we fixed it, though, several classes became unplayable; it turns out they had only been fun at all because of that broken skill. So then we had to buff those in various ways, until the next major bug was discovered, which threw our rebalancing out of whack again. And on and on, over and over, a constant dance of chaos and confusion, for over a year, before we had created our own specs and could work towards stability.
If only the skills had been affected this way, we could have dealt. But the lack of specs permeated almost every aspect of the game. The quest areas were about half-documented; the monster spec was nothing but one extremely complex spreadsheet. It’s not like the AC2 designers didn’t know how to make specs, of course: they were given impossibly small time windows to do their work, and they did their best.
If our publisher had allowed us, it would have been so much better for AC2 to have launched with only half the classes, but with docs for all of the classes. Then the live team could just implement the docs, rolling out new things every couple months, and make Turbine look super productive instead of super incompetent.
In the end, specs save money for three reasons.
- Specs allow for collaboration on a design. If it’s all in your head, nobody can point out the flaws in your plan until you’ve implemented it already.
- An MMO that is supposed to run for 5 years needs at least a rough semblance of documentation, or the game’s maintenance will cost a ton.
- Specs are also mandatory if you want to use a QA team. If you can’t afford to write specs, just fire the damned QA team, or convert them to content designers or something. QAing without specs is an amazing waste of time and just makes everybody angry.
And just to be clear: I love working with a talented QA team. I am 100% pro-QA. But you must give them the ability to succeed. Without specs, you’ve set them up to fail.
The Problem With Rapid Content Creation Tools
When I was at Perpetual, the Gods and Heroes team had a quest-development process where designers didn’t need to write out all the details of each spec. They just opened the database and plop! they dumped their quest right into the game. And man, did they hate QA. They agressively, loudly, viciously denounced the QA department. QA was always telling them the wrong thing and never finding the real bugs. The QA at Perpetual was the scapegoat for all problems. Even the QA team lead was apologetic for how shitty they were.
But how could they be successful? The only notion of how these quests were supposed to work was in one designer’s head. Eventually, the QA team wisely stopped reporting bugs about quests. But there were still bugs in the quests.
When Perpetual’s Star Trek team was planning their content pipeline, everyone just assumed the same system would be used. This was a major point of contention for me: I wanted designers to write out every detail about the quest before they implemented it. “That’s ridiculous! It will double the time needed to implement quests! We can’t budget for that!”
Okay, fine. But if you can’t budget for specs, don’t budget for QA, either.