What’s a QA team without a spec?

What’s a QA team without a spec? A goddamned nuisance and a waste of time, that’s what.

Man I hate when QA people do their jobs without specs! It’s so irritating. When Asheron’s Call 2 launched, there were thousands of outstanding bugs in the QA database that we opted not to fix before launch. Sounds bad, doesn’t it? But if you looked at them, you’d understand. Hundreds of them were bugs about how buildings were floating 2 virtual inches off the ground; if you zoomed the camera down to the floor and looked up, you could see that these structures were very slightly hovering.

Hundreds more were about items that “popped in” too soon or too late; the “art degrades” for the items weren’t set up right, so they seemed jumpy. And so on… thousands of little tiny nits.

Why does that piss me off? Surely those bugs should have been in the bug database, right? Even if they’re not fixed immediately, they’ll get fixed at some point! Right, true. Except for this:

Asheron’s Call 2 launched with over 3000 severe yet unrecognized bugs. It is not unreasonable to argue that AC2′s early failings were due to the lack of quality in things like crafting, combat, and skills. Every time the QA people entered a bug about a floating object, that was time they weren’t spending finding more serious bugs. For instance:

  • Many of the crafting recipes were about ten times harder to create than intended. The creatures that dropped the needed parts spawned incredibly rarely, due to an oversight.
  • Many of the quests could easily become broken if the steps of the quest were done in the wrong order. Fixing them then required the assistance of a customer service representative.
  • Almost every skill in the game was broken in some way. Some skills literally did nothing; others were too costly or too powerful; some started out super strong and then got weaker as you leveled up the skill.

And so on. Serious bugs, very much worth fixing. These didn’t make it into the database at all; the live team stumbled upon them when players started screaming about them. What the hell, QA? What the hell?

I’m just being a jerk to QA here. One of the reasons that QA did such a terrible job on Asheron’s Call 2 was because there were almost no design specs.

It was the extremely talented Jesse Kurlancheek (a.k.a. “Devilmouse”) who was responsible for the skills in AC2. Because of the agreement Turbine had made with their publisher Microsoft, he was obligated to create at least 600 skills for the game before it shipped, broken into 30 different skill trees. Problem was, he only had three months to design and implement them all. Jesse famously told his boss, “I can either implement the skills, or I can document the skills, but there’s no time to do both.” And he was right. But they chose to implement instead of documenting, and the result was tragedy.

Without any idea of how skills were supposed to work, QA never looked at any of them. To be fair, QA should have at least poked around with them some, but they just weren’t motivated to spend any time on them because the designer would almost always say “no, that’s not a bug, that’s how it’s supposed to work!” So the QA team spent their time wandering around looking for objects that were hovering a tiny bit off the ground, instead. At least those were irrefutable bugs!

The lack of specs was devastating. When the game launched, the live team needed to get the game into a maintainable state. Without specs, we had no idea what Jesse had intended. Worse yet, a few months later, not even Jesse remembered what the intention of each skill was. So we created elaborate analysis software in order to locate skill deviations, and slowly reverse-engineered the intentions, like archaeologists exploring an ancient culture. The lack of specs cost us thousands of man-hours. It made us look like total dicks, too, when we fixed the outrageous bugs we found.

The “reap” abilities in AC2 stole the health of an enemy and gave it to you. However, as is typical of this sort of power, you couldn’t steal more life than you were missing. So if you were only missing 100 health, you could never steal more than 100 from the enemy. Tragically, there was an accidental minus-sign in the implementation. If you reaped somebody and you were fully healthy, instead of doing no damage, you did double damage. Reaps were the most effective attacks in the game, provided you hadn’t been hurt too much. It made PvP in particular a nightmare, and players were howling for us to fix it. When we fixed it, though, several classes became unplayable; it turns out they had only been fun at all because of that broken skill. So then we had to buff those in various ways, until the next major bug was discovered, which threw our rebalancing out of whack again. And on and on, over and over, a constant dance of chaos and confusion, for over a year, before we had created our own specs and could work towards stability.

If only the skills had been affected this way, we could have dealt. But the lack of specs permeated almost every aspect of the game. The quest areas were about half-documented; the monster spec was nothing but one extremely complex spreadsheet. It’s not like the AC2 designers didn’t know how to make specs, of course: they were given impossibly small time windows to do their work, and they did their best.

If our publisher had allowed us, it would have been so much better for AC2 to have launched with only half the classes, but with docs for all of the classes. Then the live team could just implement the docs, rolling out new things every couple months, and make Turbine look super productive instead of super incompetent.

In the end, specs save money for three reasons.

  • Specs allow for collaboration on a design. If it’s all in your head, nobody can point out the flaws in your plan until you’ve implemented it already.
  • An MMO that is supposed to run for 5 years needs at least a rough semblance of documentation, or the game’s maintenance will cost a ton.
  • Specs are also mandatory if you want to use a QA team. If you can’t afford to write specs, just fire the damned QA team, or convert them to content designers or something. QAing without specs is an amazing waste of time and just makes everybody angry.

And just to be clear: I love working with a talented QA team. I am 100% pro-QA. But you must give them the ability to succeed. Without specs, you’ve set them up to fail.

The Problem With Rapid Content Creation Tools

When I was at Perpetual, the Gods and Heroes team had a quest-development process where designers didn’t need to write out all the details of each spec. They just opened the database and plop! they dumped their quest right into the game. And man, did they hate QA. They agressively, loudly, viciously denounced the QA department. QA was always telling them the wrong thing and never finding the real bugs. The QA at Perpetual was the scapegoat for all problems. Even the QA team lead was apologetic for how shitty they were.

But how could they be successful? The only notion of how these quests were supposed to work was in one designer’s head. Eventually, the QA team wisely stopped reporting bugs about quests. But there were still bugs in the quests.

When Perpetual’s Star Trek team was planning their content pipeline, everyone just assumed the same system would be used. This was a major point of contention for me: I wanted designers to write out every detail about the quest before they implemented it. “That’s ridiculous! It will double the time needed to implement quests! We can’t budget for that!”

Okay, fine. But if you can’t budget for specs, don’t budget for QA, either.

This entry was posted in Production. Bookmark the permalink.

31 Responses to What’s a QA team without a spec?

  1. Peter aka Shark (Dawnsong) says:

    As a former AC2 player I really enjoyed this article.
    I started playing in Feb of the first year(2003?). I didn’t have that many problems with the game at start. It was when the chat server went down for a month!! that I almost left the game.
    I did hang in there until the very end and in the last year the game was REALLY good.
    One thing I remember was when you added tons of armour to mobs in a patch. That almost killed the game aswell. The players went from being “uber” to being weak ants in comparison, at leat it felt that way.
    I played a “Zerker” from start and hung in there during all the nerfs and buffs, and in the end that class is still the most fun I have had in any MMO I have played.
    AC2 really had a bad start and if the game was launched just 6 months later it might have been running still since most people seem to have left during the first 6 months of the games life.
    Still playing LoTRo today and dreaming of the return of AC2 :)

  2. Stropp says:

    It’s an interesting situation because that was essentially a management failure in both cases, not a QA failure. And it happened on at least three levels too.

    1) Management failed to budget for a proper requirements/design phase.
    2) Management failed to respond to the issues arising with QA, including making life easier for the QA team by getting the process ironed out.
    3) In Perpetuals case, management failed to even recognise that there was a problem to correct.

    It’s not just the game industry where this happens, I’ve seen it in a few places that I’ve worked at. It just seems like the game industry is more open about it, post event. There’s a reason Dilbert is so popular.

  3. Fenryr says:

    Wow, this is pretty insightful. I think it was clear to anyone who played AC2 in that first year that something had gone REALLY wrong in the design process – i dont think there was a single game aspect other than the graphics that was the same 2 years on from the launch. Very interesting to read what the exact issue was – QA didnt know what they were looking for.

  4. robusticus says:

    I still think test automation would go a long way to that nerf-buff cycle of “chaos and confusion”. But yeah, how do you automate something that isn’t documented? Still, 600 skills is alot of permutations and I’m sure there are many modifiers on those skills external to the skills themselves. Would be a fun project but I don’t see any game company paying more than $10/hour to do it, and that’s fairly complex work. Maybe they could do it if they outsourced it to the third world.

  5. robusticus says:

    What I mean to say, it’s one of the few areas in all of software that I’ve seen where that part of the frAgile methodology might actually help.

  6. Cat says:

    robusticus: Farming out QA to India hasn’t worked any time I’ve been part of it (three times so far). Outsourced QA testers are less creative, less adaptive, and more literal than the in-house testers. Outsourced QA has to collaborate with in-house resources, which can be very tough if they’re nine timezones away. If you don’t give them every single test case broken down into tiny little steps with instructions and diagrams, they tend to ask hundreds of questions about the particulars of each step or just do it wrong, wasting the time of your in-house resources. If you’re going to go so far as to hand-hold outsourced QA through every single test case, you might as well do the test case yourself.

    Eric: Amen. Players scapegoat QA for being too dumb to find “obvious” bugs they find on patch day. Developers scapegoat QA as well. I have worked with designers who actively hated QA and did their best to obstruct our ability to test by refusing to give documentation. One designer went so far as to give complete specs to the playtest realm users after refusing to give them to QA! I have worked with content designers who didn’t do so much as a single test on their own content, not even to make sure it kinda worked. Hundreds of man-hours were wasted filing blocker bugs to say “your content is too broken to test”, then waiting for a fix, an overnight build, and a new day of testing just to discover more problems that prevent further testing. One notorious designer would check in dozens of tiny “fixes” per day without double-checking them, I guess in order to look super-productive. I’d have to reopen the same bug six times in order to get it properly fixed. When you’ve got to ship a patch in two weeks and you spend one week waiting for the content to be testable, you wind up shipping with obvious bugs because you run out of time. At the end, the game was to find as many bugs as possible and make them “Known Issues” so the players wouldn’t embarrass us.

    To be honest, the players yelling at us were preferable to the developers wasting our time. The players had a valid point. The developers were often playing passive-aggressive games. Not enough time was budgeted for the work that needed to be done, so the developers focused on what they liked to do (make stuff) and ignored what they didn’t like to do (write tedious documentation). QA would often chase down a developer and talk to them while taking notes just so we’d have a clue what we were supposed to test.

    In general, there seemed to be a lot of hatred between dev and QA, though I was not a dev-hater as some of my coworkers were. I was a live tester, meaning I cared about bug reports from players and reproducing them. I was not a tester who looked for tiny graphical glitches, though I’d bug report them if I came across them. I was not checking if the game was fun — how could I? We didn’t have a week to commit the entire team to running quests with every possible class. Management kept cutting QA (we finally had just two testers, one for bugs and one for content), so QA kept cutting out the niceties and concentrating on one thing — did we test everything that was officially checked in? Did it work to spec? Ok. It’s done. Is it fun? No time. Does it look polished? No time. Does it fit the lore? No time. Did it break something else? No time.

    At one desperate point, I started looking at the game data itself, comparing the old data to the fixed data. It was often faster to do this than it was to log in and test the change, as long as you could safely assume the entire involved game system wasn’t itself broken. This was a totally budget way to test monster, item, and spell fixes, but we were working overtime as it was. I begged my manager to pitch in and help. He refused. We never did fully test that patch. We pulled out all the stops and burned ourselves out.

    Anyway, you’re right. If you’re not going to have specs and you’re not going to have enough testers, why have QA? We had to pretend to the customers that we were a big company with lots of staff, all to keep stringing the players along while they slowly got disgusted with our inability to deliver functioning content on time. It was so sleazy and dishonest. I hated it.

  7. Wayne says:

    I remember beta testing AC2 and providing feedback about many issues we were seeing as players (or at least concerns as to how things appeared to work). This helps provide lots of insight into what was going on behind the scenes. I figured Microsoft pulling the strings had lots to do with the issues.

  8. SmakenDahed says:

    Ouch. Sounds like a lot of fun.

    I’m not a big fan of specs but I do realize they serve a purpose. I find it creates immediate bias towards one set of actions and use. Yes, any tester worth their pay will try to misuse it but it is really easy to fall into the ‘requirements validation’ trap – most management only care that it works, not what can be done with it.

    As a tester, it is nice to have an idea of what the feature is supposed to do but I actually prefer to be there for the design and planning stage. I get to see what the intent is, what the feature will be and I can question the design before it gets implemented. While they’re planning how to build it, I plan on how I’m going to bring it down and make sure they have plans for that.

  9. Pingback: And You Wonder Why… « Random Ogre Thoughts

  10. Pingback: FileCatalyst » QA - Come get some

  11. grudge says:

    Comment #2 by Stropp

    +1

  12. Mj says:

    Wonderful post.
    I have experience working as a QA on both MMO’s with outstanding and detailed documentation and MMO’s without even rudimentary documentation. In my experience, without clear, concise and detailed documentation we as a testing team archived less, found less bugs and covered less ground than a team half the size but with proper documentation.

    If the documentation is good you can test the basic functionality in a very short span of time, leaving a lot more time and energy to actively trying to break the system, trying to find situations where an otherwise working system can be abused and generate feedback on whether the system is actually fun and engaging when played.

    Of course the problem gets even worse once the game is 3+ years old and various new and old undocumented systems gets build on top of one another.

  13. Hawks Talon says:

    This is an interesting article and one I agree with fully. The need for documentation prior to development is critical to a successful product. I will say this though, in this case, since the publisher (e.g.: MS) was not willing to give the time required to spec out and document the needed requirements, the in my professional opinion, one or more of the managers should have stepped up to help out by doing the documentation in parallel to the development work that Jesse (and others) were doing. This is how I approach this kind of situation as both a developer my self and a project manager. I have no issues with putting in the extra effort, even in my “off” time to ensure the success of the project. I realize that I do not know every single detail about the situation or the environment, but there is always a solution to a give challenge, you just have to find it.

  14. Sandra says:

    Hawks Talon: Keep in mind that we’re not talking about documentation prior to development which implies a plan that can be written at a high level and then fleshed-out as you go, but documentation prior to testing, which instead implies a fairly detailed record of mechanics and — more importantly — intent.

    It was unrealistic for a manager to step in to document in parallel since what was needed was a detailed record of mechanics and Jesse’s intent — how are the skills supposed to work? In order for a manager to document that, they’d have to sit down with Jesse and talk about each skill in turn to learn what it did and why it did it that way.

    That might have taken less of Jesse’s time than him writing the docs himself, but not by very much, and not enough to have allowed both development and documentation to happen. And it would have taken a hefty chunk of the manager’s time as well — time the project also didn’t have.

    (At that point, most of the team, managers included, were already working 60-80 hours a week. Once you count in time for sleeping and showering, there is no more “off” time to utilize.)

    The right answer in this case may well have been for the producer to try to put the brakes on and get the time needed for both dev and docs — which may have resulted in, variously, the project getting canceled, the producer being fired, or the project seemingly getting more time until “something else” came up and more features were added or time was sacrificed for something else.

    Oh, or the company could have pulled one of my favorite tricks and gone to the team with a ‘deal’ like the following: “We know this is a really tight schedule, so we’re leaving it up to you guys. Would you rather cut all crafting professions, which will probably kill the game outright even before it launches because every other game on the market has a massive crafting system (and if this game fails the company might even go under)? Or would you rather just skip all that full, tedious documentation of skills, which, after all, hardly affects you unless you’re going to be on the live team. You? Oh no, you only have to be on the live team if you want to be on the live team. I’m sure we can find somewhere else for you.”

  15. Ibn says:

    The thing that I particularly like about this article is that it’s by no means limited to MMO development or even game development at all.

  16. Wayne says:

    True Ibn, it fits many software projects.

  17. AdamM says:

    I’ve managed both Devs and QA for a few years now. The one thing that ticks me off more than anything is when Prima donna devs spew bugs in their code and having the gall to complain about the quality of QA’s work. Remember that every QA bug is born as a dev screw-up.

    On the other hand, documentation is a cop-out proxy for actually talking to devs daily. If you need detailed docs to understand what your features are supposed to achieve so that you can test them, then maybe you should sit closer to the devs so you can ask. Because you’re obviously not getting what you should out of your feature team meetings or email.

    I’m so tired of the Dev/QA blame game going back and forth. Both of you pay a lot more attention to your features and what your customers want and you wouldn’t have half of the problems that you do today. Take some pride in ownership for goodness sake!

  18. Vahkris says:

    This is an awesome post. I started my career in QA and have participated in many Betas, so proper testing before going to production/live is ingrained into my psyche (if I started my own software company I’m probably going to piss off a lot of management because of how much emphasis I put on proper testing). I fully agree that without proper documentation, QA grinds to a standstill as ANYTHING can be considered “WAD” by a developer.

  19. Daemon says:

    It’s all about time constraints, I blame microsoft for AC2′s downfall. It tends to be publishers trying to get the game released, that, ironically, cause the game to fail.

  20. Eric says:

    There’s more to the AC2 story than “Microsoft pushed it out the door” — Turbine is hardly blameless. But I’ll save that for another post, I guess…

  21. Cat says:

    “On the other hand, documentation is a cop-out proxy for actually talking to devs daily. If you need detailed docs to understand what your features are supposed to achieve so that you can test them, then maybe you should sit closer to the devs so you can ask.”

    Good point, AdamM. Our QA department was situated on the other side of the building, involving a complicated trek to their desks if I had a question. Some room once freed up in the section with the designers and we figured our small little QA team might get to finally sit with the Live team and thus know what was going on. That fell through because the devs wanted to have nothing to do with us.

    We had two people in QA who (I guess) thought in their secret hearts that their job was really to design the game by filing tons of suggestion bugs and getting into fights with the devs over how every little thematic detail was implemented. They were notorious dev-haters. The real designers, naturally, did not enjoy having Bugzilla fights with QA over tiresome trivia. I imagine they did not want to be within verbal harassment distance of those two, since not only were they argumentative about everything they were *persistent*. So we all lost out.

    My boss was a personal friend of those two and resisted extremely strong suggestions from every other department head to boot them to the curb.

  22. Pingback: QA braucht eine Dokumentation | mmplay

  23. Ronald says:

    Basically this whole theme reads like a “10 things you never want to see in software development”. Mismanagement mostly – the managers responsible for allowing the messes above should have been replaced by more competent people long ago.

    If your QA team sucks and the manager protects them: get a better manager. Oh and a QA team that isnt involved in the design of the software, is pretty much useless from the start. A good QA team makes sure the design is clear to everyone, and is testable. Then works with the developers to make sure that bugTESTING is easy to do before something ships, and then discusses *intent* with the developers so they get the big picture. Ofcourse if you underpay your testers compared to the devs, which often happens, you get testers who cannot deal with the developers (less int, less exp) so you set yourself up for failure.

    As someone already remarked, this is one area where Extreme Programming could work quite well – because it recognizes exactly these issues as failmodes, and tries to prevent them from happening.

  24. Pingback: The Cave » Blog Archive » What’s a QA Team without a Spec?

  25. Steve L says:

    Adam M, Cat,

    “On the other hand, documentation is a cop-out proxy for actually talking to devs daily. If you need detailed docs to understand what your features are supposed to achieve so that you can test them, then maybe you should sit closer to the devs so you can ask.”

    Having QA work closely with developers is a Very Good Idea. But it’s no substitute for a formal specification — it supplements the specification. I work in QA myself, and I sit in the middle of my development team. I talk to my developers on a regular basis about the features they’re working on. [They (mostly) jokingly say "Oh no" when I walk into their offices and ask if they can take a look at something, as they've learned it usually means I've found something weird.]

    I _still_ want them to write up specifications, in part because reading the spec triggers questions in my mind that become either test cases or things to ask the developers about (“Well, the spec says that if X and Y both happen, then Z happens. It also says if just X happens but not Y, then Q happens. But what if X doesn’t happen but Y does?”) and in part because it triggers questions in _their_ minds as well, questions that could lead to preemptive bug fixing.

    Fixing a bug in a specification is cheap. Fixing a bug in the code before it ships is much more expensive. Fixing a bug in the code after it ships is the most expensive.

    Another benefit to specs is the HBAB scenario. If Jesse Kurlancheek had been Hit By A Bus (or in a less morbid example, was hired away by a rival company) the day after AC2 had shipped, doing the archaeology to determine how exactly the skills system was supposed to work would have been much, much harder.

  26. In reply to Steve L., I disagree. Collaboration doesn’t supplement a specification; a specification supplements collaboration. Specifications are stand-ins for what person would have said had you asked them at some point in the past, the point at which the specification was committed to print or file.

    Fixing a bug in a specification is cheap. Fixing a bug in the code before it ships is much more expensive. Fixing a bug in the code after it ships is the most expensive.

    Fixing a bug in a specification might be cheap, and often is. But fixing a bug in code could be far cheaper too. I think you might mean that fixing a design that expressed in specification or prototype is cheaper than fixing a design that is expressed in code; that’s true rather more often—but still not universally.

    Doing the archaeology to determine how exactly the skills system was supposed to work would have been much, much harder. Than what? Distributed knowledge before the bus accident would be cheaper still than the specification, with a shorter learning curve. That combined with a concise spec might be cheaper still.

    —Michael B.

  27. Rich says:

    Having played AC2 since launch to its death throes, I had a love hate relationship with it. But the best part of this article is the following quote:

    “Almost every skill in the game was broken in some way. Some skills literally did nothing”

    So now we know that the Bounty Hunter skill “lucky charm” indeed did nothing. This was wildly speculated for years, some thought it would increase hit, other thought it increased drop rates, but it was indeed a cruel joke on the poor bounty hunter and did nothing.

    I would also like to add that the forced grouping did as much to kill this game as anything else. Citan you were one of the proponents, you have now been proven wrong, forced grouping ONLY LIMITS your game.

  28. David Reese says:

    I agree with this post 100%. When I was on the QA team for Earth and Beyond Online, we never got a single scrap of design from the dev team. We weren’t even ALLOWED to talk to the dev team so as a result we were forced to test what we could with what little knowledge we had. The reason for this, as it turns out, is the dev team didn’t want a QA team testing their product so much as they wanted a focus group testing their product as an end-user. This approach obviously does not work. That’s what beta testers are for, yes?

    In the end, the approach we took to testing Earth and Beyond was literally documenting everything we found (including creating zone maps, cataloging mobs and drop tables, quests, etc) into what we fondly called the “X Files”. It was more like a strategy guide than anything else if you want to know the truth… but it was the only way we could do our job. We weren’t even told what changed when new builds were put out for us to test. We were basically left to our own devices and the “X Files” was our way of coping with that significant barrier. Another side effect of this was since we were never told what was fixed in each build, we had to spend an extensive length of time going through all the bug reports verifying whether those bugs were fixed or not with each build. This is to be expected of course, since that’s part of QA’s job… but without knowing what was fixed or not, we were unable to focus our attention on any given bug-type and had to go through them all.

    And of course, our particular Director of QA put a significant weight on number of bugs found per tester. It was his way of gauging how effective an individual tester was. Quality of bugs found took the back seat to quantity of bugs found and if a tester didn’t meet some unspecified number of bugs found, that tester was reprimanded and (in some cases, let go). As a result, the bug database got inflated with unimportant fluff bugs like the floating object bugs that were mentioned in the blog post. Other favorite fluff bugs among the Earth and Beyond Online test team were spelling errors and missing tooltips.

    Just some food for thought on the importance of providing QA design docs and identifying the appropriate role for your QA team. If you want your QA team to be an in-house focus group… don’t pay them QA wages and don’t call them QA. That’s a waste of money that is better spent in other areas of the development process. If you want your QA team to actually test your product, you must give them some kind of design documentation so they can test your product effectively and cut down on the existence of fluff bugs (having a Director of QA that doesn’t impart a bug quota helps here as well).

  29. Paul says:

    I still think test automation would go a long way to that nerf-buff cycle of “chaos and confusion”. But yeah, how do you automate something that isn’t documented?

    You can’t test if requirements are met if you don’t have requirements. But there are always partial requirements, even if some of them are obvious (“the server will not crash”).

    In many kinds of software, one can blindly and stupidly generate random inputs (perhaps tailored in some way to bias them towards more interesting behaviors), then see if any of these obvious conditions are violated. More often than you’d think this kind of testing will reveal bugs, often quite serious ones, that would not have been found by a more deliberate testing process. The big advantage of the random test generator is that it can spew out orders of magnitude more test cases than manual testers can (and do it far more cheaply).

    The thing that limits this kind of testing is often that once it finds a bug, that bug has to be fixed or else the tester discovers it over and over again.

  30. Logan-O says:

    I completely agree with the sentiments Eric. Without a spec you are essentially blinding your QA team and inviting them to find the nit-picky bugs that are of absolutely minimal consequence. If only more folks understood – no – actually bought into this practice.

  31. David Bowman says:

    Good documentation of how a system and the content examples that use that system are expected to function is useful in several ways. QA can help the entire team with directed testing that provides clear examples of errors in the system or content. The designers who have to support the system with content after the original designers have moved on have a clear understanding of how the system should work. End users can be given clear expectations through manuals, tutorial or content based on the system’s function. Programmers can better understand what is expected of the system. I’ve been guilty as anyone, perhaps more than most, of not providing clearly written documentation. What I am currently trying to do with a small team is to create the design goals for the systems and give some content examples. I collaborate with the programmers and technical artist on the design. We then get the system coded and I enter some content into the system and begin testing both the system and my ideas on how the content should work. Once I am convinced that the system will support the content desired, and I actually understand how this content will best work for the players, I will then update the wiki with the details of the content and how it works with the design. This will then hopefully act to meet the needs that I listed above. The writing of the documentation, including the spreadsheets of content takes time, but it’s very important that this time be spent. In a perfect world this time would always be available. Anybody wishing to make a game should include time for documentation, and we’re adjusting our schedule accordingly. Thank you Eric and Sandra for reminding me of this importance, it’s easy to get heads down and focused on just entering the content into the systems.