You can’t test if requirements are met if you don’t have requirements. But there are always partial requirements, even if some of them are obvious (“the server will not crash”).

In many kinds of software, one can blindly and stupidly generate random inputs (perhaps tailored in some way to bias them towards more interesting behaviors), then see if any of these obvious conditions are violated. More often than you’d think this kind of testing will reveal bugs, often quite serious ones, that would not have been found by a more deliberate testing process. The big advantage of the random test generator is that it can spew out orders of magnitude more test cases than manual testers can (and do it far more cheaply).

The thing that limits this kind of testing is often that once it finds a bug, that bug has to be fixed or else the tester discovers it over and over again.

In the end, the approach we took to testing Earth and Beyond was literally documenting everything we found (including creating zone maps, cataloging mobs and drop tables, quests, etc) into what we fondly called the “X Files”. It was more like a strategy guide than anything else if you want to know the truth… but it was the only way we could do our job. We weren’t even told what changed when new builds were put out for us to test. We were basically left to our own devices and the “X Files” was our way of coping with that significant barrier. Another side effect of this was since we were never told what was fixed in each build, we had to spend an extensive length of time going through all the bug reports verifying whether those bugs were fixed or not with each build. This is to be expected of course, since that’s part of QA’s job… but without knowing what was fixed or not, we were unable to focus our attention on any given bug-type and had to go through them all.

And of course, our particular Director of QA put a significant weight on number of bugs found per tester. It was his way of gauging how effective an individual tester was. Quality of bugs found took the back seat to quantity of bugs found and if a tester didn’t meet some unspecified number of bugs found, that tester was reprimanded and (in some cases, let go). As a result, the bug database got inflated with unimportant fluff bugs like the floating object bugs that were mentioned in the blog post. Other favorite fluff bugs among the Earth and Beyond Online test team were spelling errors and missing tooltips.

Just some food for thought on the importance of providing QA design docs and identifying the appropriate role for your QA team. If you want your QA team to be an in-house focus group… don’t pay them QA wages and don’t call them QA. That’s a waste of money that is better spent in other areas of the development process. If you want your QA team to actually test your product, you must give them some kind of design documentation so they can test your product effectively and cut down on the existence of fluff bugs (having a Director of QA that doesn’t impart a bug quota helps here as well).

“Almost every skill in the game was broken in some way. Some skills literally did nothing”

So now we know that the Bounty Hunter skill “lucky charm” indeed did nothing. This was wildly speculated for years, some thought it would increase hit, other thought it increased drop rates, but it was indeed a cruel joke on the poor bounty hunter and did nothing.

I would also like to add that the forced grouping did as much to kill this game as anything else. Citan you were one of the proponents, you have now been proven wrong, forced grouping ONLY LIMITS your game.

Fixing a bug in a specification is cheap. Fixing a bug in the code before it ships is much more expensive. Fixing a bug in the code after it ships is the most expensive.

Fixing a bug in a specification might be cheap, and often is. But fixing a bug in code could be far cheaper too. I think you might mean that fixing a design that expressed in specification or prototype is cheaper than fixing a design that is expressed in code; that’s true rather more often—but still not universally.

Doing the archaeology to determine how exactly the skills system was supposed to work would have been much, much harder. Than what? Distributed knowledge before the bus accident would be cheaper still than the specification, with a shorter learning curve. That combined with a concise spec might be cheaper still.

—Michael B.

“On the other hand, documentation is a cop-out proxy for actually talking to devs daily. If you need detailed docs to understand what your features are supposed to achieve so that you can test them, then maybe you should sit closer to the devs so you can ask.”

Having QA work closely with developers is a Very Good Idea. But it’s no substitute for a formal specification — it supplements the specification. I work in QA myself, and I sit in the middle of my development team. I talk to my developers on a regular basis about the features they’re working on. [They (mostly) jokingly say "Oh no" when I walk into their offices and ask if they can take a look at something, as they've learned it usually means I've found something weird.]

I _still_ want them to write up specifications, in part because reading the spec triggers questions in my mind that become either test cases or things to ask the developers about (“Well, the spec says that if X and Y both happen, then Z happens. It also says if just X happens but not Y, then Q happens. But what if X doesn’t happen but Y does?”) and in part because it triggers questions in _their_ minds as well, questions that could lead to preemptive bug fixing.

Fixing a bug in a specification is cheap. Fixing a bug in the code before it ships is much more expensive. Fixing a bug in the code after it ships is the most expensive.

Another benefit to specs is the HBAB scenario. If Jesse Kurlancheek had been Hit By A Bus (or in a less morbid example, was hired away by a rival company) the day after AC2 had shipped, doing the archaeology to determine how exactly the skills system was supposed to work would have been much, much harder.

If your QA team sucks and the manager protects them: get a better manager. Oh and a QA team that isnt involved in the design of the software, is pretty much useless from the start. A good QA team makes sure the design is clear to everyone, and is testable. Then works with the developers to make sure that bugTESTING is easy to do before something ships, and then discusses *intent* with the developers so they get the big picture. Ofcourse if you underpay your testers compared to the devs, which often happens, you get testers who cannot deal with the developers (less int, less exp) so you set yourself up for failure.

As someone already remarked, this is one area where Extreme Programming could work quite well – because it recognizes exactly these issues as failmodes, and tries to prevent them from happening.