Getting the Most Out of a Test Server

Test papers with different grades.

Zubon from Kill Ten Rats recently posted about why he doesn’t use the test server. His points, valid or not, are pretty common-place: I’ve heard similar complaints from other players over a broad swath of games.

And that’s unfortunate, because a test server can be a wonderful tool for developers to check their assumptions and straighten their collars before their work goes before the entire playerbase. On the other hand, a lot of current games are not making terribly effective use of their test servers, so I completely understand where Zubon is coming from.

So, what are the most important factors behind a successful test server?

  1. Know your goals.

    Are you setting this thing up primarily to promote new content or to find bugs? What kinds of bugs? The kind that require lots and lots of players just playing normally (i.e. a stress test)? Or the kind that require intelligent investigation by the players (for example, finding exploits)? Are you looking for feedback? On specific items or on general gameplay systems? Only on the new stuff, or on everything in the game? And if you answered “All of the above”, you need to prioritize!

    In addition, you need to communicate your goals, not only internally — QA works best when they know why you just made their lives hell by adding a test server — but also to the players who might use the test server. Set their expectations accurately and they’ll give you a lot less grief and a lot more help.

  2. Work out your processes and priorities.

    How do players report bugs? (And how do you respond when they report bugs in other ways, which they will?) Who is responsible for vetting the reports? How do the bugs from the test server line up with new development? In other words, will the development team drop everything until this update has made it out the door? Or will they continue blithely on their way developing new content while the bugs from the test server rot in a queue somewhere? Or do you hope to handle both new development and the test server at the same time — and if so, how?

    Once you have the bugs fixed, what then? How often do you push a new build through QA? What determines when you push a new build to the test server itself? A fast turn-around makes players feel good about their efforts and makes it easier for them to continue testing. But push too fast and you’ll lose control, introducing more additional new bugs than you fix. Again, setting expectations is a big help here. If you can’t update but once a week, for instance, then let players know — and perhaps post a list of issues that will be fixed in the next build so they can still see that you’re paying attention.

  3. Write decent change notes.

    I know this one is hard — believe me, I know how hard it is! — but it’s essential. I’m speaking to the producers here: If you seriously don’t know what’s in the patch that you are foisting on your players, you fail! Try again. (And if you have so many changes going in with each patch that it’s not feasible for you to keep up, you need to slow down your development cycle — your players can’t keep up either.)

    If you do know what’s in the update, you need to tell your testers. You don’t neccesarily have to describe every typo you fixed (especially if you are using the test server primarily for stress testing), but you still need to give them a decent idea of the broad areas that may have changed. Indeed, full disclosure of changes is your best route to avoiding potentially embarrassing unintended consequences.

    Now for my mini-rant: Surprising your players is not an acceptable excuse for leaving things out of the notes. Wanting to give them the joy of exploration by hiding new content is not an acceptable excuse. (If they read the patch notes, their going to read the spoilers that go up 9.5 seconds after the patch is live.) Finding and fixing bugs trumps lore.

Test servers are a useful tool, but they aren’t nearly as simple as just standing up a server and letting players poke around. You either put some thought into what you are doing (and how!) or you’re just wasting precious time and money.

This entry was posted in Production. Bookmark the permalink.

5 Responses to Getting the Most Out of a Test Server

  1. Zubon says:

    I’m going to agree under the heading “it’s all about communication.”

  2. rmckee78 says:

    You need to make sure that the people on your test server can actually reach the content you want tested. I play on the Everquest test server from time to time. The population there is fairly small making it difficult to test raiding content. In fact the test server lags far behind the rest of the servers in raid content anyway. The test server does offer extra XP and the ability to jump to level 25 but lets face it in EQ this still leaves a long, lonely haul to the cap (and on the test server it is extra lonely).

    I know some people play on test as their main so they no longer wipe it, but is this really appropriate? Perhaps you should be able to instantly make a character that could test the content. Really the test server wouldn’t even need to be up constantly in that case. You could offer veteran rewards on the accounts that did a lot of testing for you.

    At this point the test server for EQ feels like its there just for people who want to level faster. There is nothing wrong with this, but why not just make casual servers? I am sure you would get more people on it than a server that has Test written on it.

  3. You forgot ‘wipe occasionally’. Players playing on test need to have the expectation that they’re going to lose all their stuff, so you can (for example) recover if something buggy on test put everything in a bad state. I personally like how WoW does it, wiping after every major patch update, but allowing players to import characters easily.

  4. Sandra says:

    *nod* That’s a good point, rmckee, and I believe it falls out of knowing your goals. If you are going to be testing high end raiding content, for example, you need to do the things that will attract high end raiding testers and give them conditions conducive to testing. This very well may conflict with the conditions for your other goals, even testing things like high end non-raiding content. That’s what I meant about the need to prioritize your goals — some of them require mutually exclusive design decisions.

    Really, this was just a very quick riff on the whole test server topic. I’d like to come back to this with a full analysis of how different companies do test servers, but … not today. :>

    [edit]

    Damion: Wiping the test server is another of those design decisions that grows out of your goals. I wouldn’t call it a necessary condition (although I’ll give you that it might be one that is usually a good idea). :>

  5. Pingback: What Do You Want Me To Test?