Zubon from Kill Ten Rats recently posted about why he doesn’t use the test server. His points, valid or not, are pretty common-place: I’ve heard similar complaints from other players over a broad swath of games.
And that’s unfortunate, because a test server can be a wonderful tool for developers to check their assumptions and straighten their collars before their work goes before the entire playerbase. On the other hand, a lot of current games are not making terribly effective use of their test servers, so I completely understand where Zubon is coming from.
So, what are the most important factors behind a successful test server?
Know your goals.
Are you setting this thing up primarily to promote new content or to find bugs? What kinds of bugs? The kind that require lots and lots of players just playing normally (i.e. a stress test)? Or the kind that require intelligent investigation by the players (for example, finding exploits)? Are you looking for feedback? On specific items or on general gameplay systems? Only on the new stuff, or on everything in the game? And if you answered “All of the above”, you need to prioritize!
In addition, you need to communicate your goals, not only internally — QA works best when they know why you just made their lives hell by adding a test server — but also to the players who might use the test server. Set their expectations accurately and they’ll give you a lot less grief and a lot more help.
Work out your processes and priorities.
How do players report bugs? (And how do you respond when they report bugs in other ways, which they will?) Who is responsible for vetting the reports? How do the bugs from the test server line up with new development? In other words, will the development team drop everything until this update has made it out the door? Or will they continue blithely on their way developing new content while the bugs from the test server rot in a queue somewhere? Or do you hope to handle both new development and the test server at the same time — and if so, how?
Once you have the bugs fixed, what then? How often do you push a new build through QA? What determines when you push a new build to the test server itself? A fast turn-around makes players feel good about their efforts and makes it easier for them to continue testing. But push too fast and you’ll lose control, introducing more additional new bugs than you fix. Again, setting expectations is a big help here. If you can’t update but once a week, for instance, then let players know — and perhaps post a list of issues that will be fixed in the next build so they can still see that you’re paying attention.
Write decent change notes.
I know this one is hard — believe me, I know how hard it is! — but it’s essential. I’m speaking to the producers here: If you seriously don’t know what’s in the patch that you are foisting on your players, you fail! Try again. (And if you have so many changes going in with each patch that it’s not feasible for you to keep up, you need to slow down your development cycle — your players can’t keep up either.)
If you do know what’s in the update, you need to tell your testers. You don’t neccesarily have to describe every typo you fixed (especially if you are using the test server primarily for stress testing), but you still need to give them a decent idea of the broad areas that may have changed. Indeed, full disclosure of changes is your best route to avoiding potentially embarrassing unintended consequences.
Now for my mini-rant: Surprising your players is not an acceptable excuse for leaving things out of the notes. Wanting to give them the joy of exploration by hiding new content is not an acceptable excuse. (If they read the patch notes, their going to read the spoilers that go up 9.5 seconds after the patch is live.) Finding and fixing bugs trumps lore.
Test servers are a useful tool, but they aren’t nearly as simple as just standing up a server and letting players poke around. You either put some thought into what you are doing (and how!) or you’re just wasting precious time and money.