PvP: Hard to Do Well

PvP is a great example: if you’re going to have PvP in your game, it’s going to suck unless you Go Big. You need to invest a lot of time and thought and effort into it. If you don’t, you shouldn’t even bother: the PvP will be unappealing and it will hurt the rest of the game in the attempt.

That’s why my MMO “Gorgon” has no PvP. I can’t afford to “Go Big” on PvP. It’s not something I have any expertise in creating, it’s not something I particularly enjoy in an MMO… so it ended up being cut out of the schedule entirely. This will limit my audience, but that’s okay. I can’t make a game for everyone.

(Hey, I guess cutting features isn’t that hard after all! As long as they aren’t features you’re personally excited about…)

Item Systems: Hard to Do Well

In the discussion about item decay, it was pointed out that item decay is hard to do well. This is very true. I’d go so far as to say treasure systems in general are hard to do well. Most MMOs, including WoW, have rather boring item systems that don’t try very hard. They aren’t the heart of the game. A game like Diablo 2, on the other hand, where the item system is a large part of the game itself, tries very hard indeed.

A great item system requires random loot in many varieties, interesting trade-offs that are nonetheless easy to understand, the ability to customize and enhance your items, and a built in reason to keep looking for more loot. This is a very tall order. It’s easy to say you’re gonna do that, but it takes an incredibly long time to get the details right.

My current feeling is that I have to “Go Big” on my item/treasure system because the game has so many non-combat mechanics. All my systems need to feed into each other in intricate ways. Mushroom farmers, animal skinners, herbalists, necromancers, geologists — they’re all collecting or generating “stuff”, and it’s not always going to be directly useful to the skill they got it from.

At the moment, I think the item system ties the game mechanics together, so it needs to be deep… really deep. But I know firsthand how long that will really take to make happen, and I know it will be quite difficult to pull off.

Trade-Offs: Short-term vs. Long-term

Whenever I spend a day designing a system, that’s a day that I didn’t code a game mechanic or create a new quest or write a new NPC’s dialog. Being just one guy means no delegation of responsibilities. So can I really afford to “go big” on my item system? Well, not nearly as big as I want to.

But I also really hate the idea of launching the game without a robust item system — because that’s not something you can patch in later. I could launch with very little content and patch content in later (assuming the lack of content doesn’t kill the game outright). But the big-ticket game systems, like PvP or complex item systems, don’t get to change that much later, especially if your game is centered around emergent gameplay like mine is. Changing something so fundamental will alter everything, and players don’t generally enjoy this, because it destroys everybody’s game knowledge and treats them like paying beta-testers.

what I’m really deciding is: how good can my game ultimately be? When I examine each game system, I ask myself: “If the game is successful and gets thousands of paying players, am I going to hate myself for not having implemented this before the game launched?”

Go Big or Go Inevitably Sad But What Can You Do

Unfortunately, that doesn’t narrow it down enough. In my game, I want to focus on the skill system, the item system, the combat mechanics, and the NPC interaction system. If I short-change any one of these areas, I’ll be sad later. But I can’t give all four of these areas the attention they deserve.

This is a place where having a producer would be helpful: someone who could see the forest for the trees and help me decide how to spend my resources best.

I’ve talked about this problem many times on the blog: designers can’t make the best choices for the long-term when they get too invested in their game. And I’m just as susceptible to this as anyone else.

My natural inclination is to say “all four of those systems are critical. I just have to do them all to 100% perfection!” But were I to succumb to that, the game would fail. I don’t have time to implement all of those, and even if I did, it would mean the game lacked content or critical infrastructure. Something precious to me has to get cut. Many things, actually.

Seek External Help

At least I realize that I’m not able to make the best decisions in this case. So I’m not going to work on any of these systems this week: I’m going to code some other things to get my mind to switch gears. Then I’ll sit down with Sandra and pitch the problems in detail, discuss the exact plan, and get her to help me figure out what to cut.

I may have a slight advantage over the average indie developer here… but even if your wife isn’t an experienced MMO producer, I bet you can still find someone to pitch your problem to and get feedback from. It needs to be someone whose opinion you respect, and someone who can take the time to fully understand the problem. And you need to be detached enough from the problem to be able to deal with the feedback.

Not Just For Indies

I just want to reiterate that every MMO team I’ve ever seen has fallen victim to this, and sometimes they pull themselves together… and sometimes they can’t. If you’ve read this blog for any length of time you’ve seen me ranting about WoW going off the rails: biting off more than they can chew, overpromising and underdelivering. If a game with the size and success of WoW can succumb to this problem, it’s not about money or team size.

Nobody wants to make the painful cuts, the ones that make your game less perfect than what you had in your head. This is true in any MMO, be it a AAA box title, a web game, or anywhere in between.

Sometimes ignoring the resource-allocation problem works out fine. Sometimes you get lucky. But lest we forget, only one in four MMOs gets from start to completion. And that’s announced MMOs. Who knows how many MMOs die before they even make a whimper? The odds are stacked heavily against you, and part of the problem is how hard it is to allocate resources.

If you think you’re immune, you’re probably wrong. (In my experience, the sort of personalities who are immune to this problem are not the sort of people who become game designers.)

If you think it doesn’t apply to your game, you’re probably wrong, too: you just don’t realize what you’re trading off yet!

Next Week: Can Talking About Engineering Ever Be Interesting?

So I’m currently working on low-level stuff that has to get done, like chat and persistence and generators and GUI. Join me next week when I desperately attempt to make networking code sound interesting!

Elder Game is sponsored by:

Every live MMO team I’ve been on has had this debate at one time or another, and with some of them it’s an ongoing power struggle that ripples out and affects development in a huge way.

A head QA honcho that I worked with once told me that my job as producer was to produce — to push the product as spec’d out the door on time. And his job as QA was to make sure that the product I pushed out was acceptable and didn’t reflect badly on the company. That meant that the final call on pushing a patch was his responsibility, he said. He could stop the patch if he felt it was unacceptable.

But while that model of control may work perfectly well for a pre-launch MMO or a one-shot boxed game, it doesn’t make any sense for a live MMO – especially not one with a constant (and regular) content update schedule.

The producer of a live MMO is different from the producer of a pre-launch product. We have different responsibilities and goals. We answer much more directly to the customers. Our bottom line rests less on development costs and more on customer retention.

While the live producer is still in charge of scope and schedule, both the scope and schedule of a live MMO are much more flexible in the short term than those of a pre-launch product. Whether you are patching once a month or every three months, there will be another patch. If you have to, you can slip content or push your patch a week without screwing up a multi-million dollar advertising budget.

As producer, I used QA as a resource (one of the my most valuable resources – no argument there!) to tell me the state of the patch. I used that information to decide what was ready to go and what needed to slip to the next patch. And yeah, I pushed without QA sign-off once or twice. After gathering all the information and balancing all the requirements, that was the action I decided was necessary.

The live producer needs to sit down with the people who are invested in the patch – QA, CS, etc. – to work out any difficulties. But giving one of those groups a flat out veto? Not useful. Not in the live MMO world.

Elder Game is sponsored by:

Long ago, when my friends and I were playing EverQuest, one of our party members decided to do some soloing after everybody else had gone to bed. He fell into a pit and died. In EverQuest, you lost everything you owned when you died, so all of his gear was now at the bottom of a pit. He then (foolishly) tried to get his gear back by himself, and subsequently lost all his backup belongings. The next day we offered to help him get his stuff back, but it was too late: he’d already rage-quit. He never came back.

If your last memories of the game are anger, then it’s much less likely that you’ll ever be back. But you might also quit just because you got bored, or some fancy new game caught your attention, or because you ran into money problems. In these cases, you may very well come back later.

No game will keep all their players forever, but when players do quit, you want to make it likely that they’ll come back again later.

Metrics Tell You What’s Going On In Your Game

As a game producer, you’ll be keeping an eye on certain key numbers to help you steer the game. Here’s some basic ones.

If people keep paying after the first free month (or whatever freebies you offer), they’re “retained” in the game. Obviously, you want your “retention rate” to be high — otherwise you’re in big trouble. A game that launches with a poor retention rate is likely going to die: nobody wants to play it.

But assuming a large percent of your players are retained past the first month, your next goal is to keep them around as long as possible. The length of time they keep playing is their ”retention length“.

People will inevitably quit at some point. Do they quit forever? Hopefully not. The number of people who come back to your game eventually is called your rebound ratio (or rebound percentage, or so on), and how long it takes them to return is the rebound duration. A game that’s doing really well might have a rebound ratio of 25% or more, with a rebound duration of eight to twelve months.

While priority #1 is to have the longest retention you can, you should also put a priority on making the rebound ratio high. This is the key to keeping your game going for a decade or more: your best fans keep coming back year after year to help keep the servers lively.

When a game damages the rebound cycle, they shoot themselves in the foot. SWG’s “New Game Experience” update did that for a lot of people — no matter what SWG does now, they aren’t going back. EverQuest burned the bridge for many because their death penalty was so high.

On the other hand, I don’t rage-quit most MMOs. Instead, I just get bored, either from lack of content or lack of friends or lack of compulsion to play. Those are all fixable things. In Lotro I simply couldn’t find people to play with; now that it’s “free”, I intend to go check it out. They didn’t burn their bridge. I didn’t rage-quit, I yawn-quitted. Much better for rebound. Knowing how many people are coming back is really valuable info.

One last warning about these metrics: don’t make the mistake of jumbling your new players and returning players together. If you use one single statistic for “retention length” for the game, it won’t mean as much.

A great game will have an initial retention length of 6 months — that’s how long new players stay. The best I’ve ever seen is Asheron’s Call 1, back when it had very little competition. Its initial retention was nine or ten months. (That’s incredibly high, and I can’t imagine a modern game beating that average, even WoW… there’s too many other choices to switch to when you get bored.)

But even the best game will have a lower retention rate for returning players. If your initial retention rate is six months, your rebound retention rate might be three months. Keep these separate so you can tell what’s going on.

Using Your Metrics to Make The Terrifying Changes

When your game is live, it can be hard to figure out what to change. No matter what the topic, somebody will tell you that it is ruining the game. Others will tell you it’s the reason the game is still afloat. It’s easy to become paralyzed.

Suppose you know that some people are quitting because the death penalty is too high. But other people say they’re only playing because the death penalty makes the game challenging. (Or any of the other reasons that MMOs might be designed with a high death penalty.) Should you change the death penalty? Hard question.

The above metrics will get you started in answering this question. Is your retention length high and your rebound rate fantastic? If so, no — definitely don’t change the death penalty. The people who are quitting may say it’s because the death penalty is too high, but apparently they’re still coming back anyway, so this isn’t a deal-breaker for them.

On the other hand, the more likely case is that your rebound rate sucks. And it’s up to you to figure out why it sucks and fix it so that the game will continue to have players in the years to come, long after all advertising money has dried up. Returning players are really important in the later years.

Whatever metrics you have, use them. They will at least help you narrow it down. If in doubt, address the issues that will alienate the existing playerbase the least.

If you have to change the game significantly, I would always err on the side of making the game easier. That tends to cause people to bitch, and some to even rage-quit over it, but mostly they still come back. Making the game easier may hurt viralness (the odds of someone telling their friend about the game) but it won’t hurt the game’s retention or rebound ratio, in my experience.

It’s all a gamble. It’s all making moves with very limited information. It sucks, basically. And no matter what you change, you’ll hear about it from angry vocal players. Worst of all, you have to wait a long time to find out if you’re right — typically up to six months before you can see how the changes are working.

But don’t make the decisions randomly, and don’t try everything at once (as I’ve seen many teams do). Unless you’re about to go under and are doing a Hail Mary to save the game, you need to make changes slowly enough that you can tell what the results are.

Not Just For Game-Balancing

When we say that games should record a lot of metrics, people seem to assume it’s for balancing the game. Sure, metrics can help balance a game. But instrumenting your game is really time consuming and expensive! (I hope I didn’t make it sound easy. Collecting the data correctly and turning it into useful reports can easily be a full-time job.) If metrics were just for balancing, the cost may not be worth it.

But it’s almost always worth it to heavily instrument your game and billing services, because that data helps you figure out how to grow. Statistics like retention rates and rebound ratios are just a start.

(If you’re stuck on how to get your rebound ratio up, here are a few ideas.)

Elder Game is sponsored by:

1. Women gamers are a vast untapped market.
2. If we can tap into that market we can make lots of money.
3. So we better hire some women developers.

Magical Woman Knowledge

The logical connection between the first two points is pretty clear. But where does #3 comes from?

Magical woman knowledge.

There is an unspoken assumption that women devs – and all women, in fact – have magical knowledge about women that can help us tap into the market of women gamers. (You knew about that, right? It’s a back-of-the-box bullet.)

This is very useful because as game developers we don’t have to time to think about these things, especially when we’re trying to get a game out the door. Goodness knows we don’t have time to think about non-traditional audiences while we’re designing the game.

Okay – I’m getting a little snarky now. But I’m also being serious.

A lot of developers – of all genders – seem to think that being a woman in game development is an automatic ticket to understanding what women want in a game.

Lived Experience

Women devs do bring something special to the development table: lived experience.

That sentence is a bit of a cheat, though, because all developers – all people – bring their own lived experience to their work.

Each bundle of experience is different. No one has the prototypical man gamer experience any more than the prototypical working class gamer experience or the prototypical Jewish gamer experience.

But you pile enough of those bundles of different experience up together, you have a team experience pool that can help guide your development.

Piling up enough lived experience from women gamers is especially important if you want to tap into the woman gamer market because gamer culture sits in a matrix of subtle sexism that can – and does – tend to alienate women. (Yes, even women gamers.)

Subtle sexism is subtle. Neither experiencing it nor recognizing it are limited to women. But on average, women probably have more personal lived experiences that help make them a tiny bit more cognizant of subtle sexism. Sometimes. Not always.

How to Tap Into Women Gamers

If you are serious about tapping into the market of women gamers, I have two suggestions:

1. Hire a bunch of women developers to add their experiences to the experience pool of your team.
2. Do some research on women gamers instead of expecting one magical hire to bring knowledge to your door, you lazy bastard.

Elder Game is sponsored by:

At some point or other, most live MMO teams with declining player bases (which is almost everyone) are handed a mandate to increase conversion of new players.

Usually this involves someone higher up in the company looking at the number of free client downloads and comparing it to the number of new subscriptions each month.

(You do have a free client download, right?)

Once the mandate comes down, the live team kicks into gear. Since they are usually limited to working with in-game content, the live team concentrates on character creation, starter areas, the in-game newbie experience, low level quests, level 1-10 socialization, etc.

What the live team almost never does though, because often it isn’t in their purview at all, is fix the gawd-awful process that a new player goes through between the download and character creation.

Like navigating the cluttered sales-speak website to find the technical support link when the install doesn’t

• … and then getting a 404.

Like filling out a massive web form to sign up an account

• … half of which it turns out is market research anyway
• … and doing it three times because each account name you choose is already in use and the form doesn’t save your entries
• … and then doing it twice more because you also need a separate forum name
• … only to have the verification e-mail go missing three times in a row.

(And do you know how many older games still require credit card information for their free trial? Egad!)

The best the live team may be able to do is bug the installer team for a fresh client download, bug the customer service team for some knowledge base updates, and bug the web team for some link fixes.

Of course, all that assumes that the installer team still knows how to build a fresh client, that the CS team still knows how to use the old knowledge base software package that the rest of the company switched away from three years ago, and that the web team isn’t tied up for the next three months building a site for the game in development - all of which, I shudder to say, is not always the case.

(And I’m not even going to mention the games that hide the link to the client download off some back page – or worse, a post in the forums! That’s outside the scope of this post. But suffice it to say that if you have to Google for the client download, you have failed.)

That’s not to say that improving your low level content is a waste – far from it! But if your goal is conversion, even character creation may well be too late.

Elder Game is sponsored by:

Unfortunately, making a full 3D MMO on just a few days a week is very hard. We’ve tried it a few times in the past, but when it comes time to make the timeline projections, we always shelve the idea. The problem is, we have a lot of experience in the industry, which keeps us from trying to do things that are “stupid” and “irrational” and “never going to work”. Industry-naive developers can sometimes pull off amazing miracles by sheer insane doggedness — and by not knowing that what they’re doing is impossible — but we know the actual amount of time each feature is likely to take, and we know from experience that by the time you finish the “engine” parts, you’re so focused on technology that you don’t have any mental room left to make the game itself.

The thing is, players don’t give a crap about the technology. They care about the game mechanics, the graphics, the atmosphere… but not the tech. So what we need is a way to skip the “engine” part and get to the part that matters.

So naturally we keep a close eye on all the MMO-capable “engines” (or engine parts) as they showed up. Mostly these are pretty useless. There are dozens of MMO engines that sound great on a website somewhere, and then you download them and realize that step #1 is “write 75% of the engine” (such as Project Darkstar, which is not much more than a handful of low-level server-side architecture primitives) or “refactor the engine to be usable” (such as Ryzom engine, whose code is an utter mess at the moment). These projects have ongoing developer support and some day they may be amazing. But that’s not helpful now.

But one product has stood out to us as being a viable client-side 3D platform: Unity. Not only does it have a really powerful development environment, it has a web-player that lets people play Unity games in a browser. That’s huge for indie games because it removes one of the biggest hurdles: convincing people to download your product and install it. Combine Unity’s network-friendly architecture and primitives with a simple server like SmartFoxServer, and maybe you’ve got something workable after all.

Unity 2.5: Not Good Enough

Sandra and I played with Unity 2.5 back in April of last year. We were very impressed with what we could accomplish in just a single week, and we were totally jazzed about going forward with it. But it defeated us: Unity 2.5 wasn’t really usable for MMO development.

Although Unity said that Cartoon Network’s Fusion Fall MMO was written with Unity, that was a lie of omission. Sure, Fusion Fall used Unity. They just used a special version with all kinds of cool powers you couldn’t replicate in Unity 2.5. We were quite frustrated when we couldn’t do the same tricks that Fusion Fall did. For instance, we couldn’t really stream character assets the way Cartoon Network’s game managed it: the features just weren’t there. That meant we would have to be really picky with our assets, because the options for streaming were very restrictive. That was pretty depressing.

But what really killed Unity 2.5 for us was the crashes: Since Unity 2.5 didn’t support typical version control products, we had a hell of a time managing our work between two programmers. (The entire project was stored in proprietary binary files.) And sometimes when it crashed, those binary files got corrupt, and you lost all your work since the last manual backup. So we hung up the Unity project and went back to doing actual lucrative work.

Unity 2.6: Huge Improvement

Well, a couple months ago we got that itch again. Unity 2.6 has been out for a while, so we dove in — and it was a huge improvement. In fact, yesterday we bought the first of at least two Unity Pro licenses. At $1200 each, that’s not a snap decision, but we believe that this time our MMO can really happen.

We’re happy with the product, and we’re amazed at what we’ve been able to accomplish in about two hundred hours of development time: we have an entire working proto-MMO, complete with melee combat, projectiles, inventory management, skill systems, shops, harvesting, and more. By letting us spend our time on these game systems instead of the nuts and bolts of 3D displays and asset streaming and so on, we have been able to do a ton of game systems very quickly. We’ve got an indie art team making art for us and things are happening at a rapid pace, given that it’s a side-project done in our spare time.

However, working with Unity 2.6 has a down side, and that down side is that Unity is only magical when it’s magical. But first, the good stuff:

It’s got everything Unity 2.5 should have had.

Unity 2.6 inherited a bunch of nice features that were originally written for the Cartoon Network folks. You can now stream any kind of asset, and there are a lot of powerful mechanics for managing streamed data. This is a lot more like it — making a tiny lightweight game is actually possible now!

Plus, Unity 2.6 has support for actual version control systems (instead of just their overpriced proprietary solution). When enabled, Unity creates a binary data file that stores the “compiled” data for each original asset (be it a source-code file, a JPG, an animation, whatever). So to version things, you put both the original asset and the binary Unity file into your SVN repository. Ta da!

Perhaps most importantly, it’s a lot less crashtastic. Well, it still crashes plenty (mostly when your code dereferences a null pointer in a fragile state such as during a screen refresh) … but it doesn’t crash completely randomly. And when it crashes, it almost never corrupts your assets anymore. And if it does corrupt your assets, you have several ways to fix it: you can reimport the single damaged asset or you can reload stuff from your SVN repository, for instance. You aren’t just stuck with a corrupt project forever.

As an added bonus, Unity 2.6 has a built-in animation editor! (You would be forgiven for thinking that came with Unity 2.5, as I initially did. It was in earlier versions, but they actually took it out for Unity 2.5, and put it back in for Unity 2.6.) While you could theoretically animate anything in this animation-editor, it’s really designed for background animations, special effects, and the like. For instance, the fish in our lake swim about on an animated path. Certain special effects cause creatures to pulse a glowing red color, which is also done as a Unity animation. It’s a great tool for little snazzy tricks. It’s a bitch to figure out how to use it, but it’s still a million times easier to understand than 3DS Max. So that’s good.

So Unity 2.6 is a perfect tool of development wonderment, right? Well, once you get over the learning curve, yes, for the most part, it is. But outside of the product itself, things get scary.

Uh … Support?

When I was first using and learning Unity last year, the forums were abuzz with feedback from the Unity team. Unity employees actually helped people  figure out problems and worked with us. That seems to have stopped.  There’s one or two posts a week where QA will pop in and say, “Is that still happening? Send me a proper bug report,” but that’s it. For a young product, support is essential. Bug fixes are essential. There should be patches to the development platform every few months, not one per year. What the hell is going on over there?

In fact, I was pretty annoyed when they made Unity Indie a free product. I wasn’t upset because I’d already paid my $200 for an Indie license — I was upset because of the subtext: when you make your product free, your support quality goes way down. When the barrier to entry was $200, there weren’t very many 13 year olds trying it out on a whim. Now that the introductory product is free, though, there are so many newbies using the product — and let’s be quite frank, most of them are complete newbies to programming — how could Unity possibly keep up with their friendly forum support?

The answer, of course, is that they haven’t. They created Unity Answers where developers can help other developers. This works great for easy questions (newb questions and the like). It doesn’t work at all if you are having an unusual experience or trying to do something esoteric. Best case is that you will get an answer that restates the documentation, sometimes slightly more clearly. Normal case is that tough questions are ignored. Worst case is that some Internet Super Hero will ride in and tell you how you’re doing it wrong anyway (at a philosophical level that ignores the actual problem, such as “Unity isn’t ideal for manipulating textures; you should have the artists do this in Maya”), and then ride off into the sunset.

And speaking of Internet Super Heroes… there’s a lot of them now. While there are still a good number of really smart developers on the Unity forums, and I still search there when I have a problem, the fanboys are also a lot more prevalent now. For example: I found a fairly annoying bug in Unity 2.6′s streaming loader, but when I asked for a workaround or confirmation that it was a bug, a fanboy told me that I just needed a better webserver. (A better web server to keep my client from crashing?) I pointed out that Flash player’s streaming loader doesn’t have trouble on the same server (when streaming much larger files), so that ruled out my webserver as being the primary culprit. To this, I was told that I was clearly wrong, or I didn’t know how Flash worked, and there are no Flash games anywhere near as big as Unity games, and anyway Flash doesn’t cache files (!), and so on.

Basically if people bothered to talk to me at all, it wasn’t because they had advice to offer. It was because they felt they needed to stick up for Unity since the employees can’t manage to even read all the threads on the forum, let alone respond usefully to them all. Gee thanks, guys.

The kicker is that I eventually found a couple of blog posts in a corner of the internet discussing the bug, where Unity’s developers acknowledged that it was a tough issue because the fix was very browser- and OS-dependent, and that they are still trying to find a clean fix for it. That’s all I wanted to know — that, and maybe a time estimate. I understand that it’s a hard problem, and I wasn’t demanding a hot-fix that afternoon. The last thing I needed, though, after hours of banging my head on a problem, was to be considered a troll.

I’m not a troll. I’m a developer who hates spending hours and hours of his weekend researching undocumented functions because the docs are shit and the forums are unsupported. The docs are adequate for easy stuff (now), but once you leave the friendly happy land covered by the demos, the docs become more and more skeletal. You just try making a complex  graphical Editor plug-in, or doing fancy things with the “headless” command-line client, or adding GUI features like focus-management to your game. (These all require using undocumented functions and techniques. At least in the latter case, the forums have a few clever people who found the undocumented functions and showed how to use them.)

For these sorts of things, I would be happy to pay a nominal fee for support. Let me give you a few hundred bucks for you to answer my damned questions, PLEASE.

Unity for iPhone: Completely Dead or Just Mostly Dead?

Unity has been focusing a ton of their effort on iPhone support. It gets more updates than the main Unity product does, and obviously a lot of Unity’s attention is on it. The reason is obvious: developers are paying for Unity iPhone licenses.

Well … the writing is on the wall for Unity for iPhone. The next big iPhone upgrade explicitly disallows most kinds of third-party development tools, and explicitly states that:

Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

Wow. Bye-bye Unity for iPhone. The forum users are hoping this somehow doesn’t apply to them, but come on. It’s pretty darn clear. It says iPhone programs have to be originally written for the low-level Apple APIs. No matter how you phrase it, Unity apps are not written “originally” for WebKit or Objective-C. They are written in Unity and transliterated into Objective-C. This is clearly no longer allowed.

Steve Jobs weighed in on why he did this: it was to kill Adobe’s new Flash-to-Objective-C compiler product. Now, many of his arguments are bull (such as Flash not having multitouch support, or all that talk about how important “rollover” support is). But I saw Flash running on an iPhone at a conference a couple months ago, and it was really, really slow. I can see him wanting to ban products running this poorly. Unfortunately, this also quite clearly covers Unity, too. Steve goes on to say:

We know from painful experience that letting a third party layer of software come between the platform and the developer ultimately results in sub-standard apps … we cannot be at the mercy of a third party deciding if and when they will make our enhancements available to our developers.

This becomes even worse if the third party is supplying a cross platform development tool. The third party may not adopt enhancements from one platform unless they are available on all of their supported platforms. Hence developers only have access to the lowest common denominator set of features. Again, we cannot accept an outcome where developers are blocked from using our innovations and enhancements because they are not available on our competitor’s platforms.

Flash is a cross platform development tool. It is not Adobe’s goal to help developers write the best iPhone, iPod and iPad apps. It is their goal to help developers write cross platform apps. And Adobe has been painfully slow to adopt enhancements to Apple’s platforms. For example, although Mac OS X has been shipping for almost 10 years now, Adobe just adopted it fully (Cocoa) two weeks ago when they shipped CS5. Adobe was the last major third party developer to fully adopt Mac OS X.

This applies to Unity too. Third-party layer? Check. Cross-platform development tool? Check. Now it’s true that Unity apps perform a whole lot better on the iPhone than Flash does. But that doesn’t seem to matter here; Jobs’ same arguments clearly apply, in principle, to Unity. If Unity finds a technical way to weasel around this, you can be sure Adobe can do the same. And if Apple just selectively bans only Flash but not Unity that would seem to open them up to lawsuits.

Unity’s CEO has posted about this on the Unity blog, saying, in a nutshell, that Apple hasn’t explicitly told them personally that they are screwed, so they are optimistic. That sounds crazy to me, but what the heck do I know. Maybe they will be fine. Maybe they have a crazy hail-Mary plan. Maybe the Department of Justice will save the day. I just know that if I were making iPhone apps, I would be exploring new platforms right about now.

I don’t mean to say they’ve been ignoring non-iPhone development. They seem to be working hard on a new major revision, Unity 3. But if Unity’s iPhone support is killed, there is a silver lining: 1) I don’t want to make iPhone apps anyway, and 2) maybe they will start working on the web player more aggressively again. Partnering with somebody to increase the installed browser user-base would be great.

Unity 3

Very little is known about Unity 3 — which is terrifying since it ships in a few months. We know if you buy a 2.6 license now, you get a free upgrade to Unity 3. That sounds awesome! But in practice it seems to mean that they have stopped bug-fixing Unity 2.6, so we’re probably going to need to upgrade. Will that be hard? No idea. Is the API mostly backwards-compatible? No clue.

They showed off some cool GUI features on the blog. The features are cool. But what I want to know is: will I have to rewrite big chunks of code (such as my GUI code, rife with those damned undocumented-function calls)? Or will I just need to spend a day or two in code-upgrade hell? This is a pretty important question for me. If you’re worried about it, you should probably wait to buy until Unity 3 comes out. Although if you do, you will lose out on the pre-order discount pricing …

So do I hate it or love it?

If I come across here as schizophrenic in my opinion of Unity, it’s because I am. First, I love it. It is the only way a couple of developers like Sandra and I can realistically make a full 3D MMORPG on a part-time schedule. There is no other product that can realistically do this given our limited time constraints. And I am pretty confident that the performance and stability can reach satisfactory levels.

On the other hand, the lack of good docs are a severe hindrance, their sluggish bug-fixing pace is irritating, and their lack of web support presence can be infuriating. Add in my other fears: that Unity 3 will be a hellish upgrade or that Unity the company may actually go out of business if the iPhone product dies. So maybe it’s not such a good time for you to be following in our footsteps and making your own Unity MMO.

And anyway, we really don’t need the competition …

Elder Game is sponsored by:

As far as the Champions CSR team was concerned, the game had been doing really well in beta. Sure, they were collecting 1200 bug reports a day but that’s pretty normal. In fact, it suggests a very efficient CSR pipeline if they actually managed to process all those reports. But once the game launched, the number of complaints about bad behavior skyrocketed.

It turns out that — get this — juvenile males are attracted to the superhero MMO genre in large number. They cuss. They insult. They cuss with insults whenever possible. This bothered the CSR leads a lot. Perhaps they were new to the MMO business… let’s suppose the CSR leads came from running a phone bank for a major publisher. If 100+ people care enough to call in about an issue with an EA game, that gets their attention really, really fast. Or worse yet, perhaps the CSR leads came from running a call center for a non-game product. 100+ calls about the same topic would cause an overnight hotfix in most software houses.

So they raised the alarm. “Our players,” they said, “are making other players uncomfortable with language. We need to fix it.”

Now any MMO producer can tell you that you don’t piss off the CSR leads. Aside from the forums, which are extremely unreliable for data gathering, CSR is the only way you hear what’s really broken in your game. A good CSR team is way better at bug-hunting than the QA team. That’s because they hear every bug every player finds, instead of just the bugs a handful of testers find. You piss these guys off and suddenly you stop hearing what’s going on in the game. Your ears go deaf and you start losing players without being able to tell why. This is not a mistake a good producer makes more than once.

So maybe the producer jumped right on this. If he’s done this before, he probably rolled his eyes when he got the request, because he knew it wouldn’t help much, but he prioritized it anyway. He sent it to the engineering lead. “Make the CSR team happy. Add a profanity filter to the game. Make this a high priority.” Now remember, the engineering lead has TENS of THOUSANDS of bugs to triage, plus new feature requests coming in every day. So he no doubt rolled his eyes at this suddenly-important tech request.

He then handed the request to his fastest engineer. “Marty, I’m going to take you off the rewrite of the generators for half a day. You’ve got four hours to get a profanity filter in the game.” The CSR team helpfully created a massive list of cuss words (in many languages), drug references, references to sex acts, and hundreds of other things that might be even slightly icky.

Marty’s a smart engineer, and he works really fast. He gets the profanity filter working in two hours. Hell, there’s time left over to polish this feature before he breaks for lunch! So Marty adds some heuristics to detect slight cussing deviations. His algorithm assumes that if a word is one character away from being a cussword, it’s actually a cussword in disguise. He submits his fix for QA approval by lunchtime, and heads off to get his caffeine and grease fix.

Now, the QA team is almost useless. Two reasons: the experienced QA folks got shipped off to Star Trek Online the week after Champions launched. None of the remaining guys knows how to make a test plan. The best they can do is load up the new feature and poke at it for a few minutes. Worse still, these guys are demoralized, overworked, and busy looking for their next job. They may not be good QA (yet) but they aren’t stupid. They know who gets fired first when an MMO game doesn’t have a spectacular launch. Yeah, that’d be QA. Every damned time. So these guys are hardly motivated, barely trained, and tired as all hell from the double overtime they’ve been forced to work.

Jim in QA gets tasked with testing the new profanity filter. He doesn’t know how to make a test plan, and Marty the engineer didn’t have time to make one for him, so Jim just loads up the game and starts typing in cuss words. It seems to work. But he knows that he needs to file a bug on the feature. When QA is in trouble, they always file a token bug on each feature if they can. That way they look like they’re working. (In fact, you can tell if a QA department is borked by how many token bugs they submit versus the number of serious issues they find.) It takes a while but Jim finds something to complain about.

The bug gets sent back to Marty the next day. “You missed cuntsuckler” says the bug report. “I say that all time so it should be in the list.” Marty the engineer adds it to the list, grumbling all the while about QA wasting his time, and then the feature is certified as done! It goes live two days later.

The *!&^%(* Feature Is Live

Cut to me playing the game. There wasn’t any mention of a cuss filter in the patch notes, so I was quite surprised when I saw a random NPC run by, saying:

Clearly this NPC was trying to cuss me out in Spanish. Whew, thanks profanity filter!

(I’m not named Templar, by the way… the NPC’s just spout off about whoever is nearby.)

It took a while to figure out what happened. The poor NPC tried to say “put a stop to the Qularr” but the cuss filter interpreted it as “put a“, which is just one letter off from “puta“, which is a cussword in Spanish.

So it’s not bad enough that I can’t say “put a” in Champions Online anymore… the NPCs can’t say it either!

The next day it’s even worse: an evil villain tries to tell his henchmen to “put that hero in a pine box”, a reference to murdering me. But alas, he said “hero in“, which of course is a misspelled reference to drug use. A reference to murder? That’s fine. Using drugs? Whoa there! So It comes out as “put that $#@*^!& a pine box“. What a foul mouth these NPCs have!

It’s ironic, perhaps, that cussing continues unabated. I can’t say “put a” but I can still say “p-u-t-a”. Turns out 13 year olds figured this out pretty quickly too.

What’s going on here? Why did this feature get out the door like this? It’s an embarrassment; it does more harm than help.

This is just a symptom of Champions’ painful death throes.

Death By Resource Atrophy

Even before they got their first months’ numbers back, Cryptic knew the game wasn’t going to be a big chart buster. Now they’re just hoping to break even: if they can keep enough players around to keep the bill collectors away and the lights on, then the company can try again with Star Trek Online. And hey, maybe in a year or two they can launch Champions for the XBox 360 and it will be a mild success.

This is very bad news for Champions players. Champions has been relegated to the role of red-headed stepchild… it’s that crappy failure of a game that keeps stealing resources from Star Trek Online, which is the game that’s going to save the company. (The same fate befell Turbine’s Dungeons and Dragons Online when it became obvious the game wasn’t going to recover its development costs quickly… its team was rapidly stripped of resources so that Lord of the Rings Online could be a success instead.)

So now Champions has a tiny, bedraggled, demoralized team. If they’re lucky, the team has taken on that grim, “we’ll die trying” mentality that desperate live teams get. I think some of the team has that now. A few people are clearly trying hard. But there’s no support, so every few steps forward also makes them stumble backwards a step.

I’m rooting for you, guys. Yes, I’m pointing out how you’re making tons of mistakes, but I also think you’re doing the right thing. At this point, you’re not going to get a reasonable amount of QA support, so you’re best off cramming in the fixes, new bugs be damned. If you fix 20 bugs and add 3 more, that’s still better than fixing 10 bugs and adding 0. Desperate times call for desperate work.

This is also the time when the live team’s producer must shine. If Champions is around two years from now, it’s because the producer made the right calls during these first three months of the game. My suggestion: pick your goals with QA. Insist on actual test plans (that you review) for the handful of features that are really important. Having QA working closely with the engineers is ideal but not realistic now. So have your lead engineer make test plans for important features (or, if your lead engineer writes the feature, have a different engineer make the test plan.) Don’t worry about seriously testing the small features. There’s no way you can get all your new stuff covered by QA. It sucks, but it’s the facts. Focus on a feature barrage instead. (For all I know, that’s exactly what Champions is doing now. But if so, they aren’t adding the important features nearly fast enough.)

Yes, I will laugh at your embarrassing bugs, but if you address the serious issues (like useless crafting, wildly arbitrary quest difficulties and rewards, and a cruel newbie experience for people who don’t research the right build on the forums), I’ll keep playing. I think you’ve got another month to show significant progress. After that, even folks like me, who really want to play this game, will give up and quit for a while. So show us what you’ve got! Your next month’s effort can have a major impact on your population numbers. It’s do or die time for Champions.

Elder Game is sponsored by:

You might be thinking my request sounds like one of the lazy animals from the story of the Little Red Hen:

“Who will help me code the MMO?” asked the little red hen (… I mean the MMO company).

“Not me,” said the independent MMO contractor. “I’m too busy doing fun easy things!”

“Fine, I will do it myself! Ah, but now who will help me run the MMO?”

“Ooh ooh I will!” said the contractor.

“No no, you didn’t help me make the MMO, so you don’t get to do the fun part! I will run it myself!”

If this is the story that runs through your head, you are definitely not from any of the major MMO companies. For the likes of Turbine, SOE, or Blizzard, making the MMO is the fun part. Working on the MMO afterwards is the terrible part.

You might expect that the people who spent five years making the game would be excited to run it after it ships. Turns out, not really. After five years of working on the same project, they’re so sick of it they never want to work on it again. They want it to be in good hands, certainly. And they want to have some oversight to keep people from damaging their vision of how the game should run. But they sure as hell don’t want to have to do that tedious maintenance stuff themselves. So companies tend to pull the experienced staff off of the live game pretty quickly, leaving behind junior people.

Live Teams Are Not Glamorous

I like the “tedious maintenance stuff.” I actually prefer working on the live team. This makes me very unusual in the MMO industry. I am also a pretty good engineer with a lot of experience, which means I don’t often end up on live teams — too experienced. At Turbine, I had a hard time getting onto the Asheron Call 2′s Live Team, because I was expected to help develop their next generation MMO engine instead. I wanted to work on AC2 after it ships?! None of my managers could understand why I wanted to be demoted like that!

But to people who enjoy the live team, well … there is nothing as good as it. The power you have! The instant feedback! The ability to literally make hundreds of thousands of people happy with just a few weeks of work. It’s very gratifying. There’s also the tedium and frustration and lack of resources and constant fire-fighting and oh my god I can’t keep up with everything… but that’s the price of the deal.

Of course, it doesn’t just happen that you hop onto the Live Team and suddenly you’re making game-design changes. At first there are a lot of smart and talented people at the helm, helping you learn the ropes, making the hard decisions for you, keeping you from doing stupid things. But inevitably they are pulled off to other projects, and somebody relatively junior gets the helm. That’s how I got to be in charge of balancing AC2′s classes.

Fortunately, I had a decade of engineering experience and understood how to tune complex systems. I wrote analyzers, modeled usage patterns, and made corrections.

Unfortunately, my approach did not take the “human equation” into consideration very well.

Learning to Balance the Human Equation

I found that the Feral Intendant class was 30% overpowered, and that’s why so many people were playing a Feral Intendant. Yet somehow, reducing the power of the Feral Intendant to the correct level did not suddenly make the game more fun… thousands of players were complaining and nobody was telling me they were happy about the change. Weird! I double checked my calculations. They were correct. So what had gone wrong?

Turns out that the people who played the other classes available to that race had taken on an “underdog” mentality. The people who played Claw Bearers liked that they were woefully underpowered compared to Feral Intendants. It was like playing the game on Hard Mode. And the people playing Feral Intendants liked playing on Easy Mode. In balancing the game I had failed to understand the needs of the people playing it. I just ham-handedly fixed the equations, instead of solving the problem with the finesse it needed. It was one of my more serious missteps. (And it’s a great example because I think it’s pretty obvious in hindsight. Most mistakes were much more subtle.)

But man, what a fast way to learn! After just a couple years of that, I became a good game balancer. The constant feedback loop helped me learn from my mistakes in a matter of weeks! Compare that to developers on traditional games, who must wait until the sequel ships before they get to try their hand at balance again. That’s why working on a live team is such a fast way to learn your craft: the feedback is so much faster than any other gaming platform, that it accelerates learning by dozens of times.

But AC2 cost millions of dollars to create. Turbine didn’t create it as a tool to help me hone my design skills, that’s for damned sure! How did I get to do it? Simple: the designers who would have done it were burned out of working on AC2, and were called away to work on the important New Project. AC2 wasn’t a blockbuster hit, so it didn’t make sense to use the rock star designers on it. Better to let the B team step in.

The Steady Hand Has Left The Rudder

But here’s the weird thing: WoW is exhibiting the same symptoms as AC2 did when I was doing the designing. The B team is in charge.

In February, we learned that lead designer (and part-time producer?) Jeff Kaplan had stepped away from WoW, off to work on the next big Blizzard game. However, if you were watching the game before that, it was obvious that major leadership changes had already happened months earlier. My guess is that Jeff Kaplan started moonlighting on the new project long before February. And many of the other key WoW live team people have also switched over, or are working on WoW only part-time.

Now, I am not being alarmist. The ship is still in intelligent, capable hands… but clearly not as experienced ones. Just as I did when I took over AC2, WoW is making newbie design mistakes that seem like a benefit on the surface, but are really not good decisions. There have been scores of examples… I’ll pick just a few.

“It’s always been stupid, and we just need to fix it!”

A few months back, the powers that be decided that Hunter ammunition didn’t work right. Hunters have to carry an arrow for every single shot they take, and in order to get the full benefits from them, they have to carry them in a special quiver — which doesn’t let you store other items in it, only ammo. All that ammo costs money, too.  Plus, it leaves the designers unable to give out awesome “raid arrows” because you’d just shoot them all and then where would you be? Even though ammo had been a fine and fun distinguishing quirk of Hunters for years, it was time to Fix It.

The first plan was announced: WoW would no longer have consumable ammo. Instead, you would just need a single “infinite arrow” that you stuck in your ammo slot, and this would let you shoot your bow forever. Problem solved! No more quivers, no more pack space wasted, no more costs. And now raids could drop “loot arrows” that wouldn’t get used up! Perfect!

Whoops, turns out that plan would be hard. So they announced their backup plan: now ammo just stacks to very high numbers. Instead of having stacks of 200, now you can have stacks of 1000. This at least addresses the “pack space” issue. Call it a win! And they removed the magical benefits from quivers, so you no longer needed to use them. So they fixed the immediate emergency, and they’ll get to the “correct fix” later.

The thing is, there was no emergency. Sure, Hunters were happy to have a few extra pack slots. But the change threw all sorts of other things out of whack: magic quivers are still given out as quest rewards… they just aren’t magical anymore. And leathercrafters can still make them! They just can’t sell them to any sane Hunter. And so on… the game wasn’t really cleaned up after this change.

But I’m sure it felt so pressing, so urgent. So they had to address the issue, side-effects be damned.

Without somebody experienced at the helm, the voice of the myopic designer tends to be the loudest. “WE HAVE TO FIX THE HUNTER” they said. Maybe they said, “Hunters have to spend 65% more on bare essentials than any other class. I will never be able to balance class expenditures like this!” Or maybe they said, “Hunters have to waste more inventory slots than any other class. It damages quest completion rates!” Or maybe they just said, “It’s SO STUPID. It’s always been stupid, and we just need to fix it! Do it now!” Obviously, nobody thought very hard about the ramifications, and nobody spent any time easing players into the idea. And nobody stopped to make sure they did a good job.

So some tiny little mistakes crept into the game. Nothing huge. Nothing that will sink the Titanic. But mistakes nonetheless… “magical” crafted quivers that aren’t magical and can’t be sold are clearly a mistake. These little bugs accumulate, like lint on a hardwood floor.

The Lint Accumulates

When we say that WoW is “polished”, what we mean is that it is surprisingly clean of linty little bugs like these. But that’s changing.

More and more little mistakes have crept into the game recently — changes that are positive on the surface, but have not been implemented with the finesse that makes them worthwhile. Mana expenditure rates have changed, rules for dungeons have been tweaked, the cost of items has fluctuated. It all seems useful. But it’s usually full of little side effects. Worse, it doesn’t take the human equation into account: it doesn’t counter-balance for the actual needs of the players very well. There are ways to meet both goals, but you have to try a lot harder at it than WoW is.

Remember when WoW class balance happened every six to eight months? Players were actually excited when their classes’ turn came around. I remember being so astonished to see players that were actually happy to have their classes redesigned. But now, every class is fiddled with every few weeks. It’s not exciting anymore. Instead of sitting on the changes and carefully honing them, the designers are just firing out every new idea they have, willy nilly, until they get it right. But here’s the thing: it doesn’t matter if you get it right. It matters if players are excited and having fun. Balance changes are happening too fast, and for too little benefit overall.

Back in the day, QA held the game to a higher standard. Consider that there never used to be skill changes that would invalidate the client tooltips about a skill (unless it was an emergency exploit-fix). If the designer wanted to tweak a skill, they had to wait until the client could be updated. But the QA bigwigs are off doing something else now, so it’s easy for the designers to slip this stuff in. And they do. All the time. Skills are routinely incorrectly displayed now, as the designers’ need for perfect balance far outpaces the ability to do client updates.

Who’s In Charge Again?

You would never let your lead artist drive decisions for your game. Chances are, they would say “This dungeon is too brightly lit! We need to hotfix it now or the mood will be ruined forever!”

But unlike artists, designers get a free ride. They’re supposed to know what’s best for the game. If the producers are busy, they trust the designers will do good things. But designers, especially young ones, get myopic. They tune into little issues — like perfect class balance – and turn them into epic quests. If the designer could just fix this balance problem, people on the boards would stop complaining, and the game would be perfect!

No. It will not happen. Perfection will not be achieved, ever. But there’s nobody around to rein them in anymore, so they try and try and try. And leave little messes everywhere they go.

Suddenly Communications Are Open

Another surefire way to tell that upper management has left the building? The systems designer “Ghostcrawler” has suddenly started posting a lot, even about… well, nothing. For years developers were nowhere to be seen, which was a shame. And then suddenly the lead systems designer has time to play the forum game? Yeah, whoever was making employee policies just doesn’t have time for WoW anymore. Not a bad thing, in this case, but certainly a dramatic shift of policies.

Nowadays it’s common for WoW to tell people to “check the forums for game updates.” This is a total newb mistake. Only your loudest and most annoying users will check your forums for updates. So every “update” is met with derision because only assholes post on game forums. (Statistically speaking, anyway.) Game updates are specifically what the launcher’s update screen is for. If you’re outpacing the ability to update the update screen, chances are you’re changing too much too fast. Slow down and get it right the first time.

It’s likely that Ghostcrawler started posting shortly after the upper management started wandering off to other projects. Ghostcrawler’s a good guy… in fact, his posts remind me a lot of what I sounded like when I was posting about AC2′s skill balance. He knows how to balance things. But he is completely unable to see the big picture. Every tiny imperfection seems like a ruinous problem. He feels assaulted on all sides by problems, too, and doesn’t think there’s time to do things the right way. But this is an illusion that happens to Live Teams because they get so close to the product. He needs someone checking over his decisions and making sure they’re worthwhile. He doesn’t have that.

WoW: No Longer Big Kahuna at Blizzard

Ghostcrawler and the rest of the team will learn their craft soon enough. WoW will survive the experience. But what’s interesting is that it tells us quite clearly that WoW is no longer the most important thing at Blizzard… in fact, it might be third or fourth place. It’s really interesting that this happened so soon. I didn’t expect it to happen to WoW while it still had 10+ million players or more still paying. But a company has only so many top-notch people, and you always want your most-experienced people on the new thing, so it makes sense.

To be clear, this isn’t necessarily a bad thing. When the game stops being in the spotlight, the live team suddenly gets a lot more flexibility to make the game fun, instead of being forced to stick to now-outdated “design visions”. The dramatic increase in WoW’s mobility options is certainly due to the lack of oversight. But without that safety-net of supervision, they need to exercise a lot of willpower and a lot of wisdom.

Ghostcrawler, and anybody else on the design team of WoW right now, I have a little unsolicited advice from somebody who’s been there: convince your bosses to let you play a different MMO for two weeks. On the clock. Don’t touch WoW. I know it feels like there’s a disaster every day and you can’t possibly stop focusing on WoW, but you can. After you get back, play WoW with a different class than you normally play. You’ll see so many new things! Your priorities will do a 180. I guarantee you it will help your perception.

Elder Game is sponsored by:

Sandra and I worked in the mainstream MMO industry for a long time, but a couple years ago, we stopped. We stopped because we better understood what we wanted: neither of us was real happy making MMOs. What we wanted most of all is to run MMOs. Unfortunately for us, running an MMO tends to require you to make one first. This is tricky, because the traditional AAA MMO takes three or four years and 50 people, and has a 50% chance of success at best. These are not odds we like.

So we’ve done other things — consulted on MMOs, web games, and all sorts of other things, and it’s actually been a lot of fun. But in the back of the mind there is still The Calling. So we tried to make our own indie MMO on the cheap. These were 2D or text-based, and we just couldn’t get into them. We needed our MMO to be 3D. We’re spoiled like that. And for practical reasons, we needed it to be web-based, because we can’t imagine being able to get a boxed product on the shelves.

But making an MMO is a huge undertaking. It’s not just a fancy 3D client; it’s also a scalable server, tools to develop and maintain it, and infrastructure to run it. But Sandra and I are experienced server engineers; we believe we can use off-the-shelf tools and some cleverness to make a reasonable little game server. And we are becoming more comfortable with various infrastructure approaches. But how do we get a 3D game on top of it? One that’s web based, too, and one with powerful development tools already made for us?

We tried various client applications, but they sucked. However, there’s a new contender.

Enter Unity 2.5

I’ve been watching Unity for a year or so now. It’s been frustrating to watch, because the numbskull developers created their first versions only for the Mac! (The resulting applications could run on any platform, but the development tools required a Mac.) When you’re an indie, it’s hard to justify doling out a few grand for a Mac in order to test-drive a piece of software you’ve never used before. This restriction didn’t stop Cartoon Network’s Fusion Fall from using the Mac-based version of Unity, but it kept most small developers, including us, on the sidelines.

However, two weeks ago they finally got around to making an accessible version of their program, one that runs on Windows or Macs. Finally! Sandra and I reorganized our schedules so that we would have a full week to experiment with Unity and a simple off-the-shelf server product called SmartFoxServer. Basically, we spent a week prototyping an MMO. Successfully.

What makes Unity special? Three things, in order of importance:

1. An enviably powerful tools pipeline, 
2. A rendering engine that works on any platform (and can run on web pages), 
3. And a very reasonable price tag. 

Let’s go over each one.

1. The Development Pipeline

“Development pipeline?” you may be thinking. “Who cares! How many polygons can it push? How many draw calls does it take to render things? Where are the technical stats?!” That’s basically irrelevant for us. We can design our game to run well under whatever conditions the engine allows. This is fortunate, because tech-wise, the engine just doesn’t seem that amazing. If you’ve played Fusion Fall, you may have noticed the low framerate for relatively simple scenes. It’s just something that has to be worked around.

There are lots of free or cheap 3D engines out there, and many of them are far more powerful than Unity in terms of rendering. But those were completely useless to us because they had no tools pipeline. A real MMO needs a client program, sure, but it also needs dozens of man-years worth of tools to build the content for the client. Indies don’t have the resources for that.

This is where Unity 2.5 shines. The Unity development environment integrates directly with Maya, Max, or various other 3D clients, plus code editors, sounds, and Photoshop files, to make a really compelling development environment. Import your 3D characters and drop them right into the scene, then start scripting them to respond to animations. Create terrain in Maya or directly from within Unity. Configure the built-in physics engine, position lights in real time, and then run everything together, watch it work, and fiddle with things on the fly. This is a great way to prototype stuff. It’s fast, it’s efficient, it’s … pretty alien to most programmers. If you’ve learned to develop in Flash, it’s sort of that mindset: it’s more resource oriented than scripting oriented.

Placing an asset on some terrain I just made

This can take some time to get used to, but it’s plenty powerful and elegant when you do master it, and it’s sufficiently versatile that you can use it for a whole lot of games.

On the other hand, this complex development environment makes it harder for programmers to manage lots of code. For instance, scripts are attached to assets and then the script’s variables are individually configured. This is done automagically and makes for a very cool customization experience. But if you decide that you need to change the values of a variable, you may not be able to find all the uses of that variable with a text-editor search: the user may have overridden those variables in the project itself, leaving you with no way to find the values programmatically.

It also presents some co-authoring issues: you’re all working on the same assets, after all. Unity did a decent job of letting you merge projects together, but that’s only if each developer is working on completely separate parts of the client. If you’re each fiddling with the same prefabricated object, you’re screwed. You can’t merge the binary assets: somebody’s work is going to get lost.

So this pipeline is ideal for small teams, or for larger teams who have spent some serious planning time figuring out how each person is going to avoid stepping on other people’s toes.

But let me just quantify this toolset’s value: Sandra and I were able to download the demo version, learn how to use it, and then create a 3D zone with mobile, animated avatars that talked, punched things, exploded, lit on fire, and so on — in a week. We also had to learn how to use the server library we picked, too. Fortunately for us, SmartFoxServer actually comes with a demo that shows you how to synch up multiple Unity clients. We achieved pretty amazing results in a week, but we took advantage of a lot of demos and free assets to make it happen.

Still… this is an unsurpassed prototyping tool. Even if you don’t use it for the final client, just imagine that you could get your next prototype up and running in a week, then iterate on the design every day after that. Now you can. I wish we’d had this when we were trying to prototype Star Trek’s space combat.

2. Web Based 3D Out of the Box

Another important advantage is its flexible runtime environment. It runs on Macs and Windows. It can be a stand-alone program or embedded in a browser. And it isn’t hampered by the “you must support the lowest common denominator” mentality that Flash has. For instance, your game can support multi-button mice, even though Macs may not have them. Conversely, you can program for that weird meta-key (the Command key, I guess?) even though its analog on PCs is the Windows key — and when you press the Windows key in a web page, the web page loses focus. But I’m very happy that they just gave us all the obvious capabilities and left us to figure out how to sanely use them, rather than oversimplifying.

The compiled files are nice and small, for what they are. I was able to get a pretty complex scene, complete with lots of scripts, animations, and networking, into an 8mb file. (Of course, users also have to download and install the Unity plug in for their browser; that’s where the “engine” code lives.)

It also has some complex tools for data streaming, which we didn’t get around to testing out yet, but they seem pretty robust. They also require a lot of planning, but that’s still a whole lot easier than coding it ourselves from scratch.

3. Cheap Price Tag

The price is very reasonable. It’s a couple grand for Sandra and I to each get professional licenses. That’s it; no percentage cut or anything scandalous like that. You even get free minor version upgrades added in (which is good, because that’s the only way they do bug fixes).

It’s not dirt cheap, but let’s be realistic: 3D games are still expensive. Sandra and I made an off-the-cuff budget that cost $40k for 3d artwork. That’s peanuts compared to a mainstream MMO, but puts it well out of the reach of the very smallest of indies. If you can’t afford a couple grand for an engine, you can’t afford to make a 3D game just yet. Maybe in five more years it’ll be at the cheapness level that 2D games are… but it’s just not there yet.

(There is also a cheap “indie” license that costs $200. This is a good way to get started with development, but the restrictions mean it’s not too practical for developing a complete commercial MMO. It should work okay for other 3D games though.)

What’s the Down Side?

So the good news is that this is a realistic way for a small team to cheaply make an MMO. Fusion Fall already exists: it proves that it’s possible. But Unity is not without it’s painful side. Once again there are three main issues:

1. Bugs
2. Language Issues
3. Documentation Flaws

1. Bugs

The primary down side is that Unity 2.5 crashes a whole lot. It’s essentially version 1.0 of the Windows line of Unity, and it shows this in its lack of stability. As the military would say, its “mean time between failure” is about one hour. This is not good. You’ll have to get used to saving every few minutes. But worse still is that two of our crashes caused the project to become corrupted. Maybe if we’d been more advanced with Unity we could have repaired and moved on, but as newbies, this was devastating. We lost many hours of work when this happened. We eventually instigated a “back up to a new folder every few hours” policy.

Obviously this needs to get fixed. Unity 2.5 has only been out a few weeks, so I am reasonably hopeful that they won’t leave us hanging for too long.

2. Language Issues

Unity has a schitzophrenic relationship with programming languages. Officially, it supports three languages: C#, JavaScript, and a variety of Python called “Boo”. But this is basically a lie.

It supports C# because it’s written in C#. This is the language you should probably use if you’re a team of experienced developers. However, none of the examples show how to use the code in C#. You will have to muddle with it for many hours to get the nuances.

It supports “JavaScript”, and this is the preferred language. The demos are all in JavaScript, and the code examples are in it, too. However, this isn’t really JavaScript. It’s an upgraded version that takes a bunch of ECMAScript features that aren’t in JavaScript. Then it tosses in some special functionality specifically for Unity. And then… it doesn’t document any of it. There is no language reference for their made-up version of JavaScript.

The support for “boo” is entirely mythical. I’ve seen no code for it ever, nobody on their forums uses it, and it goes without saying that there is not a lick of reference to it in their help. You’d be really stupid to decide to use boo for your project.

More annoying still? The languages are poorly interoperable. We were pulling in code from lots of different demos, and needed to use both JavaScript and C# code in the same project. It turns out that when you have two languages in use, there are dependency issues that can only be worked out by sticking your code in special “load me first” directories. Very kludgey. At least it can be done. 

3. Documentation Flaws

The “Unity Manual” is a tiny wisp of a thing. There are no real docs on how this stuff works. What there is, is a massive step-by-step tutorial that teaches you how to make a platform game in Unity. This is awesome… if you’re the sort of person who learns by doing. I am the sort of person who prefers to absorb all the data available and then start exploring. I simply can’t do that with Unity. Those docs don’t exist. For a commercial product they are significantly under-documented.

Expect to spend days just screwing around with the demos in order to have any clue what’s going on. Expect to search frantically through their forums in the hopes of understanding the syntax for their scripting languages and complex GUIs. Expect a few sudden jarring inconsistencies in what is otherwise a smooth and orthogonal interface.

The docs looked especially paltry when compared to SmartFoxServer’s luxurious documentation. Yes, SmartFoxServer is a much simpler piece of software than Unity. I don’t care though. Fickle that way. Need docs.

The Bottom Line

If the question is “can Unity be a viable MMO client?”, then it’s been answered by Fusion Fall: “yes”.  But the neat thing about Unity is that after spending a week with it, you would easily come to that conclusion on your own.

It’s not for everyone, of course. If you can’t deal with the relatively paltry graphics level allowed, or if you need your tools to conform to your existing pipeline, then you’re not going to like Unity. You have to be agile enough to work with it instead of against it.

But after a week of using it, I’d have to say that Unity feels pretty good. Maybe this program is the missing piece in our indie MMO plans.

Elder Game is sponsored by:

Man I hate when QA people do their jobs without specs! It’s so irritating. When Asheron’s Call 2 launched, there were thousands of outstanding bugs in the QA database that we opted not to fix before launch. Sounds bad, doesn’t it? But if you looked at them, you’d understand. Hundreds of them were bugs about how buildings were floating 2 virtual inches off the ground; if you zoomed the camera down to the floor and looked up, you could see that these structures were very slightly hovering.

Hundreds more were about items that “popped in” too soon or too late; the “art degrades” for the items weren’t set up right, so they seemed jumpy. And so on… thousands of little tiny nits.

Why does that piss me off? Surely those bugs should have been in the bug database, right? Even if they’re not fixed immediately, they’ll get fixed at some point! Right, true. Except for this:

Asheron’s Call 2 launched with over 3000 severe yet unrecognized bugs. It is not unreasonable to argue that AC2′s early failings were due to the lack of quality in things like crafting, combat, and skills. Every time the QA people entered a bug about a floating object, that was time they weren’t spending finding more serious bugs. For instance:

• Many of the crafting recipes were about ten times harder to create than intended. The creatures that dropped the needed parts spawned incredibly rarely, due to an oversight.
• Many of the quests could easily become broken if the steps of the quest were done in the wrong order. Fixing them then required the assistance of a customer service representative.
• Almost every skill in the game was broken in some way. Some skills literally did nothing; others were too costly or too powerful; some started out super strong and then got weaker as you leveled up the skill.

And so on. Serious bugs, very much worth fixing. These didn’t make it into the database at all; the live team stumbled upon them when players started screaming about them. What the hell, QA? What the hell?

I’m just being a jerk to QA here. One of the reasons that QA did such a terrible job on Asheron’s Call 2 was because there were almost no design specs.

It was the extremely talented Jesse Kurlancheek (a.k.a. “Devilmouse”) who was responsible for the skills in AC2. Because of the agreement Turbine had made with their publisher Microsoft, he was obligated to create at least 600 skills for the game before it shipped, broken into 30 different skill trees. Problem was, he only had three months to design and implement them all. Jesse famously told his boss, “I can either implement the skills, or I can document the skills, but there’s no time to do both.” And he was right. But they chose to implement instead of documenting, and the result was tragedy.

Without any idea of how skills were supposed to work, QA never looked at any of them. To be fair, QA should have at least poked around with them some, but they just weren’t motivated to spend any time on them because the designer would almost always say “no, that’s not a bug, that’s how it’s supposed to work!” So the QA team spent their time wandering around looking for objects that were hovering a tiny bit off the ground, instead. At least those were irrefutable bugs!

The lack of specs was devastating. When the game launched, the live team needed to get the game into a maintainable state. Without specs, we had no idea what Jesse had intended. Worse yet, a few months later, not even Jesse remembered what the intention of each skill was. So we created elaborate analysis software in order to locate skill deviations, and slowly reverse-engineered the intentions, like archaeologists exploring an ancient culture. The lack of specs cost us thousands of man-hours. It made us look like total dicks, too, when we fixed the outrageous bugs we found.

The “reap” abilities in AC2 stole the health of an enemy and gave it to you. However, as is typical of this sort of power, you couldn’t steal more life than you were missing. So if you were only missing 100 health, you could never steal more than 100 from the enemy. Tragically, there was an accidental minus-sign in the implementation. If you reaped somebody and you were fully healthy, instead of doing no damage, you did double damage. Reaps were the most effective attacks in the game, provided you hadn’t been hurt too much. It made PvP in particular a nightmare, and players were howling for us to fix it. When we fixed it, though, several classes became unplayable; it turns out they had only been fun at all because of that broken skill. So then we had to buff those in various ways, until the next major bug was discovered, which threw our rebalancing out of whack again. And on and on, over and over, a constant dance of chaos and confusion, for over a year, before we had created our own specs and could work towards stability.

If only the skills had been affected this way, we could have dealt. But the lack of specs permeated almost every aspect of the game. The quest areas were about half-documented; the monster spec was nothing but one extremely complex spreadsheet. It’s not like the AC2 designers didn’t know how to make specs, of course: they were given impossibly small time windows to do their work, and they did their best.

If our publisher had allowed us, it would have been so much better for AC2 to have launched with only half the classes, but with docs for all of the classes. Then the live team could just implement the docs, rolling out new things every couple months, and make Turbine look super productive instead of super incompetent.

In the end, specs save money for three reasons.

• Specs allow for collaboration on a design. If it’s all in your head, nobody can point out the flaws in your plan until you’ve implemented it already.
• An MMO that is supposed to run for 5 years needs at least a rough semblance of documentation, or the game’s maintenance will cost a ton.
• Specs are also mandatory if you want to use a QA team. If you can’t afford to write specs, just fire the damned QA team, or convert them to content designers or something. QAing without specs is an amazing waste of time and just makes everybody angry.

And just to be clear: I love working with a talented QA team. I am 100% pro-QA. But you must give them the ability to succeed. Without specs, you’ve set them up to fail.

The Problem With Rapid Content Creation Tools

When I was at Perpetual, the Gods and Heroes team had a quest-development process where designers didn’t need to write out all the details of each spec. They just opened the database and plop! they dumped their quest right into the game. And man, did they hate QA. They agressively, loudly, viciously denounced the QA department. QA was always telling them the wrong thing and never finding the real bugs. The QA at Perpetual was the scapegoat for all problems. Even the QA team lead was apologetic for how shitty they were.

But how could they be successful? The only notion of how these quests were supposed to work was in one designer’s head. Eventually, the QA team wisely stopped reporting bugs about quests. But there were still bugs in the quests.

When Perpetual’s Star Trek team was planning their content pipeline, everyone just assumed the same system would be used. This was a major point of contention for me: I wanted designers to write out every detail about the quest before they implemented it. “That’s ridiculous! It will double the time needed to implement quests! We can’t budget for that!”

Okay, fine. But if you can’t budget for specs, don’t budget for QA, either.

Elder Game is sponsored by: