Unfortunately, making a full 3D MMO on just a few days a week is very hard. We’ve tried it a few times in the past, but when it comes time to make the timeline projections, we always shelve the idea. The problem is, we have a lot of experience in the industry, which keeps us from trying to do things that are “stupid” and “irrational” and “never going to work”. Industry-naive developers can sometimes pull off amazing miracles by sheer insane doggedness — and by not knowing that what they’re doing is impossible — but we know the actual amount of time each feature is likely to take, and we know from experience that by the time you finish the “engine” parts, you’re so focused on technology that you don’t have any mental room left to make the game itself.

The thing is, players don’t give a crap about the technology. They care about the game mechanics, the graphics, the atmosphere… but not the tech. So what we need is a way to skip the “engine” part and get to the part that matters.

So naturally we keep a close eye on all the MMO-capable “engines” (or engine parts) as they showed up. Mostly these are pretty useless. There are dozens of MMO engines that sound great on a website somewhere, and then you download them and realize that step #1 is “write 75% of the engine” (such as Project Darkstar, which is not much more than a handful of low-level server-side architecture primitives) or “refactor the engine to be usable” (such as Ryzom engine, whose code is an utter mess at the moment). These projects have ongoing developer support and some day they may be amazing. But that’s not helpful now.

But one product has stood out to us as being a viable client-side 3D platform: Unity. Not only does it have a really powerful development environment, it has a web-player that lets people play Unity games in a browser. That’s huge for indie games because it removes one of the biggest hurdles: convincing people to download your product and install it. Combine Unity’s network-friendly architecture and primitives with a simple server like SmartFoxServer, and maybe you’ve got something workable after all.

Unity 2.5: Not Good Enough

Sandra and I played with Unity 2.5 back in April of last year. We were very impressed with what we could accomplish in just a single week, and we were totally jazzed about going forward with it. But it defeated us: Unity 2.5 wasn’t really usable for MMO development.

Although Unity said that Cartoon Network’s Fusion Fall MMO was written with Unity, that was a lie of omission. Sure, Fusion Fall used Unity. They just used a special version with all kinds of cool powers you couldn’t replicate in Unity 2.5. We were quite frustrated when we couldn’t do the same tricks that Fusion Fall did. For instance, we couldn’t really stream character assets the way Cartoon Network’s game managed it: the features just weren’t there. That meant we would have to be really picky with our assets, because the options for streaming were very restrictive. That was pretty depressing.

But what really killed Unity 2.5 for us was the crashes: Since Unity 2.5 didn’t support typical version control products, we had a hell of a time managing our work between two programmers. (The entire project was stored in proprietary binary files.) And sometimes when it crashed, those binary files got corrupt, and you lost all your work since the last manual backup. So we hung up the Unity project and went back to doing actual lucrative work.

Unity 2.6: Huge Improvement

Well, a couple months ago we got that itch again. Unity 2.6 has been out for a while, so we dove in — and it was a huge improvement. In fact, yesterday we bought the first of at least two Unity Pro licenses. At $1200 each, that’s not a snap decision, but we believe that this time our MMO can really happen.

We’re happy with the product, and we’re amazed at what we’ve been able to accomplish in about two hundred hours of development time: we have an entire working proto-MMO, complete with melee combat, projectiles, inventory management, skill systems, shops, harvesting, and more. By letting us spend our time on these game systems instead of the nuts and bolts of 3D displays and asset streaming and so on, we have been able to do a ton of game systems very quickly. We’ve got an indie art team making art for us and things are happening at a rapid pace, given that it’s a side-project done in our spare time.

However, working with Unity 2.6 has a down side, and that down side is that Unity is only magical when it’s magical. But first, the good stuff:

It’s got everything Unity 2.5 should have had.

Unity 2.6 inherited a bunch of nice features that were originally written for the Cartoon Network folks. You can now stream any kind of asset, and there are a lot of powerful mechanics for managing streamed data. This is a lot more like it — making a tiny lightweight game is actually possible now!

Plus, Unity 2.6 has support for actual version control systems (instead of just their overpriced proprietary solution). When enabled, Unity creates a binary data file that stores the “compiled” data for each original asset (be it a source-code file, a JPG, an animation, whatever). So to version things, you put both the original asset and the binary Unity file into your SVN repository. Ta da!

Perhaps most importantly, it’s a lot less crashtastic. Well, it still crashes plenty (mostly when your code dereferences a null pointer in a fragile state such as during a screen refresh) … but it doesn’t crash completely randomly. And when it crashes, it almost never corrupts your assets anymore. And if it does corrupt your assets, you have several ways to fix it: you can reimport the single damaged asset or you can reload stuff from your SVN repository, for instance. You aren’t just stuck with a corrupt project forever.

As an added bonus, Unity 2.6 has a built-in animation editor! (You would be forgiven for thinking that came with Unity 2.5, as I initially did. It was in earlier versions, but they actually took it out for Unity 2.5, and put it back in for Unity 2.6.) While you could theoretically animate anything in this animation-editor, it’s really designed for background animations, special effects, and the like. For instance, the fish in our lake swim about on an animated path. Certain special effects cause creatures to pulse a glowing red color, which is also done as a Unity animation. It’s a great tool for little snazzy tricks. It’s a bitch to figure out how to use it, but it’s still a million times easier to understand than 3DS Max. So that’s good.

So Unity 2.6 is a perfect tool of development wonderment, right? Well, once you get over the learning curve, yes, for the most part, it is. But outside of the product itself, things get scary.

Uh … Support?

When I was first using and learning Unity last year, the forums were abuzz with feedback from the Unity team. Unity employees actually helped people  figure out problems and worked with us. That seems to have stopped.  There’s one or two posts a week where QA will pop in and say, “Is that still happening? Send me a proper bug report,” but that’s it. For a young product, support is essential. Bug fixes are essential. There should be patches to the development platform every few months, not one per year. What the hell is going on over there?

In fact, I was pretty annoyed when they made Unity Indie a free product. I wasn’t upset because I’d already paid my $200 for an Indie license — I was upset because of the subtext: when you make your product free, your support quality goes way down. When the barrier to entry was $200, there weren’t very many 13 year olds trying it out on a whim. Now that the introductory product is free, though, there are so many newbies using the product — and let’s be quite frank, most of them are complete newbies to programming — how could Unity possibly keep up with their friendly forum support?

The answer, of course, is that they haven’t. They created Unity Answers where developers can help other developers. This works great for easy questions (newb questions and the like). It doesn’t work at all if you are having an unusual experience or trying to do something esoteric. Best case is that you will get an answer that restates the documentation, sometimes slightly more clearly. Normal case is that tough questions are ignored. Worst case is that some Internet Super Hero will ride in and tell you how you’re doing it wrong anyway (at a philosophical level that ignores the actual problem, such as “Unity isn’t ideal for manipulating textures; you should have the artists do this in Maya”), and then ride off into the sunset.

And speaking of Internet Super Heroes… there’s a lot of them now. While there are still a good number of really smart developers on the Unity forums, and I still search there when I have a problem, the fanboys are also a lot more prevalent now. For example: I found a fairly annoying bug in Unity 2.6′s streaming loader, but when I asked for a workaround or confirmation that it was a bug, a fanboy told me that I just needed a better webserver. (A better web server to keep my client from crashing?) I pointed out that Flash player’s streaming loader doesn’t have trouble on the same server (when streaming much larger files), so that ruled out my webserver as being the primary culprit. To this, I was told that I was clearly wrong, or I didn’t know how Flash worked, and there are no Flash games anywhere near as big as Unity games, and anyway Flash doesn’t cache files (!), and so on.

Basically if people bothered to talk to me at all, it wasn’t because they had advice to offer. It was because they felt they needed to stick up for Unity since the employees can’t manage to even read all the threads on the forum, let alone respond usefully to them all. Gee thanks, guys.

The kicker is that I eventually found a couple of blog posts in a corner of the internet discussing the bug, where Unity’s developers acknowledged that it was a tough issue because the fix was very browser- and OS-dependent, and that they are still trying to find a clean fix for it. That’s all I wanted to know — that, and maybe a time estimate. I understand that it’s a hard problem, and I wasn’t demanding a hot-fix that afternoon. The last thing I needed, though, after hours of banging my head on a problem, was to be considered a troll.

I’m not a troll. I’m a developer who hates spending hours and hours of his weekend researching undocumented functions because the docs are shit and the forums are unsupported. The docs are adequate for easy stuff (now), but once you leave the friendly happy land covered by the demos, the docs become more and more skeletal. You just try making a complex  graphical Editor plug-in, or doing fancy things with the “headless” command-line client, or adding GUI features like focus-management to your game. (These all require using undocumented functions and techniques. At least in the latter case, the forums have a few clever people who found the undocumented functions and showed how to use them.)

For these sorts of things, I would be happy to pay a nominal fee for support. Let me give you a few hundred bucks for you to answer my damned questions, PLEASE.

Unity for iPhone: Completely Dead or Just Mostly Dead?

Unity has been focusing a ton of their effort on iPhone support. It gets more updates than the main Unity product does, and obviously a lot of Unity’s attention is on it. The reason is obvious: developers are paying for Unity iPhone licenses.

Well … the writing is on the wall for Unity for iPhone. The next big iPhone upgrade explicitly disallows most kinds of third-party development tools, and explicitly states that:

Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

Wow. Bye-bye Unity for iPhone. The forum users are hoping this somehow doesn’t apply to them, but come on. It’s pretty darn clear. It says iPhone programs have to be originally written for the low-level Apple APIs. No matter how you phrase it, Unity apps are not written “originally” for WebKit or Objective-C. They are written in Unity and transliterated into Objective-C. This is clearly no longer allowed.

Steve Jobs weighed in on why he did this: it was to kill Adobe’s new Flash-to-Objective-C compiler product. Now, many of his arguments are bull (such as Flash not having multitouch support, or all that talk about how important “rollover” support is). But I saw Flash running on an iPhone at a conference a couple months ago, and it was really, really slow. I can see him wanting to ban products running this poorly. Unfortunately, this also quite clearly covers Unity, too. Steve goes on to say:

We know from painful experience that letting a third party layer of software come between the platform and the developer ultimately results in sub-standard apps … we cannot be at the mercy of a third party deciding if and when they will make our enhancements available to our developers.

This becomes even worse if the third party is supplying a cross platform development tool. The third party may not adopt enhancements from one platform unless they are available on all of their supported platforms. Hence developers only have access to the lowest common denominator set of features. Again, we cannot accept an outcome where developers are blocked from using our innovations and enhancements because they are not available on our competitor’s platforms.

Flash is a cross platform development tool. It is not Adobe’s goal to help developers write the best iPhone, iPod and iPad apps. It is their goal to help developers write cross platform apps. And Adobe has been painfully slow to adopt enhancements to Apple’s platforms. For example, although Mac OS X has been shipping for almost 10 years now, Adobe just adopted it fully (Cocoa) two weeks ago when they shipped CS5. Adobe was the last major third party developer to fully adopt Mac OS X.

This applies to Unity too. Third-party layer? Check. Cross-platform development tool? Check. Now it’s true that Unity apps perform a whole lot better on the iPhone than Flash does. But that doesn’t seem to matter here; Jobs’ same arguments clearly apply, in principle, to Unity. If Unity finds a technical way to weasel around this, you can be sure Adobe can do the same. And if Apple just selectively bans only Flash but not Unity that would seem to open them up to lawsuits.

Unity’s CEO has posted about this on the Unity blog, saying, in a nutshell, that Apple hasn’t explicitly told them personally that they are screwed, so they are optimistic. That sounds crazy to me, but what the heck do I know. Maybe they will be fine. Maybe they have a crazy hail-Mary plan. Maybe the Department of Justice will save the day. I just know that if I were making iPhone apps, I would be exploring new platforms right about now.

I don’t mean to say they’ve been ignoring non-iPhone development. They seem to be working hard on a new major revision, Unity 3. But if Unity’s iPhone support is killed, there is a silver lining: 1) I don’t want to make iPhone apps anyway, and 2) maybe they will start working on the web player more aggressively again. Partnering with somebody to increase the installed browser user-base would be great.

Unity 3

Very little is known about Unity 3 — which is terrifying since it ships in a few months. We know if you buy a 2.6 license now, you get a free upgrade to Unity 3. That sounds awesome! But in practice it seems to mean that they have stopped bug-fixing Unity 2.6, so we’re probably going to need to upgrade. Will that be hard? No idea. Is the API mostly backwards-compatible? No clue.

They showed off some cool GUI features on the blog. The features are cool. But what I want to know is: will I have to rewrite big chunks of code (such as my GUI code, rife with those damned undocumented-function calls)? Or will I just need to spend a day or two in code-upgrade hell? This is a pretty important question for me. If you’re worried about it, you should probably wait to buy until Unity 3 comes out. Although if you do, you will lose out on the pre-order discount pricing …

So do I hate it or love it?

If I come across here as schizophrenic in my opinion of Unity, it’s because I am. First, I love it. It is the only way a couple of developers like Sandra and I can realistically make a full 3D MMORPG on a part-time schedule. There is no other product that can realistically do this given our limited time constraints. And I am pretty confident that the performance and stability can reach satisfactory levels.

On the other hand, the lack of good docs are a severe hindrance, their sluggish bug-fixing pace is irritating, and their lack of web support presence can be infuriating. Add in my other fears: that Unity 3 will be a hellish upgrade or that Unity the company may actually go out of business if the iPhone product dies. So maybe it’s not such a good time for you to be following in our footsteps and making your own Unity MMO.

And anyway, we really don’t need the competition …

Elder Game is sponsored by:

As far as the Champions CSR team was concerned, the game had been doing really well in beta. Sure, they were collecting 1200 bug reports a day but that’s pretty normal. In fact, it suggests a very efficient CSR pipeline if they actually managed to process all those reports. But once the game launched, the number of complaints about bad behavior skyrocketed.

It turns out that — get this — juvenile males are attracted to the superhero MMO genre in large number. They cuss. They insult. They cuss with insults whenever possible. This bothered the CSR leads a lot. Perhaps they were new to the MMO business… let’s suppose the CSR leads came from running a phone bank for a major publisher. If 100+ people care enough to call in about an issue with an EA game, that gets their attention really, really fast. Or worse yet, perhaps the CSR leads came from running a call center for a non-game product. 100+ calls about the same topic would cause an overnight hotfix in most software houses.

So they raised the alarm. “Our players,” they said, “are making other players uncomfortable with language. We need to fix it.”

Now any MMO producer can tell you that you don’t piss off the CSR leads. Aside from the forums, which are extremely unreliable for data gathering, CSR is the only way you hear what’s really broken in your game. A good CSR team is way better at bug-hunting than the QA team. That’s because they hear every bug every player finds, instead of just the bugs a handful of testers find. You piss these guys off and suddenly you stop hearing what’s going on in the game. Your ears go deaf and you start losing players without being able to tell why. This is not a mistake a good producer makes more than once.

So maybe the producer jumped right on this. If he’s done this before, he probably rolled his eyes when he got the request, because he knew it wouldn’t help much, but he prioritized it anyway. He sent it to the engineering lead. “Make the CSR team happy. Add a profanity filter to the game. Make this a high priority.” Now remember, the engineering lead has TENS of THOUSANDS of bugs to triage, plus new feature requests coming in every day. So he no doubt rolled his eyes at this suddenly-important tech request.

He then handed the request to his fastest engineer. “Marty, I’m going to take you off the rewrite of the generators for half a day. You’ve got four hours to get a profanity filter in the game.” The CSR team helpfully created a massive list of cuss words (in many languages), drug references, references to sex acts, and hundreds of other things that might be even slightly icky.

Marty’s a smart engineer, and he works really fast. He gets the profanity filter working in two hours. Hell, there’s time left over to polish this feature before he breaks for lunch! So Marty adds some heuristics to detect slight cussing deviations. His algorithm assumes that if a word is one character away from being a cussword, it’s actually a cussword in disguise. He submits his fix for QA approval by lunchtime, and heads off to get his caffeine and grease fix.

Now, the QA team is almost useless. Two reasons: the experienced QA folks got shipped off to Star Trek Online the week after Champions launched. None of the remaining guys knows how to make a test plan. The best they can do is load up the new feature and poke at it for a few minutes. Worse still, these guys are demoralized, overworked, and busy looking for their next job. They may not be good QA (yet) but they aren’t stupid. They know who gets fired first when an MMO game doesn’t have a spectacular launch. Yeah, that’d be QA. Every damned time. So these guys are hardly motivated, barely trained, and tired as all hell from the double overtime they’ve been forced to work.

Jim in QA gets tasked with testing the new profanity filter. He doesn’t know how to make a test plan, and Marty the engineer didn’t have time to make one for him, so Jim just loads up the game and starts typing in cuss words. It seems to work. But he knows that he needs to file a bug on the feature. When QA is in trouble, they always file a token bug on each feature if they can. That way they look like they’re working. (In fact, you can tell if a QA department is borked by how many token bugs they submit versus the number of serious issues they find.) It takes a while but Jim finds something to complain about.

The bug gets sent back to Marty the next day. “You missed cuntsuckler” says the bug report. “I say that all time so it should be in the list.” Marty the engineer adds it to the list, grumbling all the while about QA wasting his time, and then the feature is certified as done! It goes live two days later.

The *!&^%(* Feature Is Live

Cut to me playing the game. There wasn’t any mention of a cuss filter in the patch notes, so I was quite surprised when I saw a random NPC run by, saying:

Clearly this NPC was trying to cuss me out in Spanish. Whew, thanks profanity filter!

(I’m not named Templar, by the way… the NPC’s just spout off about whoever is nearby.)

It took a while to figure out what happened. The poor NPC tried to say “put a stop to the Qularr” but the cuss filter interpreted it as “put a“, which is just one letter off from “puta“, which is a cussword in Spanish.

So it’s not bad enough that I can’t say “put a” in Champions Online anymore… the NPCs can’t say it either!

The next day it’s even worse: an evil villain tries to tell his henchmen to “put that hero in a pine box”, a reference to murdering me. But alas, he said “hero in“, which of course is a misspelled reference to drug use. A reference to murder? That’s fine. Using drugs? Whoa there! So It comes out as “put that $#@*^!& a pine box“. What a foul mouth these NPCs have!

It’s ironic, perhaps, that cussing continues unabated. I can’t say “put a” but I can still say “p-u-t-a”. Turns out 13 year olds figured this out pretty quickly too.

What’s going on here? Why did this feature get out the door like this? It’s an embarrassment; it does more harm than help.

This is just a symptom of Champions’ painful death throes.

Death By Resource Atrophy

Even before they got their first months’ numbers back, Cryptic knew the game wasn’t going to be a big chart buster. Now they’re just hoping to break even: if they can keep enough players around to keep the bill collectors away and the lights on, then the company can try again with Star Trek Online. And hey, maybe in a year or two they can launch Champions for the XBox 360 and it will be a mild success.

This is very bad news for Champions players. Champions has been relegated to the role of red-headed stepchild… it’s that crappy failure of a game that keeps stealing resources from Star Trek Online, which is the game that’s going to save the company. (The same fate befell Turbine’s Dungeons and Dragons Online when it became obvious the game wasn’t going to recover its development costs quickly… its team was rapidly stripped of resources so that Lord of the Rings Online could be a success instead.)

So now Champions has a tiny, bedraggled, demoralized team. If they’re lucky, the team has taken on that grim, “we’ll die trying” mentality that desperate live teams get. I think some of the team has that now. A few people are clearly trying hard. But there’s no support, so every few steps forward also makes them stumble backwards a step.

I’m rooting for you, guys. Yes, I’m pointing out how you’re making tons of mistakes, but I also think you’re doing the right thing. At this point, you’re not going to get a reasonable amount of QA support, so you’re best off cramming in the fixes, new bugs be damned. If you fix 20 bugs and add 3 more, that’s still better than fixing 10 bugs and adding 0. Desperate times call for desperate work.

This is also the time when the live team’s producer must shine. If Champions is around two years from now, it’s because the producer made the right calls during these first three months of the game. My suggestion: pick your goals with QA. Insist on actual test plans (that you review) for the handful of features that are really important. Having QA working closely with the engineers is ideal but not realistic now. So have your lead engineer make test plans for important features (or, if your lead engineer writes the feature, have a different engineer make the test plan.) Don’t worry about seriously testing the small features. There’s no way you can get all your new stuff covered by QA. It sucks, but it’s the facts. Focus on a feature barrage instead. (For all I know, that’s exactly what Champions is doing now. But if so, they aren’t adding the important features nearly fast enough.)

Yes, I will laugh at your embarrassing bugs, but if you address the serious issues (like useless crafting, wildly arbitrary quest difficulties and rewards, and a cruel newbie experience for people who don’t research the right build on the forums), I’ll keep playing. I think you’ve got another month to show significant progress. After that, even folks like me, who really want to play this game, will give up and quit for a while. So show us what you’ve got! Your next month’s effort can have a major impact on your population numbers. It’s do or die time for Champions.

Elder Game is sponsored by:

You might be thinking my request sounds like one of the lazy animals from the story of the Little Red Hen:

“Who will help me code the MMO?” asked the little red hen (… I mean the MMO company).

“Not me,” said the independent MMO contractor. “I’m too busy doing fun easy things!”

“Fine, I will do it myself! Ah, but now who will help me run the MMO?”

“Ooh ooh I will!” said the contractor.

“No no, you didn’t help me make the MMO, so you don’t get to do the fun part! I will run it myself!”

If this is the story that runs through your head, you are definitely not from any of the major MMO companies. For the likes of Turbine, SOE, or Blizzard, making the MMO is the fun part. Working on the MMO afterwards is the terrible part.

You might expect that the people who spent five years making the game would be excited to run it after it ships. Turns out, not really. After five years of working on the same project, they’re so sick of it they never want to work on it again. They want it to be in good hands, certainly. And they want to have some oversight to keep people from damaging their vision of how the game should run. But they sure as hell don’t want to have to do that tedious maintenance stuff themselves. So companies tend to pull the experienced staff off of the live game pretty quickly, leaving behind junior people.

Live Teams Are Not Glamorous

I like the “tedious maintenance stuff.” I actually prefer working on the live team. This makes me very unusual in the MMO industry. I am also a pretty good engineer with a lot of experience, which means I don’t often end up on live teams — too experienced. At Turbine, I had a hard time getting onto the Asheron Call 2′s Live Team, because I was expected to help develop their next generation MMO engine instead. I wanted to work on AC2 after it ships?! None of my managers could understand why I wanted to be demoted like that!

But to people who enjoy the live team, well … there is nothing as good as it. The power you have! The instant feedback! The ability to literally make hundreds of thousands of people happy with just a few weeks of work. It’s very gratifying. There’s also the tedium and frustration and lack of resources and constant fire-fighting and oh my god I can’t keep up with everything… but that’s the price of the deal.

Of course, it doesn’t just happen that you hop onto the Live Team and suddenly you’re making game-design changes. At first there are a lot of smart and talented people at the helm, helping you learn the ropes, making the hard decisions for you, keeping you from doing stupid things. But inevitably they are pulled off to other projects, and somebody relatively junior gets the helm. That’s how I got to be in charge of balancing AC2′s classes.

Fortunately, I had a decade of engineering experience and understood how to tune complex systems. I wrote analyzers, modeled usage patterns, and made corrections.

Unfortunately, my approach did not take the “human equation” into consideration very well.

Learning to Balance the Human Equation

I found that the Feral Intendant class was 30% overpowered, and that’s why so many people were playing a Feral Intendant. Yet somehow, reducing the power of the Feral Intendant to the correct level did not suddenly make the game more fun… thousands of players were complaining and nobody was telling me they were happy about the change. Weird! I double checked my calculations. They were correct. So what had gone wrong?

Turns out that the people who played the other classes available to that race had taken on an “underdog” mentality. The people who played Claw Bearers liked that they were woefully underpowered compared to Feral Intendants. It was like playing the game on Hard Mode. And the people playing Feral Intendants liked playing on Easy Mode. In balancing the game I had failed to understand the needs of the people playing it. I just ham-handedly fixed the equations, instead of solving the problem with the finesse it needed. It was one of my more serious missteps. (And it’s a great example because I think it’s pretty obvious in hindsight. Most mistakes were much more subtle.)

But man, what a fast way to learn! After just a couple years of that, I became a good game balancer. The constant feedback loop helped me learn from my mistakes in a matter of weeks! Compare that to developers on traditional games, who must wait until the sequel ships before they get to try their hand at balance again. That’s why working on a live team is such a fast way to learn your craft: the feedback is so much faster than any other gaming platform, that it accelerates learning by dozens of times.

But AC2 cost millions of dollars to create. Turbine didn’t create it as a tool to help me hone my design skills, that’s for damned sure! How did I get to do it? Simple: the designers who would have done it were burned out of working on AC2, and were called away to work on the important New Project. AC2 wasn’t a blockbuster hit, so it didn’t make sense to use the rock star designers on it. Better to let the B team step in.

The Steady Hand Has Left The Rudder

But here’s the weird thing: WoW is exhibiting the same symptoms as AC2 did when I was doing the designing. The B team is in charge.

In February, we learned that lead designer (and part-time producer?) Jeff Kaplan had stepped away from WoW, off to work on the next big Blizzard game. However, if you were watching the game before that, it was obvious that major leadership changes had already happened months earlier. My guess is that Jeff Kaplan started moonlighting on the new project long before February. And many of the other key WoW live team people have also switched over, or are working on WoW only part-time.

Now, I am not being alarmist. The ship is still in intelligent, capable hands… but clearly not as experienced ones. Just as I did when I took over AC2, WoW is making newbie design mistakes that seem like a benefit on the surface, but are really not good decisions. There have been scores of examples… I’ll pick just a few.

“It’s always been stupid, and we just need to fix it!”

A few months back, the powers that be decided that Hunter ammunition didn’t work right. Hunters have to carry an arrow for every single shot they take, and in order to get the full benefits from them, they have to carry them in a special quiver — which doesn’t let you store other items in it, only ammo. All that ammo costs money, too.  Plus, it leaves the designers unable to give out awesome “raid arrows” because you’d just shoot them all and then where would you be? Even though ammo had been a fine and fun distinguishing quirk of Hunters for years, it was time to Fix It.

The first plan was announced: WoW would no longer have consumable ammo. Instead, you would just need a single “infinite arrow” that you stuck in your ammo slot, and this would let you shoot your bow forever. Problem solved! No more quivers, no more pack space wasted, no more costs. And now raids could drop “loot arrows” that wouldn’t get used up! Perfect!

Whoops, turns out that plan would be hard. So they announced their backup plan: now ammo just stacks to very high numbers. Instead of having stacks of 200, now you can have stacks of 1000. This at least addresses the “pack space” issue. Call it a win! And they removed the magical benefits from quivers, so you no longer needed to use them. So they fixed the immediate emergency, and they’ll get to the “correct fix” later.

The thing is, there was no emergency. Sure, Hunters were happy to have a few extra pack slots. But the change threw all sorts of other things out of whack: magic quivers are still given out as quest rewards… they just aren’t magical anymore. And leathercrafters can still make them! They just can’t sell them to any sane Hunter. And so on… the game wasn’t really cleaned up after this change.

But I’m sure it felt so pressing, so urgent. So they had to address the issue, side-effects be damned.

Without somebody experienced at the helm, the voice of the myopic designer tends to be the loudest. “WE HAVE TO FIX THE HUNTER” they said. Maybe they said, “Hunters have to spend 65% more on bare essentials than any other class. I will never be able to balance class expenditures like this!” Or maybe they said, “Hunters have to waste more inventory slots than any other class. It damages quest completion rates!” Or maybe they just said, “It’s SO STUPID. It’s always been stupid, and we just need to fix it! Do it now!” Obviously, nobody thought very hard about the ramifications, and nobody spent any time easing players into the idea. And nobody stopped to make sure they did a good job.

So some tiny little mistakes crept into the game. Nothing huge. Nothing that will sink the Titanic. But mistakes nonetheless… “magical” crafted quivers that aren’t magical and can’t be sold are clearly a mistake. These little bugs accumulate, like lint on a hardwood floor.

The Lint Accumulates

When we say that WoW is “polished”, what we mean is that it is surprisingly clean of linty little bugs like these. But that’s changing.

More and more little mistakes have crept into the game recently — changes that are positive on the surface, but have not been implemented with the finesse that makes them worthwhile. Mana expenditure rates have changed, rules for dungeons have been tweaked, the cost of items has fluctuated. It all seems useful. But it’s usually full of little side effects. Worse, it doesn’t take the human equation into account: it doesn’t counter-balance for the actual needs of the players very well. There are ways to meet both goals, but you have to try a lot harder at it than WoW is.

Remember when WoW class balance happened every six to eight months? Players were actually excited when their classes’ turn came around. I remember being so astonished to see players that were actually happy to have their classes redesigned. But now, every class is fiddled with every few weeks. It’s not exciting anymore. Instead of sitting on the changes and carefully honing them, the designers are just firing out every new idea they have, willy nilly, until they get it right. But here’s the thing: it doesn’t matter if you get it right. It matters if players are excited and having fun. Balance changes are happening too fast, and for too little benefit overall.

Back in the day, QA held the game to a higher standard. Consider that there never used to be skill changes that would invalidate the client tooltips about a skill (unless it was an emergency exploit-fix). If the designer wanted to tweak a skill, they had to wait until the client could be updated. But the QA bigwigs are off doing something else now, so it’s easy for the designers to slip this stuff in. And they do. All the time. Skills are routinely incorrectly displayed now, as the designers’ need for perfect balance far outpaces the ability to do client updates.

Who’s In Charge Again?

You would never let your lead artist drive decisions for your game. Chances are, they would say “This dungeon is too brightly lit! We need to hotfix it now or the mood will be ruined forever!”

But unlike artists, designers get a free ride. They’re supposed to know what’s best for the game. If the producers are busy, they trust the designers will do good things. But designers, especially young ones, get myopic. They tune into little issues — like perfect class balance – and turn them into epic quests. If the designer could just fix this balance problem, people on the boards would stop complaining, and the game would be perfect!

No. It will not happen. Perfection will not be achieved, ever. But there’s nobody around to rein them in anymore, so they try and try and try. And leave little messes everywhere they go.

Suddenly Communications Are Open

Another surefire way to tell that upper management has left the building? The systems designer “Ghostcrawler” has suddenly started posting a lot, even about… well, nothing. For years developers were nowhere to be seen, which was a shame. And then suddenly the lead systems designer has time to play the forum game? Yeah, whoever was making employee policies just doesn’t have time for WoW anymore. Not a bad thing, in this case, but certainly a dramatic shift of policies.

Nowadays it’s common for WoW to tell people to “check the forums for game updates.” This is a total newb mistake. Only your loudest and most annoying users will check your forums for updates. So every “update” is met with derision because only assholes post on game forums. (Statistically speaking, anyway.) Game updates are specifically what the launcher’s update screen is for. If you’re outpacing the ability to update the update screen, chances are you’re changing too much too fast. Slow down and get it right the first time.

It’s likely that Ghostcrawler started posting shortly after the upper management started wandering off to other projects. Ghostcrawler’s a good guy… in fact, his posts remind me a lot of what I sounded like when I was posting about AC2′s skill balance. He knows how to balance things. But he is completely unable to see the big picture. Every tiny imperfection seems like a ruinous problem. He feels assaulted on all sides by problems, too, and doesn’t think there’s time to do things the right way. But this is an illusion that happens to Live Teams because they get so close to the product. He needs someone checking over his decisions and making sure they’re worthwhile. He doesn’t have that.

WoW: No Longer Big Kahuna at Blizzard

Ghostcrawler and the rest of the team will learn their craft soon enough. WoW will survive the experience. But what’s interesting is that it tells us quite clearly that WoW is no longer the most important thing at Blizzard… in fact, it might be third or fourth place. It’s really interesting that this happened so soon. I didn’t expect it to happen to WoW while it still had 10+ million players or more still paying. But a company has only so many top-notch people, and you always want your most-experienced people on the new thing, so it makes sense.

To be clear, this isn’t necessarily a bad thing. When the game stops being in the spotlight, the live team suddenly gets a lot more flexibility to make the game fun, instead of being forced to stick to now-outdated “design visions”. The dramatic increase in WoW’s mobility options is certainly due to the lack of oversight. But without that safety-net of supervision, they need to exercise a lot of willpower and a lot of wisdom.

Ghostcrawler, and anybody else on the design team of WoW right now, I have a little unsolicited advice from somebody who’s been there: convince your bosses to let you play a different MMO for two weeks. On the clock. Don’t touch WoW. I know it feels like there’s a disaster every day and you can’t possibly stop focusing on WoW, but you can. After you get back, play WoW with a different class than you normally play. You’ll see so many new things! Your priorities will do a 180. I guarantee you it will help your perception.

Elder Game is sponsored by:

Sandra and I worked in the mainstream MMO industry for a long time, but a couple years ago, we stopped. We stopped because we better understood what we wanted: neither of us was real happy making MMOs. What we wanted most of all is to run MMOs. Unfortunately for us, running an MMO tends to require you to make one first. This is tricky, because the traditional AAA MMO takes three or four years and 50 people, and has a 50% chance of success at best. These are not odds we like.

So we’ve done other things — consulted on MMOs, web games, and all sorts of other things, and it’s actually been a lot of fun. But in the back of the mind there is still The Calling. So we tried to make our own indie MMO on the cheap. These were 2D or text-based, and we just couldn’t get into them. We needed our MMO to be 3D. We’re spoiled like that. And for practical reasons, we needed it to be web-based, because we can’t imagine being able to get a boxed product on the shelves.

But making an MMO is a huge undertaking. It’s not just a fancy 3D client; it’s also a scalable server, tools to develop and maintain it, and infrastructure to run it. But Sandra and I are experienced server engineers; we believe we can use off-the-shelf tools and some cleverness to make a reasonable little game server. And we are becoming more comfortable with various infrastructure approaches. But how do we get a 3D game on top of it? One that’s web based, too, and one with powerful development tools already made for us?

We tried various client applications, but they sucked. However, there’s a new contender.

Enter Unity 2.5

I’ve been watching Unity for a year or so now. It’s been frustrating to watch, because the numbskull developers created their first versions only for the Mac! (The resulting applications could run on any platform, but the development tools required a Mac.) When you’re an indie, it’s hard to justify doling out a few grand for a Mac in order to test-drive a piece of software you’ve never used before. This restriction didn’t stop Cartoon Network’s Fusion Fall from using the Mac-based version of Unity, but it kept most small developers, including us, on the sidelines.

However, two weeks ago they finally got around to making an accessible version of their program, one that runs on Windows or Macs. Finally! Sandra and I reorganized our schedules so that we would have a full week to experiment with Unity and a simple off-the-shelf server product called SmartFoxServer. Basically, we spent a week prototyping an MMO. Successfully.

What makes Unity special? Three things, in order of importance:

1. An enviably powerful tools pipeline, 
2. A rendering engine that works on any platform (and can run on web pages), 
3. And a very reasonable price tag. 

Let’s go over each one.

1. The Development Pipeline

“Development pipeline?” you may be thinking. “Who cares! How many polygons can it push? How many draw calls does it take to render things? Where are the technical stats?!” That’s basically irrelevant for us. We can design our game to run well under whatever conditions the engine allows. This is fortunate, because tech-wise, the engine just doesn’t seem that amazing. If you’ve played Fusion Fall, you may have noticed the low framerate for relatively simple scenes. It’s just something that has to be worked around.

There are lots of free or cheap 3D engines out there, and many of them are far more powerful than Unity in terms of rendering. But those were completely useless to us because they had no tools pipeline. A real MMO needs a client program, sure, but it also needs dozens of man-years worth of tools to build the content for the client. Indies don’t have the resources for that.

This is where Unity 2.5 shines. The Unity development environment integrates directly with Maya, Max, or various other 3D clients, plus code editors, sounds, and Photoshop files, to make a really compelling development environment. Import your 3D characters and drop them right into the scene, then start scripting them to respond to animations. Create terrain in Maya or directly from within Unity. Configure the built-in physics engine, position lights in real time, and then run everything together, watch it work, and fiddle with things on the fly. This is a great way to prototype stuff. It’s fast, it’s efficient, it’s … pretty alien to most programmers. If you’ve learned to develop in Flash, it’s sort of that mindset: it’s more resource oriented than scripting oriented.

Placing an asset on some terrain I just made

This can take some time to get used to, but it’s plenty powerful and elegant when you do master it, and it’s sufficiently versatile that you can use it for a whole lot of games.

On the other hand, this complex development environment makes it harder for programmers to manage lots of code. For instance, scripts are attached to assets and then the script’s variables are individually configured. This is done automagically and makes for a very cool customization experience. But if you decide that you need to change the values of a variable, you may not be able to find all the uses of that variable with a text-editor search: the user may have overridden those variables in the project itself, leaving you with no way to find the values programmatically.

It also presents some co-authoring issues: you’re all working on the same assets, after all. Unity did a decent job of letting you merge projects together, but that’s only if each developer is working on completely separate parts of the client. If you’re each fiddling with the same prefabricated object, you’re screwed. You can’t merge the binary assets: somebody’s work is going to get lost.

So this pipeline is ideal for small teams, or for larger teams who have spent some serious planning time figuring out how each person is going to avoid stepping on other people’s toes.

But let me just quantify this toolset’s value: Sandra and I were able to download the demo version, learn how to use it, and then create a 3D zone with mobile, animated avatars that talked, punched things, exploded, lit on fire, and so on — in a week. We also had to learn how to use the server library we picked, too. Fortunately for us, SmartFoxServer actually comes with a demo that shows you how to synch up multiple Unity clients. We achieved pretty amazing results in a week, but we took advantage of a lot of demos and free assets to make it happen.

Still… this is an unsurpassed prototyping tool. Even if you don’t use it for the final client, just imagine that you could get your next prototype up and running in a week, then iterate on the design every day after that. Now you can. I wish we’d had this when we were trying to prototype Star Trek’s space combat.

2. Web Based 3D Out of the Box

Another important advantage is its flexible runtime environment. It runs on Macs and Windows. It can be a stand-alone program or embedded in a browser. And it isn’t hampered by the “you must support the lowest common denominator” mentality that Flash has. For instance, your game can support multi-button mice, even though Macs may not have them. Conversely, you can program for that weird meta-key (the Command key, I guess?) even though its analog on PCs is the Windows key — and when you press the Windows key in a web page, the web page loses focus. But I’m very happy that they just gave us all the obvious capabilities and left us to figure out how to sanely use them, rather than oversimplifying.

The compiled files are nice and small, for what they are. I was able to get a pretty complex scene, complete with lots of scripts, animations, and networking, into an 8mb file. (Of course, users also have to download and install the Unity plug in for their browser; that’s where the “engine” code lives.)

It also has some complex tools for data streaming, which we didn’t get around to testing out yet, but they seem pretty robust. They also require a lot of planning, but that’s still a whole lot easier than coding it ourselves from scratch.

3. Cheap Price Tag

The price is very reasonable. It’s a couple grand for Sandra and I to each get professional licenses. That’s it; no percentage cut or anything scandalous like that. You even get free minor version upgrades added in (which is good, because that’s the only way they do bug fixes).

It’s not dirt cheap, but let’s be realistic: 3D games are still expensive. Sandra and I made an off-the-cuff budget that cost $40k for 3d artwork. That’s peanuts compared to a mainstream MMO, but puts it well out of the reach of the very smallest of indies. If you can’t afford a couple grand for an engine, you can’t afford to make a 3D game just yet. Maybe in five more years it’ll be at the cheapness level that 2D games are… but it’s just not there yet.

(There is also a cheap “indie” license that costs $200. This is a good way to get started with development, but the restrictions mean it’s not too practical for developing a complete commercial MMO. It should work okay for other 3D games though.)

What’s the Down Side?

So the good news is that this is a realistic way for a small team to cheaply make an MMO. Fusion Fall already exists: it proves that it’s possible. But Unity is not without it’s painful side. Once again there are three main issues:

1. Bugs
2. Language Issues
3. Documentation Flaws

1. Bugs

The primary down side is that Unity 2.5 crashes a whole lot. It’s essentially version 1.0 of the Windows line of Unity, and it shows this in its lack of stability. As the military would say, its “mean time between failure” is about one hour. This is not good. You’ll have to get used to saving every few minutes. But worse still is that two of our crashes caused the project to become corrupted. Maybe if we’d been more advanced with Unity we could have repaired and moved on, but as newbies, this was devastating. We lost many hours of work when this happened. We eventually instigated a “back up to a new folder every few hours” policy.

Obviously this needs to get fixed. Unity 2.5 has only been out a few weeks, so I am reasonably hopeful that they won’t leave us hanging for too long.

2. Language Issues

Unity has a schitzophrenic relationship with programming languages. Officially, it supports three languages: C#, JavaScript, and a variety of Python called “Boo”. But this is basically a lie.

It supports C# because it’s written in C#. This is the language you should probably use if you’re a team of experienced developers. However, none of the examples show how to use the code in C#. You will have to muddle with it for many hours to get the nuances.

It supports “JavaScript”, and this is the preferred language. The demos are all in JavaScript, and the code examples are in it, too. However, this isn’t really JavaScript. It’s an upgraded version that takes a bunch of ECMAScript features that aren’t in JavaScript. Then it tosses in some special functionality specifically for Unity. And then… it doesn’t document any of it. There is no language reference for their made-up version of JavaScript.

The support for “boo” is entirely mythical. I’ve seen no code for it ever, nobody on their forums uses it, and it goes without saying that there is not a lick of reference to it in their help. You’d be really stupid to decide to use boo for your project.

More annoying still? The languages are poorly interoperable. We were pulling in code from lots of different demos, and needed to use both JavaScript and C# code in the same project. It turns out that when you have two languages in use, there are dependency issues that can only be worked out by sticking your code in special “load me first” directories. Very kludgey. At least it can be done. 

3. Documentation Flaws

The “Unity Manual” is a tiny wisp of a thing. There are no real docs on how this stuff works. What there is, is a massive step-by-step tutorial that teaches you how to make a platform game in Unity. This is awesome… if you’re the sort of person who learns by doing. I am the sort of person who prefers to absorb all the data available and then start exploring. I simply can’t do that with Unity. Those docs don’t exist. For a commercial product they are significantly under-documented.

Expect to spend days just screwing around with the demos in order to have any clue what’s going on. Expect to search frantically through their forums in the hopes of understanding the syntax for their scripting languages and complex GUIs. Expect a few sudden jarring inconsistencies in what is otherwise a smooth and orthogonal interface.

The docs looked especially paltry when compared to SmartFoxServer’s luxurious documentation. Yes, SmartFoxServer is a much simpler piece of software than Unity. I don’t care though. Fickle that way. Need docs.

The Bottom Line

If the question is “can Unity be a viable MMO client?”, then it’s been answered by Fusion Fall: “yes”.  But the neat thing about Unity is that after spending a week with it, you would easily come to that conclusion on your own.

It’s not for everyone, of course. If you can’t deal with the relatively paltry graphics level allowed, or if you need your tools to conform to your existing pipeline, then you’re not going to like Unity. You have to be agile enough to work with it instead of against it.

But after a week of using it, I’d have to say that Unity feels pretty good. Maybe this program is the missing piece in our indie MMO plans.

Elder Game is sponsored by:

Man I hate when QA people do their jobs without specs! It’s so irritating. When Asheron’s Call 2 launched, there were thousands of outstanding bugs in the QA database that we opted not to fix before launch. Sounds bad, doesn’t it? But if you looked at them, you’d understand. Hundreds of them were bugs about how buildings were floating 2 virtual inches off the ground; if you zoomed the camera down to the floor and looked up, you could see that these structures were very slightly hovering.

Hundreds more were about items that “popped in” too soon or too late; the “art degrades” for the items weren’t set up right, so they seemed jumpy. And so on… thousands of little tiny nits.

Why does that piss me off? Surely those bugs should have been in the bug database, right? Even if they’re not fixed immediately, they’ll get fixed at some point! Right, true. Except for this:

Asheron’s Call 2 launched with over 3000 severe yet unrecognized bugs. It is not unreasonable to argue that AC2′s early failings were due to the lack of quality in things like crafting, combat, and skills. Every time the QA people entered a bug about a floating object, that was time they weren’t spending finding more serious bugs. For instance:

• Many of the crafting recipes were about ten times harder to create than intended. The creatures that dropped the needed parts spawned incredibly rarely, due to an oversight.
• Many of the quests could easily become broken if the steps of the quest were done in the wrong order. Fixing them then required the assistance of a customer service representative.
• Almost every skill in the game was broken in some way. Some skills literally did nothing; others were too costly or too powerful; some started out super strong and then got weaker as you leveled up the skill.

And so on. Serious bugs, very much worth fixing. These didn’t make it into the database at all; the live team stumbled upon them when players started screaming about them. What the hell, QA? What the hell?

I’m just being a jerk to QA here. One of the reasons that QA did such a terrible job on Asheron’s Call 2 was because there were almost no design specs.

It was the extremely talented Jesse Kurlancheek (a.k.a. “Devilmouse”) who was responsible for the skills in AC2. Because of the agreement Turbine had made with their publisher Microsoft, he was obligated to create at least 600 skills for the game before it shipped, broken into 30 different skill trees. Problem was, he only had three months to design and implement them all. Jesse famously told his boss, “I can either implement the skills, or I can document the skills, but there’s no time to do both.” And he was right. But they chose to implement instead of documenting, and the result was tragedy.

Without any idea of how skills were supposed to work, QA never looked at any of them. To be fair, QA should have at least poked around with them some, but they just weren’t motivated to spend any time on them because the designer would almost always say “no, that’s not a bug, that’s how it’s supposed to work!” So the QA team spent their time wandering around looking for objects that were hovering a tiny bit off the ground, instead. At least those were irrefutable bugs!

The lack of specs was devastating. When the game launched, the live team needed to get the game into a maintainable state. Without specs, we had no idea what Jesse had intended. Worse yet, a few months later, not even Jesse remembered what the intention of each skill was. So we created elaborate analysis software in order to locate skill deviations, and slowly reverse-engineered the intentions, like archaeologists exploring an ancient culture. The lack of specs cost us thousands of man-hours. It made us look like total dicks, too, when we fixed the outrageous bugs we found.

The “reap” abilities in AC2 stole the health of an enemy and gave it to you. However, as is typical of this sort of power, you couldn’t steal more life than you were missing. So if you were only missing 100 health, you could never steal more than 100 from the enemy. Tragically, there was an accidental minus-sign in the implementation. If you reaped somebody and you were fully healthy, instead of doing no damage, you did double damage. Reaps were the most effective attacks in the game, provided you hadn’t been hurt too much. It made PvP in particular a nightmare, and players were howling for us to fix it. When we fixed it, though, several classes became unplayable; it turns out they had only been fun at all because of that broken skill. So then we had to buff those in various ways, until the next major bug was discovered, which threw our rebalancing out of whack again. And on and on, over and over, a constant dance of chaos and confusion, for over a year, before we had created our own specs and could work towards stability.

If only the skills had been affected this way, we could have dealt. But the lack of specs permeated almost every aspect of the game. The quest areas were about half-documented; the monster spec was nothing but one extremely complex spreadsheet. It’s not like the AC2 designers didn’t know how to make specs, of course: they were given impossibly small time windows to do their work, and they did their best.

If our publisher had allowed us, it would have been so much better for AC2 to have launched with only half the classes, but with docs for all of the classes. Then the live team could just implement the docs, rolling out new things every couple months, and make Turbine look super productive instead of super incompetent.

In the end, specs save money for three reasons.

• Specs allow for collaboration on a design. If it’s all in your head, nobody can point out the flaws in your plan until you’ve implemented it already.
• An MMO that is supposed to run for 5 years needs at least a rough semblance of documentation, or the game’s maintenance will cost a ton.
• Specs are also mandatory if you want to use a QA team. If you can’t afford to write specs, just fire the damned QA team, or convert them to content designers or something. QAing without specs is an amazing waste of time and just makes everybody angry.

And just to be clear: I love working with a talented QA team. I am 100% pro-QA. But you must give them the ability to succeed. Without specs, you’ve set them up to fail.

The Problem With Rapid Content Creation Tools

When I was at Perpetual, the Gods and Heroes team had a quest-development process where designers didn’t need to write out all the details of each spec. They just opened the database and plop! they dumped their quest right into the game. And man, did they hate QA. They agressively, loudly, viciously denounced the QA department. QA was always telling them the wrong thing and never finding the real bugs. The QA at Perpetual was the scapegoat for all problems. Even the QA team lead was apologetic for how shitty they were.

But how could they be successful? The only notion of how these quests were supposed to work was in one designer’s head. Eventually, the QA team wisely stopped reporting bugs about quests. But there were still bugs in the quests.

When Perpetual’s Star Trek team was planning their content pipeline, everyone just assumed the same system would be used. This was a major point of contention for me: I wanted designers to write out every detail about the quest before they implemented it. “That’s ridiculous! It will double the time needed to implement quests! We can’t budget for that!”

Okay, fine. But if you can’t budget for specs, don’t budget for QA, either.

Elder Game is sponsored by:

I agree with Lum’s conclusion that Tabula Rasa failed because it took too long, spent too much money, and became incapable of meeting its own expectations. But let’s not beat that dead horse — I’d rather beat the dead horse that Adam brings up:

Re-reading this as I go, I remember there was another big excuse that I never gave credence to: “I’ll bury my head in real work and make *my* parts as good as I possibly can, and hope if everyone else does the same, it will All Come Together In The End”. This one wasn’t voiced so much, but is simply what people did, in some cases.

Hope is not a strategy. Whenever your attempt to avoid disaster revolves around the H-word instead of a concrete averting action, you are doomed.

In my experience this happens an awful lot. Is this you? Are you doing this right now? Hoping that by working harder you will be able to stave off the doom you see coming? Take a step back and look at what you’re working on. “This is going to be the best monster ever. It will save my shitty game.” “This new twist on crafting is going to keep people distracted from the tedious combat!” “With enough back story I can distract players from the lack of content!” Come on. That’s not a plan. It’s hoping for a miracle.

It’s easy to hope against hope that it will all work out, and it’s so hard to rock the boat, risk your job and friendships, and push to make the critical changes happen. But if you can’t or won’t do that, then you need to at least leave. I know it’s hard to leave now, you’re bailing out on your doomed comrades, you’re sealing their fate, you’re… yeah yeah. Whatever. Down deep you know your game is going to die no matter what, and yep, it’s going to hurt like hell knowing that you wasted years of your life. But what would going down with the ship achieve?

I’m not speaking from a higher position, I’ve been plenty guilty of this. And I gotta tell you, changing everybody’s opinion about something they care strongly about is not my idea of a fun time. I have never signed up to change a company, I sign up to make a game! Yet most MMO companies are rudderless ships of overworked, desperately hoping, doomed people. It really hurts me to think about some of the failures I’ve not been willing or able to avert. (This is why I haven’t signed up full time on a new MMO. I would love to work on a live game again more than anything else. But I can’t bring myself to jump back into the doom yet.)

Hope isn’t a strategy, and you and I both know when we’re hoping instead of strategizing. You’re a professional game developer, and you got to this point in life by making sacrifices. You made these decisions in order to make great games. Are you making a great game now?

You’re a brilliant person, and I expect more from you than keeping your head down and hoping for a miracle.

Elder Game is sponsored by:

Well, Scaleform does deliver what it advertises. But it’s not a no-brainer decision. Here’s why.

Do you really want to use the Flash development tools?

Scaleform supports ActionScript 2, not the more modern ActionScript 3. This means that all of Adobe’s neat Flex development tools are completely off the table. It also means you can’t use all the nice GUI widgets made for AS3. (Scaleform told me they are developing their own custom GUI library specifically for Scaleform development, but I don’t believe it’s available yet.)

So this means that you will need to use the Flash 2 IDE. If you are a traditional programmer, you will find this torturous. The Flash IDE is not for you. It is made for 2D animation artists who might know a little programming on the side. The workflow is not like anything you are used to.

Engineers will inevitably want to mod the living crap out of the Flash workflow. You can create plug-ins for Flash, but you will have to do a lot of work to get it to the point that an engineer can use it comfortably.

Can your artists actually use the Flash tools?

“Silly blogger,” you say, “the Flash IDE won’t be used by our ENGINEERS… it will be used by our ARTISTS!” Well, better start training them now, because artists who have the necessary skillset are very hard to find. Oh, you can find Flash 2 developers by the yard… but you need good ones, don’t you? You need developers who can animate stuff well, and then wire the animations together with code, and connect it all to a custom interface your engineers make for them. This significantly raises the skill bar. Now you’re not looking for a mere animator. You’re looking for a technical artist who also happens to be good at 2D animation and coding.

But it gets worse! Scaleform is optimized for GPUs, not CPUs. This has numerous ramifications: bitmaps need to be certain sizes, and not too big; scripting needs to be kept to an absolute minimum in order to not bottleneck the pipeline, so you’ll need to know clever timeline tricks to get good performance; and some common Flash effects are just not available at all (because they can’t be done efficiently on consoles’ GPUs).

So you need a really experienced 2D animator who is not afraid of code or technical details at all. The folks at Scaleform told me that the best way to look for Scaleform developers is to look for Flash Lite developers. Flash Lite is the version of Flash for cell-phones; these developers are used to working within limitations that are very similar to Scaleform’s limitations. I haven’t personally looked for Flash Lite developers, but I do know that very few Flash Lite games exist, so devs may be very hard to find.

Can you actually use Scaleform for your GUI?

Scaleform works best for console games. Civilization Revolutions is a great example of a Scaleform game. It doesn’t have a ton of resizable windows and it doesn’t have widgets with dozens of functions. It has a really flashy GUI but simple, clean interfaces. MMORPGs tend to have much more complex GUIs.

Now you can easily argue that this is a flaw in MMO design, but let’s face it: if you’re pitching an MMO today, you are probably pitching a game with resizable chat windows, dozens of floating interface panes, macro-scriptable interfaces, and elaborate drag-and-drop systems. You can do all this with Scaleform — but not for free. You’ll have to do some heavy lifting in the engineering department.

Don’t count on “free” user modding

Lots of developers see Scaleform as the secret to free user-generated content. Users can just use Flash to create new replacement SWF files for your game, and can make it do anything! Free mods! Well, yes and no.

You can definitely make that happen, but you will face three big problems. First, you still need a well-documented interface to the game systems proper — otherwise, their SWF files won’t be able to make the game do anything.

Second, you will have to relinquish a lot of control over what users can and can’t do. The modding system in World of Warcraft is carefully crafted to keep users from being able to make combat macros. You would have a very hard time pulling this off if you let users plug in their own custom SWF files.

Finally, you have the problem that Scaleform is not appropriate for typical Flash developers. The Scaleform guys said that they currently actively discourage companies from using Scaleform for user-modding. Suppose a user creates a popular mod with a 2000-pixel texture in it — your game’s perf goes to hell, and you turn around and say that Scaleform is crap. They don’t want that, and neither do you.

When to use Scaleform

Use Scaleform when:

• You are absolutely, definitely creating SKUs of your game on multiple platforms (preferably three or more).
• You have procured one or more people with just the right experience and skillset to use it.
• You have relatively simple interaction interfaces, or can live with keeping a very streamlined UI everywhere.
• You don’t intend to let users mod your GUI.

The Future Probably is Scaleform… but…

ActionScript is a really good choice for a multi-platform GUI development tool, but traditional MMO interfaces will be constantly straining at the limitations of AS2. When you can use AS3 instead, you’ll have more options. Scaleform is working on this, as well as a system that uses the CPU for more of the processing, which (among other things) will make it more practical to have user-created content in your game. But that’s not now: that’s in a year or two.

In the mean time, you need to look carefully at this choice from every angle. Don’t jump on it just because you heard some other MMO is using it. Remember, most MMOs you hear about don’t actually ship, so you can’t use them as a barometer.

I like Flash, and so does my company

I have been particularly interested in Scaleform because (along with regular contract work, which I’m still available for), I’m also the part-time CTO of an internet startup that brokers Flash games. FlashGameLicense.com is all about pairing Flash developers with people who need games. It’s been around for about 10 months now, and we’re growing very quickly — we now have about 2000 games for sale, the majority of which have never been seen anywhere on the internet yet. Flash is definitely heating up as a gaming platform.

Anyway, it’s sort of tangential to this post, but I figured this was a good time to plug my company! :)  If my dire warnings about using Scaleform in your MMO haven’t scared you off, we might be able to help you find a developer that fits your needs, so feel free to contact me. Most developers on our site are only looking for work-from-home situations, so it’s hard to tell how many bites you’ll get, but it’s probably a better bet than a blind post on Gamasutra or monster.com. (And if you find yourself needing some fun Flash mini-games for a project, definitely visit!)

So I’m confident that Flash is the future. But Flash embedded into MMOs? That’s the bleeding edge of the future. If you’re bleeding edge, go for it. Just know what you’re getting into.

Elder Game is sponsored by:

• As an industry developer, I found Mythic’s fear of running a forum to be a bad business decision. As a user who is trying to get Warhammer to install and run, I found Mythic’s lack of a forum to be anger inducing. At worst it feels like a sleazy move to distance themselves from angry players; at best it comes off as a lack of confidence in their product. Sorry guys, you can’t not have a forum when everybody else has one. You look like idiots, or like sleazebags… take your pick. Talking with folks here at the Austin Game Conference, it sounds like this is mostly Mark Jacobs’ handiwork. I don’t care that people hurt your feelings when they post, Mark. Get a damned forum, and get some good moderators.
• Speaking of Mythic, this Gamasutra article by Mythic’s Paul Barnett shows just how terrible a workplace Mythic is. Paul (the creative director for Warhammer) hates it when developers play WoW because they get ideas from it. He much prefers barely-competent sheep-like workers who will do what he says rather than experienced and outspoken team members. He thinks people who disagree with the game’s design should be publicly humiliated in front of the team (“burned at the stake”). Do you think I am exagerrating or misrepresenting Paul here? Go ahead and read the article. I’ve talked with people from Mythic, and it’s true: you don’t want to work at Mythic.
• Age of Conan made an embarrassing, newbie-developer mistake a while back when their female avatars were dramatically slower than the male avatars. This was due to the animations being longer. Chalk it up to a lack of communication between teams. But then they compounded their failure when they announced that it wouldn’t be fixed until new art was created, which would take a month to make. This is something that an engineer could have hacked a fix for during a single all-night session. The fact that they couldn’t or wouldn’t band-aid this problem tells us that Funcom’s intra-team communication skills (or perhaps their programming skills) are embarrassingly bad.
• Hmm… the rest, I’ll save for another day when I can be a bit more upbeat!

Elder Game is sponsored by:

There is a point in any multiplayer game where the world suddenly feels barren and empty. And then suddenly people start disappearing even more rapidly, and then the game boils down to the most dedicated of fans only… and then you’re stuck with just those people forever.

It’s often very sudden. Your game is just cruising along, losing a few hundred people per month, and then suddenly the number of people leaving your game goes through the roof! It doubles, triples, quadruples all of a sudden.

The first instinct is, “OH MY GOD THE LAST UPDATE RUINED THE GAME!” and then there’s panicking and freaking out. The level designers are saying “I told you we needed to add three new instances each month, they got bored and they’re quitting!” And the systems designer is thinking, “I must have done the math wrong when I redid the taunt aggro system! Where’s the error?! I have to find it so we can hotfix!” and the producer starts yelling at the marketing guys because obviously the ad campaigns are less effective this month…

Odds are everybody’s barking up the wrong tree. This is one of those places where being on a live team gives you tunnel vision — it’s nearly impossible to see the big picture when you spend your day creating the tiniest details. But odds are, you need to step back and fix a really big problem. If you’re losing critical mass, there’s only one fix. You need more players on each world, STAT. If you have an expansion coming out in a month, maybe you’re going to be okay. Otherwise, you have to bite the bullet and merge worlds.

For the record, EQ2 needs to merge worlds. I know what the ramifications are in terms of publicity. But the fact is that there are fewer and fewer players of the game every day. The expansions help certain level ranges (either the <40 range or the > 70 range) but there are virtually NO people playing the game between 40 and 70. It’s a black hole where people disappear forever, and it’s not about lack of content. It’s about lack of critical mass.

There are plenty of other games that need to merge worlds, too. But EQ2 is the one that is closest to me, since I like EQ2. I would love to be playing it nightly, but it’s too dispiriting to live in a world where there are only 20 other people within 10 levels of you. Especially given that my character class, like over half the others, is only really effective in groups.

Don’t let a game die because you can’t see the big picture. If you don’t have critical mass, nothing else matters until you fix that.

Elder Game is sponsored by:

Look at this list and tell me if you recognize anything that applies to your company. (This is not a complete list, but I think it will give you the general idea of what to look for.)

Give yourself a point if your game has finished pre-production and…

• Your company has hired a half-dozen level designers already, but the level-design tools aren’t finished being coded yet. So those level designers are ”working on paper” or making mock ups in 3D Max. And playing lots of games on the clock.
• You still don’t have a playable demo version of your game that supports even 50 people. (“We’ll add the ‘massive’ part during production! It’s just a matter of scaling.”)
• The only part of your game design that’s actually fleshed out is the back-story. Most of your system design docs just say, “implement this more or less like WoW does it.”
• You’re still trying to decide the right “look” for the art style so you can get the final assets started.

Give yourself a point if your game is more than a year out of pre-production and…

• Nobody’s ever discussed things like “customer support tools”, “live development cycle”, “expansion pack infrastructure”, “billing system”, or “login interface”. You’ve just assumed that’s the publisher’s job.
• You’re desperately pitching your game to prospective publishers each week.
• The design team and the engineering team don’t talk to each other. The animosity is growing constantly, but nobody can quite understand why.
• Lunchtime conversation inevitably degenerates into “such-and-such idea is insane, and there’s no way that’s gonna work,” but despite this, the team just keeps their heads down and does the best they can.
• The team is on its third Lead Designer (or Producer)… and you’re not sure how long this new guy will last, either.
• Your “lead designer” is actually a committee of five people, only one of whom is an actual designer.
• The engineering and/or design teams don’t test their own work at all. “That’s what QA is for!” However, everyone constantly complains about how many bugs QA is missing.
• QA has no specs to work from, and the only bugs they enter are things like, “Wouldn’t this font look better if it was blue?”
• This is your company’s second game (or attempt at a game). The first one failed, but nobody has addressed the reason why, and you’re seeing the exact same mistakes made again.
• This is your company’s second game, but the engineering team decided to throw out all the old code from the first game because it was too crufty, so they’re starting from scratch again.
• The very core of your game has recently been fundamentally changed by forces outside of your team.

Give yourself a point if your game is six months away from beta and…

• You can’t point to five people on the team and say, “These five people are truly EXCITED about how this game is turning out.” I don’t mean five people who are accepting of the design, or amenable to it… I mean EXCITED. They want to play it so bad RIGHT NOW that their enthusiasm comes through all the time.
• The production schedule requires you to create content and add features during the beta process.
• The engineering team still hasn’t stress-tested the server to prove it can support 1000+ people.
• If you’re looking for the assistant producer (or some other middle manager type), you no longer even check at their desk… they’re more likely to be found at the foosball table in the break room.
• It seems like management (and anyone else who can get away with it) is taking longer and longer lunch breaks every day.

How many of these apply to your team? One? Two? God help you, three or more? None of these are made up. I’ve seen every one of these problems… many of them repeatedly. These are all symptoms of a game that will end up being mediocre to poor… if the game launches at all.

And my list is far from complete… are there other looming problems that nobody’s addressing?

Every MMO development cycle has troubles. This is a massive undertaking and it’s harder than anybody expected. But things aren’t going to magically get better if you just wait it out. You may end up with a game to your name, but you aren’t going to end up with a game you’re proud of. And it’s going to cost you three to five years of your life.

So you either need to identify and FIX these problems NOW, or you need to walk away. Don’t keep doing something that isn’t working. Have the company form a “tiger team” or a “cross-departmental group” or have an “emergency meeting” or whatever you have to do, but you need to address these issues this week. By Friday you need to have some plans on how things are going to get better. You need to PUSH to make things better now, before it’s too late. This is way more important than your next made-up milestone.

If nobody is letting you push…

If nobody is willing to change, and you can see the problems are growing…

Then you need to leave.

Go find another job. Look at it this way: on one of the games I was indirectly associated with, the team had to crunch for so long that by the end, their effective hourly wage was $4 per hour. Their souls were beaten too: they were angry and sullen and miserable. These were not people who were having fun, and they would have been making more money as a Burger King shift manager. The saddest part? The game didn’t even ship. The team wouldn’t address their fundamental problems. They kept their heads down and worked really hard at their particular areas and tried not to think about the looming failure. Guess what? Ignoring the problems didn’t work.

What are you doing?! Either you fix the problems, or you get the heck out of there. Decide!

Elder Game is sponsored by: