User Generated Quests and the Ruby Slippers
Do you remember this part from the Wizard of Oz movie? It’s my favorite part:
GLINDA You don't need to be helped any longer. You've always had the power to go back to Kansas. DOROTHY I have? SCARECROW Then why didn't you tell her before? GLINDA Because she wouldn't have believed me. She had to learn it for herself.
It turns out that Dorothy could have gone home at any time during the movie! But if Glinda had just told her that clicking her ruby slippers together would teleport her home, Dorothy would have been unable to believe it. She had to learn it for herself or she could never learn it.
We’ve all been there plenty of times, right? The Ruby Slippers Phenomenon is part of human nature. Of course I had to date that girl even though everyone told me it would end badly. Of course I had to make an indie casual game even though everyone said it would be a flop. No amount of talking would ever convince me.
In my professional life, I’ve made conscious effort to avoid this problem — that is, I’ve tried very hard to learn from the experiences of others. And I’ve had, eh… sub-par results. It’s really hard to believe in the slippers if you didn’t figure it out for yourself. So I’m not pointing fingers at other people who have the same issue. But we do need to try to avoid learning every lesson the hard way.
The #1 reason we dismiss other people’s lessons is by pretending that they “aren’t applicable here.” User-created quests are a great example. No achievement-oriented MMORPG has ever had user-created quests before, so there’s “no possible way anybody could know if it would work”. (Let’s pretend that Anarchy Online didn’t have a simple custom mission generator … remember, most game developers burn out within 5 years, so very few working designers were around for AO!)
When designers would bring up this feature (and yes, it’s been brought up on every game I’ve worked on), the veteran designers would tell them, “That’s going to backfire tremendously. People will exploit it to make the easiest possible missions, and you won’t like the results.” This is always countered by some variety of “you can’t possibly know that for sure!” But actually, working on a live team teaches that lesson very quickly. From AC2, I learned:
- Players subconsciously calculate the cost-to-benefit ratio of content when deciding if it’s fun. For most MMO players, more reward = more fun. (This is a bitch of a lesson to learn, too. “My custom-scripted quest was so incredibly cool! Why aren’t players doing the quest? Well, yes, the reward was a little sub-par, but so what? You’re telling me they aren’t playing it because of THAT? Players can’t be THAT shallow!” Ha ha, newb.)
- Players aren’t objective reviewers. If you ask them to grade content, they will grade more rewarding content higher than other content even if it isn’t as good by other metrics (like plot, writing, annoyance factor, or originality).
- Many players spend incredible amounts of time finding ways to min-max the system so they can get more power for less effort. That’s part of the fun for many players. So there are tens of thousands of people actively looking for mistakes, loopholes, and gray areas in your game. All the time.
“Yes yes,” the other designers would say, “those lessons from the live team are interesting, but that isn’t exactly the same situation as user-created content, is it? Nobody can say for sure if user-created quests are problematic.” Maybe, just maybe, users could be convinced to grade content fairly. Maybe they would discover how fun it is to run really well-plotted quests instead of just trying to level up as fast as possible. Maybe players can change their stripes. Nope. MMORPG players are as predictable as the sunrise.
When City of Heroes released its user-created mission generator, it was mere hours before highly exploitative missions existed. Players quickly found the way to min-max the system, and started making quests that gave huge rewards for little effort. These are by far the most popular missions. Actually, from what I can tell, they are nearly the only missions that get used. Aside from a few “developer’s favorite” quests, it’s very hard to find the “fun but not exploitative” missions, because they get rated poorly by users and disappear into the miasma of mediocrity.
This was not what the designers hoped for. Somehow they had convinced themselves that the number of exploiters would be relatively low — certainly not the vast majority of the users. But they were wrong, and now they’re stuck between a rock and a hard place. They feel they must counteract these abusive quests, “for the sake of balance”. But how? Well the first step is to ban people who make cheaty content. But what’s cheaty? Do they explicitly list every possible exploit condition? What if they miss one? Nah, then the problem would start all over again. Instead, how about if they just issue blanket threats that they’ll ban missions that seem “exploitative”, without actually explaining what is and isn’t “exploitative”? They went with the latter.
So now, any user-created mission that is “exploitative” will get deleted, and users who played it will get their XP retroactively lowered, or even lose their character. So what counts as exploitative? One or two of the “exploits” are pretty obvious, but it’s really unclear where the line is drawn. Their forums are struggling with this very problem:
Are all-boss maps ok? Are all-AV maps ok? Are custom enemy groups with only one mob ok? Two mobs? Three? Five, but with no minions?
Or do we not get to know before they sack us?
Bingo. You don’t know if you’re breaking the rules until you get punished. So the developers are creating a chilling effect on their own content generator. Now it’s risky for players to even use user-created quests. What if some customer service rep decides the quest is exploitative? You’d retroactively lose your XP. It’s best to just to stick to the old dev-made quests, the ones you know won’t get you punished.
They made the wrong call here. Without some guidelines about what’s legit and what isn’t, I would certainly keep away from most user missions. Their lead designer reinforced that they won’t be giving useful guidelines out, saying:
I would say that a good interpretation of abuse is “Disregard for the risk and/or time to reward ratio”.
This is startlingly unhelpful to people trying to figure out how to make ban-safe, but fun, content. To keep this fiasco from chilling the buzz, they need to publish guidelines about what is and isn’t “fair”, or better yet, code this fairness into their tools. As I write this, pick-up groups are running user-generated quests consisting of nothing but max-level boss monsters, so that doesn’t seem to be “unfair”… of course, since there’s no guidelines, who knows if those quests are about to get banned? Since deletion only happens after an “abusive” quest is reported to customer service, it could just be a matter of time before any quest you play gets banned and your hard work gets reversed. Worse yet, since the rules are secret and enforced by numerous people, it is very likely that they will be enforced semi-arbitrarily, and will tend to become more aggressive over time.
But the thing is, even if they make the rules explicit, it’s not going to help the “power-leveling problem” which is ostensibly the reason for all of this grief. Unless they remove all difficulty options from the system, there will always be easier and harder ways to level. And remember what I said above: users tend to prefer easier content with better rewards. This isn’t limited to user-created content — it’s true for designer-made content, also. But designer-made quests don’t get graded by the players. Player-voted content like this will always gravitate towards easy. And pick-up groups will always be picking the most rewarding content with the least annoyance. And the game devs will keep being unhappy about it.
I hope they can find a compromise that makes this all worthwhile, but even when they do, the costs will be huge. All the tweaking, pleading, balancing, and customer service time involved is hard to imagine. Man, the customer service costs alone are tremendous! Think about it: CoH now has customer service personnel evaluating tons of content and deciding if it’s “fair” or not. Plus they have to deal with “incorrectly flagged” content, plus handling the thousands of additional complaint calls … unless they make clever decisions quickly, the labor and maintenance costs of their system will be in the millions of dollars over the next few years. This is not what game owners like to hear. And to add insult to injury, what started out as a PR win seems to be turning into a PR failure almost overnight. Personally, I think the most tragic cost is that their developers will have to continue to tweak this system for months or years to come. They could be adding other features, but instead, they have to try to bandage this system over and over again. Even if they win the battle, they may lose the resource war.
I’m not saying CoH is doomed — this won’t kill them or anything, even if their user-content tools aren’t a success in the long run. I’m not even saying they were dumb for trying this. Every game has “proven the ruby slippers” about a few things. These are missteps that seem really obvious in hindsight, and were pretty obvious beforehand, too, but somebody had to try them… because they couldn’t quite believe it if they didn’t learn it for themselves. So I’m really happy that CoH added this. They’re proving the ruby slippers by showing that this sort of system takes tremendous effort — Herculean effort — to be successful. And I’m pretty sure the CoH team will never be happy with the level of “exploitation” that happens with the system.
Now maybe we won’t have to debate whether user-created quests in an achievement-oriented game are a good idea or not. Oh, who am I kidding? In just a few years all the designers will be new again and nobody will remember CoH’s hard-earned lessons. Sigh…


