SmartFoxServer: The MMO Engine for Indies?

Run Screaming…

If I were to tell you that I was thinking of making an MMO with a database-centric design, using a DB to serialize entities from one sub-server to another, you would be in good company if you thought I was a dumbass.

Darrin West, the architect at Emergent Game Technologies, does not waver in his disdain

There are hidden costs of a DB-centric approach for migration. … In my experience, DB throughput is the limiting factor in scaling a shard. Please, please, run screaming from DB-centric!

No? If that doesn’t convince you, how about Bryant Durrell, former Tech Ops director for Turbine? Let’s see what he thinks

Relational databases: please no!

OK. It is completely obvious that any MMO is going to need a way to store data. I understand that the instinctive reaction is to use a relational database, because that’s what relational databases are for. However, I beg of you as the guy who needs to keep the things running fast and smooth, think twice.  

However, Emergent’s MMO engine still isn’t ready for development, and Bryant doesn’t seem to have leaked Turbine’s server technology to the internet before he left. So I know there are better ways, and yet I can do nothing about that.

… Or Embrace Your Destiny!

Sandra and I, along with a few others, are plotting out what it would take to make an MMO on our own. And if I plug realistic timelines into a realistic engineering schedule, it appears that two or three engineers working half-time on an MMO will never ever finish it if they have to develop the server tech themselves (or the client tech, or the tools pipeline, but those are a different post). Every single hour that is not spent on gameplay reduces the chance the game will ever come to market. This is the bitter pill to swallow: the best practices of MMO server development simply aren’t available to tiny indie teams, no matter how experienced they are.

There are some really cool technologies that could help, like Terracotta. This sucker is just what we need! Wait, you say it would cost us how much?! Oh. I see… these solutions are designed for rich companies, not indie developers. Well, maybe we can use that for our second MMO. 

SmartFoxServer: The Glorified, Extensible Chat Room

In the mean time, we can use SmartFoxServer to make a simple database-centric MMO. It’s got a decent track record: Habbo Hotel runs on it. SmartFoxServer (SFS) is really a simple piece of software: it’s a glorified chat room. This is, contrary to what you’d expect, its strength. See, it’s a really full-featured chat system. Authentication, multiple rooms, private messages, friends lists, ignore lists, bad-word filters, you name it. But that’s all it is. It doesn’t try to be a full MMO engine. And that means it’s relatively fast, elegant, and stable at what it does.

You plug extra modules into the back of your “rooms” to add extra functionality. You can implement entire MMOs as plug-in modules to SFS. And that is one road we’re considering. 

If you’re thinking to yourself, “Chat?! Come on, I can write a full-featured super-powered chat service in a month!” then you aren’t getting the “tiny indie MMO team” part. But more importantly, you’re wrong. 

There are a ton of details that getcha when you want to make a professional-quality MMO. SFS also provides logging, banning, moderation, customer support tools, and flood prevention. It’s all little detail stuff, none of it hard, but all of it time-consuming. You couldn’t actually code all the features of SmartFoxServer in a month.

SmartFoxServer Plus A Database Equals “MMO” (With Air Quotes)

Using multiple SFS Zones to run the areas of your MMO

Using multiple SFS Zones to run the areas of your MMO

Okay, so how does it work? Well, you use SmartFoxServer as your “front door”: it helps with authentication and, of course, chat. Then for the game logic, it calls your plug-in module, which in turn loads the player’s character from a central database. Each geographic section of your world is run in a separate SFS “zone”. Multiple zones may be on the same physical server computer, or they might not.

A client only talks to one “zone” at a time (well, maybe a couple at a time. But the point is, it doesn’t talk to all of them at once). When the player moves to a different area, the client literally connects to a different zone.

It’s really simple to code — that’s the whole point. In fact, if you’ve got a good handle on SQL, this is probably the simplest possible MMO server. When you’re a tiny indie company, simplicity is key. But there are down sides.

Embracing Your Destiny Sucks Sometimes

What are the down sides? Well, we’ve basically invented the server technology for EverQuest 1. But EQ1 wasn’t technologically advanced even when it launched, ten years ago. This is a poor man’s engine in every respect.

The biggest limitation is that your world will be segmented into very tightly-defined geographical areas. When the player goes from one geographic area to the next, they will have noticeable delays. This engine isn’t going to win any tech awards.

The physical size of each geographic area is unimportant to the server. Instead, the limiting factor will be the number of players that can be in a given area at once. The exact number of people allowed in one area will vary from game to game, but a good guess is about 50 players per area, max.

And lastly, you have a big bottleneck in your design: the database. It will be the limiter on how big each game shard can grow. And you will need shards: you won’t be able to have one world in which every player exists simultaneously.

The Poor Man’s MMO Is Good Enough

Gee, what if I have so many people playing my game that I can’t support them all without more hardware expenditures? Man, as a tiny indie, I’d love to have that problem. It means I have a hit on my hands, and in that case I will be able to find the financial support I need to get more hardware.

The flip side of the coin is more dangerous: “what if I spend months or years making cool server tech and then never make an MMO with it?” Wait… I already did that before. In fact it’s the easiest trap in the world for an engineer to fall into. But it will doom your project, indie or not.

As long as the game looks good and is fun, players aren’t gonna care that there are load times occasionally. Lots of games have geographic zones, actually. Don’t get hung up on it.

We haven’t settled on SmartFoxServer yet. There are several other simple server engines that we’re going to examine before making a decision. But none of them are going to be amazing MMO masterpieces.

There’s a simple indie mantra that I’ve been trying to take to heart: “design around the limitations.” Don’t try to remove the limitations: get creative and figure out ways to minimize their effect, instead.

This entry was posted in Business. Bookmark the permalink.

16 Responses to SmartFoxServer: The MMO Engine for Indies?

  1. Bryant says:

    Well, that’s a convincing argument.

    At this point, I’d recommend trying to design the data store layer such that you’d be able to swap in a more polished plug-in to replace their ODBC layer when and if the time comes, which would save some time in the hypothetical case. But you’re absolutely right; getting the thing out the door is a better choice than piddling around forever and not finishing it.

  2. Tim says:

    Awesome to read this. I too have been in the “how to build an MMO with spare parts” mood, and had also zeroed in on SmartFoxServer as a very cool tool with an amazingly good price.

    One correction maybe? I think it’s Club Penguin that uses SFS, not Habbo?

  3. I evaluated SFS before I started my current MMO, and dismissed it. There’s very little there, and the “extensibility” is pretty crap. They’re using their own homegrown method instead of one of the many robust Java standards.

    I’m now 4 months into development (one man team) and am just polishing the engine (server and client) and starting to think about how I’m going to create all the content.

    As for saying no to DB-based engines, that’s more bullshit. There are so many ways to make a DB real time it’s not even funny. At that point it becomes a matter of your server software being smart enough.

    Keep in mind, all of my comments concern a Flex client/Java server based free to play mmo. That’s synchronous too, not post and refresh or poll. :)

    Anyway, I know enough about DB performance from my previous life to be able to make the servers support a pretty massive load – DBs are not by any means the kiss of death, it’s just that most game devs don’t know much about them.

  4. Darrin West says:

    Clarification for Jason: “DB centric” is not the same thing as “uses a DB”. In MMO architecture terms, DB-centric means using the transactionality of the DB to solve the transactionality needs in the game. Doing that puts the latency you get through a DB into the user’s game response latency. I’m sure you can put together a database that has *massive* throughput. But I’d like to see you do that at the same time you guarantee all response times are below a millisecond when you have 5000 players connected through to that DB.

    Eric’s post actually touches on the same issue. He has decided to embrace a room-based game design so that there are no real transactionality problems. All real-time interactions are within code that runs in a single-thread. The DB is used for persistence, and for transfering to another room. That approach is fine for 50 player zones, but can’t support seamless worlds. If your game design fits that limitation, you can definitely go with the low-tech (low-price) architecture. You might say it is embarassingly scalable :) But it can be pretty hard on players that want to quest with their buddies, and can’t get into the same instance of a zone.

    I’ve found it kind of challenging to convince a game designer the magnitude of the technical development cost of some design decisions. “We need to get rid of the loading screen.” “Any two players have to be able to quest together whenever they want.” “The power of the spell you are casting is determined by the number of logged in players game-wide that are part of your guild.” It would be great to have a designer that is aware of, and will not push on the limitations of the tech adopted at the beginning of the N-year project.

  5. Aaron says:

    You should take a look at Project Darkstar. It is an open source Java-based server which is being actively developed by Sun. It provides very robust base on which to build an MMO. A good overview of the project can be found at: Developing with Project Darkstar

  6. jason says:

    Terracotta is free and open source. There is a commercial license, but that is mostly for support services. The free version doesn’t lack any mission critical features, and the license is very liberal. Please correct me if I’m wrong, cause I’ll have to change plans if that’s not the case! :)

    I’m researching if I can dispense with a database entirely by using Terracotta (with SmartFox). It keeps the whole state on disk and updates in real time behind the scenes. This solves a lot of big problems. That would also eliminate the need for shards, and a seamless world becomes possible. I’m not really interested in making a seamless world, but lots of people are, so I figured I should say that. :)

  7. Sorry Darrin, I don’t buy it.

    DB clustering (MySQL Cluster in my case) and ram based caching (Cluster does it, plus you can layer on more for each of the server machines in the non-DB cluster, either Hibernates 1st and 2nd level cache or memcached or what have you) makes a hell of a screaming machine.

    As for 1 millisecond response times – lol. What do you think we’re making here, an MMOFPS? There’s very, VERY few MMOs that require that kind of speed.

    As for 5000 users, no problem. Cluster will do 100k replicated transactions per second with around 5-10ms of latency with 4 nodes in the cluster. You want more, add more machines. There’s an upper limit of course, but it’s high.

    Anyway, I’m not just some DB guy spouting. I’ve done many, many years of game work as well. Almost 2 decades worth now.

  8. Tim says:

    Terracotta and DarkStar aren’t MMO solutions per se, just tech packages that have some overlap or application. Can anyone point me at some MMOs that actually use them?

  9. Darrin West says:

    Jason…consider an entity behavior that is interacting with another entity, and has to have those interactions be transactional (or across a host boundary). If there are multiple steps in the interaction before the user gets feedback that can be multiple trips through the DB. Using your numbers from a clustered parallel DB engine, maybe the latency injected by the DB approach is now 50ms. Then add network, behavior execution, frame delays, etc. And suddenly you’re talking about real money.

    My big concern is not throughput. It is spikes of latency and outright deadlock. WoW had an issue where mining would lock the user up, and they would have to log out. What I heard was it was a DB deadlock or a transaction on the row containing that rock that never completed. By being DB-centric and relying on the DB’s transactionality, the operation could never complete. Took a long time before they fixed it. You would also see pretty significant delays when scrolling through items the auction house.

    Another example: What happens when you have to do DB maintenance? With a non-DB centric solution, you could keep running the game and only risk losing some game time if something crashed, as opposed to losing revenue (assuming you charge per minute), or ticking off the community if you closed the game.

    I can think lots of reasons to not rely on the DB for synchronization and state transmission. I understand that Eric feels it is a matter of money and a continuum of “quality”, not a digital must/must-not. I can respect that. Just be aware of the risks/costs/consequences. It is nice that Eric or Sandra are the game designer. They “get” the trade off, and can design around it. I’m designing for those less tolerant, and who have money.

  10. Eric says:

    Terracotta is open source? Very interesting! I’ll have to check into this more!

    Project Darkstar is several years from being useful for development. It promises to eventually simplify multi-sub-server development, but I think it’s quite a way from being really helpful in that arena. (I’ll talk about more in a while, probably.) I am really happy to see it exist, but regardless of who you talk to, the .9 releases available now are single-server only, meaning their key feature is missing. You’d be better off with SmartFoxServer, just for its niceties like friends lists, IP bans, and other minutia.

  11. Bryant says:

    Darrin’s covering this better than I would, so mostly “what he said.” The followup question is this: what are you getting out of using a DB? Is it worth the cost of the cluster, plus the DBA expertise needed to keep it running? Jason, do you want to get stuck in an operations role post-launch? And yeah, you can find an operations equivalent, but that’s a critical skillset and another point of failure unless you hire two of that person, etc., etc.

    Eric says that he’s getting ease of development out of a DB and he’s thought about the tradeoffs, which is all I can ask for! My rant’s targeted at people who just assume a DB is the way to go.

  12. Hendy Irawan says:

    Hey, try Project Darkstar at http://www.projectdarkstar.com/

    who knows it helps your MMO

  13. Pingback: MMORTS mechanics – hypertable « Mental Meanderings

  14. hmmm says:

    I’ve been developing an indie MMO for (ugh) 2 years now and have been using SmartFox all the way. I dig it. It does have some silly limitations that the authors don’t seem eager to fix…and instead waste their time on useless gimmicks (Streaming video support….uh…WHY?!!?). But it’s robust, simple, and cheap.

    Also, considering I have yet to register the full version (probably when I go “alpha extreme” next month) I’ve gotten great support from the developers in their forums on the freebie 20 user license.

    There’s no way I’d be as far as I am today with this game if I had to write all that myself. I personally think SmartFox 2.0 is going to be a killer update.

    Anyway–I’ll probably launch the public alpha in a month….but then again I’ve been saying that for a year now. :) Thank you SmartFox!

  15. Darrin – It’d be awesome if you suggested alternatives. My suggestion would be to only use storage for backup persistence and not depend on it for computations, and to read this: http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

    If you want a system where a server going down means nothing goes down then you should be paying more for your developers+operations than your software because NO system has that out of a box. Even if it did, without users, you’re burning cash. That’s really not an option anymore for the next say, 5 years. If you *happen* to have a lot of funding then you’re 1 in a million and I’d recommend taking advantage of that and again, spending more on your developers, marketing, and game quality than on architecture. Architecture and systems designers (data layer) EXPECT critical scaling problem with operations because that’s their job; the people who will make a game worth being bought don’t work that way.

    I think anyone who’s starting an MMO w/an rdbms knows it’ll be replaced and if they don’t then I doubt they’ll end up here.

  16. Realist says:

    The best place to start is http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect here.

    After you have assimilated the full knowledge of that highly relevant article, you’re ready to move on to http://sol.gfxile.net/mmorpg.html.