CCP Lays Off 120, Announces Refocus On Eve, World Of Darkness Delayed Amid Staff Cuts

Ouch.

At CCP we have been working hard to expand the gaming landscape by applying the knowledge and expertise we’ve built up with EVE Online to create a new experience within the EVE Universe in DUST 514, as well as a new video game franchise in World of Darkness.

During the last few months, as evidenced by our interaction with the community, we made some missteps on that journey.

As we reexamine our outward relations, we are also taking time to reevaluate our internal goals. In doing so, we have come to the conclusion that we are attempting too many things for a company our size. Developing EVE expansions, DUST 514 and World of Darkness has stretched our resources too thin.

More detail here.

A Brief Programming Note

I’ve moved on from my position at NCsoft and am now a design lead with these guys.

Hopefully will have more to share on what I’m working on soon. For now, just leave the LUM SOLD OUT notes right here!

Moneypet! Blizzard Adds Convertible Currency to World of Warcraft

"Do You Still worry about Buying Gold? Old brand provides cheapest and favorable price for you! Fastest delivery Speed! WWW.BLIZZARD.COM Discount code: CUB Welcome!"

Well, this is fairly large news. Blizzard is set to add a new vanity pet for World of Warcraft to its out-of-game shop – the difference being that unlike all the others they sell, this one is freely transferable in-game. And Blizzard is not shy about pointing out exactly what that means:

While our goal is to offer players alternative ways to add a Pet Store pet to their collection, we’re ok with it if some players choose to use the Guardian Cub as a safe and secure way to try to acquire a little extra in-game gold without turning to third-party gold-selling services. However, please keep in mind that there’s never any guarantee that someone will purchase what you put up for sale in the auction house, or how much they’ll pay for it. Also, it’s important to note that we take a firm stance against buying gold from outside sources because in most cases, the gold these companies offer has been stolen from compromised accounts. (You can read more about our stance here.) While some players might be able to acquire some extra gold by putting the Guardian Cub in the auction house, that’s preferable to players contributing to the gold-selling “black market” and account theft.

If this back door to monetizing in-game gold transfers looks familiar – well, it should, as Eve Online has a similar scheme where players can buy monthly game-time codes and then sell them for ISK (in-game currency) – CCP even has an out-of-game secure transfer website set up to facilitate this.  Trade in these GTCs are very active, and the player base has essentially embraced this pegging of in-game ISK to out-of-game currency value (currently US$ 1 / ISK 40m). It hasn’t eliminated in-game botting but it has put a stake in RMT dealers – why bother, when everyone is an RMT dealer. It also, ironically, at the highest end of the game turned Eve into a free-to-play game since it is possible in nullspace to earn more than 400m ISK / monthly and thus pay your own way via purchasing game time from people who don’t want to bother to.

Blizzard’s moneypet isn’t the same, and that may be its weakness. For while Eve’s GTC traffic is based on an actual good (a monthly subscription, which already has implicit value, at least to people who want to play Eve), WoW’s moneypet is literally created from thin air. Which, while better financially for Blizzard in that it literally is printing gold and/or money, it also inherently has less value than a more tangible good. There is also the small matter of the moneypet being an in-game manifestation of “Hi, I like to sell gold”, and that, to put it gently, may not be a popular opinion.

So, unlike Eve’s GTCs, investing in Blizzard moneypets for resale is far from a sure thing. The irony may well be if a game company sells a moneypet and no one actually buys it, does anyone care? It would, after all, be a market-driven response of disapproval that would say more than any message board post…

…oh, who am I kidding. Blizzard is going to sell 9 trillion of these in an hour.

A Decade Of Camelot

Ten years ago, I began my career in the gaming industry by signing on to help with the imminent launch of Dark Age of Camelot. That game and its team still holds a special place in my heart, though most of the founders from that day have gone their separate ways. Matt Firor, the original producer, asked me to post his recollections of that launch.

(Note that he does make me look like a bit of a doof at one point. I have three things to say in response: 1 – I’m still pretty happy with how we managed to get a working CS front-end and back-end system up and running in about six weeks, 2 – it wasn’t *quite* as doof-tastic as Firor makes out, as I explain in annotations to his tale, 3 – I am in fact a bit of a doof.)

And with that, I give you Matt Firor.

We went into October 9 relatively calm and serene. Vivendi, our distribution partner, had forecast 100,000 sales of the game lifetime, with about 50,000 coming in the first couple of days after launch, and as such, they only “sold in” a very limited number of boxes into the retail channel. We were very comfortable that we could handle those numbers, as we had just had a very successful beta program.

Then, just before launch day, Vivendi got in touch with us and said because customer/retail demand was so high, they were going to release all 100,000 boxes into retail. Fortunately, with delivery times, these extra copies would be delivered to stores a couple of days after the initial 50,000 boxes. This was very helpful to us, as all the boxes were not available on the same day which spread out the “opening day” crush of users over five days. Camelot would go on to sell more than a million boxes in the next couple of years.

Camelot’s official launch day (as in boxes were in the stores) would be October 9 2001, but everyone that had a free account (lots of media, some friends/family, and of course all of us) were able to play starting October 8, as soon as we put our first seven servers online starting around 5:00pm that day.

At the time, Mythic’s offices were located in a townhouse office community near the middle of Fairfax, VA. We were about 30 developers in October 2001, and about 40 customer support. We didn’t have enough space for CS in our original office, so we had to lease space in another building in the same complex, about 50 yards from the developer wing of the company, in a basement space.

Everything was done on a shoestring at that time, so to get internet access over to the CS “center” 50 yards away, Rob Denton, the Development Head of Mythic at the time, and an electrical engineer by training, set up a pair of IR “guns” to make a link between the two spaces. We put one “gun” in a window in the development office, and another in a basement window, pointing up, in the CS center. The link worked very well, and allowed us to share our one Internet line with both spaces. However, because the CS Center was below grade, we had one problem: if a vehicle over a certain height (about 5 feet) parked in a particular parking spot, the link would be broken. We lived in fear those couple of weeks that a delivery truck would park in that spot and cut off Internet access to the CS center. We arranged a quasi-official parking schedule to ensure that an employee car (a short one) be parked in that spot 24/7. The link, fortunately, was never broken.

It's on.

All day on the 9th, we watched as the server numbers grew and grew. I ran the login utility on my laptop all day, just so I could see the population numbers of each server, real time. The population numbers started small around 10:00am on the 9th, and grew slowly but steadily until around 5:00pm, when they exploded. All servers in the space of about an hour after 5:00pm were jammed full – and we had a very large server population setting (about 3,000 players). Even with full servers, everything ran smoothly.

By about 8:00 we were jubilant. Everything was smooth and easy. CS was functioning, and had already responded to many trouble tickets and issues. People were playing, the servers were up.

A group of us formed in Rob’s office, talking and generally basking in the glory of the moment. Each of those 20,000 (max peak players that night) was a paying customer, and each represented significant revenue to us (remember we were very small at the time). It appeared that we finally were going to make money on one of our products. We were giddy with excitement – everything was going awesomely.

Brian Axelson, the 21 year old whiz-kid programmer/designer who had been working for us since he was 16 – responsible for inventing, implementing, and designing Camelot’s combat system, including Combat Styles – was so happy he slammed his fist down on Rob’s desk and said, “Ain’t nothing going to bring this house down!”.

At that moment, all the servers crashed, simultaneously.

We all looked at one another in dread, and sprinted back to our offices, each checking on the part of the game we were responsible for. Everything checked out – nothing seemed wrong. But the servers were down and wouldn’t reboot.

All the programmers were summoned to Rob’s office – I was a fly on the wall – and he walked them through the problem. It wasn’t a code problem, although that wasn’t immediately obvious. It was something keeping the servers from booting and authenticating properly.

After about an hour, the problem was traced to the Customer Support tool – that very day programmer Scott Jennings had made a small modification to the CS tool to take advantage of a database feature buried in MySQL to make database queries work faster. That change did in fact significantly increase the speed at which the tool made queries to the database – but at full load, the index that he built quickly became overloaded, and started to time out and lock out other queries.  (Editor’s note: This change was actually fully tested… with one low-population test server running. Guess what changed on launch day!)  Because the game servers relied on access to the database as well (for player authentication, etc.), they couldn’t keep up with the crush of players logging on and off – and they crashed like the proverbial house of cards. And, because the database was locked up, when they rebooted, they immediately ran into the same problem and crashed again.

The helpful message I added to greet CS reps on launch day.

Once the problem was found, it was very easy to fix, for the moment. The CS tool was modified to not make any of those specific types of queries, the database server was rebooted, the index rebuilt, and everything came up again – this time smoothly and without error – and ran flawlessly until the next afternoon, when we had our first bug-fix patch. Scott had fixed the DB/authentication problem in the meantime (Editor’s note: and said programmer slept three days later), and that functioned properly as well.

In the end, a very smooth launch, but a lesson was definitely learned that when you’re dealing with something as complex as a MMOG launch, you never know what is going to take you down.

There are many more stories to tell about the early days of the service – like how we had to expand servers quickly because of demand, but couldn’t get them delivered from Dell because we had no credit rating. All our purchases up until that point had been made on the spot with no leasing. We had no leasing history, so Dell wouldn’t ship us servers quickly. We were forced to drive to MicroCenter (in Fairfax) and buy a dozen or so desktops, quickly installed Linux, and then drove them (in a pickup truck) to our colocation facility, and stacked them up like firewood in a cage. Those two servers clusters (lovingly called the “gimp servers”) ran for at least a year with no problems, at which point they were swapped out with standard Dell rack-mounted models.

I remember walking into the office one morning towards the end of October. By that time it was obvious we had a smash hit on our hands. Our marketing/sales consultant, Eugene Evans (now the GM of the studio) had a whiteboard near his desk (right by the front door) where he jotted down sales numbers. By October 27 or so, it showed that we were not only the #1 selling PC game for October, but also the #1 selling PC product for that month. Since this was the first boxed retail product Mythic had, I asked him if this success was normal. Eugene, and old industry veteran, looked at me like I was insane and replied, “No, this isn’t typical.” He then broke out in laughter. It seemed so easy at the time: you make a game, put it in a box, and it sells like hotcakes.

Most everyone knows the rest of the story from here – Mythic quickly outgrew its space and in 2002 relocated a few miles away to new mid-rise building, where it grew to take over three floors. A buyout by EA followed in 2006, and the studio is now known as Bioware Mythic. Dark Age of Camelot’s numbers have dwindled down to a fraction of what they were in those heady days of 2001-2003, but it is still up and running, ten years later.