A "small number" of players are currently locked out of RuneScape as of writing this. Estimates from Jagex suggest that number is close to 1% and the entire company is taking an all hands on deck approach to getting everyone back online. I am personally struggling to remember a time we had a game update delayed 2 weeks (maybe more?) in the almost 20 years I've been playing. Put simply, we are in unprecedented times.
What happened? On March 4th a number of players were unable to log into the game. Players were advised that those logged in would not be affected. After the login servers were brought back online it was determined that a "small portion" of players would see a rollback of their progress. Players reported incorrect rollbacks and new reports of login issues surfaced. Continuing through the end of March 4th we saw continued outages for players and portions of the RuneScape website were also taken offline. By 8:35pm game time on the 4th the login servers were back online and we were left with a small number of accounts unable to log in.
The story became more interesting and more concerning at this point since some players attempting to log in were greeted with an "error loading your profile" message. This would frighten any RuneScape player given the substantial time investment we all put into the game. By 11pm there was a support article up with the option to report issues if your profile was locked. The process began to restore these accounts but unlike most other RuneScape outages, it wasn't fixed within hours.
Starting March 5th we saw daily updates, including two messages from Mod Warden. We also had a livestream on March 9th featuring Jagex CEO Phil Mansell, otherwise known as Mod Pips. Also joining this livestream were Mods ID0 and ZZ. Mod ID0 is Jagex's Senior Vice President of Technology and Mod ZZ is Senior Director of Analytics, Data Science & Engineering. This crew updated the community on March 9th and assured players that the top priority was restoring service.
It was revealed that some game saves were lost starting February 8th running up to March 4th. The plan was to utilize the telemetry data generated when you get a drop, complete a quest, or do pretty much anything in RuneScape to rebuild the saves. This process involves building a process to reconstruct the saves, a quality assurance process, and an account integrity beta where some affected players will be able to sample their account to ensure the save has been rebuilt successfully.
As of March 25th there have been a number of beta invites sent out to affected players. The game update slated for March 22nd was going to ahead but community solidarity encouraged Jagex to hold off another week on shipping game updates. Players involved in the account integrity beta have seen levels, items, and quests intact. However, other items such as Invention perks have been lost. The commitment made by the team is that players will be done right. There will be no half measures and everyone will have everything restored to the condition it was in.
I have long maintained that the RuneScape community is different than the community you will find surrounding any other game. On the March 9th livestream Mod Warden became visibly choked up when describing the vast support the team has received from the community. RuneScape is a living game meaning that with its updates it changes and evolves. But one thing that's constant is the player community surrounding it.
There are two ways to go: understanding or pitchforks. The pitchforks approach would look like vocal mob incited by a number of hot headed players or community representatives. They would say that they know better and everything that we see on the surface is smoke and mirrors to mask the dumpster fire internally. Because of course if you had proper backups and a proper IT infrastructure this would never have happened. And anyone who has set up online infrastructure should be aware of this or else they're the devil incarnate!
I'm personally aware of or have at least talked to a dozen plus RuneScape players who are affected. They're the understanding type and aren't brandishing pitchforks. Anyone who claims they know better than Jagex on this doesn't know what they're talking about and shouldn't be listened to for the following reasons:
1.) Nobody except the internal team in charge of the data and servers knows how the RuneScape architecture is built, without having that knowledge it's foolish to assume.
2.) For any organization or system that is inherently complex and large, it's a non-human ask to ask someone to watch over such a system and ensure it doesn't fail. You can build in redundancies and monitoring but there will always be something that's missed.
If one were to speculate based on what was said in the March 9th livestream, here's what likely happened
- There was a hardware failure somewhere on the systems that deal with game saves in February.
- It cascaded and caused issues for the game save system without being detected.
- It didn't manifest itself until March 4th with players being unable to log in.
- March 4th looks like an issue with the login servers until a rollback is requested and then the true depth of the problem becomes known.
I've long talked about and foreshadowed a Jagex and RuneScape team that can do what they want and has the backing of the community. This started with a build up of trust over time and a return to basics. This was well underway before Mod Warden joined as Executive Producer but his arrival accelerated it. It is my belief that had we not built up an impressive amount of good will over the last three years or so (minus the Anachronia summer drought), we wouldn't be where we are today with the reaction to the login lockout issue.
Without the support of the community this issue could have caused massive damage to the reputation of RuneScape and Jagex as a company. This shows why it's important to lead and be on your toes in the best of times. Their plans going forward to do right, be transparent, and focus on the players affected over expediency is the right move. With that we eagerly anticipate the return to normal game updates.