The v0.4 Update – Postmortem and Next Steps

Hey folks!

It’s been a little over two weeks since we launched the v0.4.0.0 update, and boy did everything go wrong. Today we’ll explain what failed and why, how we fixed it, and our near-term plans to help get everything to run 100% smoothly again.

 

First off, the dreaded content server.

Our ol’ faithful server had been with us since early 2020, when we scrambled to find a new server after hosting them on our website was no longer a viable option. We set up a small Ec2 instance on AWS, threw a 500GB file system on it, and put all of our content there. Download speeds weren’t the best, averaging a couple of mb/s. Content was downloaded sporadically as we gained new users, new packs were released, or patches came out. Everything was pretty damn stable, so cue the update release.

This wasn’t just any update. Because the v0.4 system used entirely new file types and folder structures, all packs on our store (over 100) required updating. That’s gigabytes of data that had to be downloaded by all users, all at roughly the same time. On its own, this was a problem, but we also timed the 0.4 release with the release of our first tier of rewards for our Sci Fi Kickstarter. That’s hundreds of new users downloading the Toolkit and packs at the same time, including a 7GB Sci-Fi Essentials pack. Safe to say that our servers couldn’t handle the load. For a week or so users were sitting at download speeds of under 100kb/s, experiencing frequent time-outs as the server would drop the connection causing downloads to fail.

The hero of this story is Alex from Fantasy Calendar (https://app.fantasy-calendar.com/ Plugging his site is the least we can do). He reached out and helped us set up a proper AWS setup, complete with CDN for fast content delivery worldwide. Now our servers are more reliable, download speeds significantly faster, and we’re now confident we can handle a huge influx of users without negatively affecting the experience of the rest of our userbase.

 

Next up, content packs.

As we mentioned earlier, every content pack required updating to the new system for its content to be detected. Suffice to say, this was a process and a half. The process of converting assets initially used functions that our beta users had been using for the last year to upgrade their legacy content into the Toolkit. This worked fine for imported content, but the content packs were a different issue entirely. Various references to filepaths weren’t updated as expected, which caused most soundscape content to show all assets as missing. Some map assets inexplicably disappeared, and are yet to be recovered. In the last two weeks, we’ve reached a state where the vast majority of content is useable again, and are working on another round of pack updates that will further improve usability.

 

And finally, the Toolkit itself.

This was the one that really surprised us. We’d been working towards the v0.4 Toolkit for just over a year, and things were looking quite stable. People were able to build maps, create notes, use audio, and run their games with little to no issues. It turns out however, that thousands of people are able to find bugs a lot easier than a small collection of beta testers, and tend to do things that surprisingly neither ourselves nor our beta users had done. It also turns out that attempting to move to a new system that’s hundreds of patches ahead of your current version causes some issues that don’t appear when you gradually update your software over a year of development.

Right off the bat, the content upgrade process broke for quite a lot of users. This meant that their imported content wasn’t available for use, and so their maps would be filled with missing assets. Not good. Canceling some prompts, attempting to access certain filepaths, or sometimes just right-clicking on the wrong thing would cause an error; putting the Toolkit into an unreliable state and requiring a restart. We started having content disappearing seemingly randomly when moving or renaming it. Renaming a folder would just straight up delete everything in that folder. After hours or days of updates, users didn’t have content or know how the new version of the software worked. Not to mention the fact that users were dealing with a completely new UI –  people weren’t happy, and rightfully so.

We’ve released 21 patches since the 0.4 release, putting the Toolkit into a relatively stable state. There’s still a few lingering bugs that are causing issues, but things are actually useable now.

 

Next steps

We’ll continue to release (hopefully daily) patches to continue removing bugs from the Toolkit. We identified quite a few places where the user experience can be improved and will be working to make the new user experience a lot more seamless, reduce the amount of guesswork required and reduce the number of clicks required to perform various functions.

We’re also working on upgrading all of our Fantasy packs with clear filepaths, tags, and some light asset renaming. This will make it as easy as possible to find the assets you’re looking for. This process will likely take a month, possibly two.

We should have the Essentials Pack up later this week. After this point we’ll start to hammer out tutorials and documentation to help with learning the new system.

 

And that’s where we are right now. Pretty much everything failed in pretty much every way for the launch, but things are pretty well under control now.

We sincerely apologise for the issues that everyone has experienced with the v0.4 update. It’ll be a little bit before things are 100%, we’re very confident that it won’t be long before we have things in an even better state than they were in before the v0.4 update.

One of our amazing community members Timor has been producing great tutorials, you can see them here: https://www.youtube.com/playlist?list=PLt1_QRzANzG9vS4W6Fc-ZaJM5mIN72Q1e
The Master’s Toolkit can be downloaded from our store here: https://arkenforge.com/shop/

Cheers,

The Arkenforge Team