The v0.4 Update – Postmortem and Next Steps

Hey folks!

It’s been a little over two weeks since we launched the v0.4.0.0 update, and boy did everything go wrong. Today we’ll explain what failed and why, how we fixed it, and our near-term plans to help get everything to run 100% smoothly again.

 

First off, the dreaded content server.

Our ol’ faithful server had been with us since early 2020, when we scrambled to find a new server after hosting them on our website was no longer a viable option. We set up a small Ec2 instance on AWS, threw a 500GB file system on it, and put all of our content there. Download speeds weren’t the best, averaging a couple of mb/s. Content was downloaded sporadically as we gained new users, new packs were released, or patches came out. Everything was pretty damn stable, so cue the update release.

This wasn’t just any update. Because the v0.4 system used entirely new file types and folder structures, all packs on our store (over 100) required updating. That’s gigabytes of data that had to be downloaded by all users, all at roughly the same time. On its own, this was a problem, but we also timed the 0.4 release with the release of our first tier of rewards for our Sci Fi Kickstarter. That’s hundreds of new users downloading the Toolkit and packs at the same time, including a 7GB Sci-Fi Essentials pack. Safe to say that our servers couldn’t handle the load. For a week or so users were sitting at download speeds of under 100kb/s, experiencing frequent time-outs as the server would drop the connection causing downloads to fail.

The hero of this story is Alex from Fantasy Calendar (https://app.fantasy-calendar.com/ Plugging his site is the least we can do). He reached out and helped us set up a proper AWS setup, complete with CDN for fast content delivery worldwide. Now our servers are more reliable, download speeds significantly faster, and we’re now confident we can handle a huge influx of users without negatively affecting the experience of the rest of our userbase.

 

Next up, content packs.

As we mentioned earlier, every content pack required updating to the new system for its content to be detected. Suffice to say, this was a process and a half. The process of converting assets initially used functions that our beta users had been using for the last year to upgrade their legacy content into the Toolkit. This worked fine for imported content, but the content packs were a different issue entirely. Various references to filepaths weren’t updated as expected, which caused most soundscape content to show all assets as missing. Some map assets inexplicably disappeared, and are yet to be recovered. In the last two weeks, we’ve reached a state where the vast majority of content is useable again, and are working on another round of pack updates that will further improve usability.

 

And finally, the Toolkit itself.

This was the one that really surprised us. We’d been working towards the v0.4 Toolkit for just over a year, and things were looking quite stable. People were able to build maps, create notes, use audio, and run their games with little to no issues. It turns out however, that thousands of people are able to find bugs a lot easier than a small collection of beta testers, and tend to do things that surprisingly neither ourselves nor our beta users had done. It also turns out that attempting to move to a new system that’s hundreds of patches ahead of your current version causes some issues that don’t appear when you gradually update your software over a year of development.

Right off the bat, the content upgrade process broke for quite a lot of users. This meant that their imported content wasn’t available for use, and so their maps would be filled with missing assets. Not good. Canceling some prompts, attempting to access certain filepaths, or sometimes just right-clicking on the wrong thing would cause an error; putting the Toolkit into an unreliable state and requiring a restart. We started having content disappearing seemingly randomly when moving or renaming it. Renaming a folder would just straight up delete everything in that folder. After hours or days of updates, users didn’t have content or know how the new version of the software worked. Not to mention the fact that users were dealing with a completely new UI –  people weren’t happy, and rightfully so.

We’ve released 21 patches since the 0.4 release, putting the Toolkit into a relatively stable state. There’s still a few lingering bugs that are causing issues, but things are actually useable now.

 

Next steps

We’ll continue to release (hopefully daily) patches to continue removing bugs from the Toolkit. We identified quite a few places where the user experience can be improved and will be working to make the new user experience a lot more seamless, reduce the amount of guesswork required and reduce the number of clicks required to perform various functions.

We’re also working on upgrading all of our Fantasy packs with clear filepaths, tags, and some light asset renaming. This will make it as easy as possible to find the assets you’re looking for. This process will likely take a month, possibly two.

We should have the Essentials Pack up later this week. After this point we’ll start to hammer out tutorials and documentation to help with learning the new system.

 

And that’s where we are right now. Pretty much everything failed in pretty much every way for the launch, but things are pretty well under control now.

We sincerely apologise for the issues that everyone has experienced with the v0.4 update. It’ll be a little bit before things are 100%, we’re very confident that it won’t be long before we have things in an even better state than they were in before the v0.4 update.

One of our amazing community members Timor has been producing great tutorials, you can see them here: https://www.youtube.com/playlist?list=PLt1_QRzANzG9vS4W6Fc-ZaJM5mIN72Q1e
The Master’s Toolkit can be downloaded from our store here: https://arkenforge.com/shop/

Cheers,

The Arkenforge Team

 

3 thoughts on “The v0.4 Update – Postmortem and Next Steps”

  1. Hey don’t sweat it. I just rolled back to an earlier version i saved on another computer. Problem fixed. And you guys fixed the new one. I’m using it now. Only one crash caused by windows updates. But so far pretty useable.

    Though for a few suggestions, the box with the 4 white squares on them (Scaling tool) I don’t use it that often except for placing tokens and adjusting the size of them. It’s not regularily used as much as the rotate button which isn’t easily seen. I’d make the token rotation easier to see and put the scaling tool more in the background somehow, but not totally hidden. Or it can be Control/Shift/Alt hotkey for quick use. Moving and rotating dozens of pieces is quite a hassle already.

    Keep up the good work.

    Wishlist;
    Initiative Tracker
    Dice Rolls in a Chat/Text Display

  2. Hey guys, as bumpy as this rollout proved to be, I was deeply impressed by your team’s performance. I was regularly surfing the discord channels and Nathan just seemed to be working 24/7, always staying nice and calm and dealing with issues the instant they popped up. And I give him and the whole team kudos for that.

    So no matter how bad things turned out, I think you guys handled this like professionals. And I’m sure you have assembled quite a nice file containing “lessons learned” in the past four weeks that will help you make the next big release much more painless.

    All in all, you’ve done a great job. And remember, it’s not “all work, no play”. Make sure you celebrate a little for yourselves, as you have delivered a very nice product that is just getting better every day.

    Thanks for your dedication and hard work!
    Mark

  3. DustyLensCap42

    You guys rock! Your commitment to your community and trying to bring a very ambitious grand map maker to life is outstanding. Plus yeah, of course, I keep backups of my many hours of map-making work. My players absolutely love the maps I’ve been able to create with Arkenforge to the point of squee’ing with excitement when the loading logo comes up in my screen-share. Thanks for all of your hard work!

Leave a Comment

Scroll to Top