Hey there folks!
With the recent launch of the new Content Library, I’m going to be diving into the history of how we’ve managed content in the Masters Toolkit.
Original .rwpack system
The year was 2017. Our Kickstarter had just finished, and we were getting ready to release our v0.1. Back in these days, the Essentials pack was less than half the size it is now, approx. 1GB, and was a single .rwpack file. This pack, however, fully loaded its contents directly into RAM, taking up 2 – 3GB when uncompressed. On the plus side, content loaded instantly! But it meant that any other packs we released would balloon this quite quickly, and with three other packs announced, it was clear that we’d have to do something about it.
I figured we had a decent amount of time to do something about it, until in early 2018, we put a vote to the community: Continue on with our planned packs, or double the size of the Essentials Pack.
To nobody’s great surprise, doubling the Essentials Pack won out (who doesn’t like more free content?), and we were faced with a pack that was soon to be 2GB in physical HD size, and likely 5 – 6GB in RAM. It was time for a new solution, and after some time and research, that solution came in the form of lazy loading.
Lazy Loading is the process of only loading content when you need it, and having it sit on the hard drive doing nothing for the rest of the time. This is a godsend when dealing with large quantities of content, as you can pick and choose what gets loaded into your precious RAM.
Lazy Loading and Meta files
You see, you can’t just read SOME of a file. It has to be all or nothing. When you’re dealing with a 7 minute song with 6 individual, you don’t want to have to read the full song into RAM just to see the name and the artist, and when you’re trying to build a soundscape preset and want to browse some 50 different tracks, that’s enough RAM use to crash the Toolkit on most high end PCs.
The way around this conundrum is to create what’s called a ‘meta’ file. A file that’s just a few kB in size that tells you all the important information about the big file you want to know about. You can then load all these meta files in, using only a few hundred MB of RAM at the absolute most, and use them to browse your content without needing to actually LOAD all of your content. Throw in some simple code to find the content file based on the meta file information, and you’ve got yourself a nice low-resource content system that can handle significantly more content than it could otherwise.
Animation – Large files
In early 2018, Unity had the problem that it couldn’t load compressed content from memory, which meant that all tiles had to be stored as a raw image. No JPG or PNG compression at all, just the raw pixel values. This makes an image significantly larger than you would expect, to the effect of a 1mb PNG file sometimes expanding up to 30 – 40MB! It was into this environment that we released our first animated content.
Due to the large file sizes of uncompressed content, our first animated asset (torch fire anim – in the Essentials Pack) was around 100mb in size. With our first run of 10 animations, we managed to add 1GB to the Essentials Pack.
Building content – Pack Creator
We needed a way to easily create packs. Given the nightmare that the Essentials Pack was to build, the Pack Creator was conceived. Those who have used it extensively know the pain and anguish it takes to arrange a sizeable content pack, but believe it or not it was a large step up from the previous tools we had. At least with this one, you could actually SEE what was in a pack, could modify the properties of each content type, and you could even come back and edit it later! It still built everything into a single .rwpack file which needed to be fully loaded into RAM, but our computers could take it.
On the Toolkit front, The Launcher would read this .rwpack file and generate the content and meta files, so as big as the .rwpack file was, it had no effect on Toolkit performance.
After Dante created the Wild Pack, it was quite clear that updates would need to be made. The default properties of an object could only be edited one at a time, and each object had to be individually added to a preset. Some simple quality of life updates were made to our private build, but it wasn’t until a year after the release of The Wild that our hand was forced.
Cartographer Upgrade
In late 2019, we released v0.3.1.123, and with it came a large rework of the Cartographer module. This rework completely changed the back-end of the Cartographer, unifying a lot of classes into one generic, modular class that different components would attach to to determine function. This meant that all map content could now have its hue and opacity adjusted, any placement type could be animated, and any content could be placed as any placement type. To facilitate this, Cartographer content had a third file type added, on top of the content/meta file system that had worked so well up until this point: the Tile Preset file.
Tile Presets
Tile Presets are essentially a meta file, containing information about the default placement properties of a given object. While these had traditionally been stored in the main content file, moving them out to the new preset file had one big advantage: Multiple preset files could point to the same object. For example, you could have a tree image, and multiple presets of that tree with different hue, saturation and brightness values to create a large variety of trees from the one base model, WITHOUT needing to duplicate the base model itself. Safe to say, this is an incredibly powerful feature, and it allowed users to be able to have even greater control of their content than ever before.
However, this radical change in content flexibility required a new system to manage it. Rather than try to upgrade the Pack Creator, a bug-ridden mess that really would only get worse (and to not have to continue duplicating code), we decided to build a new system directly into the Toolkit (ironically what we’d done in the first place), and the Content Library module was born.
The Content Library
The most recent iteration of our content management system, the Content Library module allows users to easily import and modify any content in the Toolkit. All content that a user imports is now stored in a MyAssets folder to distinguish it from pack-based content, and is effectively treated like a pack that the user owns (from a back-end, content management perspective). This also lets us roll out similar content in future packs as well, where we’ll be able to bundle maps, encyclopedia content, and any future types of content we end up supporting.
The Future
We don’t know if this will be the final version of content management for the Toolkit, but it’s definitely the best we’ve come up with so far. each iteration improves things significantly, and we’re excited to see how it evolves!