Technical Debt is a handy metaphor that has been used in the programming community for awhile now. I sort of think it’s one of those things that probably got coined by somebody in the Lean Startup community, or the Agile community, or something like that. The idea of the metaphor is simple: you incur technical debt at the start of a project for a variety of reasons – getting your software started or out the door, for instance; or meeting a release target. At some point, however, the debt collector comes calling. This week, I seem to be paying my debts off: in the case of Clockwork Empires, we incurred a lot of technical debt getting the game into Early Access, with the knowledge that at some point during the process we would have to pay it off in order to get out of Early Access. Well, the point of paying it off seems to be this week.

Let’s examine some of our debts:

• The speed of loading save games. Save games save very quickly, but they load very slowly. In a previous version of the game, I switched how what the save system calls “nodes” were stored, from an STL container (bloated!) to a simple list in order to save memory and ensure that the game could load without crashing due to running out of memory. We fixed that, but the cost of the linked list approach is, of course, that load times are slower. A new game, saved, and then loaded takes two minutes on my machine. Pretty rough. What motivated me to fix this was a user commenting that typically when the game was loading a save game was when he’d go get stoned.

  Well, that’s one form of user feedback…For 49C my first thought was to implement a balanced binary tree to speed up the search for nodes with specific names (where the bottleneck lies.) I spent Friday doing that, to not much success; in fact, it made things slower. I threw that out, and tried something else: just getting rid of a bunch of the general case stuff that went through the save system and should, probably, have been saved as binary files. Much faster. Now saves load in 30 seconds or so on my machine. Still not fast enough, though. The next step is to eliminate the amount of time we spend allocating and deallocating memory in the loader – doing one trip to and from the allocator every time we load an individual object is slow, especially when we know at the start of the load time how many objects we saved. Now we allocate all of that at the start of the program. After much running about figuring out where the heck all the allocations and deallocations in the load/save code actually were, save game performance is now hopefully much better with none of the memory hiccups of previous versions.

  “The first living thing to go through the device was a small white rat. I still have him, in fact. As you can see, the damage was not so great as they say.”

• Memory usage in general. For some reason, our new biome code blows the roof off of our memory footprint. In the process of auditing this, I made two unpleasant re-discoveries: every object in the world allocates its own tooltip, and those tooltips took up a lot of memory. I mean a lot. 300 megabytes worth. Well, that’s getting killed off; we now have one tooltip per type of interactive object, and can share them between them. (You figure out which tooltip you’re using for your string using a hash table. I don’t use hash tables enough and I should!)
  

  “What actually transpires beneath the veil of an event horizon? Decent people shouldn’t think too much about that.”

• I also discovered that we were allocating 150 megabytes for a largely empty data structure that is mainly used to find the top of the world. Well, that can be sparsified, and that will save some memory as well.We also allocate memory for some data structures on rendered models that is never used anywhere (edge adjacency information.) Bye-bye.
  

  “No matter how beautiful the theory, one irritating fact can dismiss the entire formulism, so it has to be proven.”

• Glow is broken. This is important for a thing David is doing. It got broken when the FBO format was changed around for the multipass rendering for the… uh, umpteenth time or so. Less than ideal. Lights cannot be attached to static props. This is important for standalone things that you might want to cast light, like lamps. This also needs to be fixed with some expediency.
  

  “Humans are natural-born scientists. When we’re born, we want to know why the stars shine. We want to know why the sun rises.”

And the list goes on and on. This is all stuff that has worked well enough up until Revision 49, and frankly could probably keep working for a bit longer; however, it’s stuff that needs to be fixed so we can get out of Early Access and actually ship the full version of the game. It’s not stuff that is necessarily *directly* obvious to the user, but it does exhibit itself as various symptoms. A lot of the time, this sort of technical work is invisible even to members of the same team as a systems programmer, but it does pile up and it does need to be done and done well.

“What we usually consider as impossible are simply engineering problems… there’s no law of physics preventing them.”

At the end of the day, nearing release, development becomes a question of “what things are still good enough to ship, even though they aren’t ideal programming practice?” That’s a fine and interesting line!

“The popular stereotype of the researcher is that of a skeptic and a pessimist. Nothing could be further from the truth! Scientists must be optimists at heart, in order to block out the incessant chorus of those who say ‘It cannot be done.'”