Waterfall Isn’t Going Away
But We Can Get Better At It
The Great Wall Of Audio
In 2016 I was internally offered a role as an “Associate Development Manager” at Creative Assembly, marking the end of my QA career, and the beginning of my production journey. My first responsibilities were managing the VFX, Audio, and localisation pipelines on Halo Wars 2. All of these teams are at the very end of the development sequence, and are never in control of the order of the work to come. As we started to approach Alpha we were shown the release roadmap, with the initial release coming first, then another seven releases, including two additional campaigns. This would have been mid 2016, and we already had a detailed schedule leading up to September 2017! We were going to have to deliver on time, every month, without missing a beat as work moved from each department to us.
This was a lot to take in, and I felt that we needed a clear visual representation of this year of lock-step development, and an opportunity promptly arose. We had just moved into a new building in Horsham, complete with floor-to-ceiling whiteboard walls on which we could draw, and one of these walls was right by our audio team. I took over this wall wholesale, drawing out dates along the top and features down the side, turning it into a seven-foot tall development calendar. We then started adding all the feature releases as post-it notes and then worked backwards, adding dependencies like recording sessions, mixes and dialogue drops. We would have our stand-ups in front of this every day, where we could point out immediately if a problem was being created several months down the line, allowing me to run off to the production team and flag up a warning.
This is the most literal vision of waterfall management I have ever built. It was beautiful, I tell you. I wish I had a photo.
Err Yeah…Nice Wall, Mate. But Why Waterfall? Why Not Agile?
That’s the interesting thing; we were working in Agile…sort of. We were following a lot of SCRUM procedures, like running our project in two-week sprints, planning at the start and reviewing/retrospecting at the end, and the production directors said they wanted us to work in an agile framework. However, this is miles away from “pure” agile in that we are deciding our work months in advance, instead of choosing at the beginning of the sprint. Moreover, our team is not making independent decisions on the product increment, but simply applying our craft to the work passed to us from the department before, and prepping it for the department that follows.
One of the core aspects of Agile is “inspect and adapt”, and when you inspect a AAA product like Halo Wars 2 what becomes clear is that the product itself necessitates waterfall thinking. The reason for this is that delivering a product increment (even a single soldier) requires a sequence of large, high-fidelity tasks to be completed, in order, by multiple departments. In order to plan for any kind of content delivery for a game like this, you will need to map out the dependencies.
Upstream And Downstream: The Reality Of Dependencies
Making a large game with a lot of content is more like a production line than an independent agile team; a lot of content needs to be made, the results need to be consistent in quality and style, and it will take a lot of different disciplines acting in sequence to produce it. A feature that is designed in January may not reach the audio team until September, having passed through concept art, 3D art, animation, scripting, VFX and lighting in the intervening months. The exact sequence of teams will differ between studios and features, but the fundamental chain of dependency generally follows this order for game content (which you can remember with the acronym “O.B.A.M.A.”...I guess…):
Objective - A feature needs a brief, often from a director or area lead. Examples could be “a gun that shoots water balloons” or “a level set on a lava moon.”
Behaviours - The brief will then be broken down into specific behaviours, such as how fast a weapon fires, how the projectiles travel, what they do when they hit the ground, and the effect they have on enemies. This will also include the required player inputs and feedback. This generally involves design, and sometimes code. In a level, the behaviour can be considered the shape of the environment, such as sightlines, cover, and vertical advantages.
Appearance - Once the behaviours are defined, the artistic teams will then need to define the visual treatment. This usually starts with a concept artist proposing variants to directors, until they sign off on a chosen look. In a 3D game, this will then go to a 3D modelling team to create and place the final models.
Motion - Once the behaviours and appearance are locked down, any visual motion will need to be authored. This step generally includes animation and VFX teams. In an environment, this is also where you would do your lighting work.
Audio - Any sound effects will need to follow all of the above, as they will need to communicate the behaviours, in respect to the appearance, and in perfect synchronisation with the motion, to deliver on the objective.
It’s very common for game developers to use the words “upstream” and “downstream” to refer to the teams that come before and after them in the working order, respectively. Producers need to work with their teams to map out these dependencies, and build schedules that accommodate the required sequence, working back from the delivery date, figuring out the required deadlines for each team, making sure to leave room for downstream teams to do all of their work. A common visual method for this is the Gantt chart:
In the Gantt chart above you can see that each team works on a level, passes it to the next one, then begins their next task, much like a production line in a factory. In this case, each team only works on one feature at a time, and we can see that each level takes several months to get from design to audio. We can also see that environment art takes triple the time, so they require triple the team size to balance the throughput.
The notion of these dependencies is fairly intuitive, but executing this at a project level is easier said than done…
Estimates, Bottlenecks and Iterations: The Challenges of Waterfall
Much like in real-life, riding down a waterfall is not without risks. One clear issue is that the entire model is based on a huge amount of estimated work. Estimates are notoriously difficult to give accurately, even on a project with experienced developers and a wealth of prior data, there is no guarantee that a feature will take the same amount of time as it did last time, and that uncertainty is multiplied many times by the amount of work and teams. For example, you may estimate that you can deliver ten levels in four years, but if each level takes 10% longer than you estimated, then you will finish that four year stretch with only nine finished levels or - perhaps worse - ten unfinished levels that each need another five months to complete! All it takes for this to occur is for your 10-day tasks to take 11 days instead.
Another challenge of waterfall environments is the presence of “bottlenecks”, where one part of the team has a lower throughput than the others, so all the work ends up piling up on them, while the teams downstream of the bottleneck can find themselves running out of work, losing time and money. Ideally you will have structured your teams to have equivalent standard throughput, but these bottlenecks can come from anywhere; sick staff and technical issues will reduce a team’s velocity, and a lack of information leads one team to estimate less than they need, or maybe one team lead takes much longer than expected to review and sign-off on work, preventing the team from handing off the feature to the next department. The Gantt chart makes everything look so tidy, but the downside is that anyone’s problem becomes everyone’s problem.
One other challenge may not be obvious when looking at the Gantt chart, but will be familiar to any experienced game developer; iteration. Games are nigh-impossible to build perfectly on the first try. The player’s experience is such a complex amalgamation of different game elements that it is almost certain that something will need rework once the game is fully built. Generally these iteration calls start from a director playthrough, where someone with product authority can declare that something isn’t good enough. This could be anything from changing a single line of dialogue to re-working the layout of a whole multiplayer level. If we think back to our OBAMA framework, an iteration on a feature Behaviour will most likely require work on the Appearance, Motion and Audio as well, so it can take a long time to action and validate that iteration, and the cost of each iteration makes this very normal part of game development highly risky.
Despite these challenges, we know we are in a dependency-based environment. So how can we handle this properly?
What Does Good Waterfall Management Look Like?
The way to figure out a good waterfall is to map out the ideal scenario; estimate the work, set up the hand-off processes and draw up your Gantt chart to lay it all out. That’s the ideal scenario, now what you have to do is figure out how to respond to disruptions. There is a project management methodology called PRINCE2 that offers a great wealth of tools for understanding this kind of project control, about which I am likely to write a separate article. But right now, let’s focus on the problems listed above.
A ritual that is very helpful in waterfall is known as a “backlog groom”, and this can help mitigate the risk of uncertain estimates. This is a regular meeting that a producer will have with a department lead where you look over the tasks and estimates in the backlog, and make changes based on anything you have learned from the project so far. This allows you to continually improve the accuracy of your estimates, and allows the producer to flag up if the re-estimated workload presents a schedule risk.
Regarding bottlenecks, we can find some tools from the world of Agile that can help us with bottleneck prevention. SCRUM can offer some useful processes for stabilising the day-to-day: a daily stand-up means that any problems found never wait more than a day before being discussed, and running a department retrospective every couple of weeks ensures that you get regular opportunities to hear the team’s ideas for improving development. Many teams in waterfall also like to use a Kanban board to give themselves a clear understanding of the work to come, and it can also make it easy for a producer to measure the task throughput and velocity. Additionally, you could consider asking your team regularly to suggest ideas for quality-of-life tasks (e.g. content polish, technical debt management, pipeline research etc) that could go into a separate list, which can be a secondary source of backup work in case the team is blocked by a bottleneck upstream.
Lastly, to handle the iteration challenges, you can do a number of things. Firstly, you should consult the team leads in pre-production about any iteration they are already anticipating, and specifically design pipelines to accommodate them. Common examples are damage values and ammo counts in weapons, environmental dressing, and text for in-game event communications. If these are known, you should try to architect the game so that these iterations will impact the waterfall as little as possible. Secondly, whatever your plan is, you should build in the assumption that there will need to be some major iteration. Most studios aim to build polish time for their teams, but it’s important to respect the dependency chain for these polish items. Consider your longest iteration line (likely a level) and assume you may need at least one re-do, from “Behaviour” all the way to “Audio”.
Another element of managing iterations is making sure that calls for iteration are informed decisions. When a director wants to change something, you should be able to quickly advise them what the change is likely to cost. Key thing here: don’t tell a director their idea is impossible! They aren’t asking your permission for their ideas, but they are interested in costs. The CEO at FundamentalVR would often follow up his asks with “I’m not asking yes or no, just tell me the cost, and I will decide.” In order to know this cost quickly, at all times, you will often need to have built a solid information database, with output designed specifically to answer this question. I have repeatedly built tools to achieve this, but I’m going to need another article to explain all that in detail…
The Jabberslythe And The Charlemagnes
Whenever I feel I need to challenge the universal application of agile game development the first example that comes to my mind is that of the Jabberslythe. This huge, multi-limbed, multi-headed monster is one of the flagship creatures of the Beastmen faction in Warhammer, but the Total War team omitted it from the initial release of the Beastmen faction, citing that it was too expensive, and that making this one unit would cost the same as doing all the art for the entire Charlemagne release (an earlier DLC for Total War: Rome II). The community even adopted the word “Charlemagne” as a unit of cost for something expensive (put that in your story points!). It was a whole five years later when Creative Assembly finally unleashed the Jabberslythe, but I know from my contacts that it had taken six months alone just to animate the beast!
Some features are just so large that you will need to plan well in advance for their arrival, coordinating schedules so that everyone in the sequence can jump in at the right time, one after the other. When a product declares itself to be waterfall, your job will be to model that waterfall, expose the strained areas to the team, and guide it to the finish line.
Outside of his day job as a game producer, Doug Pennant is a writer and speaker with a dream of growing a healthier and more confident game industry.
Subscribe for weekly game production stories, tips and advice.
Great piece, Doug. The dependency chain framing maps cleanly onto what Lean calls a value stream, and your OBAMA sequence is essentially a value stream map waiting to be drawn. One thing Lean adds to the waterfall toolkit that doesn't get enough attention in games is the distinction between cycle time and wait time. In most AAA pipelines, the majority of elapsed time on any asset isn't work time. It's queue time between handoffs. The bottleneck problem you describe is real, but the bigger silent killer is assets sitting idle waiting for the next department to pick them up. Takt time thinking (matching each team's output rate to the downstream team's intake rate) can help teams design their capacity to reduce that accumulation before it becomes a crisis. The wall you built at Creative Assembly was essentially doing this manually and visually, which is exactly what Lean practitioners would recommend. The Jabberslythe is a perfect example of a high-WIP, low-flow feature, and the five-year wait for it suggests nobody had sized the constraint correctly at the start.