This presentation from MIGS 2008 explains how we use SCons to build game data from source assets.

1. Data Massage Therapy:
Pipelines, SCons and 'SOCOM: Confrontation‘
November 18th 2008

2. Introduction
Who Are We? Who are you?

3. Introduction
Who Are We?
Dean Giberson:
• Started out in 1999 as tool developer for 4 years working on several StarWars titles
• Currently Sr. Tools & Pipeline Engineer at Slant Six Games
• Specialist areas; data build management and pipelines, …
• After a short period teaching and contracting he returned to professional game development working on Sports titles for 3 years
• Credits:
– NBA Live 08 (2007) Xbox 360/PS3
– NBA Live 07 (2006) Xbox 360/PS22/XBox/PC
– NBA Live 06 (2005) PS2/XBox/Gamecube/PC
– Midway Arcade Treasures 2 (2004) Gamecube
– Rogue Squadron 3: Rebel Strike (2003) Gamecube
– Rogue Squadron 2: Rogue Leader (2001) Gamecube
– Battle for Naboo (2000) N64
Paul Martin:
• Started out in the industry 1996 graphics programmer for PlayStation
• Currently a Technical Director and one of the principals of Slant Six games
• Specialist areas are in graphics rendering, special effects and data build pipelines.
• Credits:
– SOCOM: Tactical Strike (2007) PSP, Slant Six Games
– Syphon Filter: Dark Mirror (2006) PSP, Sony Computer Entertainment America, Inc.
– Syphon Filter: The Omega Strain (2004) PS2, SCEA
– Sled Storm (2002) PS2, Electronic Arts, Inc.
– SSX (2000) PS2, Electronic Arts, Inc.
– Need for Speed III: Hot Pursuit (1998) PS1, Electronic Arts, Inc.
– [bunch of unreleased stuff prior to this…]

4. Introduction
Who Are We?
Slant Six Games – Intro:
• Slant Six Games is one of the largest video game
development studios within the Vancouver game
development hub. We were established in February
2005 and specialize in developing games for the
PlayStation 3 and PlayStation Portable.
• We currently have 110 full time staff and occupy
20,000 square feet in our studio located at the mouth
of Granville Island in Vancouver, British Columbia.

5. Introduction
Who Are We?
Slant Six Games – Our Games:
• First project was the creation of the graphics rendering engine for Sony Computer Entertainment
America’s (SCEA) Syphon Filter: Dark Mirror (developed by Sony Bend, Oregon).
• Inaugural title, SOCOM: Tactical Strike, published by SCEA, in October 2007. Was subsequently named
the Best PSP Multiplayer Experience at E3 2007 and in February 2008 it was awarded the Elan Award
for the Best Handheld Game of the Year. To date it has sold over 400,000 copies worldwide. Also in
February 2008 we were awarded the prestigious Elan Award for Best New Video Game Company.
• Just completed development of our largest and most significant project to date, SOCOM: Confrontation,
for the PS3, in late summer of 2008. SOCOM: Confrontation has been in production for just under two
years and was one of the more highly anticipated titles of this fall.
• Currently working on
• Shipping SOCOM: Confrontation in other worldwide regions
• Unannounced PSP title for SCEA which is scheduled for completion in June of 2009
• Expansion packs and other new downloadable content for SOCOM: Confrontation on PS3
• Another unannounced new concept game for PS3

6. Introduction
Who Are You?
• Programmers/Technical Artists/Bunch of other people,
hopefully interested in how to build an efficient data build
pipeline for current/next generation game content
• In the wrong room?

7. Data Build System Requirements
• Fast!
• Correct
• Extensible
• Debuggable
• Asset management integration

8. Our Data Builder System
Features:
• Dependency Tracking
• Shared Build Cache
• Tight integration with Perforce
• Partial Builds (build only: Textures, Collision,
World Descriptors, etc)
Built on top of Scons…

9. SCons
What is it?
• From SCons.org:
SCons is an Open Source software construction tool—that is, a next-generation build tool.
Think of SCons as an improved, cross-platform substitute for the classic Make utility with
integrated functionality similar to autoconf/automake and compiler caches such as ccache.
In short, SCons is an easier, more reliable and faster way to build software.
Where did it come from?
• Also from SCons.org:
SCons began life as the ScCons build tool design which won the Software Carpentry SC
Build competition in August 2000. That design was in turn based on the Cons software
construction utility. This project has been renamed SCons to reflect that it is no longer
directly connected with Software Carpentry (well, that, and to make it slightly easier to
type...).

10. SCons
Why are we using it?
…back to Data Build System Requirements
• Fast!
• Correct
• Extensible
• Debuggable
• Asset management integration

11. SCons
Pros:
• Robust
• Reliable
• Rich Scripting Interface (Python!)
• Shared Data Caching
• Multi-processor support

12. SCons
Cons:
• Phase 1 dependency & job building can be
slow
• Core built on non-trivial Python code
• Can make non-standard extensions to the
system tricky to implement (e.g. dependency
insertion, dependency pruning)

13. SConstruct
What is it?
• Equivalent of a Makefile
• The input file that SCons uses to control the build
• A Python script. With a one main difference
• Definition order of SCons builder functions, does not
dictate execution order
• Execution order of non-builder functions is as expected
• Two-step processing
• Regular Python (non-SCons Builder) functions first
• Then SCons builders invoked as deemed necessary by
the dependency engine

14. SConstruct
Some usage examples:
• Texture processing
• SSG (wad/pak/zip/rx2/etc) processing
• Tool Target Scanner (examples)

15. Pretty It Up – Use a GUI!
Nature abhors a vacuum

16. Pretty It Up – Use a GUI!
Artists and Designers abhor a command-line interface

17. Pretty It Up – Use a GUI!
Undesirable Desirable

18. Pretty It Up – Use a GUI!

19. Not all roses!
• Staged building (the graph problem)
• Maya
• Large trees have large dependency
graphs

20. Not all roses! Staged building
• You can’t know what Maya files to build
until you have scanned your level files.
• You can’t know what textures to process
until you have scanned your Maya files.
• You don’t know what assets to place into
you ZIP until you have everything else
first.

21. Solution. Staged building
• We manually scanned the level files using
Python XML
• Texture build can still be a problem
• ZIPs get rebuilt if there is any change
• We have alpha level code for staged
building for automatic LOD generation, but
it was not exercised enough before we
changed LOD methods

22. Not all roses! Maya
• We use MBs as our source file format for
3D art.
• You can only have one instance of Maya
running (and keep it reliable).
• Maya can take 30 seconds to launch.
• Bottleneck!

23. Solution. Maya
• Going to use gzip MAs, and write a text
scanner for texture links, or write a simple
MB scanner for textures
• SN-DBS MayaBatch
• Stripping Maya’s default launch scripts can
get ~3 second launches
• Still bottleneck!

24. Not all roses! Big tree
• Building the dependency tree for the whole
game could take 1.5 minutes for all assets
• MD5 sum of many files can be slow
• Artists that only want to change one
texture (a 2 second problem) should not
have to wait on this
• But it works great for build servers
• Balance?

25. Solution. Big tree
• Internally trim the tree for some assets
based on type
• ResourceBuilder tells SCons the sub
portion of the tree the artist is interested in
and we only build that graph
• New versions of SCons have the `--
interactive’ flags to scan the tree once and
let you run multiple commands on it
• Balance

26. Not all roses! Other platforms
• Make XNA Ant Cmake YABS
• All have similar problems
• SCons’ MD5 checking gives best results
so far
• …and Python script, not <gripe> XML
</gripe> 

27. The rebuild story

28. SOCOM: Confrontation

29. SOCOM: Confrontation
Some Stats:
• Source data total size: 200 Gb
• Target (PS3) data total size: 3 Gb
• World models for 1 world:49
• Unique prop models for 1 world: 167
• Prop instances for 1 world: 1901
• Unique textures for 1 world: 1209
• Lightmaps for 1 world: 1581 (x2 – day+night)

30. What Next?
• Debugger
• Staged building
• Link to SN-DBS
• Code?

31. References
• SCons: http://www.scons.org/
• Slant Six Games: http://slantsixgames.com
• SOCOM: Confrontation:
http://www.us.playstation.com/PS3/Games/SOCOM_U_S_Navy_SEALs_Confrontation

32. Talk to Us!
Paul Martin: paul@slantsixgames.com
Dean Giberson: dgiberson@slantsixgames.com
Slant Six Games
3rd Floor,
1523 West 3rd Avenue,
Vancouver, BC V6J 1J8
http://www.slantsixgames.com

33. Bye For Now!