1. Data Massage Therapy:Pipelines, SCons and SOCOM: Confrontation‘November 18th 2008
2. IntroductionWho Are We? Who are you?
3. IntroductionWho Are We?Dean Giberson:• Started out in 1999 as tool developer for 4 years working on several StarWars titles• Currently Sr. Tools & Pipeline Engineer at Slant Six Games• Specialist areas; data build management and pipelines, …• After a short period teaching and contracting he returned to professional game development working on Sports titles for 3 years• Credits: – NBA Live 08 (2007) Xbox 360/PS3 – NBA Live 07 (2006) Xbox 360/PS22/XBox/PC – NBA Live 06 (2005) PS2/XBox/Gamecube/PC – Midway Arcade Treasures 2 (2004) Gamecube – Rogue Squadron 3: Rebel Strike (2003) Gamecube – Rogue Squadron 2: Rogue Leader (2001) Gamecube – Battle for Naboo (2000) N64Paul Martin:• Started out in the industry 1996 graphics programmer for PlayStation• Currently a Technical Director and one of the principals of Slant Six games• Specialist areas are in graphics rendering, special effects and data build pipelines.• Credits: – SOCOM: Tactical Strike (2007) PSP, Slant Six Games – Syphon Filter: Dark Mirror (2006) PSP, Sony Computer Entertainment America, Inc. – Syphon Filter: The Omega Strain (2004) PS2, SCEA – Sled Storm (2002) PS2, Electronic Arts, Inc. – SSX (2000) PS2, Electronic Arts, Inc. – Need for Speed III: Hot Pursuit (1998) PS1, Electronic Arts, Inc. – [bunch of unreleased stuff prior to this…]
4. IntroductionWho Are We?Slant Six Games – Intro:• Slant Six Games is one of the largest video game development studios within the Vancouver game development hub. We were established in February 2005 and specialize in developing games for the PlayStation 3 and PlayStation Portable.• We currently have 110 full time staff and occupy 20,000 square feet in our studio located at the mouth of Granville Island in Vancouver, British Columbia.
5. IntroductionWho Are We?Slant Six Games – Our Games:• First project was the creation of the graphics rendering engine for Sony Computer Entertainment America’s (SCEA) Syphon Filter: Dark Mirror (developed by Sony Bend, Oregon).• Inaugural title, SOCOM: Tactical Strike, published by SCEA, in October 2007. Was subsequently named the Best PSP Multiplayer Experience at E3 2007 and in February 2008 it was awarded the Elan Award for the Best Handheld Game of the Year. To date it has sold over 400,000 copies worldwide. Also in February 2008 we were awarded the prestigious Elan Award for Best New Video Game Company.• Just completed development of our largest and most significant project to date, SOCOM: Confrontation, for the PS3, in late summer of 2008. SOCOM: Confrontation has been in production for just under two years and was one of the more highly anticipated titles of this fall.• Currently working on • Shipping SOCOM: Confrontation in other worldwide regions • Unannounced PSP title for SCEA which is scheduled for completion in June of 2009 • Expansion packs and other new downloadable content for SOCOM: Confrontation on PS3 • Another unannounced new concept game for PS3
6. IntroductionWho Are You?• Programmers/Technical Artists/Bunch of other people, hopefully interested in how to build an efficient data build pipeline for current/next generation game content• In the wrong room?
7. Data Build System Requirements• Fast!• Correct• Extensible• Debuggable• Asset management integration
8. Our Data Builder SystemFeatures:• Dependency Tracking• Shared Build Cache• Tight integration with Perforce• Partial Builds (build only: Textures, Collision, World Descriptors, etc)Built on top of Scons…
9. SConsWhat is it?• From SCons.org: SCons is an Open Source software construction tool—that is, a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.Where did it come from?• Also from SCons.org: SCons began life as the ScCons build tool design which won the Software Carpentry SC Build competition in August 2000. That design was in turn based on the Cons software construction utility. This project has been renamed SCons to reflect that it is no longer directly connected with Software Carpentry (well, that, and to make it slightly easier to type...).
10. SConsWhy are we using it?…back to Data Build System Requirements• Fast!• Correct• Extensible• Debuggable• Asset management integration
11. SConsPros:• Robust• Reliable• Rich Scripting Interface (Python!)• Shared Data Caching• Multi-processor support
12. SConsCons:• Phase 1 dependency & job building can be slow• Core built on non-trivial Python code • Can make non-standard extensions to the system tricky to implement (e.g. dependency insertion, dependency pruning)
13. SConstructWhat is it?• Equivalent of a Makefile• The input file that SCons uses to control the build• A Python script. With a one main difference • Definition order of SCons builder functions, does not dictate execution order • Execution order of non-builder functions is as expected• Two-step processing • Regular Python (non-SCons Builder) functions first • Then SCons builders invoked as deemed necessary by the dependency engine
15. Pretty It Up – Use a GUI!Nature abhors a vacuum
16. Pretty It Up – Use a GUI!Artists and Designers abhor a command-line interface
17. Pretty It Up – Use a GUI!Undesirable Desirable
18. Pretty It Up – Use a GUI!
19. Not all roses!• Staged building (the graph problem)• Maya• Large trees have large dependency graphs
20. Not all roses! Staged building• You can’t know what Maya files to build until you have scanned your level files.• You can’t know what textures to process until you have scanned your Maya files.• You don’t know what assets to place into you ZIP until you have everything else first.
21. Solution. Staged building• We manually scanned the level files using Python XML• Texture build can still be a problem• ZIPs get rebuilt if there is any change• We have alpha level code for staged building for automatic LOD generation, but it was not exercised enough before we changed LOD methods
22. Not all roses! Maya• We use MBs as our source file format for 3D art.• You can only have one instance of Maya running (and keep it reliable).• Maya can take 30 seconds to launch.• Bottleneck!
23. Solution. Maya• Going to use gzip MAs, and write a text scanner for texture links, or write a simple MB scanner for textures• SN-DBS MayaBatch• Stripping Maya’s default launch scripts can get ~3 second launches• Still bottleneck!
24. Not all roses! Big tree• Building the dependency tree for the whole game could take 1.5 minutes for all assets• MD5 sum of many files can be slow• Artists that only want to change one texture (a 2 second problem) should not have to wait on this• But it works great for build servers• Balance?
25. Solution. Big tree• Internally trim the tree for some assets based on type• ResourceBuilder tells SCons the sub portion of the tree the artist is interested in and we only build that graph• New versions of SCons have the `-- interactive’ flags to scan the tree once and let you run multiple commands on it• Balance
26. Not all roses! Other platforms• Make XNA Ant Cmake YABS• All have similar problems• SCons’ MD5 checking gives best results so far• …and Python script, not <gripe> XML </gripe>
27. The rebuild story
28. SOCOM: Confrontation
29. SOCOM: ConfrontationSome Stats:• Source data total size: 200 Gb• Target (PS3) data total size: 3 Gb• World models for 1 world:49• Unique prop models for 1 world: 167• Prop instances for 1 world: 1901• Unique textures for 1 world: 1209• Lightmaps for 1 world: 1581 (x2 – day+night)
30. What Next?• Debugger• Staged building• Link to SN-DBS• Code?