• Teilen
  • E-Mail
  • Einbetten
  • Gefällt mir
  • Speichern
  • Privater Inhalt
SCons an Introduction
 

SCons an Introduction

on

  • 1,168 Views

SCons is a software construction tool (similar to “Make”) that we use at Slant Six to drive our data build pipeline. Here is an introduction to it.

SCons is a software construction tool (similar to “Make”) that we use at Slant Six to drive our data build pipeline. Here is an introduction to it.

Statistiken

Views

Gesamtviews
1,168
Views auf SlideShare
1,168
Views einbetten
0

Actions

Gefällt mir
0
Downloads
8
Kommentare
0

0 Einbettungen 0

No embeds

Zugänglichkeit

Details hochladen

Uploaded via as Adobe PDF

Benutzerrechte

© Alle Rechte vorbehalten

Report content

Als unangemessen gemeldet Als unangemessen melden
Als unangemessen melden

Wählen Sie Ihren Grund, warum Sie diese Präsentation als unangemessen melden.

Löschen
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Ihre Nachricht erscheint hier
    Processing...
Kommentar posten
Kommentar bearbeiten

    SCons an Introduction SCons an Introduction Document Transcript

    • SCons: an introduction Dean Giberson <dean@deangiberson.com> Dec 17, 2008 If youve tried putting a build system together for a modern game you knowthat you are facing a monumental, never ending task. Once you do get a basicpipeline working, modern games require source assets that are in the range of100 Gb of raw data. Tracking this amount of data takes time and resources. SCons (http://www.scons.org) is a Python based make replacement; withit you can tame your dependencies and data sizes. What follows is quick artcentric introduction.1 The Canonical ExampleThe SCons documentation1 would give an example like this: env = Environment() env.Program( HelloWorld, [HelloWorld.cpp]) follow this and provide a compilable HelloWorld.cpp and you would have aprogram ready to go. But how would you extend this to use your own pipelinetools?2 Adding Compressed TexturesLets assume that you want to compress textures for use in a game. NVidiaTextures2 Tools can be used on your TGA/PNG/JPG/DDS images. For exam-ple you want to convert this image3 into a DXT5 compressed image for use ontarget: nvcompress -color -bc3 gunmap.png gunmap.dds 1 http://www.scons.org/doc/0.97/HTML/scons-user/book1.html 2 http://developer.nvidia.com/object/texture_tools.html 3 Images provide by G3D Engine Data Collection http://g3d-cpp.sourceforge.net/. 1
    • Once we understand how to do this from the command line adding it toSCons is quite easy. import SCons env = Environment(ENV = os.environ) env[NVCOMPRESS] = nvcompress env[NVCOMPRESSTYPE] = SCons.Util.CLVar(-color) env[NVCOMPRESSFLAGS] = SCons.Util.CLVar(-bc3) env[NVCOMPRESSCOM] = $NVCOMPRESS $NVCOMPRESSFLAGS $SOURCE $TARGET NvCompressAction = Action( $NVCOMPRESSCOM) NvCompressBuilder = Builder( action=NvCompressAction, suffix=.dds) env[BUILDERS][NvCompress] = NvCompressBuilder env.NvCompress( image, image.png) 2
    • 3 A Quick walk-through to understand what is happening import SCons We import the SCons python module into our script. Normally you donthave to do this, but Im using the SCons.Util.CLVar class provided by SConsso I import the module to gain access to it. env = Environment(ENV = os.environ) Construct a SCons Environment object. SCons does not copy the systemenvironment by default, this is by design as a build environment should be asexplicit as possible. For now Ill just copy the system environment, but pleasenote that this is bad practice. env[NVCOMPRESS] = nvcompress env[NVCOMPRESSTYPE] = SCons.Util.CLVar(-color) env[NVCOMPRESSFLAGS] = SCons.Util.CLVar(-bc3) env[NVCOMPRESSCOM] = $NVCOMPRESS $NVCOMPRESSFLAGS $SOURCE $TARGET Now begin populating the Environment with data. Environment objectsare class instances that behave like dictionaries. We can add to them throughassignment, and query the contents using normal Python dictionary functions. In our case Im lling four slots with: • the name of the executable (NVCOMPRESS), • the default type ag (NVCOMPRESSTYPE), • the default compression ag(NVCOMPRESSFLAGS), • a template command line (NVCOMPRESSCOM). SCons uses late binding of variables found within environment strings, any sub-string that starts with a $ character is interpreted at the calling site for the current value within the current environment. This acts like a controlled dynamic scoping system. Ill come back to what this means in practical terms in a moment, but for now accept that our default command line evaluates to: nvcompress -color -bc3 $SOURCE $TARGET . 3
    • NvCompressAction = Action( $NVCOMPRESSCOM) NvCompressBuilder = Builder( action=NvCompressAction, suffix=.dds) These three lines are the core of our SCons extension; an Action object iscreated, and the command line string is set. The evaluation of this commandfollows the same rules as for environment variables set earlier. Then a Builder is constructed using the Action object just created. We alsoset the default extension for all targets. env[BUILDERS][NvCompress] = NvCompressBuilder The Environment is then extended with this builder, and a name is givenfor calls to it. env.NvCompress( image, image.png) The only thing left is to construct a Node in the dependency tree for thetarget le. SCons is a three stage process: 1. Create or gather a collection of tools and set the environment, 2. Create a tree of dependencies with sources and targets, 3. The dependencies are scanned, then for any out of date data, the action is called. Each of these steps is discrete, and must be occur in this order. Step 1 happens during the setup phase of SCons (default tools are scanned for and basic environments are constructed) and to a lesser extent within the running SConstruct script itself (as Ive just shown). Step 2 happens within the SConstruct script and continues until the end of the script. After control returns to the SCons system the Step 3 begins and the dependency tree is scanned and the actions are triggered. Always in this order. Its for this reason that I say that a dependency node is constructed from the last line of the SConstruct. It doesnt actually run nvcompress at this point. Only an object, representing the potential to run nvcompress, is constructed and added to the system. Work is done later by an internal SCons class, Taskmaster. 4
    • 4 More les, more controlsOne of the great things about SCons is that its embedded with Python. Aconsequence of this choice is that you, the tool writer, have access to all ofPythons functions. Ive shown you how to add one Builder and access that Builder for a singletarget texture. Games are not made from one texture, youre going to wantaccess to many textures and that means several targets. Glob to the rescue. from glob import glob for tex in glob( r./*/*.png): target = tex.replace(.png,.dds) env.NvCompress( target, tex) This will nd all of the source textures in subdirectories, and add them tothe dependency graph. Any textures that are added are found automatically. The same strategy can be applied to other types of data as well. Pythonhas great libraries for XML, SQL, even direct in memory structure access orpeeking in compressed les. You will not have to drop out of Python for verymany reasons. Check the module documentation4 for details.5 Other compression typesThe previous method of adding dependencies will add each texture into thepipeline with the same compression options. Normally you will want a rangeof compression methods for textures. I want to focus on the clearest method,changing the values in the Environment. I mentioned earlier that the evaluation of strings set for Builders and Actionsacts like dynamic scoping for variables. This feature allows us to change thefunctionality of a call by changing values when the dependency node is built. env.NvCompress( image2, image.png, NVCOMPRESSFLAGS=-bc2) Which will result in this command: nvcompress -color -bc2 image.pngimage2.dds. Adding a method to match lename patterns in a database (or text le)gives us a simple way to control the compression of individual textures. # Global texture compression options 4 http://docs.python.org/modindex.html 5
    • # format glob,opts .envmaps*.png,-bc1 # Does not include cubemaps .nmaps*.png,-bc3n This simple text le has a line for each le pattern, a comma (,) andthe compression option for the command line. Comments start with a hashcharacter (#) and continue to the end of the line. A parser for this format iseasy to write. from glob import glob from fnmatch import fnmatch gCompressionOptions = [] f = open(texture_options.txt,rt) try: for line in f: line = line.split(#)[0] if line != : (pattern,options) = line.split(,) gCompressionOptions.append( (pattern,options)) finally: f.close() for tex in glob( r.**.png): hasCustomPattern = False target = tex.replace(.png,.dds) for pat,opt in gCompressionOptions: if fnmatch(tex,pat): opt = opt.strip() env.NvCompress( target, tex, NVCOMPRESSFLAGS=opt) hasCustomPattern = True if not hasCustomPattern: env.NvCompress( target, tex) Once we have the patterns into an array its simple to check if any les foundby the glob matches a given pattern. If there is a match, set the compressionoptions for that texture. If not the default is used.6 Exploring dependenciesA core strength of SCons is its dependency system. This system is not normallybased on time stamps but on a one way hash of the le contents (MD5). Usinghash values allows for stronger connections between assets. 6
    • In order to follow what SCons is doing with dependency checking you canuse the debug=explain command line option. This option will print outinformation about dependency checks and commands being run. The interesting thing about using hash values for dependency check is thatyou cant use touch to force a recompile of an asset, you must change thecontents of the le or force SCons to rebuild an asset. On the ip side of this, you get control of the derived le from both thecontents of the source les and the contents of the command line used to buildthe derived le. SCons combines all of these values into a string that representsthe derived target asset, if any sources have changed that targets action isinvoked. To get a view of the dependency tree use the tree=derived option. scons: Reading SConscript files ... special .envmapsgunmap.dds .envmapsgunmap.png -bc1 special .nmapsgunmap.dds .nmapsgunmap.png -bc3n normal .texgunmap.dds .texgunmap.png scons: done reading SConscript files. scons: Building targets ... scons: `. is up to date. +-. +-envmaps | +-envmapsgunmap.dds +-nmaps | +-nmapsgunmap.dds +-tex +-texgunmap.dds scons: done building targets.7 Sharing the resultsStrong dependency systems are great for personal development, you can be surethat you only need to build the minimum following a change. This doesnt helpon a large team; if an artist makes a change then once submitted every personon the team needs to build that same asset. In my case building a compressedtexture takes 9.5 seconds, if you have 20 other people on your team, you teamwill spend 190 seconds for each texture change. The odd part of this result isthat every person is trying to get the same result from the same source. SCons took the strong hash key info and took it to the next logical level. Ifeveryone calculates the hash the same way, then its possible to store copies ofthe result in a shared location under that name. Then when you go to build, do 7
    • a quick check rst. If the asset exists under that name in this cache then justcopy it; if not then build the asset and place it into the cache under its hashname. The result is a distributed build cache. And you can take advantage of itout of the box. Create a location with a lot of available disk space, available foreveryone on your team, some server. Next place this line into you SConstructle. env.CacheDir(rx:Locationofcachedir) Now your derived les are shared, and only one person, normally the origi-nator of the source art, needs to build the nal result. In most cases, this saveshours of cumulative time for a large team of people. You results will be betterif you have a continuous integration server for art.8 The script we haveTaking all of these changes into account we get the following script. import SCons env = Environment(ENV = os.environ) env.CacheDir(rx:Locationofcachedir) env[NVCOMPRESS] = nvcompress env[NVCOMPRESSTYPE] = SCons.Util.CLVar(-color) env[NVCOMPRESSFLAGS] = SCons.Util.CLVar(-bc3) env[NVCOMPRESSCOM] = $NVCOMPRESS $NVCOMPRESSFLAGS $SOURCE $TARGET NvCompressAction = Action( $NVCOMPRESSCOM) NvCompressBuilder = Builder( action=NvCompressAction, suffix=.dds) env[BUILDERS][NvCompress] = NvCompressBuilder from glob import glob from fnmatch import fnmatch gCompressionOptions = [] f = open(texture_options.txt,rt) try: 8
    • for line in f: line = line.split(#)[0] if line != : (pattern,options) = line.split(,) gCompressionOptions.append( (pattern,options)) finally: f.close() for tex in glob( r.**.png): hasCustomPattern = False target = tex.replace(.png,.dds) for pat,opt in gCompressionOptions: if fnmatch(tex,pat): opt = opt.strip() env.NvCompress( target, tex, NVCOMPRESSFLAGS=opt) hasCustomPattern = True if not hasCustomPattern: env.NvCompress( target, tex) This will build all of your textures found in sub directories, using the com-pression options found in the texture_options.txt le, and share the resultswith colleagues using a shared drive. I hope you see how easy it is to build a strong build system for art given theright tools. SCons is not just for code, but can be used for art builds as well. Coupledwith a continuous integration server, having a quick, robust pipeline is withinyour grasp. 9