How will you reduce bugs and ship your game on time? This is what we did. Annotated notes for C++ game programmers.

1. C++ Coder Meeting
Aug 17
1

2. I do not make fancy slides
2

3. C++ Coding Restrictions
C++ is a general purpose programming language
C++ provides features for many different hardware and software problems
Our game is a specific subset of problems
Our team is a specific set of individuals
We must agree a subset of the C++ language which is helpful to our
specific requirements
C++ 11, 14 and 17 standards expand the language to cater for an ever increasing set
of use cases. We must look at the modern C++ standards and decide which features
are a benefit and which features are a burden.
Game development has similar constraints to embedded system development. Our
primary constraints are CPU operation and memory.
By example
Game developers will disable C++ exception handling to avoid the CPU cost. We will
also use all available memory.
This is the opposite to general programming where CPU and memory plentiful.
When C++ new fails it throws an exception.
STL interfaces are designed using exceptions. Disabling exceptions compromises STL's
usability and will result in more bugs. Is it still sensible to use STL under these
constraints?
3

4. Coding Ethos
“Make it hard to do the wrong thing.”
Programmers will take the path of least resistance. Design your code and interfaces
so the correct way to use them is obvious and easy.
Assert if your code is not used as designed.
Assert if calling code is using bad patterns.
Complexity will always propagate through the code. A complex interface will increase
the complexity of the calling code. Which will in turn increase the complexity of other
code.
4

5. Public / Private Headers
• Goal
• Keep compilation speed fast.
• By reducing the number of header files parsed.
• By enabling unity builds.
• Rules
• Cpp files must only include a single pch file.
• H files must not include other .h files.
• Public PCH must only include headers from the current folder.
• Private PCH files explicitly include all dependencies.
Under normal development conditions (e.g. not using LTO) parsing header files will
account for >90% of the object file compilation time.
Enforcing strict rules for header file usage is the most effective way to keep build
iteration times fast.
The next significant factor in fast compilation is the number of compiler invocations.
The OS overheads is launching a new process for each .obj file are large. These
overheads are orders of size worse for distributed builds.
Unity builds as a simple and effective way to reduce compiler invocation and number
of head files parsed.
5

6. Public / Private Headers
• Code smell ;-(
• Changes which touch lots of private PCH files are a code smell.
• Private PCH files which are very large need reviewing.
• Code should build without using unity build.
Our strict header file rules have extra advantages.
The private PCH file explicitly list all external dependencies to the module (no
recursion).
At a glace it is easy to see which modules have a large number of dependencies.
Simply list *.pch by size!!
At a glance it is easy to see relevance of dependencies. ( e.g. Why does GameCamera
include AIBrain ?? )
Because header file recursion is not allowed. Adding new dependencies to modules
requires that each affected private PCH is edited.
This is an example of "make it hard to do the wrong thing".
If you find yourself editing 20+ PCH files stop and think. "Why am I adding a new
dependency to 20+ modules"
6

7. Memory Architecture
• Goal
• Avoid time wasted on difficult to reproduce bugs due to memory usage.
• Avoid expensive alloc/free in game loop.
• Design for efficient cache friendly allocations.
• All memory failures to be during load and 100% reproducible.
• Never fail on pool allocations during game loop.
• How
• Memory context explicitly define lifetime of memory allocations.
• Separate memory allocation lifetime from object lifetime.
"Make it hard to do the wrong thing."
Construction Phase.
Calls to new/alloc are allowed. Calls to delete/free will assert.
Preventing calls to delete/free prevents temporary allocations leaving unusable holes
of free memory.
All memory allocations must happen in the constructions phase. Memory budgets are
asserted at the end of the Construction phase.
Running Phase.
Calls to new/alloc will assert. Calls to delete/free will assert.
This prevents allocating inside the main game loop.
Given memory budgets are asserted in the Construction phase, if we are Running
then we can run for ever without a memory exception.
Teardown Phase.
Calls to new/alloc will assert. Calls to delete/free are allowed.
This must mirror the construction phase. Zero allocated memory will be asserted at
the end of the Teardown phase.
7

8. Asserting new/alloc prevents early setup of the next phase which is a common cause
of bugs.
7

9. Clear Code
• Goal
• Main loop reads like pseudo code.
• Easy to understand.
• Easy to change.
• Easy to optimise.
• Rules
• No abstract base class at top level.
• Main game update directly calls functions with ‘highly readable names’.
• Top level function names read as sentences.
We have chosen not to use an abstract game component as a top level base class.
We decided to avoid a main loop which iterates through a list of abstract game
components. This can make execution order difficult to control and understand. Can
make optimising for multi-core difficult. Can hide dependencies between modules.
Instead we have a main loop which reads like a few pages of pseudo code. Highly
readable, easy to change, easy to optimise.
Descriptive function names means the main update is readable. Dependencies
between modules are clearly visible. Logical execution order is as written. Optimising
with tools like 'parrallel_invoke' is easy to experiment and iterate.
Previous game engine used top level abstract base class. This required complicated
supporting systems to manage execution order, dependencies and data access.
This is an example of complexity generating more complexity. By radically simplifying
the top level we are able to completely remove related complex systems.
8

10. Clear naming
• Goal
• Readable code without requiring the ‘tomato’ to navigate overloaded names.
• Rules
• Within our code base we should have unique naming.
• Namespaces and classes enable duplicated naming. Avoid.
• File scope using <namespace> obfuscates code. Avoid.
• Filename must be unique within our codebase.
When is it a good idea to have different things with the same name?
Avoid generic non-descriptive function names like update().
Do not use namespaces or classes to allow different objects, methods etc, with the
same name.
Do not have files with the same name in different directories.
9

11. Strong naming
• Overloading removes the compilers ability to check errors.
• Example 1
Stream.Write(brain.humour);
Stream.Write(brain.intelligence);
Stream.Write(brain.empathy);
struct brain {
int humour;
int intelligence;
int empathy;
}
Typical example of a stream class with polymorphic Write methods.
The compiler will pick the correct method from the types used in the call.
10

12. Strong naming
Page intentionally left blank.
<blank page to prevent visual comparison between previous and next slides>
11

13. Strong naming
• Overloading removes the compilers ability to check errors.
• Example 2
Stream.Read(brain.humour);
Stream.Read(brain.intelligence);
Stream.Read(brain.empathy);
struct brain {
int humour;
char intelligence;
int empathy;
}
This is the Stream Read counterpart.
Again the compiler will pick the correct method based on the calling types.
Can you spot the mistake ?
12

14. Strong naming
• Overloading removes the compilers ability to check errors.
• Example
Stream.Read(brain.humour);
Stream.Read(brain.intelligence);
Stream.Read(brain.empathy);
struct brain {
int humour;
int intelligence;
int empathy;
}
struct brain {
int humour;
char intelligence;
int empathy;
}
When you change the type from int to char the compiler will change the Stream
methods called. The program will compile without error.
Serialising an existing file with the previous types will result in a run time error. This
run time error could manifest in any manner of subtle and none obvious ways.
Resulting in significant time lost to QA and debugging.
13

15. Strong naming
• Allow the compiler to find your mistakes.
• Example
Stream.ReadInt(brain.humour);
Stream.ReadInt(brain.intelligence);
Stream.ReadInt(brain.empathy);
struct brain {
int humour;
char intelligence;
int empathy;
}
By choosing to have unique methods names we allow the compiler to find the error
immediately. Saving down stream QA and debugging time. The developer can then
plan how best to version and migrate the file format.
In large teams this is an significant issue. In large teams it is likely the person changing
the type is not the only developer using the type. Another team member might be
the owner of file serialisation. These classes might be in common code and used in
another application.
Neither develop has perfect knowledge of the code. Let the compiler find the errors.
14

16. Action at a distance
• Do not use ‘auto’
• Auto askes the compiler to make assumptions about type.
• Auto removes the compilers ability to find mistakes.
• Auto removes the programmers ability to see mistakes.
• When using auto it is easy to ‘copy by value’ instead of ‘reference’.
auto has the same problems as polymorphic function overloading.
auto enables action at a distance in code you have no knowledge of.
auto is a example of complexity leading to more complexity.
A common use case for auto is typing-convenience for long and complex type names.
An example is the STL iterator types. The real issue is complicated and difficult to use
type names. Using auto is masking complexity behind greater complexity.
Tackle the real issue. Design your types to be easy to use. And easy to type!
15

17. Anti-patterns
• Patterns which are statistically more likely to cause bugs.
• We should avoid these patterns.
• even if in your use case it will be fine ☺
Specifically, programming patters which tend to result in difficult and costly bugs.
Bugs which take disproportionally large amount of engineering time away from polish
and finishing. Bugs which make your game late!
See the appendix for background information.
16

18. Anti-patterns - Callbacks
• Unscoped callback (usually notify event), Just no.
• Results in ‘callback from space’ at unexpected times.
• Callback lifetime and scope decoupled from client code.
• Pushes complexity to client code.
• Give calling code control over timing and threads
• Buffer data inside API to remove the need for callback.
• Prefer polling to give client control. ( e.g. GetMyEvents() )
• Use invoke callback if necessary. ( e.g. DoMyCallbacksNow() )
Long lived unscoped callbacks are a cause of high cost bugs. By using unscoped
callbacks you are saying.
"I will invoke this callback at an undefined time, on an undefined thread and you (the
client code) will have to handle it without error"
This is an unreasonable constraint and will lead to difficult to reproduce errors.
Ideally design systems to buffer data inside the API. Manage the complexity yourself.
You have the best knowledge of the problems and requirements.
Give the calling code control of when to get or put data.
A buffer full error will be far easier to debug than an 'callback from space' which
corrupts memory.
17

19. Anti-patterns - Lambdas
• Lambdas (and other delegates)
• Premature Generalisation.
• Overly generic interfaces do not give usage guidance to calling code.
• Use concrete functions and types to show intended use cases.
• Tends towards inefficient ‘one of’ usage.
Aside from the efficiency issues with Lambda capture allocating memory. Which is
reason enough not to use Lambdas.
The last time you used a lambda how many actual use cases where there? More than
50? More than 10? More than 2?
Lets not kid ourselves. We are making games. We are not making generic APIs for
unknown 3rd parties. We have the luxury of control over the entire code base.
Ask yourself. "Am I using a Lambda because there are going to be 10's or 100's of
valid use cases?" or "Am I using a Lambda because I do not fully understand the
requirements?". If you do not understand the requirements stop and figure them out.
Understand the use case of the systems and write the most specific, strongly typed
and strongly named function possible. In large teams no one has perfect knowledge.
Overly generic APIs force other develops to make assumptions. Assumptions which
will lead to errors.
A well defined interface will be self describing and "makes it hard to do the wrong
18

20. thing".
Examples of valid use cases for Lambda are parallel_invoke({}) and parallel_for({})
which provide highly readable parallelism.
18

21. Anti-patterns - Mutex
• Mutex and Semaphores
• Complicated logic which is easy to get wrong.
• Deadlocks and wait on resource errors.
• Difficult to reproduce bugs.
Find another way
• No threading primitives available for general programming.
• Redesign to avoid tight coupling of resources between threads.
• Single thread ownership of data.
Reasoning about concurrent thread is difficult and you will get it wrong. We abstract
usage of threads behind the game framework. We do not expose mutex, semaphore
or thread outside the API.
The game framework manages long lived tasks.
We use TBB for sub frame wide execution.
An over view of the Rust programming language is useful when talking about threads,
lifetimes and scope.
19

22. Stateful Logic
• Prefer to evaluate logic conditions every frame.
• Avoid transient logic as hard to reproduce and debug.
• Prefer explicit logic which is visible on the callstack.
switch(mystate) {
…
case badstate:
assert(“Irrational state error”);
break;
…
}
If(weight>100) {
if(num_legs==4) {
if(has_trunk==true) {
if(has_wings>0) {
assert(“Irrational state error”);
}
}
}
The switch statement on the left is a simple state machine. When an error occurs,
determining how, why and when mystate was set to a bad value is difficult. Test cases
will have to be re-run to find the exact moment mystate was set to the bad value.
Debugging this type of state transition can have a high cost in time. We refer to this
as stateful logic.
The code on the right executes the conditional logic every frame. Current state is not
tracked from frame to frame. When an error occurs the logic and parameters are
visible on the callstack. Debugging is easier because we are only looking for how and
why the logic failed. We have constrained the when to be this frame. We can usually
step the code along in the debugger and watch the same error happen again next
frame. This gives many attempts at understanding the problem as it happens. We
refer to this as stateless logic.
When performance allows prefer stateless logic.
20

23. Defensive Programming
• Defensive
• Defensive programming is just 'if's in code when they're not need.
• Easy to add when not sure some object/system/state is available.
• Bad defensiveness propagates to other code.
• Lots of code passing along bad game state.
• Resulting observed error can be long distance away from root cause.
• Ultimately making debugging harder.
• Assertive
• If in doubt. Assert.
• Use references for required parameters.
• Use pointers only if NULL is a valid parameter.
Question. Your code relies on several other modules to complete the required task. If
these systems are not available what do you do?
In large teams and large projects no one has perfect knowledge of the code. This is
normal. If dependant systems are not available it is tempting to do nothing. Return
some safe value and proceed. This is defensive programming. If everyone writes
defensive code the result is a game which runs without error, yet doesn't do anything.
Each system in turn silently doing nothing passing along an invalid state.
An working program which is doing the wrong thing is difficult to debug. Investigating
where and when things went wrong will have a high cost.
The correct answer is simple. Your code has a job to do. The code needs to perform
some function. If the code can not perform it's function for ANY reason you should
assert(). Catch errors early and fail hard. When everyone writes assertive code we
increase the probability the actual problem is near the assert. Easy and quick to find.
21

24. C++11/14
• Check before using C++11/14 features.
• Do not assume you are ahead of the curve.
• Only using
• New initialiser syntax.
• Const Expr.
• Iterator for loop syntax.
• Custom literal types.
• Override and Final. (Final offers performance benefits)
• Emplace back.
Using features which are an advantage.
Where C++ has multiple ways of doing the same things, we choose the one which
makes interfaces readable to others.
22

25. C++11/14
Do not use auto
Just in case you missed it.
23

26. Design for QA
• Design the bugs you will get back from QA¬!
• Design your code to fail in obvious and easy to reproduce ways.
• E.g.
• Fail on load, fail 100% or run forever.
• Prefer stateless logic to make better use of crash dumps.
If you get a 1 in 100 bug entered by QA this is your fault!
Difficult to reproduce bugs are a failure in the design of your code.
Everything we have covered in these slides is aimed towards reducing bugs. Reducing
bugs saves engineering time. Saving engineering waste allows more time for features
and polish.
When designing a system consider how it will fail. Consider how QA will enter the
resulting bugs. Design the system to fail in easy to understand ways. Ways which will
fail before you submit your changes.
Catching bugs in QA is slow and expensive. Design systems which fail hard or run
forever.
"The compiler is your friend, let it find the errors."
"Make it hard to do the wrong thing."
"Make it hard to waste time!"
24

27. General Reading List
• Premature Generalization
• You Arent Gonna Need It
• Typical C++ Bullshit
• Data Oriented Design
• Functional programming in C++
25

28. Appendix
C++ Coder Meeting Aug 17
26

29. Applying UX research to Code
How do we ask the team to write fewer bugs?
• Apply UX research techniques to the ‘user experience’ of programmers,
artists and designers.
• Analyse data from JIRA, Perforce, Resource plans and anecdotal feedback.
• Combine analytical research and technical knowledge.
27

30. Hypotheses
By analysing bugs, source code and resource allocations from previous
projects we have developed 2 hypotheses which underpin all technical
decisions.
• Hypothesis 1 - Specific programming patterns will be statistically more likely
to cause bugs in a large software project.
• Hypothesis 2 - Time spent fixing bugs throughout the project will be reduced
if we avoid the use of programming patterns identified by (1).
28

31. Hypotheses
When we talk to the team the message is very clear.
"Having analysed our previous projects we have made the follow
hypothesis...."
“Based on data. We believe by avoiding these specific patterns we will
reduce the time fixing bugs and improve quality"
29

32. High risk patterns
• A small amount of patterns account for a significant amount time
fixing bugs.
• Coupling memory allocation lifetime with object construction and lifetime.
• Overloading operators and de-normalised naming conventions.
• ‘auto’ and the removal of type safety.
• Pushing complexity upwards by dependency infection, callbacks, lamdbas,
threading.
30