Low-Level Windows
Debugging with
WinDbg
ARNO HUETTER

About the Author
Arno Huetter
Arno wrote his first lines of code on a Sinclair ZX80 in
1984.
Over the years, he has been programming in C/C++,
Java and C#, and also did quite some database
development.
Today he is Development Lead at Dynatrace (APM
vendor).

Dijkstra on Debugging

The Unsung Hero of Debugging

WinDbg Core Features
 WinDbg 1.0 released in 2000 (rewrite of previous WinDBG)
 Usermode Debugging
 Kernel/Remote Debugging
 Post-mortem Debugging
 Huge set of commands (some quite cryptical)
 UI (oh well...)
 Debugging Extensions (.NET: SOS, SOSEX, Psscor2|4, etc)
 Script Engine
 32bit vs. 64bit Version: Use the WinDbg executable matching your process bitness

What about Visual Studio Debugger?
 Visual Studio Debugger?
 Easier to use
 Sufficient for many debugging problems
 Also supports remote, kernel and post-mortem debugging
 WinDbg helps on the really tough cases (and most interesting ones)
 For example, one might just have a function offset from an eventlog entry (no symbols), or some
bytes from the stack, or a dump from an old binary without symbols, or...
 Problem might be not reproducible and only happening in production in border case scenario, or at
Windows startup, in a driver, etc. WinDbg is well suited for post-mortem, can also connect over
RS232, USB. Lightweight installation, WinDbg even runs from USB stick.
 WinDbg Extensions and Scripts provide powerful means, e.g. to analyze heap corruption, etc.

WinDbg Download
 Part of Windows SDK/WDK, choose Debugging Tools within SDK Download
Installer
 https://msdn.microsoft.com/en-
us/library/windows/hardware/ff551063%28v=vs.85%29.aspx

Tweak your UI

Set your Symbol Path

Demo 1: Finding a Memory Leak
 Enable allocation backtracking (gflags is part of Debugging Tools; be careful
though, backtracking adds considerable runtime overhead)
gflags /i plauscherl.exe +ust
 Display heap summary
0:001> !heap -s
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
039c0000 08000002 16420 8520 16364 158 26 5 0 0 LFH
03c10000 08001002 60 12 60 3 2 1 0 0
-----------------------------------------------------------------------------

Finding a Memory Leak
 Display heap status by entry size
0:001> !heap -stat -h 039c0000
heap @ 039c0000
group-by: TOTSIZE max-display: 20
size #blocks total ( %) (percent of total busy bytes)
fa0 158 - 14ff00 (97.12)
17d8 1 - 17d8 (0.43)
10d2 1 - 10d2 (0.30)
1000 1 - 1000 (0.29)
20 68 - d00 (0.23)

Finding a Memory Leak
 Filter by entry size
0:001> !heap -flt s fa0
_HEAP @ 39c0000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
039d1190 01f7 0000 [00] 039d11a8 00fa0 - (busy)
039d2148 01f7 01f7 [00] 039d2160 00fa0 - (busy)
039d3100 01f7 01f7 [00] 039d3118 00fa0 - (busy)
039d40b8 01f7 01f7 [00] 039d40d0 00fa0 - (busy)

Finding a Memory Leak
 Inspect entry and get allocation backtrace
0:001> !heap -p -a 039d1190
address 039d1190 found in
_HEAP @ 39c0000
HEAP_ENTRY Size Prev Flags UserPtr UserSize - state
039d1190 01f7 0000 [00] 039d11a8 00fa0 - (busy)
77b3cdb3 ntdll!RtlpCallInterceptRoutine+0x00000026
77af71f1 ntdll!RtlAllocateHeap+0x000461d1
73f3f7cb ucrtbase!malloc+0x0000002b
d112d plauscherl!operator new+0x0000002c
d1027 plauscherl!allocateStuff+0x00000027
d12f6 plauscherl!__scrt_common_main_seh+0x000000ff
77447c04 KERNEL32!BaseThreadInitThunk+0x00000024
77acad1f ntdll!__RtlUserThreadStart+0x0000002f

Demo 2: Analyzing Crash by Offset
 We might only have something like this eventlog entry
Faulting application name: plauscherl.exe, version: 0.0.0.0, time stamp: 0x5669ec61
Faulting module name: plauscherl.exe, version: 0.0.0.0, time stamp: 0x5669ec61
Exception code: 0xc0000005
Fault offset: 0x00011a2e
 Simply attach WinDbg to .exe or .dll only
windbg -z plauscherl.exe
ModLoad: 00400000 00420000 plauscherl.exe

Analyzing Crash by Offset
 Disassemble at offset
0:000> uf 0x00411a2e
plauscherl!doSomethingWrong [@ 25]:
25 00411a10 55 push ebp
25 00411a11 8bec mov ebp,esp
25 00411a13 81ecc0000000 sub esp,0C0h
25 00411a19 53 push ebx
25 00411a1a 56 push esi
25 00411a1b 57 push edi
25 00411a1c 8dbd40ffffff lea edi,[ebp-0C0h]
25 00411a22 b930000000 mov ecx,30h
25 00411a27 b8cccccccc mov eax,0CCCCCCCCh
25 00411a2c f3ab rep stos dword ptr es:[edi]
26 00411a2e c7050000000000000000 mov dword ptr ds:[0],0

Demo 3: Analyzing a Crashdump
 With dump file: Show stack of selected thread
0:000:x86> k
# ChildEBP RetAddr
00 0100f97c 003d19e3 plauscherl!doSomethingWrong+0x1e
01 0100fa50 003d1b86 plauscherl!doSomething+0x23
02 0100fb24 003d238e plauscherl!main+0x76
03 0100fb38 003d21da plauscherl!invoke_main+0x1e
04 0100fb90 003d206d plauscherl!__scrt_common_main_seh+0x15a
05 0100fb98 003d23a8 plauscherl!__scrt_common_main+0xd
06 0100fba0 77447c04 plauscherl!mainCRTStartup+0x8
07 0100fbb4 77acad1f kernel32!BaseThreadInitThunk+0x24
08 0100fbfc 77acacea ntdll_77a70000!__RtlUserThreadStart+0x2f
09 0100fc0c 00000000 ntdll_77a70000!_RtlUserThreadStart+0x1b

Analyzing a Crashdump
 Or: Run full-fledged analysis
0:000:x86> !analyze –v
CONTEXT: (.ecxr)
eax=cccccccc ebx=7f14b000 ecx=00000000 edx=0ff4f4b8 esi=007efaa8 edi=007ef9cc
eip=012a151e esp=007ef900 ebp=007ef9cc iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
plauscherl!doSomethingWrong+0x1e:
012a151e c7050000000000000000 mov dword ptr ds:[0],0 ds:002b:00000000=????????
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 012a151e (plauscherl!doSomethingWrong+0x0000001e)
ExceptionCode: c0000005 (Access violation)
Attempt to write to address 00000000

Analyzing a Crashdump
STACK_TEXT:
007ef9cc 012a1563 007efb74 007efaa8 7f14b000 plauscherl!doSomethingWrong+0x1e
007efaa0 012a160a 00000000 00000000 7f14b000 plauscherl!doSomething+0x23
007efb74 012a1bff 00000002 00bb3f98 00bb1f58 plauscherl!main+0x7a
007efbc4 012a1a2f 007efbd8 758c919f 7f14b000 plauscherl!__tmainCRTStartup+0x1bf
007efbcc 758c919f 7f14b000 007efc1c 77e6a8cb plauscherl!mainCRTStartup+0xf
007efbd8 77e6a8cb 7f14b000 b92e2b36 00000000 kernel32!BaseThreadInitThunk+0xe
007efc1c 77e6a8a1 ffffffff 77e5f67f 00000000 ntdll!__RtlUserThreadStart+0x20
007efc2c 00000000 012a1113 7f14b000 00000000 ntdll!_RtlUserThreadStart+0x1b
FAULTING_SOURCE_FILE: d:etcplauscherlplauscherlplauscherl.cpp
FAULTING_SOURCE_LINE_NUMBER: 26

Demo 4: Analyzing a .NET Deadlock
 Enter: SOS (Son Of Strike) .NET Debugging Extension
 Load SOS (part of every CLR)
0:008> .loadby sos clr
0:008> .cordll -ve -u -l
Automatically loaded SOS Extension
CLRDLL: Loaded DLL C:WindowsMicrosoft.NETFrameworkv4.0.30319mscordacwks.dll
CLR DLL status: Loaded DLL
C:WindowsMicrosoft.NETFrameworkv4.0.30319mscordacwks.dll
 Memdumps: Always use sos.dll / mscordacwks.dll from identical CLR (version /
bitness). Latest WinDbg releases can load matching sos.dll from Microsoft symbol
server.

Analyzing a .NET Deadlock
 View managed thread
0:004> !clrstack
OS Thread Id: 0x1e70 (4)
Child SP IP Call Site
056cf7dc 77aaca2c [GCFrame: 056cf7dc]
056cf8b4 77aaca2c [GCFrame: 056cf8b4]
056cf8d0 77aaca2c [HelperMethodFrame_1OBJ: 056cf8d0] System.Threading.Monitor.ReliableEnter(System.Object,
Boolean ByRef)
056cf94c 7042a287 System.Threading.Monitor.Enter(System.Object, Boolean ByRef)
056cf95c 02ca0741 netplauscherl.Program.DoSomethingImportant() 056cf98c 02ca06bb
netplauscherl.Program+c.b__4_0()
056cf998 70446274 System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
056cf9a4 7042a417 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object, Boolean)
056cfa10 7042a366 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object, Boolean)
056cfa24 7042a321 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
System.Threading.ContextCallback, System.Object)
056cfa3c 704461fc System.Threading.ThreadHelper.ThreadStart()
056cfb80 73181396 [GCFrame: 056cfb80]
056cfd64 73181396 [DebuggerU2MCatchHandlerFrame: 056cfd64 ]

Analyzing a .NET Deadlock
 Check who owns the locks
0:004> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock
Owner
3 011d3eac 3 1 011d37e8 1e70 4 031f3104 System.Object
4 011d3ee0 3 1 011d51f0 1864 5 031f3110 System.Object
-----------------------------
Total 4
CCW 0
RCW 0
ComClassFactory 0
Free 0

Debugger Markup Language

Some WinDbg commands
 Go
g
 Set breakpoint at address
bp [Addr]
 Set breakpoint at symbol (pattern)
bm SymPattern
 Set breakpoint on access
ba [r|w|e] [Size] Addr

More WinDbg commands
 Display callstack for selected thread
k
 Display local variables and parameters
dv [/i /t /V /a /n /z] [Pattern]
 Display type
dt [-n|y] [mod!]Name [-n|y] [Field] [Addr] -abcehioprsv
0:000> dt 0x0100297c MYTYPE1
+0x000 a : 22
+0x004 b : 43 '+'
+0x006 c : 0x0
+0x008 d : 0x0
+0x00c gn : [6] 0x0

More WinDbg commands
 Display callstacks for all threads
~* k
 Dump all registers
r
 Show process environment block
!peb
 Display time consumption per thread
!runaway
 Search memory for pattern
s -[Flags][b|w|d|q|a|u] Range Pattern

Selected SOS commands
 Display managed callstack
!CLRStack [-a] [-l] [-p] [-n]
 Display managed / unmanaged callstack for all threads (see: !DumpStack)
!EEStack [-short] [-EE]
 Display IL associated with managed method
!DumpIL <Managed DynamicMethod object> | <MethodDesc pointer>
 Heap object statistics
!DumpHeap –stat
 Display object at specified address
!DumpObj [-nofields] <object address>

Selected SOSEX commands
 Display deadlocks between SyncBlocks and/or ReaderWriterLocks
!dlk [-d]
 Display GC roots for the specified object
!mroot <ObjectAddr> [-all]
 Print a stack trace of managed and unmanaged frames
!mk
 Search the managed heap for strings
strings [ModuleAddress] [Options]

Selected PSSCOR commands
 Displays all objects on stack
!DumpStackObjects
 Shows detailed info about running HttpRuntime (ASP.NET)
!DumpHttpRuntime
 Save all .NET assemblies to local disk
!SaveAllModules folderName
 Dump all exceptions found in heap (incl. callstacks)
!dae
 Find managed MethodDesc and JITted code address
!Name2EE mscorlib.dll System.String.ToString

For your Toolbox
 MS DebugDiag
 http://blogs.msdn.com/b/debugdiag/
 Process-specific rules for crashes,
perfcounters, memory usage, exceptions
 Creates full dumps, mini dumps, stack traces
 Built-in auto-analysis

MS DebugDiag

For your Toolbox
 SysInternals ProcDump
 https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
 Commandline tool for dump creation
 Can be event-triggered, e.g. CPU usage
 MS AppVerifier
 https://msdn.microsoft.com/en-us/library/ms220948(v=vs.90).aspx
 Runtime verification for native code, e.g. memory corruption

For your Toolbox
 VerySleepy
 http://www.codersnotes.com/sleepy/
 CPU profiler
 Samples native callstacks, aggregates results over time

Online Resources
 Common WinDbg Commands (Thematically Grouped)
 http://windbg.info/doc/1-common-cmds.html
 SOS.dll (SOS Debugging Extension)
 https://msdn.microsoft.com/en-us/library/bb190764(v=vs.110).aspx
 "If broken it is, fix it you should" - Tess Ferrandez Blog
 https://blogs.msdn.microsoft.com/tess/
 Software Diagnostics Institute
 http://www.dumpanalysis.org/

Books
 Hewardt, Pravat: “Advanced Windows Debugging”
 Vostokov, D.: “Memory Dump Analysis”

Thank you!
Twitter: https://twitter.com/ArnoHu
Blog: http://arnosoftwaredev.blogspot.com