Debugging NET Applications With WinDBG

Debugging .NET Applications with WinDBG Cory Foy http://coryfoy.com | @cory_foy

Agenda .NET Fundamentals Debugging Crashes Debugging Memory Leaks Debugging Hangs WinDBG Tips and Tricks Other Tools

.NET Fundamentals CLR Fundamentals Memory Management Basics Debugging Fundamentals

CLR Fundamentals Managed Assembly Layout PE Header CLR Header IL (Intermediate Language) Metadata Header Metadata

CLR Fundamentals .NET Runtime in a Process

Demo: Viewing Managed Objects Cory Foy http://coryfoy.com | @cory_foy

Viewing Managed Objects Start up the app and click on the Managed Objects button Start up WinDBG and attach to the executable (click “No” if it asks you to save the workspace) then load SOS by typing “.loadbysosmscorwks” and hitting enter. Also load symbols by typing “.symfix C:ymbols” and hitting enter, then typing “.reload” and hitting enter

Viewing Managed Objects View All Threads by typing in “~” and hitting enter. View all managed threads by typing in “!threads” and hitting enter. Note there are 5 native threads, but only 2 managed threads. Also note that thread 2 is marked as (Finalizer) – that’s the thread responsible for finalization Change to the main thread by typing “~0s” (tilde zero s). This tells WinDBG to switch to thread 0. Then type “!clrstack” to see where we are in the application. Note we are in the Main method right now.

Viewing Managed Objects Next, let’s look at the objects on the heap. Type “!dumpheap –stat”. You’ll see many objects. You can filter by typing things like -min or –max as parameters to dumpheap. Now find all of our objects on the heap by typing “!dumpheap –type WinDBG”. This filters for any object with the text “WinDBG” in it’s name. The top list are the actual objects and their memory locations, while the bottom list are the type names. You can figure out which is which by matching up the MethodTable addresses (MT Column)

Viewing Managed Objects We can view a specific object using “!dumpobj” (or “!do” for short). We’ll look at the HelloWorld Object by typing “!do <address>” where address is from the !dumpheap command earlier. We can see we have a string field called “hello” and a world property. We can then look at the string by typing “!do <address>”. In this example, the address would be 263a000. We can see that field contains the string “Hello”.

Viewing Managed Objects When we did a !do on the HelloWorld object, one piece of information was the MethodTable address. We can use the !dumpmt command to see what methods the object exposes. Type “!dumpmt –md <methodtableaddress>”. If the JIT Column has “PreJIT” then the method came from an Ngen’d object. If it has JIT, then the method has been called and JIT’d. If it has NONE, the method hasn’t been called. You can dissassemble IL that has been JIT’d by passing in the MethodDesc address to “!U”

Viewing Managed Objects If we’ve attached to the process doing a live debug (which we’re doing here), then you can set breakpoints using !bpmd. For example, we can have it breakpoint just before the MessageBox shows up by passing in the assembly name and fully qualified type. You can use bl to see the breakpoints. We then type “g” to release the current break we have on the app, and click the Managed Objects button again. We’ll then see WinDBG hit the breakpoint. We can run !clrstack to see what led us to that call

CLR Fundamentals Threads Managed Threads is an object – it lives on native system threads CLR Threads Finalizer Garbage Collector (Server GC) Debugger Timer Threadpool (I/O and Worker)

CLR Fundamentals Just-In-Time compilation Program makes a call to a method The .NET Runtime checks to see if the method has been called before If so, it executes the JIT’d code If not, it compiles the IL code and stores it in memory, updating the MethodDesc

Memory Management Stacks versus Heaps Stack – First in / First Out Heap – access by address Garbage Collector Heap Where all objects are stored Broken into 3 generations and one Large Object Heap Large Object > 85,000 bytes

Memory Management Garbage Collector Sequence Suspend Execution Engine Mark objects without roots Plan (budgets, fragmentation) Sweep (delete market objects) Compact (move leftover objects to back of heap) Restart Execution Engine

Memory Management Memory Leaks Possible in both Managed and Unmanaged Code Use Perfmon to check for symptoms Unmanaged Leak Private Bytes Increase, #Bytes In All Heaps stays flat Managed Leak Both Private Bytes and Bytes In All Heaps increase Need multiple dump files

Memory Management http://blogs.msdn.com/tess/archive/2008/03/17/net-debugging-demos-lab-6-memory-leak-review.aspx

Memory Management Why do we leak managed memory? Objects not being released “Pinned” memory Finalized Objects (Destructors on Managed Objects) Finalized objects require an additional GC cycle to be cleaned since they have to go in the finalizer thread to run

Memory Management Exception Handling Workflow Exception Occurs (create an exception object) Notify Debugger (1st Chance Exception) Look for a handler by walking up the call stack If handler found, let it handle exception If not, throw a 2nd Chance Exception and terminate the process

Debugging Fundamentals Typical Problems in Production System hangs or deadlocks Fatal Exceptions Data Loss or inconsistency Performance Problems Excessive Memory Usage App Pool Restarts (slow access to ASP.NET pages)

Debugging Fundamentals Approaching the problem Be the application (visualize what could cause the problem) Use Application and System Logs Try to reproduce in Development or Staging environments Create a hypothesis and use WinDBG to validate

Debugging Fundamentals Debugging Production Apps Typically don’t have Visual Studio installed, or access to the remote debugger Attaching a debugger freezes all threads Capture memory dumps At time of crash / exception Over time to troubleshoot hangs / leaks

Debugging Crashes Application seems to work fine, but something happens Unhandled Exception Dialog App “disappears” Steps Desktop App: Attach and stop on Exceptions Web App: Use DebugDiag to capture memory dump

Demo: Debugging Crashes Cory Foy http://coryfoy.com | @cory_foy

Debugging Crashes Open Application and click the “Crash” button”. Unhandled Dialog should appear and kill the app. Start app back up and attach to it with WinDBG, then load SOS. Once SOS is loaded, type “sxeclr” to tell WinDBG to break on all .NET Exceptions, then type “g” and click on the Crash button again. Because we’ve enabled exception notification, WinDBG breaks on the first-chance exception.

Debugging Crashes The first place to look is the exception object itself. We can either do a “!do” on the object address listed, or simply type “!pe”. You can also pass an address to !pe if you need to view other than the last exception thrown. We can see there is a file not found exception. Let’s see where we were at. So it looks like the app calls “ReadAllText” from a method called “Crash_Click” from the Form1 object. If we have access to Source, we’d start there. If not, we can find the method address and dissassemble

Debugging Crashes Note that if we click “g” at this point, WinDBG breaks again. This is because the exception we first saw was *not* the crash reason. Like the previous exception, this one is listed as a First Chance Exception. If we click “g” again, we’ll see WinDBG breakpoint again. Note that it clearly tells us this is a second chance exception – the next thing that will happen is process termination.

Demo: Debugging Crashes on Application Startup Cory Foy http://coryfoy.com | @cory_foy

Debugging Crashes On Startup Open the App and check the “Crash on Startup” option, then close the app and restart. Notice it immediately crashes. The challenge is that we won’t have time to attach a debugger. You can set registry keys to automatically launch a debugger, or we could launch the app from the debugger, which we’ll do here. Start up WinDBG and go to File->Open Executable. Browse to the WinDBGDemo executable and select it.

Debugging Crashes On Startup At this point, we are in the PE load of the application, meaning it hasn’t even decided if this is a native or managed app yet. Which also means no .NET libraries are loaded, so we can’t load SOS. What we can do is set a breakpoint to fire when the .NET runtime is loaded, at which point we can set the handlers we need. Type “sxe ld mscorwks” which means set WinDBG to break when mscorwks is loaded. We can now load SOS and set the CLR Exception breakpoints. Now type “g” till our app breaks. You can now debug it as a crash.

Debugging Crashes On Startup If you have a Windows Service which is crashing on startup, then you’ll need to modify some registry keys and permissions on the service. For more information, see the following KB article, or the blog post from our site (Note: When you finish this demo, go to your user directoryppDataocalicrosoftinDBGDemo.exe.0.0.0 and modify the user.config file to have CrashOnStartup to be False) http://support.microsoft.com/kb/824344 http://blog.coryfoy.com/2008/06/debugging-a-net-windows-service-that-is-crashing-on-startup/

Debugging Memory Leaks Memory Usage Continues to grow in the app May cause app pool restarts in ASP.NET Out of Memory exceptions when you still have memory free Several Reasons Unreleased objects “Pinned” memory You have plenty of memory! (GC only runs when it feels memory pressure)

Debugging Memory Leaks Diagnosing Memory Leaks PerfMon to determine if it is a managed leak Take multiple memory dumps to determine what is happening For the objects on the heap, find out why they are sticking around Check the GC Generation for the objects

Demo: Debugging Memory Leaks Cory Foy http://coryfoy.com | @cory_foy

Debugging Memory Leaks Start the app and task manager, then click on the Memory Leak button. Note that the memory is still increasing. Fire up Perfmon to see if this is a managed or native leak. Start->Run->Perfmon. You can remove the Processor Time counter and add the following counters: .NET CLR Memory -> #Bytes in all Heaps -> WinDBGDemo and Process -> Private Bytes -> WinDBGDemo. You should see that both are growing at about the same rate indicating a managed leak

Debugging Memory Leaks To debug this, we’ll need to take two snapshots of the application far enough apart to see memory difference. You can do this right from Task Manager. Create two, about 15 seconds apart. You can now kill the application. Open two instances of WinDBG and open each memory dump file you created in a different instance of WinDBG. Load up SOS in each one as well.

Debugging Memory Leaks What we’re wanting to do is compare the two memory dumps to see if we can spot the culprit. Using !VMStat on each file we can see that memory usage is certainly increasing. And if we do a “!dumpheap –stat” on each one, we can see something quite startlingly – 230,000 additional objects are on the heap!

Debugging Memory Leaks Normally you’d have to dig through the heap to find which objects increased, but we see something interesting. An object called “LeakForm” has had 900 instances created. Perhaps those aren’t being cleaned up? Let’s look at one. We’ll see all the instances by doing a “!dumpheap -type WinDBGDemo.LeakForm” then picking any object at random to do a !do on With that object address, let’s see what is holding on to it with “!GCRoot <address>”

Debugging Memory Leaks We see that the LeakForm is being held on to an object, which is of type System.Object[]. If we do a !do on it, we don’t see much of interest But if we compare the arrays between the first dump and the second dump using the “!da <address>” command, we find a clue. Between the first and second memory dumps, all of the extra LeakForms were added to this array! At this point, we examine the source code to see what is going on.

Debugging Hangs Two types of hangs Low CPU Hang Generally caused by a deadlock High CPU Hang App in a tight loop, or under heavy load

Debugging Hangs Diagnosing Hangs High CPU Hang Multiple memory dumps Compare which thread is getting processor time Low CPU Hang Walk through the waiting threads and match locks Very tedious process

Demo: Debugging High CPU Hangs Cory Foy http://coryfoy.com | @cory_foy

Debugging High CPU Hangs Start the app and Task Manager. Click on the High CPU Hang. You should see a single processor spike (100% on 1CPU). In this example, I have a 4CPU system, so a CPU of 25% indicates it is fully utilizing one processor. Take two memory dumps from Task Manager about 15 seconds apart. You can then kill the application.

Debugging High CPU Hangs Open the two memory dump files in different WinDBG instances and load SOS. The first thing we need to know is if a single thread is using up all of the time, indicating a possible loop. We can use the command !runaway to see which threads are getting CPU time We can see from this that thread 0 was using up all of the time in between the two memory dumps

Debugging High CPU Hangs So what is thread 0 doing? By running !clrstack on each memory dump, we can see that the call stacks are the same, possibly indicating that this is the culprit. I say possibly because if a bunch of calls are happening to different methods, we very well could have gotten “lucky” and seen the same method. Always verify your assumptions. In this case, looking at the source (either real source, or through reflector) we see the following. I think we have our culprit

Demo: WinDBG Tips and Tricks Cory Foy http://coryfoy.com | @cory_foy

WinDBG Tips and Tricks You can get quite a bit of help with both WinDBG and SOS. For the WinDBG help content, type “.hh” and it will open the WinDBG CHM file. You can also type “.hh <command>” and it will take you to that section For SOS, you can type “!help” to see all available commands, or “!help <command>” to see a specific one

WinDBG Tips and Tricks You can use the shell command to run the output of a WinDBG command to an external process and the results are displayed back into the UI. For example, to quickly find the shared domain, you can run ‘.shell –ci “!dumpdomain” find /i “shared domain”’ You can also take advantage of this to do integrations with PowerShell

WinDBG Tips and Tricks When viewing stacks, you can use !clrstack to view the managed stack. You can also use the command “k” to view the native stack You can also output the stack for all threads by combining the “~” command with a star (for all threads) and the command. For example “~*k” or “~*e!clrstack” (which means execute the command !clrstack)

WinDBG Tips and Tricks You can do a lot with automation of WinDBG. For example, you can have WinDBG execute commands when certain breakpoints are hit You can also automate what happens when CLR Exceptions are hit by using the –c option with sxe. In this example, every time we hit an exception we would spit out the exception and CLR stack trace.

WinDBG Tips and Tricks You can also loop through various elements. For example, if we wanted to dump out all strings greater than 5k that were on the heap, we could do the following with the “.foreach” command Finally, if you want to adjust the colors, you can do that in View->Options

Additional Tools DebugDiag Used to automate the capture of exceptions for IIS processes ADPlus Used to capture dumps from command line Managed Debuggers (CorDbg, MDbg) Managed Debugging Assistants (Visual Studio) Profilers

More Information http://blogs.msdn.com/tess microsoft.com/whdc/DevTools/Debugging http://windbg.info http://www.coryfoy.com foyc at coryfoy dot com @cory_foy on Twitter Slides will be posted on CoryFoy.com

Debugging NET Applications With WinDBG

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (17)

Ähnlich wie Debugging NET Applications With WinDBG

Ähnlich wie Debugging NET Applications With WinDBG (20)

Mehr von Cory Foy

Mehr von Cory Foy (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Debugging NET Applications With WinDBG

Hinweis der Redaktion