Visual Studio undocumented native debugging tricks presentation. By Ofek Shilon.

1. Little Known
Native Debugging Tricks
Ofek Shilon
Director of Algorithms @ Sarin Technologies
www.thetweaker.wordpress.com

3. DISCLAIMER

4. VS AS AN INTERPRETER

5. Expression Evaluator
• Demo
• Scope
– Watch, QuickWatch,
– Immediate,
– BP conditions, ‘when hit’
• Call Functions from the debugger!

6. Expression Evaluator
• Context Operator
– Documentation is wrong.
– Might require decorated names.
– VS2010 blocked WIN-API, but eased usage of
others.
– VS11B supports the simpler syntax
msvcr110d.dll!fabs(-8.0f)

7. Expression Evaluator
• Thread context
• Access insensitive
• Blind to inlined functions
• Things to try when evaluation fails:
– Very explicit casting,
– Explicit addresses.

8. Edit and Continue
• Demo
• Stale code

9. Edit and Continue
• Preparation: /ZI, /INCREMENTAL

10. Edit and Continue
• Limitations:
– Inconsistent with /RELEASE, /OPT:REF, /OPT:ICF,
/ORDER, /FORCE,
– Does not invoke custom build steps,
– Isn’t supported on attached processes,
– Can’t change object layout,
– Doesn’t regenerate IDL files,
– Can’t change resource files,
– Does not build static libs.

11. CLICK SAVERS

12. StepOver
• Demo

13. StepOver
• VC6: autoexp section [ExecutionControl]
• VC7: Reg key -
HKCUSoftwareMicrosoftVisualStudio7.0NativeDEStepOver
• VC8:
HKLMSoftwareMicrosoftVisualStudio8.0NativeDEStepOver
• VC10, VC11B:
HKCUSoftwareMicrosoftVisualStudio10.0_ConfigNativeDEStepOver
Dropped ‘=NoStepInto’ syntax

14. StepOver
• Warning!
When 2 or more
instances of VS
are active, occasionally
it would create and use
a duplicate reg key
… and all ’10.0_Config’ modifications would have no effect !

15. StepOver
• Credit: Andy Pennell
http://blogs.msdn.com/b/andypennell/archive/2004/02/06/69004.aspx
• A request to make StepOver official:
http://visualstudio.uservoice.com/forums/121579-visual-
studio/suggestions/2668765-support-nostepinto-registry-mechanism

16. autoexp
• How’s the debugger ‘aware’ of STL containers?
• autoexp.dat
– By default at
%VS folder %Common7PackagesDebugger

17. autoexp
• Demo
• Specialized syntax for common containers:
Array, List, Map, Tree.
• Re-parsed on every debugging session
• Extremely fragile, yet excellent investment

18. autoexp
• Status limbo:

19. autoexp
• Bypass file path via env var: _vcee_autoexp
• Bypass visualizer locally: xxx,!
• Bypass
globally
in
Tools/Options:

20. autoexp
• Resources:
– Avery Lee
http://virtualdub.org/blog/pivot/entry.php?id=120
– Stephan T. Lavavej BoostCon presentation
http://filetolink.com/d/?h=a456fc7fd3d35644edcbe5e764d1ba63&t=1315076450&f=17ad36ef
– My blog: complete MFC containers visualization, Matrix 2D
visualization, and more
www.thetweaker.wordpress.org
– A request to make autoexp official:
http://visualstudio.uservoice.com/forums/121579-visual-
studio/suggestions/2056273-support-debug-visualizers-through-
autoexp-dat

21. Sound Alerts
• Win7:
Control Panel
Hardware and Sounds
Sound
Change system sounds
• WinXP:
Control Panel
Sounds and Audio...
Program Events –
Microsoft Developer

22. TRACING ERRORS

23. GetLastError Interactively
• GetLastError implementation:
• (int*)($fs)+0x34
• (int*)($tib)+0x34
• $err !
• Credit: msdn
http://msdn.microsoft.com/en-us/library/dtw169z6.aspx

24. Extensions
• Customizing ‘,hr’ display in autoexp.dat:
• Other TEB data: (_TEB*)$tib

25. Breaking on Errors
• BP location: (int*)($tib)+0x34
• Specific error: BP condition
– Slooooooooow
• g_dwLastErrorToBreakOn !
– (int*){,,ntdll.dll}_g_dwLastErrorToBreakOn
– Fails in VC10 over Win7. Restored in VC11B
• Credit: Daniel Pearson,
http://blogs.msdn.com/b/danpear/archive/2007/04/06/2033100.aspx

26. INVESTIGATING CODE

27. Breaking on all class methods
• Break at function:
Class::*
• Credit: Habib Heydarian
http://blogs.msdn.com/b/habibh/archive/2009/09/10/class-breakpoint-
how-to-set-a-breakpoint-on-a-c-class-in-the-visual-studio-debugger.aspx

28. Naming Native Threads
• Demo
• ‘Documented hack’
Credit: MSDN
http://msdn.microsoft.com/en-us/library/xcb2z8hs.aspx

29. Breaking on Data Read
• Some code required.
• Links to 4 implementations:
http://thetweaker.wordpress.com/2011/05/22/breaking-on-data-read/
• Request to make it official:
http://visualstudio.uservoice.com/forums/121579-visual-
studio/suggestions/2668700-expose-break-on-read-breakpoints

30. Static Object Layout
• /d1reportSingleClassLayout
• /d1reportAllClassLayout
• Credit: Andy Rich
http://blogs.msdn.com/b/vcblog/archive/2007/05/17/diagnosing-hidden-
odr-violations-in-visual-c-and-fixing-lnk2022.aspx

31. ENHANCED DIAGNOSIS

32. Useful Pseudo-registers
• $tid
-> very useful as a BP condition.
• $env=0
-> view debugee environment variables
– In VC11: just $env in watch
• $cmdline = 0
-> view launching command line
Credit: John Cunningham,
http://blogs.msdn.com/b/ms_joc/archive/2004/08/23/218912.aspx

33. Useful Pseudo-registers
• $handles
-> narrows down handle leaks
• $clk
-> rough profiling
• $user
-> view permissions info
• Credits: Steve Steiner, Gregg Miskelly, et. al.
http://blogs.msdn.com/b/stevejs/
http://blogs.msdn.com/b/greggm/

34. Searching Through Memory
• Credit: msdn
http://msdn.microsoft.com/en-us/library/ms171363.aspx

35. www.thetweaker.wordpress.com
ofekshilon@gmail.com
THANK YOU.