Philip Paeps - FreeBSD, Detangling and debugging

Philip recommends debugging without using the debug tools.

"Debugging is universally anticipated with distaste, performed with reluctance and bragged about forever"  -- anonymous.

One of the biggest drawbacks to using the debug tools is losing an entire day rebuilding the system to include the debug symbols and then to figure out that the problem was a simple typo that you could have caught with five minutes of critical thinking and some code review.

Suggestions to debug without the debugger

• Printf's are boring.  Instead when your program crashes, have it print a stack trace.
•  Cookies -- Write an unsigned long as a global variable and use it as a poor man's running stack trace.  Write to it  (fiddle with the bits) in the different subsystems to keep track of where you have been.  Works great for embedded systems.
• GCC is your friend.  Don't silence the debugger with a cast, fix the problem.
• use GCC -E  -- It goes through the pre-processor and prints out the info.
• Know your -w flags.  Use -w Error to stop the program on warnings and fix them.  Lots of problems can go away when you fix the warnings.
• Use GCC instrumentations  -- Very useful in userspace, not so much in kernel.
• Do an object dump. -- Useful, but you need to know a lot to use it.  Also you have to remove the -fomit-frame-pointer flag on intel platforms, or this process is useless.  You can use this method to disassemble  your program and figure out where the program crashed.  Very useful in trace analysis. 

Summary

Try not to debug, try to think first.

Take shortcuts.  You have already broken something, cheating won't make it worse.

Remember who your friends are, like nm and object dump.

Document your clever tricks.