This is an HTML rendering of a working paper draft that led to a publication. The publication should always be cited in preference to this draft using the following reference:

Citation(s): 3 (selected).

This document is also available in PDF format.

The document's metadata is available in BibTeX format.

Find the publication on Google Scholar

This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Diomidis Spinellis Publications

© 2006 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

Tools of the Trade

Debuggers and Logging Frameworks

Diomidis Spinellis

As soon as we started programming, we found to our surprise that it wasnít as easy to get programs right as we had thought.Debugging had to be discovered.

ó Maurice Wilkes discovers debugging, 1949


The testing, diagnostic, and repair equipment of many professions is horrendously expensive.Think of logic analyzers, CAT scanners, and dry docks.For us the cost of debuggers and logging frameworks is minimal; some of them are even free.All we need to become productive, is to invest some time and effort to learn how to use these tools in the most efficient and effective way.

Assuming that the bug-finding systems we discussed in our previous column have given our programís code a clean bill of health, using debugger or logging instrumentation is the most productive way for pinpointing errors that have managed to creep in our code.With these tools we can often get a starting point for locating a bug, and then also verify our hypotheses on what is going wrong.As one would expect, adopting an appropriate strategy and mastering the corresponding techniques are the important factors for making the best out of these tools.

Debugging strategies

The most efficient debugging strategy is a bottom-up one: we start from the symptom and look for the cause.The symptom can be a memory access violation (for example the dereferencing of a NULL pointer), an endless loop, or an uncaught exception.A debugger will typically allow us to get a snapshot of the program at the point where the symptom occurred.From that snapshot we can examine the programís stack frame: the sequence of function or method invocations that led to the execution of the problematic code.At the very least we thus obtain an accurate picture of our programís runtime behavior.Even better, we can also examine the values of variables at each level of the stack frame to really understand what brought our program belly-up.

Unfortunately, there are times when we canít adopt a bottom-up strategy.This situation crops up when the bugís symptom canít be precisely tied to a debugger event.†† Our program may cause a problem in another application, or the contents of a variable may be wrong for reasons we canít explain.In such cases top-down is the name of the game.Debuggers allow us to step through the code, stepping over or into functions and methods.When we debug in a top-down fashion we initially step over bodies of code we consider irrelevant, narrowing down our search as we come nearer the problemís manifestation.This strategy requires patience and persistence.Often we step-over a crucial function and find ourselves having to repeat the search aiming to step-into the function the next time round.This process can be tiring, but sooner or later will produce results.

There are also cases where we may have to debug a program at the level of assembly code: either because we donít trust the compiler, or because we donít have access to the programís source code.What Iíve found over the years is that assembly code is a lot less intimidating than it appears.Even if we donít know the processorís architecture, a few educated guesses and a bit of luck often allow us to decipher the instructions needed to pinpoint the problem.


Debugging techniques

Code and data breakpoints

Stack frame printouts and stepping commands are the basic and indispensable debugging tools, but there are more powerful commands that can often help us locate a tricky problem.A code breakpoint allows us to stop the programís execution at a specific line.We often use those to expedite a top down bug search, by placing a breakpoint before the point where we think the problem lies.In such cases we use the breakpoint as a bookmark for the location where we want to look at the programís operation in more detail.

Less known, but no less valuable, are data breakpointsóalso known as watchpoints.Many modern processors provide hardware support that will interrupt a programís execution when the code accesses the contents of some specified memory locations.Data breakpoints leverage this support allowing us programmers to specify that the programís execution will stop when its code reads or writes a variable, an array or an object.Note that debuggers that implement such commands without hardware support slow down the programís execution to a crawl rendering thiscommand almost useless (Java tool builders take note).

Live, post mortem and remote debugging

Although the typical set-up involves us starting the misbehaving program under a debugger, there are also other debugging options that can often help us escape a tight corner.

Consider non-reproducible bugs, also known as Heisenbugs, because they make our program appear as if itís operating under the spell of Heisenbergís uncertainty principle.We can often pinpoint those by debugging a program after it has crashed.On typical Unix systems crashed programs will leave behind them an image of their memory, the core dump.By running a debugger on this core dump we get a snapshot of the programís state at the point of the crash.Windows, on the other hand, offers us the possibility to launch a debugger immediately after a program has crashed.In both situations we can then look at the location of the crash, and examine the values the variables had at the time.If the program hasnít crashed but is acting weirdly, we can attach a debugger to that running process, and examine its operation from that point onward using the debuggerís commands.

Another class of applications that are difficult to debug are those with an interface thatís incompatible with the debuggerís.Embedded systems, operating system kernels, games, and applications with a cranky GUI fall in this category.Here the solution is remote debugging.We run the process under a debugger, but interact with the debuggerís interface on another system, connected through the network or a serial interface.This leaves the target system almost undisturbed, but still allows us to issue debugging commands and view their output from our debugging console.

The Logging Controversy

Instructions in the programís code that generate logging and debug messages allow us to inspect a programís behavior without using a debugger.Some believe that logging statements are only employed by those who donít know how to use a debugger.There may be cases where this is true, but it turns out that logging statements offer a number of advantages over a debugger session, and therefore the two approaches are complimentary.

First of all, the location and output of a logging statement is program-specific.Therefore, it can be permanently placed at a strategic location, and will output exactly the data we require.A debugger, as a general purpose tool, requires us to follow the program's control flow, and manually unravel complex data structures.

Furthermore, the work we invest in a debugging session only has ephemeral benefits.Even if we save our set-up for printing a complex data structure in a debugger script file, it would still not be visible or easily accessible to other people maintaining the code.I have yet to encounter a project that distributes debugger scripts together with its source code.On the other hand, because logging statements are permanent we can invest more effort than we could justify in a fleeting debugging session to format their output in a way that will increase our understanding of the programís operation and, therefore, our debugging productivity.

Finally, logging statements are inherently filterable.Many logging environments, such as the Unix syslog library, Javaís util.logging framework, and the log4j Apache logging services, ( offer facilities for identifying the importance and the domain of a given log message.More impressively, Appleís OS X logging facility stores log results in a database and allows us to run sophisticated queries on them.We can thus filter messages at runtime to see exactly those that interest us.Of course, all these benefits are only available to us when we correctly use an organized logging framework, not simple println statements.


As you can see our tool-bag is full of useful debugging tools.Being an expert user of a debugger and a logging framework is a sign of professional maturity.So, next time you encounter a bug, select the appropriate tool, go out, and slay it.


Diomidis Spinellis is an associate professor in the Department of Management Science and Technology at the Athens University of Economics and Business and the author of Code Quality: The Open Source Perspective (Addison-Wesley, 2006).Contact him at