Linux.conf.au 2003 | Abstracts

Dynamic Probes - Debugging by Stealth

Suparna Bhattacharya

Bio

Paper

Audio

Dynamic Probes has been around in the Linux Community for a couple years. Originally it was conceived as a stealth system debugger - i.e. one that you might be more happy to use in a production environment - its capability and application has expanded way beyond these modest beginnings. Since its initial release it has undergone continual development and enhancement. During 2002 DProbes has undergone major surgery: not only have we enhancing its capability, but we have streamlined the kernel patch and making it more generic. In June 2002 we released a major usability tool - the DProbes C Compiler (dpcc). DProbes was also ported to Power and zSeries architectures in 2002. Finally but not least DProbes was made a ready candidate for the 2.5 kernel and for several major distributions and has in fact been accepted by TurboLinux and MontaVista at the time of writing. So much has changed it seems an opportune moment to report on recent developments and work in progress. In this paper we will cover the following topics:

• The essential mechanisms that allow DProbes to be a system-wide debugger (kernel and multiple user contexts).
• How global probes are implemented using breakpoint mechanisms but without impacting the physical memory footprint through SOW. The techniques used here are applicable to kernel debuggers in general and would provide a means of allowing them to become system-wide debuggers.
• The new patch organisation into Kernel Probes; User Probes and Watchpoint Probes will be discussed in detail. These patches provide interfaces that allow any kernel module to define a probe and probe handler. It is by this means that we have been able to separate as an entirely optional component the original Reverse Polish Notation (RPN) interpreter that was previously an integral part of DProbes.
• How new pagepoint extension: we have the option of exploiting the processor's paging mechanism to extend numbers of concurrent watchpoints beyond the few that are permitted by processor debugging hardware.
• The new dpcc compiler: this allows probepoints to be defined using c-like language that references the variables and structures present in the code which is being probed. The Probe definition appears as an insert to the original source, without the need to edit or recompile the original source.
• We will also spend some time discussing how DProbes is able to be used as a general debugging engine to trigger crash dumps and core dumps from arbitrary code locations; to provide an on-the-fly tracing and logging mechanism (dynamic trace), which has found particular favour with some kernel developers whose needs was to add temporary printks for debugging; and finally we will show how DProbes can be used to exercise error code paths through fault injection.