manpage for each, and try running each on a couple of different a.out's—the "hello world" program, and a big program.
Go through these carefully—investing 15 minutes now in trying each one will save you hours later in solving a hard bug.
Table 6-1. Tools to Examine Source Tool Where to
Find It
What It Does
cb
Comes withthe compiler
C program beautifier. Run your source through this filter to put it in a standard layout and indentation. Comes from Berkeley.
indent
Does the same thingscb
does. Comes from AT & T.cdecl
This book Unscrambles C declarations.cflow
Comes with the compilerPrints the caller/callee relationships of a program.
cscope
Comes with the compilerAn interactive
ASCII
-based C program browser. We use it in the OS group to check the impact of changes to header files. It provides quick answers to questions like: "How many commands use libthread?" or "Who are all the kmem readers?"ctags
/usr/bin Creates a tags file for use in vi editor. A tags file speeds up examining program source by maintaining a table of where most objects are located.lint
Comes with the compilerA C program checker.
sccs
/usr/ccs/bin A source code version control system.vgrind
/usr/bin A formatter for printing nice C listings.Doctors can use x-rays, sonograms, arthroscopes, and exploratory operations to look inside their patients. These tools are the x-rays of the software world.
Table 6-2. Tools to Examine Executables Tool Where to
Find It
What It Does
dis
/usr/ccs/bin Object code disassemblerdump -Lv
/usr/ccs/bin Prints dynamic linking information
ldd
/usr/bin Prints the dynamic libraries this file needsnm
/usr/ccs/bin Prints the symbol table of an object filestrings
/usr/bin Looks at the strings embedded in a binary. Useful for looking at the error messages a binary can generate, built-in file names, and (sometimes) symbol names or version and copyright information.sum
/usr/bin Prints checksum and block count for a file. An-swers questions like: "Are two executables the same version?" "Did the transmission go OK?"Table 6-3. Tools to Help with Debugging
Tool Where to Find It
What It Does
truss
/usr/bin The SVr4 version oftrace
. This tool prints out the system calls that an executable makes. Use it to see what a binary is doing, and why it's stuck or failing. This is a great help!ps
/usr/bin Displays process characteristics.ctrace
Comes with compilerModifies your source to print lines as they are executed. A great tool for small programs!
debugger
Comes with compilerInteractive debugger.
file
/usr/bin Tells you what a file contains (e.g., executable, data, ASCII, shell script, archive, etc.).Table 6-4. Tools to Help with Performance Tuning Tool Where to Find
It
What It Does
collector
Comes withdebugger
(SunOS only) Collects runtime performance data under the control of the debugger.
analyzer
Comes with debugger(SunOS only) Analyzes collected performance data.
gprof
/usr/ccs/bin Displays the call-graph profile data (identifies the compute-intensive functions).prof
/usr/ccs/bin Displays the percentage of time spent in each routi ne.tcov
Comes withcompiler
Displays a count of how often each statement is executed (identifies the compute-intensive loops within a function).
time
/usr/bin/time Displays the total real and CPU time used by a program.If you're working on the OS kernel, most of the runtime tools are not available to you, because the kernel does not run as a user process. The compiletime tools, like lint, work, but otherwise we have to use the stone knives and flint axes: putting nonrandom patterns in memory to see when they are overwritten (two favorites are the hex constants
dead-beef
andabadcafe
), using printf 's or their equivalent, and logging trace information.Software Dogma
Debugging the Ker nel with grep
A kernel "panics", or comes to an abrupt halt, when it detects a situation that "cannot" arise.
For example, it finds a null pointer when looking for some essential data. Since there is no way it can recover from this, the safest course is to halt the processor before more data disappears. To solve a panic, you must first consider what happened that could possibly
frighten an operating system.
The kernel development group at Sun had one obscure bug that was very difficult to track down. The symptom was that kernel memory was getting overwritten at random,
occasionally panicking the system.
Two of our top engineers worked on this, and they noticed that it was always the 19th byte from the beginning of a memory block that was being creamed. This is an uncommon offset, unlike the more usual 2, 4, or 8 that occur everywhere. One of the engineers had a brainwave and used the offset to home in on the bug. He suggested they use the kernel debugger kadb to disassemble the kernel binary image (it took an hour!) into an ASCII file.
They then grepped the file, looking for "store" instructions where the operand indicated offset 19! One of these was almost guaranteed to be the instruction causing the corruption.
There were only eight of these, and they were all in the subsystem that dealt with process control. Now they were pretty sure where the problem was, and just needed to find out what it was. Further effort eventually yielded the cause: a race condition on a process control structure. This resulted in one thread marking the memory for return to the system before another thread had truly finished with it. Result: the kernel memory allocator then gave the memory away to someone else, but the process control code thought it still held it, and wrote into it, causing the otherwise impossible-to-find corruption!
Debugging an OS kernel with grep—what a concept. Sometimes even source tools can help solve runtime problems!
While on the subject of useful tools, Table 0-5 lists some ways to see exactly what the configuration of a Sun system is. However, none of these tools can help you unless you practice using them.
Table 6-5. Tools to Help Identify Your Hardware
What It Identifies Typical Output How to Invoke It
Kernel architecture sun4c
/usr/kvm/arch -k
Any OS patches applied
no patches are installed
/usr/bin/showrev -p
Various hard warethings
lots
/usr/sbin/prtconf
CPU clock rate 40MHz processor
/usr/sbin/psrinfo -v
hostid 554176fe
/usr/ucb/hostid
Memory 32Mb Displayed on power up
Serial number 4290302 Displayed on power up
ROM revision 2.4.1 Displayed on power up
Mounted disk 198Mb disk
/usr/bin/df -F ufs -k
Swap space 40Mb
/etc/swap -s
Ethernet address 8:0:20:f:8c:60
/usr/sbin/ifconfig -a
The ethernet address is built into the machine
IP address le0=129.144.248.36
/usr/sbin/ifconfig -a
The IP address is built into the network Floating-pointhardware
FPU's frequency appears to be 38.2
MHz