Archive for the ‘Debugging’ Category

Why are these learning approaches important to a system/embedded programming student

Monday, May 4th, 2015

These approaches are extremely important for any student looking to learn Linux system programming, Kernel progrmming, Linux Device Drivers or Embedded Linux. Following these tips will surely make learning more effective and help to build the right foundation and direction for a satisfying career as an Embedded Software Developer. These learning approaches include (more…)

Linux Driver & Embedded Batch starts 9th Feb.

Tuesday, January 20th, 2015

Linux Driver & Embedded Developer

Course starts 9th February, 7:15am to 8:40am

Now fully revised and updated

 Program Highlights

  • Fully revised for the latest 3.15 kernel
  • Added new sessions on Linux Kernel and Drivers
  • Fully revised Embedded Linux content
  • Easily switch from microcontroller backgrounds to OS based embedded systems
  • Placement support
  • Fully delivered by Raghu Bharadwaj (India’s leading corporate trainer on Linux system programming)
Full Course Contents http://techveda.org/linux-driver-embedded-developer-2/
Program Schedule and Fee

9th Feb., 7:15am to 8:40am. Fee: Rs.25,000/-. Register Now http://techveda.org/enroll-for-our-courses/

About the Speaker

Raghu Bharadwaj is India’s leading corporate trainer on Linux system, Kernel and Drivers programming. Delivered training to over 25 companies including Xilinx, Sasken, NCR, Cognizant, Broadcom, UTC, GE, Tech Mahindra etc.

Participant Feedback on this course

I consider this course as a boon for both Freshers & Experienced who want to build their career in Linux Drivers or Kernel. The entire course is designed in a very conceptual-way that even if you are a beginner, it won’t take much time to build the fundamentals. The course is more of a practical than just theoretical and has provided me a wide-exposure and clarity on various concepts than I have ever got through any book or course”

— Kuldeep Dave (C-DAC)

Registration Procedure

Please pay online here http://techveda.org/enroll-for-our-courses/

Or contact 9885808505 or info@techveda.org for more details

 

Veda Solutions

301, Prashanthiram Towers, Sarathi Studio Lane, Ameerpet, Hyderabad

www.techveda.org   Ph:66100265   Email:info@techveda.org

 

Upcoming Course Schedules

Wednesday, May 28th, 2014

Announcing our latest course schedules for you to choose based on your learning convenience. We are also for the first time introducing Instructor led virtual training on Embedded Linux for the convenience of outstation participants.

Schedules for June and July

Course Date Duration Delivery Mode
Embedded Linux June 7 & 8- (9:30am to 5:30pm) 2 full days Weekend Workshop- Classroom (Hyderabad)
Embedded Linux (New) June 16th (7am to 9am) 2 weeks Instructor led virtual training (Now learn from your home)
Linux System and Driver Developer(@ Bangalore) June 27,28,29 – July 4,5,6 (9:30am to 5:30pm) Spread over 2 weekends Classroom @ Bangalore
Linux Driver and Embedded Developer July 14th (7:20 pm to 8:45pm) 3 months Classroom @ Hyderabad

 

For registrations or for any further details regarding the courses please feel free to mail to info@techveda.org or call 040-66100265

 

Sincerely,

Sajith Kumar,

Operations Manager.

www.techveda.org

Linux Kernel Debugging Workshop

Thursday, August 29th, 2013

For the first time in India, Veda Solutions is launching an advanced and practical oriented weekend workshop on Linux Kernel Debugging. There was a reasonable delay in launching this course due to the complexity associated with a program like this, however we are happy that this delay has helped us to launch this much anticipated and one of its kind core training programs in much technical detail.

This workshop focuses on detailed debugging scenarios and debugging approaches which would vastly improve the debugging skills of a kernel level programmer. Course contents http://techveda.org/linux-kernel-debugging/

We would like to thank all the kernel programmers and our well wishers who incessantly requested to conduct this program.

Workshop Schedule: 31st Aug. and 1st Sep.

Time: 10am to 5pm

Fee: Rs.5000/- only

Registrations are open till 31st Aug. 10:30am

We request all the interested Linux kernel programmers to attend this workshop.

This weekend program will now be a regular feature of our training calendar

 

Gdb Tutorial-II

Thursday, November 1st, 2012

In the previous Tutorial (GDB Tutorial-I)we looked into basic environment and commands of gnu source debugger gdb,  we have also looked into how gnu debugger works , lets now run some debug sessions using sample buggy programs

(more…)

Using Valgrind: Video Tutorial

Saturday, October 13th, 2012

Crash Dump analysis Howto

Sunday, September 30th, 2012

Video Tutorial  on Application Crash  Dump Analysis

Note: It is recommeneded to watch them in High Definition Mode 720p or 1080p  with full screen.

Look here http://www.labnol.org/internet/youtube-video-hdplayback-quality/13873/ for hd playback settings.

 

How source debuggers work?

Sunday, September 23rd, 2012

Introduction

Application binaries are a result of compile and build operations performed on a single or a set of source files. Program Source files contain functions, data variables of various types (local, global, register, static), and abstract data objects, all written and neatly indented with nested control structures as per high level programming language syntax (C/C++).  Compilers translate code in each source file into machine instructions (1’s and 0’s) as per target processors Instruction set Architecture and bury that code into object files. Further, Linkers integrate compiled object files with other pre-compiled objects files (libraries, runtime binaries) to create end application binary image called executable.

Source debuggers are tools used to trace execution of an application executable binary. Most amazing feature of a source debugger is its ability to list source code of the program being debugged; it can show the line or expression in the source code that resulted in a particular machine code instruction of a running program loaded in memory. This helps the programmer to analyze a program’s behavior in the high-level terms like source-level flow control constructs, procedure calls, named variables, etc,  instead of machine instructions and memory locations. Source-level debugging also makes it possible to step through execution a line at a time and set source-level breakpoints. (If you do not have any prior hands on experience with source debuggers I suggest you to look at this before continuing with following.)

lets explore how source debuggers like gnu gdb work ? So how does a debugger know where to stop when you ask it to break at the entry to some function? How does it manage to find what to show you when you ask it for the value of a variable? The answer is – debug information.  All modern compilers are designed to generate Debug information together with the machine code of the source file. It is a representation of the relationship between the executable program and the original source code. This information is encoded as per  a pre-defined format and stored alongside the machine code. Many such formats were invented over the years for different platforms and executable files (aim of this article isn’t to survey the history of these formats, but rather to show how they work). Gnu compiler and ELF executable on Linux/ UNIX platforms use DWARF, which is widely used today as default debugging information format.

Word of Advice : Does an Application/ Kernel programmer need to know Dwarf?

Obvious answer to this question is a big NO.  It is purely subject matter for developers involved in implementation of a Debugger tool. A normal Application developer using debugger tools would never need to learn or dig into binary files for debug information. This in no way adds any edge to your debugging skills nor adds any new skills into your armory. However, if you are a developer using debuggers for years and curious about how debuggers work read this document for an outline into debug information.  If you are a beginner to systems programming or fresher’s learning programming I would  suggest  not to waste your time as you can safely ignore this.

ELF -DWARF sections

Gnu compiler generates debug information which is organized into various sections of the ELF object file. Let’s use the following source file for compiling and observing DWARF sections

root@techveda:~# vim sample.c

#include <stdio.h>
#include <stdlib.h>

int add(int x, int y)
{
        return x + y;
}
int main()
{
        int a = 10, b = 20;
        int result;
        int (*fp) (int, int);

        fp = add;
        result = (*fp) (a, b);
        printf(" %d\n", result);
}

root@techveda:~# gcc -c -g sample.c -o sample.o
root@techveda:~# objdump -h sample.o | more

sample.o:     file format elf32-i386

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         0000005f  00000000  00000000  00000034  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .data         00000000  00000000  00000000  00000094  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  2 .bss          00000000  00000000  00000000  00000094  2**2
                  ALLOC
  3 .debug_abbrev 000000a2  00000000  00000000  00000094  2**0
                  CONTENTS, READONLY, DEBUGGING
  4 .debug_info   00000114  00000000  00000000  00000136  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  5 .debug_line   00000040  00000000  00000000  0000024a  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  6 .rodata       00000005  00000000  00000000  0000028a  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .debug_loc    00000070  00000000  00000000  0000028f  2**0
                  CONTENTS, READONLY, DEBUGGING
  8 .debug_pubnames 00000023  00000000  00000000  000002ff  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  9 .debug_pubtypes 00000012  00000000  00000000  00000322  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
 10 .debug_aranges 00000020  00000000  00000000  00000334  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
 11 .debug_str    000000b0  00000000  00000000  00000354  2**0
                  CONTENTS, READONLY, DEBUGGING
 12 .comment      0000002b  00000000  00000000  00000404  2**0
                  CONTENTS, READONLY
 13 .note.GNU-stack 00000000  00000000  00000000  0000042f  2**0
                  CONTENTS, READONLY
 14 .debug_frame  00000054  00000000  00000000  00000430  2**2
                  CONTENTS, RELOC, READONLY, DEBUGGING

All of the sections with naming debug_xxx are debugging information sections.  Information in these sections is interpreted by source debugger like gdb.  Each debug_ section holds specific information like

.debug_info               core DWARF data containing DIEs
.debug_line               Line Number Program
.debug_frame              Call Frame Information
.debug_macinfo            lookup table for global objects and functions
.debug_pubnames           lookup table for global objects and functions
.debug_pubtypes           lookup table for global types
.debug_loc                Macro descriptions
.debug_abbrev             Abbreviations used in the .debug_info section
.debug_aranges            mapping between memory address and compilation
.debug_ranges             Address ranges referenced by DIEs
.debug_str                String table used by .debug_info

Debugging Information Entry (DIE)

Dwarf format organizes debug data in all of the above sections using special objects (program descriptive entities) called Debugging Information Entry (DIE).  Each DIE has a tag filed whose value specifies its type, and a set of attributes. DIEs are interlinked via sibling and child links, and values of attributes can point at other DIEs. Now let’s dig into ELF file to view how a DIE looks like. We will begin our exploration with .debug_info section of the ELF file since core DIE’s are listed in it.

root@techveda:~# objdump --dwarf=info . /sample

Above operation shows long list of DIE’s. Let’s limit ourselves to relevant information

./sample:     file format elf32-i386

Contents of the .debug_info section:

  Compilation Unit @ offset 0x0:
   Length:        0x110 (32-bit)
   Version:       2
   Abbrev Offset: 0
   Pointer Size:  4
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    < c>   DW_AT_producer    : (indirect string, offset: 0xe): GNU C 4.5.2	
    <10>   DW_AT_language    : 1	(ANSI C)
    <11>   DW_AT_name        : (indirect string, offset: 0x44): sample.c	
    <15>   DW_AT_comp_dir    : (indirect string, offset: 0x71): /root	
    <19>   DW_AT_low_pc      : 0x80483c4	
    <1d>   DW_AT_high_pc     : 0x8048423	
    <21>   DW_AT_stmt_list   : 0x0

Each source file in the application is referred in dwarf terminology as a “compilation unit”.   Dwarf data for each compilation unit (source file) starts with a compilation unit DIE. Above dump shows the first DIE’s and tag value “DW_TAG_compile_unit “.  This DIE provides general information about compilation unit like source file name (DW_AT_name   : (indirect string, offset: 0x44): sample.c),  high level programming language used to write source file(DW_AT_language    : 1    (ANSI C))  ,   directory of the source file(DW_AT_comp_dir    : (indirect string, offset: 0x71): /root) ,  compiler and producer of dwarf data(  DW_AT_producer    : (indirect string, offset: 0xe): GNU C 4.5.2) , start virtual address of the compilation unit  (DW_AT_low_pc      : 0x80483c4), end virtual address of the unit (DW_AT_high_pc     : 0x8048423).

Compilation Unit DIE is the parent for all the other DIE’s that describe elements of source file. Generally, the list of DIE’s that follow will describe data types, followed by global data, then the functions that make up the source file. The DIEs for variables and functions are in the same order in which they appear in the source file.

How does debugger locate Function Information ?

 While using source debuggers we often instruct debugger to insert or place break point at some function, expecting the debugger to pause program execution at functions. To be able to perform this task, debugger must have some mapping between a function name in the high-level code and the address in the machine code where the instructions for this function begin. For this mapping information debuggers rely on DIE’s that describes specified function. DIE’s describing functions in a compilation unit are assigned tag value DW_TAG_subprogram” subprogram as per dwarf terminology is a function.

In our sample application source we have two functions (main, add), dwarf should generate a “DW_TAG_subprogramDIE’s for each function, these DIE attributes would define function mapping information that debugger needs for resolving machine code addresses with function name.

Each “DW_TAG_subprogam” DIE contains

  1. function scope
  2. function name
  3. source file or compilation unit in which function is located
  4.  line no in the source file where the function starts
  5. Functions return type
  6. Start address of the fucntion
  7. End address of the function
  8. Frame information of the function.

 

 root@techveda:~# objdump --dwarf=info ./sample | grep "DW_TAG_subprogram"
 <1><72>: Abbrev Number: 4 (DW_TAG_subprogram)
 <1><a8>: Abbrev Number: 6 (DW_TAG_subprogram)

<1><72>: Abbrev Number: 4 (DW_TAG_subprogram)
    <73>   DW_AT_external    : 1	
    <74>   DW_AT_name        : add	
    <78>   DW_AT_decl_file   : 1	
    <79>   DW_AT_decl_line   : 4	
    <7a>   DW_AT_prototyped  : 1	
    <7b>   DW_AT_type        : <0x4f>	
    <7f>   DW_AT_low_pc      : 0x80483c4	
    <83>   DW_AT_high_pc     : 0x80483d1	
    <87>   DW_AT_frame_base  : 0x0	(location list)
    <8b>   DW_AT_sibling     : <0xa8>	

<1><a8>: Abbrev Number: 6 (DW_TAG_subprogram)
    <a9>   DW_AT_external    : 1	
    <aa>   DW_AT_name        : (indirect string, offset: 0x1a): main	
    <ae>   DW_AT_decl_file   : 1	
    <af>   DW_AT_decl_line   : 8	
    <b0>   DW_AT_type        : <0x4f>	
    <b4>   DW_AT_low_pc      : 0x80483d2	
    <b8>   DW_AT_high_pc     : 0x8048423	
    <bc>   DW_AT_frame_base  : 0x38	(location list)
    <c0>   DW_AT_sibling     : <0xf8>

 

We now have accessed DIE description of function’s main and addLet’s analyze attribute information of add fucntions DIE.

Function scope:    DW_AT_external    : 1 (scope external)

Function name:     DW_AT_name        : add

Source file or compilation unit in which function is located:  DW_AT_decl_file   : 1 (indicates 1st compilation unit which is sample.c)

line no in the source file where the function starts:  DW_AT_decl_line   : 4 ( indicates line no 4 in source file)

#include <stdio.h>
#include <stdlib.h>

int add(int x, int y)
{
        return x + y;
}
int main()
{
        int a = 10, b = 20;
        int result;
        int (*fp) (int, int);

        fp = add;
        result = (*fp) (a, b);
        printf(" %d\n", result);
}

Function’s source line no matched with DIE description of line no. let’s continue with rest of the attribute values

Functions return type:  DW_AT_type        : <0x4f>

As we have already understood that values of attributes can point to other DIE , here is an example of it.  Value   DW_AT_type     : <0x4f> indicates that return type description is stored in other DIE at offset 0x4f.

<4f>: Abbrev Number: 3 (DW_TAG_base_type)
    <50>   DW_AT_byte_size   : 4	
    <51>   DW_AT_encoding    : 5	(signed)
    <52>   DW_AT_name        : int

This DIE describes data type and composition of return type of the function add, as per DIE attribute values return type is signed int  of size 4 bytes.

Start address of the function : DW_AT_low_pc   : 0x80483c4

End address of the function: DW_AT_high_pc     : 0x80483d1

Above values indicate start and end virtual address of the machine instructions of add function, we can verify that with binary dump of the function

080483c4 <add>:
 80483c4:	55                   	push   %ebp
 80483c5:	89 e5                	mov    %esp,%ebp
 80483c7:	8b 45 0c             	mov    0xc(%ebp),%eax
 80483ca:	8b 55 08             	mov    0x8(%ebp),%edx
 80483cd:	8d 04 02             	lea    (%edx,%eax,1),%eax
 80483d0:	5d                   	pop    %ebp
 80483d1:	c3                   	ret

How does debugger find program data (variables…) Information?

When the program hits assigned break point in a function, debugger pauses the program execution, at this time we can instruct debugger to show or print values of variables, by using debugger commands like print or display followed by variable name (ex: print a) How does debugger know where to find memory location of the variable ? Variables can be located in global storage, on the stack, and even in registers.The debugging information has to be able to reflect all these variations, and indeed DWARF does. As an example let’s take a look at complete DIE information set for main function.

<1><a8>: Abbrev Number: 6 (DW_TAG_subprogram)
    <a9>   DW_AT_external    : 1	
    <aa>   DW_AT_name        : (indirect string, offset: 0x1a): main	
    <ae>   DW_AT_decl_file   : 1	
    <af>   DW_AT_decl_line   : 8	
    <b0>   DW_AT_type        : <0x4f>	
    <b4>   DW_AT_low_pc      : 0x80483d2	
    <b8>   DW_AT_high_pc     : 0x8048423	
    <bc>   DW_AT_frame_base  : 0x38	(location list)
    <c0>   DW_AT_sibling     : <0xf8>	
 <2><c4>: Abbrev Number: 7 (DW_TAG_variable)
    <c5>   DW_AT_name        : a	
    <c7>   DW_AT_decl_file   : 1	
    <c8>   DW_AT_decl_line   : 10	
    <c9>   DW_AT_type        : <0x4f>	
    <cd>   DW_AT_location    : 2 byte block: 74 1c 	(DW_OP_breg4 (esp): 28)
 <2><d0>: Abbrev Number: 7 (DW_TAG_variable)
    <d1>   DW_AT_name        : b	
    <d3>   DW_AT_decl_file   : 1	
    <d4>   DW_AT_decl_line   : 10	
    <d5>   DW_AT_type        : <0x4f>	
    <d9>   DW_AT_location    : 2 byte block: 74 18 	(DW_OP_breg4 (esp): 24)
 <2><dc>: Abbrev Number: 8 (DW_TAG_variable)
    <dd>           : (indirect string, offset: 0x6a): result	
    <e1>   DW_AT_decl_file   : 1	
    <e2>   DW_AT_decl_line   : 11	
    <e3>   DW_AT_type        : <0x4f>	
    <e7>   DW_AT_location    : 2 byte block: 74 10 	(DW_OP_breg4 (esp): 16)
 <2><ea>: Abbrev Number: 7 (DW_TAG_variable)
    <eb>   DW_AT_name        : fp	
    <ee>   DW_AT_decl_file   : 1	
    <ef>   DW_AT_decl_line   : 12	
    <f0>   DW_AT_type        : <0x10d>	
    <f4>   DW_AT_location    : 2 byte block: 74 14 	(DW_OP_breg4 (esp): 20)

Note the first number inside the angle brackets in each entry. This is the nesting level – in this example entries with <2> are children of the entry with <1>. main function has three integer variables a,b and result each of these variables are described with DW_TAG_variable nested DIE’s (0xc4, 0xd0, 0xdc). main function also has a function pointer  fp described  in DIE 0xea . Variable DIE attributes specify variable name (DW_AT_name), declaration line no in source function (DW_AT_decl_line ), pointer to address of DIE describing variables data type (DW_AT_type) and relative location of the variable within function’s frame (DW_AT_location).

To locate the variable in the memory image of the executing process, the debugger will look at the DW_AT_location attribute of DIE. For a its value is    DW_OP_fbreg4 (esp):28. This means that the variable is stored at offset 28  from the top in the frame of  containing function. The DW_AT_frame_base attribute of main has the value 0x38(location list), which means that this value actually has to be looked up in the location list section. Let’s look at it:

root@techveda:~# objdump --dwarf=loc sample

sample:     file format elf32-i386

Contents of the .debug_loc section:

    Offset   Begin    End      Expression
    00000000 080483c4 080483c5 (DW_OP_breg4 (esp): 4)
    00000000 080483c5 080483c7 (DW_OP_breg4 (esp): 8)
    00000000 080483c7 080483d1 (DW_OP_breg5 (ebp): 8)
    00000000 080483d1 080483d2 (DW_OP_breg4 (esp): 4)
    00000000 <End of list>
    00000038 080483d2 080483d3 (DW_OP_breg4 (esp): 4)
    00000038 080483d3 080483d5 (DW_OP_breg4 (esp): 8)
    00000038 080483d5 08048422 (DW_OP_breg5 (ebp): 8)
    00000038 08048422 08048423 (DW_OP_breg4 (esp): 4)
    00000038 <End of list>

Offset column 0x38 values are the entries for main function variables. Each entry here describes possible frame base address with respect to where debugger may be paused by break point within function instructions; it specifies the current frame base from which offsets to variables are to be computed as an offset from a register. For x86, bpreg4 refers to esp and bpreg5 refers to ebp.  Before analyzing further lets look at disassemble dump for main function

080483d2 <main>:
 80483d2:	55                   	push   %ebp
 80483d3:	89 e5                	mov    %esp,%ebp
 80483d5:	83 e4 f0             	and    $0xfffffff0,%esp
 80483d8:	83 ec 20             	sub    $0x20,%esp
 80483db:	c7 44 24 1c 0a 00 00 	movl   $0xa,0x1c(%esp)
 80483e2:	00 
 80483e3:	c7 44 24 18 14 00 00 	movl   $0x14,0x18(%esp)
 80483ea:	00 
 80483eb:	c7 44 24 14 c4 83 04 	movl   $0x80483c4,0x14(%esp)
 80483f2:	08 
 80483f3:	8b 44 24 18          	mov    0x18(%esp),%eax
 80483f7:	89 44 24 04          	mov    %eax,0x4(%esp)
 80483fb:	8b 44 24 1c          	mov    0x1c(%esp),%eax
 80483ff:	89 04 24             	mov    %eax,(%esp)
 8048402:	8b 44 24 14          	mov    0x14(%esp),%eax
 8048406:	ff d0                	            call   *%eax
 8048408:	89 44 24 10          	mov    %eax,0x10(%esp)
 804840c:	b8 f0 84 04 08       	mov    $0x80484f0,%eax
 8048411:	8b 54 24 10          	mov    0x10(%esp),%edx
 8048415:	89 54 24 04          	mov    %edx,0x4(%esp)
 8048419:	89 04 24             	mov    %eax,(%esp)
 804841c:	e8 d3 fe ff ff       	call   80482f4 <printf@plt>
 8048421:	c9                   	leave  
 8048422:	c3                   	ret

First two instructions deal with function’s preamble, function’s stack frame base pointer is determined after pre-amble instructions are executed.  Ebp remains constant throughout function’s execution and esp keeps changing with data being pushed and popped from the stack frame. From the above dump instructions at offset 80483db and 80483e3 are assigning values 10 and 20 to variables a, and b. These variables are being accessed their offset in stack frame relative to location of current esp( variable a: 0x1c(%esp), variable b: 0x18(%esp)). Now let’s assume that break point was set after initializing a and b variables and program paused, and we have run print command to view conents of a or b variables. Debugger would access 3rd record of the main function’s dwarf debug.loc table since our break point falls between 080483d5 – 08048422 region of the function code.

00000038 080483d2 080483d3 (DW_OP_breg4 (esp): 4)
00000038 080483d3 080483d5 (DW_OP_breg4 (esp): 8)
00000038 080483d5 08048422 (DW_OP_breg5 (ebp): 8)--- 3rd record
00000038 08048422 08048423 (DW_OP_breg4 (esp): 4)

Now as per records debugger will locate a with esp + 28 , b with esp +24 and so on…

Looking up line number information

We can set breakpoints mentioning line no’s Lets now look at how debuggers resolve line no’s to machine instruction’s? DWARF encodes a full mapping between lines in the C source code and machine code addresses in the executable. This information is contained in the .debug_line section and can be extracted using objdump

root@techveda:~# objdump --dwarf=decodedline ./sample

./sample:     file format elf32-i386

Decoded dump of debug contents of section .debug_line:

CU: sample.c:
File name                            Line number    Starting address					
sample.c                                       5          0x80483c4
sample.c                                       6          0x80483c7
sample.c                                       7          0x80483d0
sample.c                                       9          0x80483d2
sample.c                                      10          0x80483db
sample.c                                      14          0x80483eb
sample.c                                      15          0x80483f3
sample.c                                      16          0x804840c
sample.c                                      17          0x8048421

This dump shows machine instruction line no’s for our program. It is quite obvious that line no’s for non-executable statements of the source file need not be tracked by dwarf .we can map the above dump to our source code for analysis.

  1 #include <stdio.h>
  2 #include <stdlib.h>
  3 
  4 int add(int x, int y)
  5 {
  6         return x + y;
  7 }
  8 int main()
  9 {
 10         int a = 10, b = 20;
 11         int result;
 12         int (*fp) (int, int);
 13 
 14         fp = add;
 15         result = (*fp) (a, b);
 16         printf(" %d\n", result);
 17 }

From the above it should be clear of what debugger does it is instructed to set breakpoint at entry into function add, it would insert break point at line no 6 and pause after pre-amble of function add is executed.

What’s next?

if you are into implementation of debugging tools or involved in writing programs/tools  that simulate debugger facilities/ read binary files , you may be interested in specific programming libraries libbfd or libdwarf.

Binary File Descriptor library (BFD) or libbfd as it is called provides ready to use functions to read into ELF and other popular binary files. BFD works by presenting a common abstract view of object files. An object file has a “header” with descriptive info; a variable number of “sections” that each has a name, some attributes, and a block of data; a symbol table; relocation entries; and so forth. Gnu binutils package tools like objdump, readelf and others have been written using these libraries.

Libdwarf is a C library intended to simplify reading (and writing) applications built with DWARF2, DWARF3 debug information.

Dwarfdump is an application written using libdwarf to print dwarf information in a human readable format. It is also open sourced and is copyrighted GPL. It provides an example of using libdwarf to read DWARF2/3 information as well as providing readable text output.

References

  1. Dwarf Standard
  2. Libdwarf Wiki
  3. Libbfd draft

 

Gdb Tutorial-I

Sunday, September 16th, 2012

Debugger is a tool to test and debug other programs. A debugger is called a source code Debugger or symbolic debugger if it can show the current point of execution, or the Location of a program error, inside the program’s source code.

With a source code debugger, you can step through your code line by line, see what path is taken through the program’s conditional and loop statements, show what functions are called, and Where in the function call stack you are. You can inspect the values of variables. You can set breakpoints on individual lines of code, and then let the program run until it reaches this line; this is convenient for navigating through complicated programs.

Getting started with Gdb

Gnu Gdb is a source level debugger, that is widely used on *nix platforms. This part of the tutorial we intend to provide an overview of widely used gdb commands. Gdb kind of Source level debuggers require some debugging information built into the binary application. That means additional symbol and metadata information has to be built into the executable binaries. This can be achieved by instructing  the compiler to include symbol, metadata  information describing the syntax of our source program.

info@techveda.org:~# gcc -g -c sample.c –o sample.o
info@techveda.org:~# gcc sample.o –o sample

-g flag instructs gdb to generate and add debug information. Let’s run objdump on relocatable object file and find additional debug sections generated for gdb’s use.

info@techveda.org:~# objdump -h sample.o 

sample.o:     file format elf32-i386

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         000000e8  00000000  00000000  00000034  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .data         00000000  00000000  00000000  0000011c  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  2 .bss          00000000  00000000  00000000  0000011c  2**2
                  ALLOC
  3 .debug_abbrev 000000a2  00000000  00000000  0000011c  2**0
                  CONTENTS, READONLY, DEBUGGING
  4 .debug_info   00000180  00000000  00000000  000001be  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  5 .debug_line   00000050  00000000  00000000  0000033e  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  6 .rodata       00000005  00000000  00000000  0000038e  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .debug_loc    000000e0  00000000  00000000  00000393  2**0
                  CONTENTS, READONLY, DEBUGGING
  8 .debug_pubnames 00000033  00000000  00000000  00000473  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  9 .debug_pubtypes 00000012  00000000  00000000  000004a6  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
 10 .debug_aranges 00000020  00000000  00000000  000004b8  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
 11 .debug_str    000000b0  00000000  00000000  000004d8  2**0
                  CONTENTS, READONLY, DEBUGGING
 12 .comment      0000002b  00000000  00000000  00000588  2**0
                  CONTENTS, READONLY
 13 .note.GNU-stack 00000000  00000000  00000000  000005b3  2**0
                  CONTENTS, READONLY
 14 .debug_frame  00000094  00000000  00000000  000005b4  2**2
                  CONTENTS, RELOC, READONLY, DEBUGGING

All the sections starting with .debug have been generated at compile time and include debug information, the contents of the above sections are arranged/ organized as per Debugging information formats, and these formats are specifications that describe layout of debug info in an object file. There are several debugging formats: stabs, COFF, PE-COFF, OMF, IEEE-695, and three versions of DWARF, to name a few.

Dwarf is the most widely used debug information format, In Linux, it is the default debugging format. Description of Dwarf standard and how the above sections are organized can be found here . Let’s conclude for now that additional debug sections identified are dwarf sections, and gdb interprets dwarf information in those debug sections. Let’s initiate debug session using gdb with an executable image

info@techveda.org:~#  gdb ./sample
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/sample...done.
(gdb)

Gdb starts by reading dwarf debug symbol information from the presented executable and shows up gdb prompt. Using gdb commands we can instruct gdb to run, stop, continue and perform various operations on the executable binary. Lets begin our session with  help command

(gdb) help
List of classes of commands:

aliases -- Aliases of other commands
breakpoints -- Making program stop at certain points
data -- Examining data
files -- Specifying and examining files
internals -- Maintenance commands
obscure -- Obscure features
running -- Running the program
stack -- Examining the stack
status -- Status inquiries
support -- Support facilities
tracepoints -- Tracing of program execution without stopping the program
user-defined -- User-defined commands

Type "help" followed by a class name for a list of commands in that class.
Type "help all" for the list of all commands.
Type "help" followed by command name for full documentation.
Type "apropos word" to search for commands related to "word".
Command name abbreviations are allowed if unambiguous.
(gdb)

Gdb supports various operations on the executable and commands have been classified into groups shown above, help command can also be used to print more commands in each category list shown above. Help can also be used to get documentation on specific gdb command. Gdb also supports execution of shell commands from gdb prompt, this facility comes very handy while inspecting multi-process, multi-thread, network communication applications and to run various tools without terminating current debug session.

(gdb) shell ls -l /usr/src
total 115188
drwxrwxr-x 25 root root     4096 2012-09-06 12:53 linux-3.2.9
-rw-r--r--  1 root src  78132997 2012-04-08 22:12 linux-3.2.9.tar.bz2
drwxr-xr-x 24 root root     4096 2011-04-26 04:38 linux-headers-2.6.38-8
drwxr-xr-x  7 root root     4096 2011-04-26 04:38 linux-headers-2.6.38-8-generic
-rw-r--r--  1 root src  39797000 2012-04-09 18:46 linux-image-3.2.9.debug_3.2.9.debug-10.00.Custom_i386.deb
drwxr-xr-x  5 root root     4096 2012-03-25 21:17 linux-source-2.6.38
(gdb)
(gdb) shell objdump -h ./sample.o

./sample.o:     file format elf32-i386

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         000000e8  00000000  00000000  00000034  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .data         00000000  00000000  00000000  0000011c  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  2 .bss          00000000  00000000  00000000  0000011c  2**2
                  ALLOC
  3 .debug_abbrev 000000a2  00000000  00000000  0000011c  2**0
                  CONTENTS, READONLY, DEBUGGING
  4 .debug_info   00000180  00000000  00000000  000001be  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  5 .debug_line   00000050  00000000  00000000  0000033e  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  6 .rodata       00000005  00000000  00000000  0000038e  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .debug_loc    000000e0  00000000  00000000  00000393  2**0
                  CONTENTS, READONLY, DEBUGGING
  8 .debug_pubnames 00000033  00000000  00000000  00000473  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  9 .debug_pubtypes 00000012  00000000  00000000  000004a6  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
 10 .debug_aranges 00000020  00000000  00000000  000004b8  2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
 11 .debug_str    000000b0  00000000  00000000  000004d8  2**0
                  CONTENTS, READONLY, DEBUGGING
 12 .comment      0000002b  00000000  00000000  00000588  2**0
                  CONTENTS, READONLY
 13 .note.GNU-stack 00000000  00000000  00000000  000005b3  2**0
                  CONTENTS, READONLY
 14 .debug_frame  00000094  00000000  00000000  000005b4  2**2
                  CONTENTS, RELOC, READONLY, DEBUGGING

List command can used to view source instructions of the executable attached to debugger,  we can access documentation on list command using help command sample executable that we have currently attached to gdb, has one source file sample.c.

(gdb) help list

List specified function or line.
With no argument, lists ten more lines after or around previous listing.
"list -" lists the ten lines before a previous ten-line listing.
One argument specifies a line, and ten lines are listed around that line.
Two arguments with comma between specify starting and ending lines to list.
Lines can be specified in these ways:
  LINENUM, to list around that line in current file,
  FILE:LINENUM, to list around that line in that file,
  FUNCTION, to list around beginning of that function,
  FILE:FUNCTION, to distinguish among like-named static functions.
  *ADDRESS, to list around the line containing that address.
With two args if one is empty it stands for ten lines away from the other arg.
(gdb)

(gdb) list 1

1          #include <stdio.h>
2          #include <stdlib.h>
3
4          int add(int x, int y)
5          {
6                      return (x + y);
7          }
8
9          int sub(int x, int y)
10        {
(gdb) <Enter key>
11                    return (x - y);
12        }
13
14        int mul(int x, int y)
15        {
16                    return (x * y);
17        }
18
19        int main()
20        {
(gdb) <Enter key>
21                    int a = 10, b = 20;
22                    int result;
23                    int (*fp) (int, int);
24
25                    fp = add;
26                    result = (*fp) (a, b);
27                    printf(" %d\n", result);
28
29                    fp = sub;
30                    result = fp(a, b);

(gdb)

To instruct gdb to run current program executable attached use run command

(gdb) run
Starting program: /root/sample
 30
 -10
 200

Program exited normally.
(gdb)

Break points

The principal purposes of using a debugger is to inspect elements of a running program, Breakpoints are strategic instructions in your program where debugger would stop “running program”, and would not allow program to continue until instructed. Break points aid inspection of stack frames, local data values, global data values, processor register dump and so on. A program can be set with any number of break points.

(gdb) help break
Set breakpoint at specified line or function.
break [LOCATION] [thread THREADNUM] [if CONDITION]
LOCATION may be a line number, function name, or "*" and an address.
If a line number is specified, break at start of code for that line.
If a function is specified, break at start of code for that function.
If an address is specified, break at that exact address.
With no LOCATION, uses current execution address of the selected
stack frame.  This is useful for breaking on return to a stack frame.

THREADNUM is the number from "info threads".
CONDITION is a boolean expression.

Multiple breakpoints at one place are permitted, and useful if their
conditions are different.

Do "help breakpoints" for info on other commands dealing with breakpoints.
(gdb)

Above Documentation clearly states how break command can be used to set breakpoints at a Particular line no, function, or an address. Break points can be joined with conditional expressions which are often referred as conditional breakpoints.

(gdb) break 1
Breakpoint 1 at 0x80483c7: file sample.c, line 1.
(gdb) break 4
Note: breakpoint 1 also set at pc 0x80483c7.
Breakpoint 2 at 0x80483c7: file sample.c, line 4.
(gdb) list 1
1	#include <stdio.h>
2	#include <stdlib.h>
3
4	int add(int x, int y)
5	{
6		return (x + y);
7	}
8
9	int sub(int x, int y)
10	{
(gdb) break add
Note: breakpoints 1 and 2 also set at pc 0x80483c7.
Breakpoint 3 at 0x80483c7: file sample.c, line 6.
(gdb)

We have now set three break points, first and second break points have been set using line no parameter and third break point has been set using fuction name. gdb maintains break point information in a table which can be viewed with  command info breakpoints

 

(gdb) info breakpoints
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   0x080483c7 in add at sample.c:1
2       breakpoint     keep y   0x080483c7 in add at sample.c:4
3       breakpoint     keep y   0x080483c7 in add at sample.c:6
(gdb)

Gdb’s breakpoint table provides attributes for each break point. Each breakpoint has a unique identifier  (see “Num” column  the above table), break points can be managed using these unique identifiers.

Type field of the table mentions the type of break point, gdb categorizes break points into three types called breakpoints, watchpoints, and catchpoints.

Dispostion field(Disp) indicates what happens to the breakpoint after the next time it causes gdb to pause the programs execution. There are three possible disposition values

Keep: breakpoint will be retained after the next time it is reached. This is default value for all new    breakpoints

Del: breakpoint will be deleted after the next time it is reached

Dis: breakpoint will be disabled the next time it is reached.

Enable status(Enb) field indicated whether the breakpoint is currently enabled or disabled

Address field specifies the location in memory where the breakpoint is set.

Location (What) field shows the line number and source filename of the breakpoints

We will run the program with breakpoints set

(gdb) run
Starting program: /root/sample 

Breakpoint 1, add (x=10, y=20) at sample.c:6
6		return (x + y);
(gdb)

First break point is now hit and program has stopped in the function add at line no 6 of source file.  As per breakpoint table record first break point should have been hit at line 1, second break point at line no 4 of the source file , but here we see gdb reports that first break point is hit and line no shown is 6

(gdb) list 1
1	#include <stdio.h>
2	#include <stdlib.h>
3
4	int add(int x, int y)
5	{
6		return (x + y);
7	}
8
9	int sub(int x, int y)
10	{

Line no 1 and 4 of the source file at which breakpoints 1 and 2 were set do not contain processor executable instructions as breakpoints are set at machine instruction level, so first breakpoint set at line 1 is automatically moved to first executable instruction after line 1 which happens to be line 5 that contains preamble to add function, program executed preamble instruction and break point was hit. This test clearly shows that when break points are set using line no as parameter, may not break precisely at the specified source line no.

Now since program has hit the first break point lets probe program data variables, function arguments and register values using gdb commands .To print the list of functions executed until this breakpoint we can use gdb command “backtrace”

 

(gdb) backtrace
#0  add (x=10, y=20) at sample.c:6
#1  0x08048425 in main () at sample.c:26
(gdb)

Above trace shows functions executed so far until this breakpoint and their stack frames. #0 entry is the current function and signifies current stack frame, #1 signifies the caller function. Main function has called add function with 2 integer arguments and we can see values passed

Print command can show status of parameters and local variables of the current frame. It can be used with parameter/variable name or address, it can also be used access address of local variables and parameters in stack frame

 

Set command lets us alter values of local variables or parameters of current frame, again we can use set command with name of the symbol or address of symbol

(gdb) print x
$1 = 10
(gdb) print y
$2 = 20
(gdb) print &x
$3 = (int *) 0xbffff3c0
(gdb) print &y
$4 = (int *) 0xbffff3c4
(gdb) print *(0xbffff3c0)
$5 = 10
(gdb) print *(0xbffff3c4)
$6 = 20
(gdb) set x=40
(gdb) print
$11 = 40
(gdb) set *(0xbffff3c0)=50
(gdb) print x
$12 = 50
(gdb)

We can access  processor register values using command info registers, this command can be used to access all register values or a specific register.

(gdb) info registers
eax            0x80483c4	134513604
ecx            0xcca81810	-861399024
edx            0x1	1
ebx            0xb7fcaff4	-1208176652
esp            0xbffff3b8	0xbffff3b8
ebp            0xbffff3b8	0xbffff3b8
esi            0x0	0
edi            0x0	0
eip            0x80483c7	0x80483c7 <add+3>
eflags         0x286	[ PF SF IF ]
cs             0x73	115
ss             0x7b	123
ds             0x7b	123
es             0x7b	123
fs             0x0	0
gs             0x33	51
(gdb) info register
eax            0x80483c4	134513604
ecx            0xcca81810	-861399024
edx            0x1	1
ebx            0xb7fcaff4	-1208176652
esp            0xbffff3b8	0xbffff3b8
ebp            0xbffff3b8	0xbffff3b8
esi            0x0	0
edi            0x0	0
eip            0x80483c7	0x80483c7 <add+3>
eflags         0x286	[ PF SF IF ]
cs             0x73	115
ss             0x7b	123
ds             0x7b	123
es             0x7b	123
fs             0x0	0
gs             0x33	51
(gdb) info register ebp
ebp            0xbffff3b8	   0xbffff3b8
(gdb) info register esp
esp            0xbffff3b8 	   0xbffff3b8

(gdb) info frame
Stack level 0, frame at 0xbffff3c0:
 eip = 0x80483c7 in add (sample.c:6); saved eip 0x8048425
 called by frame at 0xbffff3f0
 source language c.
 Arglist at 0xbffff3b8, args: x=50, y=20
 Locals at 0xbffff3b8, Previous frame's sp is 0xbffff3c0
 Saved registers:
  ebp at 0xbffff3b8, eip at 0xbffff3bc
(gdb)

Info frame command shows details of current stack frame and processors registers like eip, esp and so on. Data shown includes argument addresses, values and caller frames base pointer, and stack pointer. We can move down into caller functions stack frame or any other stack frame show in the back trace using select-frame command and examine its contents

(gdb) select-frame 1
(gdb) info frame
Stack level 1, frame at 0xbffff3f0:
 eip = 0x8048425 in main (sample.c:26); saved eip 0xb7e83e37
 caller of frame at 0xbffff3c0
 source language c.
 Arglist at 0xbffff3e8, args:
 Locals at 0xbffff3e8, Previous frame's sp is 0xbffff3f0
 Saved registers:
  ebp at 0xbffff3e8, eip at 0xbffff3ec
(gdb)

Lets examine local data of the main function

(gdb) list main
15	{
16		return (x * y);
17	}
18
19	int main()
20	{
21		int a = 10, b = 20;
22		int result;
23		int (*fp) (int, int);
24
(gdb)
25		fp = add;
26		result = (*fp) (a, b);
27		printf(" %d\n", result);
28
29		fp = sub;
30		result = fp(a, b);
31		printf(" %d\n", result);
32
33		fp = mul;
34		result = (*fp) (a, b);
(gdb)
35		printf(" %d\n", result);
36		return 0;
37	}
(gdb)

Main has symbol a, b, result as local integer variables, lets access their values from stack.

(gdb) print a
$13 = 10
(gdb) print b
$14 = 20
(gdb) print result
$15 = -1209414347
(gdb) print *(0xbffff3d0)
$17 = -1209414347
(gdb)

Main function also has the function pointer “fp” which should be currently initialized with address of add function, since add is already called and we stepped down from add functions frame into main, we should get add function address in the function pointer

(gdb) print fp
$18 = (int (*)(int, int)) 0x80483c4 <add>

we can probe the current execution state of the program using info program command

(gdb) info program
	Using the running image of child process 5589.
Program stopped at 0x80483c7.
It stopped at breakpoint 1.
It stopped at breakpoint 2.
It stopped at breakpoint 3.
(gdb)
(gdb) info breakpoints
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   0x080483c7 in add at sample.c:1
	breakpoint already hit 1 time
2       breakpoint     keep y   0x080483c7 in add at sample.c:4
	breakpoint already hit 1 time
3       breakpoint     keep y   0x080483c7 in add at sample.c:6
	breakpoint already hit 1 time
(gdb)

Dump shows we are at break point , it shows all the three breakpoints with hit status, there is no literal effect of breakpoint 1 and 2 for the reasons we have understood earlier. We can delete or disable breakpoints using delete and disable commands

(gdb) disable 1
(gdb) disable 2
(gdb) info breakpoints
Num     Type           Disp Enb Address    What
1       breakpoint     keep n   0x080483c7 in add at sample.c:1
	breakpoint already hit 1 time
2       breakpoint     keep n   0x080483c7 in add at sample.c:4
	breakpoint already hit 1 time
3       breakpoint     keep y   0x080483c7 in add at sample.c:6
	breakpoint already hit 1 time
(gdb) enable 1
(gdb) info breakpoints
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   0x080483c7 in add at sample.c:1
	breakpoint already hit 1 time
2       breakpoint     keep n   0x080483c7 in add at sample.c:4
	breakpoint already hit 1 time
3       breakpoint     keep y   0x080483c7 in add at sample.c:6
	breakpoint already hit 1 time
(gdb)

Disabling a breakpoint will change its Enb disposition value to n, such breakpoints will have no effect on the program until enabled again. Disabled Breakpoints can be enabled back using enable command. All active breakpoints can be set to disabled

(gdb) disable breakpoints
(gdb) info breakpoints
Num     Type           Disp Enb Address    What
1       breakpoint     keep n   0x080483c7 in add at sample.c:1
	breakpoint already hit 1 time
2       breakpoint     keep n   0x080483c7 in add at sample.c:4
	breakpoint already hit 1 time
3       breakpoint     keep n   0x080483c7 in add at sample.c:6
	breakpoint already hit 1 time
(gdb)
(gdb) delete 1
(gdb) delete 2
(gdb) info breakpoints
Num     Type           Disp Enb Address    What
3       breakpoint     keep n   0x080483c7 in add at sample.c:6
	breakpoint already hit 1 time
(gdb) delete breakpoints
Delete all breakpoints? (y or n) y
(gdb) info breakpoints
No breakpoints or watchpoints.
(gdb)

Similarly breakpoints can be deleted selectively or all breakpoints can be removed with single command.

Program execution can be resumed from a breakpoint using continue which runs the program until next breakpoint is hit or we can execute program in single instruction step mode using step command, or execute next source operation and stop using next command. We will get acquaintance with next, step and further commands in our next part of this article, for now will close the current debug session issuing continue command and let the program run to completion.

(gdb) continue
Continuing.
 70
 -10
 200

Program exited normally.
(gdb)

References

1. Introduction to the DWARF format

2. Gdb quick reference

System-call tracing with strace

Saturday, September 15th, 2012

System calls are kernel interfaces through which applications enter kernel mode to run kernel services. Since system calls are in kernel address space applications cannot directly call or step into a system call routine. To facilitate applications invoke system calls Glibc provides applications special functions called API’s(Application programming interfaces). API’s can be understood as user mode functions  that contain assembly instructions to invoke system calls. API functions are made available to application programmers through shared/ static libraries. Gnu Glibc  distribution includes various api routines that applications can link.

read (2), write(2), open(2), Ioctl(2), mmap(2), fork(2) are few examples of api’s. API’s invoke system calls using a special interrupt with the number of system call and its parameters stored in CPU registers. Since  an Api  is bound to kernel specific system calls, an application program code calling API will end up being operating system specific, and is not portable. Programmers use varieties to techniques to achieve code portability across operating systems, one of the widely used method is using standard library functions provided by the compiler distributions like GCC. Standard C / C++ libraries provide ready to link functions to perform operations like file I/O, dynamic memory allocations and so on. Applications can be written to use these functions instead of API’s to achieve code portability.

Strace is an open source tool that can trace an application execution and show up list of API’s being invoked at runtime. This information helps us discover which API’s are being invoked with what arguments and return status of the API. Since API’s invoke system calls we can conclude on what system calls succeeded and which ones failed.

Strace is very handy to find bugs that may cause exceptions during system call execution, or detect signal events that are delivered to the app.

System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them.  Students, hackers and the overly-curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary programs, And programmers will find that since system calls and signals are events that happen at the user/kernel interface, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.

Let’s trace few sample applications to understand how to use strace

root@techveda:~# vim test.c
/* strace test code
 * Author: techveda.org
 */

# include<stdio.h>

int main()
{
        int a;
        a++;
        return 0;
}

root@techveda:~# gcc test.c -o test
root@techveda:~# strace  ./test
execve("./test", ["./test"], [/* 41 vars */]) = 0
brk(0)                                  = 0x906e000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7726000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=63579, ...}) = 0
mmap2(NULL, 63579, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7716000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220o\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1442372, ...}) = 0
mmap2(NULL, 1448456, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb75b4000
mmap2(0xb7710000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb7710000
mmap2(0xb7713000, 10760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7713000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb75b3000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb75b38d0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb7710000, 8192, PROT_READ)   = 0
mprotect(0x8049000, 4096, PROT_READ)    = 0
mprotect(0xb7745000, 4096, PROT_READ)   = 0
munmap(0xb7716000, 63579)               = 0
exit_group(0)                           = ?

Above trace dump shows API’s invoked during application initialization and system calls to set up process address space components and load time libraries, arguments being passed to each apis and their return status are also being shown up.  System calls invoked by applications code would be shown up after all startup calls, since our test application did not invoke any API’s above dump ends with description of startup calls.

Let‘s modify our application to include a c standard library function call

/* strace test code
 * Author: techveda.org
 */

# include <stdio.h>
int main()
{
        printf("hello strace\n");
        return 0;
}

root@techveda:~# gcc test.c  -o test
root@techveda:~# strace ./test
execve("./test", ["./test"], [/* 41 vars */]) = 0
brk(0)                                  = 0x9e9f000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7742000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=63579, ...}) = 0
mmap2(NULL, 63579, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7732000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220o\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1442372, ...}) = 0
mmap2(NULL, 1448456, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb75d0000
mmap2(0xb772c000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb772c000
mmap2(0xb772f000, 10760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb772f000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb75cf000
set_thread_area({entry_number:-1 -&amp;gt; 6, base_addr:0xb75cf8d0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb772c000, 8192, PROT_READ)   = 0
mprotect(0x8049000, 4096, PROT_READ)    = 0
mprotect(0xb7761000, 4096, PROT_READ)   = 0
munmap(0xb7732000, 63579)               = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7741000
write(1, "hello strace\n", 13hello strace
)          = 13
exit_group(0)                           = ?

Printf function has invoked write API to direct string constant to terminal device, we can see write being called with stdout as first argument.

 

/* Strace test code
 * Author: techveda.org
 */

# include <stdio.h>
# include <malloc.h>
# include <string.h>
# include <stdlib.h>

int main()
{
        void *ptr;
        printf("hello strace\n");
        ptr = (void *)malloc( 2048);
        if( ptr == NULL){
                perror("malloc");
                exit(1);
        }
        memset( ptr, 0, 2048);
        free( ptr);
        return 0;
}

root@techveda:~# gcc test2.c -o test2
root@techveda:~# strace ./test2
execve("./test2", ["./test2"], [/* 41 vars */]) = 0
brk(0)                                  = 0x89c2000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb777e000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=63579, ...}) = 0
mmap2(NULL, 63579, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb776e000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220o\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1442372, ...}) = 0
mmap2(NULL, 1448456, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb760c000
mmap2(0xb7768000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb7768000
mmap2(0xb776b000, 10760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb776b000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb760b000
set_thread_area({entry_number:-1 -&gt; 6, base_addr:0xb760b8d0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb7768000, 8192, PROT_READ)   = 0
mprotect(0x8049000, 4096, PROT_READ)    = 0
mprotect(0xb779d000, 4096, PROT_READ)   = 0
munmap(0xb776e000, 63579)               = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb777d000
write(1, "hello strace\n", 13hello strace
)          = 13
brk(0)                                  = 0x89c2000
brk(0x89e3000)                          = 0x89e3000
exit_group(0)                           = ?

Dynamic memory allocation request of malloc was processed by brk system call; brk alters the program break point. Increasing the program break has the effect of allocating memory to the process; decreasing the break deallocates memory. The first brk call with 0 argument returns current program break point and value read is 0x89c2000, second brk call is being invoked with new break point as an argument, the value 0x89e3000 indicates new break point, and return value of this call confirms that it succeeded in altering program break point to specified location.

Also upon close observation of the dump we can find that applications heap allocation request was for only 2048(2k) bytes, but the dump shows around 135k block is allocated (0x89e3000 – 0x89c2000 = 0x21000[135168 bytes]). This is an optimization of glibc implementation of malloc. Further details show Applications free call to release memory did not alter program break point, so we could conclude that freeing dynamic memory allocated using malloc doesn’t release allocated heap back to system.

/* Strace test code
 * Author: techveda.org
 */

#include<stdio.h>
#include<string.h>
int main()
{
        char ch[100];
        FILE *fp;
        fp = fopen("gnu","r");
        fread(&ch,100,1,fp);
        fclose(fp);
}

root@techveda:~# gcc test.c -o test
root@techveda:~# strace  ./test
execve("./test", ["./test"], [/* 41 vars */]) = 0
brk(0)                                  = 0x831a000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb76ed000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=63579, ...}) = 0
mmap2(NULL, 63579, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb76dd000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220o\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1442372, ...}) = 0
mmap2(NULL, 1448456, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb757b000
mmap2(0xb76d7000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb76d7000
mmap2(0xb76da000, 10760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76da000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb757a000
set_thread_area({entry_number:-1 -&amp;gt; 6, base_addr:0xb757a8d0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb76d7000, 8192, PROT_READ)   = 0
mprotect(0x8049000, 4096, PROT_READ)    = 0
mprotect(0xb770c000, 4096, PROT_READ)   = 0
munmap(0xb76dd000, 63579)               = 0
brk(0)                                  = 0x831a000
brk(0x833b000)                          = 0x833b000
open("gnu", O_RDONLY)                   = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=44541, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb76ec000
read(3, "\n\nThe GNU Manifesto\n\nCopyright ("..., 4096) = 4096
close(3)                                = 0
munmap(0xb76ec000, 4096)                = 0
exit_group(0)                           = ?

fopen used to open file has used open api to intitate   file open, open returned with file descriptor no 3.  Our next operation in the program was fread to read 100 bytes of file data. Dump shows clearly how the operation was carried out. First fstat64 call was use to get file stats including its total size, then 4k (4096) buffer block is allocated using anonymous memory map. mmap returned start address of the region (0xb76ec000), subsequently read system call was initiated to read 4096 bytes of file data from file descriptor 3 into memory mapped buffer. This again clearly shows optimization of file read operations, tough applications requested to read 100 bytes, full page was read. Such optimization may consume additional memory , but minimizes system calls, since 4096 data is pre-fetched into application memory map, subsequent fread call does not require system call(if req is less than 4096 bytes).

We will insert one more fread operation into source application to confirm this optimization

#include<stdio.h>
#include<string.h>
int main()
{
        char ch[100];
        FILE *fp;
        fp = fopen("gnu","r");
        fread(&ch,100,1,fp);
        fread(&ch,100,1,fp);
        fclose(fp);
}

root@techveda:~# gcc test.c -o test
root@techveda:~# strace  ./test
execve("./test", ["./test"], [/* 41 vars */]) = 0
brk(0)                                  = 0x9474000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77ab000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=63579, ...}) = 0
mmap2(NULL, 63579, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb779b000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220o\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1442372, ...}) = 0
mmap2(NULL, 1448456, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7639000
mmap2(0xb7795000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb7795000
mmap2(0xb7798000, 10760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7798000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7638000
set_thread_area({entry_number:-1 -&gt; 6, base_addr:0xb76388d0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xb7795000, 8192, PROT_READ)   = 0
mprotect(0x8049000, 4096, PROT_READ)    = 0
mprotect(0xb77ca000, 4096, PROT_READ)   = 0
munmap(0xb779b000, 63579)               = 0
brk(0)                                  = 0x9474000
brk(0x9495000)                          = 0x9495000
open("gnu", O_RDONLY)                   = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=44541, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77aa000
read(3, "\n\nThe GNU Manifesto\n\nCopyright ("..., 4096) = 4096
close(3)                                = 0
munmap(0xb77aa000, 4096)                = 0
exit_group(0)                           = ?

Above log confirms that each call to fread, need not necessarily invoke system calls  and shows how library routines are written to perform various optimizations. Call  fclose resulted in close system call and then anonymous map used to hold file data was unmapped since application has closed file descriptor.

While running strace on large executables it may generate huge trace, we can instruct strace to redirect trace data into a file like this

#strace -o outfile ./a.out

To attach and trace a running program use

#strace –p <pid>

If you use the -t option, then strace will prefix each line of the trace with the time of day. We can even specify the system call functions to trace using the -e option. For example, to trace only open() and close() function system calls use the following command

#strace -o outfile  -e trace=open,close ./a.out