Archive for September, 2012

Crash Dump analysis Howto

Sunday, September 30th, 2012

Video Tutorial  on Application Crash  Dump Analysis

Note: It is recommeneded to watch them in High Definition Mode 720p or 1080p  with full screen.

Look here for hd playback settings.


Linux Char driver Basics

Sunday, September 30th, 2012

Live Class Videos on Linux Character Drivers

Note: It is recommeneded to watch them in High Definition Mode 720p or 1080p  with full screen.

Look here for hd playback settings.



Linux Driver Architecture

Sunday, September 30th, 2012

Live class videos on Linux Driver Model

Note: It is recommeneded to watch them in High Definition Mode 720p or 1080p  with full screen.

Look here for hd playback settings.



part 2


Part 3 

How source debuggers work?

Sunday, September 23rd, 2012


Application binaries are a result of compile and build operations performed on a single or a set of source files. Program Source files contain functions, data variables of various types (local, global, register, static), and abstract data objects, all written and neatly indented with nested control structures as per high level programming language syntax (C/C++).  Compilers translate code in each source file into machine instructions (1’s and 0’s) as per target processors Instruction set Architecture and bury that code into object files. Further, Linkers integrate compiled object files with other pre-compiled objects files (libraries, runtime binaries) to create end application binary image called executable.

Source debuggers are tools used to trace execution of an application executable binary. Most amazing feature of a source debugger is its ability to list source code of the program being debugged; it can show the line or expression in the source code that resulted in a particular machine code instruction of a running program loaded in memory. This helps the programmer to analyze a program’s behavior in the high-level terms like source-level flow control constructs, procedure calls, named variables, etc,  instead of machine instructions and memory locations. Source-level debugging also makes it possible to step through execution a line at a time and set source-level breakpoints. (If you do not have any prior hands on experience with source debuggers I suggest you to look at this before continuing with following.)

lets explore how source debuggers like gnu gdb work ? So how does a debugger know where to stop when you ask it to break at the entry to some function? How does it manage to find what to show you when you ask it for the value of a variable? The answer is – debug information.  All modern compilers are designed to generate Debug information together with the machine code of the source file. It is a representation of the relationship between the executable program and the original source code. This information is encoded as per  a pre-defined format and stored alongside the machine code. Many such formats were invented over the years for different platforms and executable files (aim of this article isn’t to survey the history of these formats, but rather to show how they work). Gnu compiler and ELF executable on Linux/ UNIX platforms use DWARF, which is widely used today as default debugging information format.

Word of Advice : Does an Application/ Kernel programmer need to know Dwarf?

Obvious answer to this question is a big NO.  It is purely subject matter for developers involved in implementation of a Debugger tool. A normal Application developer using debugger tools would never need to learn or dig into binary files for debug information. This in no way adds any edge to your debugging skills nor adds any new skills into your armory. However, if you are a developer using debuggers for years and curious about how debuggers work read this document for an outline into debug information.  If you are a beginner to systems programming or fresher’s learning programming I would  suggest  not to waste your time as you can safely ignore this.

ELF -DWARF sections

Gnu compiler generates debug information which is organized into various sections of the ELF object file. Let’s use the following source file for compiling and observing DWARF sections

root@techveda:~# vim sample.c

#include <stdio.h>
#include <stdlib.h>

int add(int x, int y)
        return x + y;
int main()
        int a = 10, b = 20;
        int result;
        int (*fp) (int, int);

        fp = add;
        result = (*fp) (a, b);
        printf(" %d\n", result);

root@techveda:~# gcc -c -g sample.c -o sample.o
root@techveda:~# objdump -h sample.o | more

sample.o:     file format elf32-i386

Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         0000005f  00000000  00000000  00000034  2**2
  1 .data         00000000  00000000  00000000  00000094  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  2 .bss          00000000  00000000  00000000  00000094  2**2
  3 .debug_abbrev 000000a2  00000000  00000000  00000094  2**0
  4 .debug_info   00000114  00000000  00000000  00000136  2**0
  5 .debug_line   00000040  00000000  00000000  0000024a  2**0
  6 .rodata       00000005  00000000  00000000  0000028a  2**0
  7 .debug_loc    00000070  00000000  00000000  0000028f  2**0
  8 .debug_pubnames 00000023  00000000  00000000  000002ff  2**0
  9 .debug_pubtypes 00000012  00000000  00000000  00000322  2**0
 10 .debug_aranges 00000020  00000000  00000000  00000334  2**0
 11 .debug_str    000000b0  00000000  00000000  00000354  2**0
 12 .comment      0000002b  00000000  00000000  00000404  2**0
                  CONTENTS, READONLY
 13 .note.GNU-stack 00000000  00000000  00000000  0000042f  2**0
                  CONTENTS, READONLY
 14 .debug_frame  00000054  00000000  00000000  00000430  2**2

All of the sections with naming debug_xxx are debugging information sections.  Information in these sections is interpreted by source debugger like gdb.  Each debug_ section holds specific information like

.debug_info               core DWARF data containing DIEs
.debug_line               Line Number Program
.debug_frame              Call Frame Information
.debug_macinfo            lookup table for global objects and functions
.debug_pubnames           lookup table for global objects and functions
.debug_pubtypes           lookup table for global types
.debug_loc                Macro descriptions
.debug_abbrev             Abbreviations used in the .debug_info section
.debug_aranges            mapping between memory address and compilation
.debug_ranges             Address ranges referenced by DIEs
.debug_str                String table used by .debug_info

Debugging Information Entry (DIE)

Dwarf format organizes debug data in all of the above sections using special objects (program descriptive entities) called Debugging Information Entry (DIE).  Each DIE has a tag filed whose value specifies its type, and a set of attributes. DIEs are interlinked via sibling and child links, and values of attributes can point at other DIEs. Now let’s dig into ELF file to view how a DIE looks like. We will begin our exploration with .debug_info section of the ELF file since core DIE’s are listed in it.

root@techveda:~# objdump --dwarf=info . /sample

Above operation shows long list of DIE’s. Let’s limit ourselves to relevant information

./sample:     file format elf32-i386

Contents of the .debug_info section:

  Compilation Unit @ offset 0x0:
   Length:        0x110 (32-bit)
   Version:       2
   Abbrev Offset: 0
   Pointer Size:  4
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    < c>   DW_AT_producer    : (indirect string, offset: 0xe): GNU C 4.5.2	
    <10>   DW_AT_language    : 1	(ANSI C)
    <11>   DW_AT_name        : (indirect string, offset: 0x44): sample.c	
    <15>   DW_AT_comp_dir    : (indirect string, offset: 0x71): /root	
    <19>   DW_AT_low_pc      : 0x80483c4	
    <1d>   DW_AT_high_pc     : 0x8048423	
    <21>   DW_AT_stmt_list   : 0x0

Each source file in the application is referred in dwarf terminology as a “compilation unit”.   Dwarf data for each compilation unit (source file) starts with a compilation unit DIE. Above dump shows the first DIE’s and tag value “DW_TAG_compile_unit “.  This DIE provides general information about compilation unit like source file name (DW_AT_name   : (indirect string, offset: 0x44): sample.c),  high level programming language used to write source file(DW_AT_language    : 1    (ANSI C))  ,   directory of the source file(DW_AT_comp_dir    : (indirect string, offset: 0x71): /root) ,  compiler and producer of dwarf data(  DW_AT_producer    : (indirect string, offset: 0xe): GNU C 4.5.2) , start virtual address of the compilation unit  (DW_AT_low_pc      : 0x80483c4), end virtual address of the unit (DW_AT_high_pc     : 0x8048423).

Compilation Unit DIE is the parent for all the other DIE’s that describe elements of source file. Generally, the list of DIE’s that follow will describe data types, followed by global data, then the functions that make up the source file. The DIEs for variables and functions are in the same order in which they appear in the source file.

How does debugger locate Function Information ?

 While using source debuggers we often instruct debugger to insert or place break point at some function, expecting the debugger to pause program execution at functions. To be able to perform this task, debugger must have some mapping between a function name in the high-level code and the address in the machine code where the instructions for this function begin. For this mapping information debuggers rely on DIE’s that describes specified function. DIE’s describing functions in a compilation unit are assigned tag value DW_TAG_subprogram” subprogram as per dwarf terminology is a function.

In our sample application source we have two functions (main, add), dwarf should generate a “DW_TAG_subprogramDIE’s for each function, these DIE attributes would define function mapping information that debugger needs for resolving machine code addresses with function name.

Each “DW_TAG_subprogam” DIE contains

  1. function scope
  2. function name
  3. source file or compilation unit in which function is located
  4.  line no in the source file where the function starts
  5. Functions return type
  6. Start address of the fucntion
  7. End address of the function
  8. Frame information of the function.


 root@techveda:~# objdump --dwarf=info ./sample | grep "DW_TAG_subprogram"
 <1><72>: Abbrev Number: 4 (DW_TAG_subprogram)
 <1><a8>: Abbrev Number: 6 (DW_TAG_subprogram)

<1><72>: Abbrev Number: 4 (DW_TAG_subprogram)
    <73>   DW_AT_external    : 1	
    <74>   DW_AT_name        : add	
    <78>   DW_AT_decl_file   : 1	
    <79>   DW_AT_decl_line   : 4	
    <7a>   DW_AT_prototyped  : 1	
    <7b>   DW_AT_type        : <0x4f>	
    <7f>   DW_AT_low_pc      : 0x80483c4	
    <83>   DW_AT_high_pc     : 0x80483d1	
    <87>   DW_AT_frame_base  : 0x0	(location list)
    <8b>   DW_AT_sibling     : <0xa8>	

<1><a8>: Abbrev Number: 6 (DW_TAG_subprogram)
    <a9>   DW_AT_external    : 1	
    <aa>   DW_AT_name        : (indirect string, offset: 0x1a): main	
    <ae>   DW_AT_decl_file   : 1	
    <af>   DW_AT_decl_line   : 8	
    <b0>   DW_AT_type        : <0x4f>	
    <b4>   DW_AT_low_pc      : 0x80483d2	
    <b8>   DW_AT_high_pc     : 0x8048423	
    <bc>   DW_AT_frame_base  : 0x38	(location list)
    <c0>   DW_AT_sibling     : <0xf8>


We now have accessed DIE description of function’s main and addLet’s analyze attribute information of add fucntions DIE.

Function scope:    DW_AT_external    : 1 (scope external)

Function name:     DW_AT_name        : add

Source file or compilation unit in which function is located:  DW_AT_decl_file   : 1 (indicates 1st compilation unit which is sample.c)

line no in the source file where the function starts:  DW_AT_decl_line   : 4 ( indicates line no 4 in source file)

#include <stdio.h>
#include <stdlib.h>

int add(int x, int y)
        return x + y;
int main()
        int a = 10, b = 20;
        int result;
        int (*fp) (int, int);

        fp = add;
        result = (*fp) (a, b);
        printf(" %d\n", result);

Function’s source line no matched with DIE description of line no. let’s continue with rest of the attribute values

Functions return type:  DW_AT_type        : <0x4f>

As we have already understood that values of attributes can point to other DIE , here is an example of it.  Value   DW_AT_type     : <0x4f> indicates that return type description is stored in other DIE at offset 0x4f.

<4f>: Abbrev Number: 3 (DW_TAG_base_type)
    <50>   DW_AT_byte_size   : 4	
    <51>   DW_AT_encoding    : 5	(signed)
    <52>   DW_AT_name        : int

This DIE describes data type and composition of return type of the function add, as per DIE attribute values return type is signed int  of size 4 bytes.

Start address of the function : DW_AT_low_pc   : 0x80483c4

End address of the function: DW_AT_high_pc     : 0x80483d1

Above values indicate start and end virtual address of the machine instructions of add function, we can verify that with binary dump of the function

080483c4 <add>:
 80483c4:	55                   	push   %ebp
 80483c5:	89 e5                	mov    %esp,%ebp
 80483c7:	8b 45 0c             	mov    0xc(%ebp),%eax
 80483ca:	8b 55 08             	mov    0x8(%ebp),%edx
 80483cd:	8d 04 02             	lea    (%edx,%eax,1),%eax
 80483d0:	5d                   	pop    %ebp
 80483d1:	c3                   	ret

How does debugger find program data (variables…) Information?

When the program hits assigned break point in a function, debugger pauses the program execution, at this time we can instruct debugger to show or print values of variables, by using debugger commands like print or display followed by variable name (ex: print a) How does debugger know where to find memory location of the variable ? Variables can be located in global storage, on the stack, and even in registers.The debugging information has to be able to reflect all these variations, and indeed DWARF does. As an example let’s take a look at complete DIE information set for main function.

<1><a8>: Abbrev Number: 6 (DW_TAG_subprogram)
    <a9>   DW_AT_external    : 1	
    <aa>   DW_AT_name        : (indirect string, offset: 0x1a): main	
    <ae>   DW_AT_decl_file   : 1	
    <af>   DW_AT_decl_line   : 8	
    <b0>   DW_AT_type        : <0x4f>	
    <b4>   DW_AT_low_pc      : 0x80483d2	
    <b8>   DW_AT_high_pc     : 0x8048423	
    <bc>   DW_AT_frame_base  : 0x38	(location list)
    <c0>   DW_AT_sibling     : <0xf8>	
 <2><c4>: Abbrev Number: 7 (DW_TAG_variable)
    <c5>   DW_AT_name        : a	
    <c7>   DW_AT_decl_file   : 1	
    <c8>   DW_AT_decl_line   : 10	
    <c9>   DW_AT_type        : <0x4f>	
    <cd>   DW_AT_location    : 2 byte block: 74 1c 	(DW_OP_breg4 (esp): 28)
 <2><d0>: Abbrev Number: 7 (DW_TAG_variable)
    <d1>   DW_AT_name        : b	
    <d3>   DW_AT_decl_file   : 1	
    <d4>   DW_AT_decl_line   : 10	
    <d5>   DW_AT_type        : <0x4f>	
    <d9>   DW_AT_location    : 2 byte block: 74 18 	(DW_OP_breg4 (esp): 24)
 <2><dc>: Abbrev Number: 8 (DW_TAG_variable)
    <dd>           : (indirect string, offset: 0x6a): result	
    <e1>   DW_AT_decl_file   : 1	
    <e2>   DW_AT_decl_line   : 11	
    <e3>   DW_AT_type        : <0x4f>	
    <e7>   DW_AT_location    : 2 byte block: 74 10 	(DW_OP_breg4 (esp): 16)
 <2><ea>: Abbrev Number: 7 (DW_TAG_variable)
    <eb>   DW_AT_name        : fp	
    <ee>   DW_AT_decl_file   : 1	
    <ef>   DW_AT_decl_line   : 12	
    <f0>   DW_AT_type        : <0x10d>	
    <f4>   DW_AT_location    : 2 byte block: 74 14 	(DW_OP_breg4 (esp): 20)

Note the first number inside the angle brackets in each entry. This is the nesting level – in this example entries with <2> are children of the entry with <1>. main function has three integer variables a,b and result each of these variables are described with DW_TAG_variable nested DIE’s (0xc4, 0xd0, 0xdc). main function also has a function pointer  fp described  in DIE 0xea . Variable DIE attributes specify variable name (DW_AT_name), declaration line no in source function (DW_AT_decl_line ), pointer to address of DIE describing variables data type (DW_AT_type) and relative location of the variable within function’s frame (DW_AT_location).

To locate the variable in the memory image of the executing process, the debugger will look at the DW_AT_location attribute of DIE. For a its value is    DW_OP_fbreg4 (esp):28. This means that the variable is stored at offset 28  from the top in the frame of  containing function. The DW_AT_frame_base attribute of main has the value 0x38(location list), which means that this value actually has to be looked up in the location list section. Let’s look at it:

root@techveda:~# objdump --dwarf=loc sample

sample:     file format elf32-i386

Contents of the .debug_loc section:

    Offset   Begin    End      Expression
    00000000 080483c4 080483c5 (DW_OP_breg4 (esp): 4)
    00000000 080483c5 080483c7 (DW_OP_breg4 (esp): 8)
    00000000 080483c7 080483d1 (DW_OP_breg5 (ebp): 8)
    00000000 080483d1 080483d2 (DW_OP_breg4 (esp): 4)
    00000000 <End of list>
    00000038 080483d2 080483d3 (DW_OP_breg4 (esp): 4)
    00000038 080483d3 080483d5 (DW_OP_breg4 (esp): 8)
    00000038 080483d5 08048422 (DW_OP_breg5 (ebp): 8)
    00000038 08048422 08048423 (DW_OP_breg4 (esp): 4)
    00000038 <End of list>

Offset column 0x38 values are the entries for main function variables. Each entry here describes possible frame base address with respect to where debugger may be paused by break point within function instructions; it specifies the current frame base from which offsets to variables are to be computed as an offset from a register. For x86, bpreg4 refers to esp and bpreg5 refers to ebp.  Before analyzing further lets look at disassemble dump for main function

080483d2 <main>:
 80483d2:	55                   	push   %ebp
 80483d3:	89 e5                	mov    %esp,%ebp
 80483d5:	83 e4 f0             	and    $0xfffffff0,%esp
 80483d8:	83 ec 20             	sub    $0x20,%esp
 80483db:	c7 44 24 1c 0a 00 00 	movl   $0xa,0x1c(%esp)
 80483e2:	00 
 80483e3:	c7 44 24 18 14 00 00 	movl   $0x14,0x18(%esp)
 80483ea:	00 
 80483eb:	c7 44 24 14 c4 83 04 	movl   $0x80483c4,0x14(%esp)
 80483f2:	08 
 80483f3:	8b 44 24 18          	mov    0x18(%esp),%eax
 80483f7:	89 44 24 04          	mov    %eax,0x4(%esp)
 80483fb:	8b 44 24 1c          	mov    0x1c(%esp),%eax
 80483ff:	89 04 24             	mov    %eax,(%esp)
 8048402:	8b 44 24 14          	mov    0x14(%esp),%eax
 8048406:	ff d0                	            call   *%eax
 8048408:	89 44 24 10          	mov    %eax,0x10(%esp)
 804840c:	b8 f0 84 04 08       	mov    $0x80484f0,%eax
 8048411:	8b 54 24 10          	mov    0x10(%esp),%edx
 8048415:	89 54 24 04          	mov    %edx,0x4(%esp)
 8048419:	89 04 24             	mov    %eax,(%esp)
 804841c:	e8 d3 fe ff ff       	call   80482f4 <printf@plt>
 8048421:	c9                   	leave  
 8048422:	c3                   	ret

First two instructions deal with function’s preamble, function’s stack frame base pointer is determined after pre-amble instructions are executed.  Ebp remains constant throughout function’s execution and esp keeps changing with data being pushed and popped from the stack frame. From the above dump instructions at offset 80483db and 80483e3 are assigning values 10 and 20 to variables a, and b. These variables are being accessed their offset in stack frame relative to location of current esp( variable a: 0x1c(%esp), variable b: 0x18(%esp)). Now let’s assume that break point was set after initializing a and b variables and program paused, and we have run print command to view conents of a or b variables. Debugger would access 3rd record of the main function’s dwarf debug.loc table since our break point falls between 080483d5 – 08048422 region of the function code.

00000038 080483d2 080483d3 (DW_OP_breg4 (esp): 4)
00000038 080483d3 080483d5 (DW_OP_breg4 (esp): 8)
00000038 080483d5 08048422 (DW_OP_breg5 (ebp): 8)--- 3rd record
00000038 08048422 08048423 (DW_OP_breg4 (esp): 4)

Now as per records debugger will locate a with esp + 28 , b with esp +24 and so on…

Looking up line number information

We can set breakpoints mentioning line no’s Lets now look at how debuggers resolve line no’s to machine instruction’s? DWARF encodes a full mapping between lines in the C source code and machine code addresses in the executable. This information is contained in the .debug_line section and can be extracted using objdump

root@techveda:~# objdump --dwarf=decodedline ./sample

./sample:     file format elf32-i386

Decoded dump of debug contents of section .debug_line:

CU: sample.c:
File name                            Line number    Starting address					
sample.c                                       5          0x80483c4
sample.c                                       6          0x80483c7
sample.c                                       7          0x80483d0
sample.c                                       9          0x80483d2
sample.c                                      10          0x80483db
sample.c                                      14          0x80483eb
sample.c                                      15          0x80483f3
sample.c                                      16          0x804840c
sample.c                                      17          0x8048421

This dump shows machine instruction line no’s for our program. It is quite obvious that line no’s for non-executable statements of the source file need not be tracked by dwarf .we can map the above dump to our source code for analysis.

  1 #include <stdio.h>
  2 #include <stdlib.h>
  4 int add(int x, int y)
  5 {
  6         return x + y;
  7 }
  8 int main()
  9 {
 10         int a = 10, b = 20;
 11         int result;
 12         int (*fp) (int, int);
 14         fp = add;
 15         result = (*fp) (a, b);
 16         printf(" %d\n", result);
 17 }

From the above it should be clear of what debugger does it is instructed to set breakpoint at entry into function add, it would insert break point at line no 6 and pause after pre-amble of function add is executed.

What’s next?

if you are into implementation of debugging tools or involved in writing programs/tools  that simulate debugger facilities/ read binary files , you may be interested in specific programming libraries libbfd or libdwarf.

Binary File Descriptor library (BFD) or libbfd as it is called provides ready to use functions to read into ELF and other popular binary files. BFD works by presenting a common abstract view of object files. An object file has a “header” with descriptive info; a variable number of “sections” that each has a name, some attributes, and a block of data; a symbol table; relocation entries; and so forth. Gnu binutils package tools like objdump, readelf and others have been written using these libraries.

Libdwarf is a C library intended to simplify reading (and writing) applications built with DWARF2, DWARF3 debug information.

Dwarfdump is an application written using libdwarf to print dwarf information in a human readable format. It is also open sourced and is copyrighted GPL. It provides an example of using libdwarf to read DWARF2/3 information as well as providing readable text output.


  1. Dwarf Standard
  2. Libdwarf Wiki
  3. Libbfd draft


Adding new system-calls Linux kernel 3.5.x

Thursday, September 20th, 2012


System-calls are kernel functions that  serve as an interface (for user mode applications ) to invoke kernel services like drivers, file-systems, Network stacks  and others. system calls are also referred as “kernel entry points” since applications can enter kernel mode only through a valid system call interface. Applications can step into system calls using special processor specific soft interrupt instructions. Linux kernel is widely being deployed and used in virtually every computing platform from desktops, servers, enterprise platforms to Mobile, Deep embedded, Robotics, consumer electronics, medical device platforms  and the list can go on and on.

while customizing and deploying Linux on to various platforms  there may be need for adding new services and system calls to kernel sources.  Fresh programmers to Kernel programming concepts  may find it more interesting to explore system call , API concepts practically by adding new system call, and implementing applications to invoke those calls . (watch linux system calls video tutorial here.) This document provides details of how to add  new system calls to Linux kernel Sources (3.3 version on-words) for x86 32 & 64 bit arch.

Getting kernel sources

we will start by downloading latest stable kernel from kernel community web-site  it is a common standard to host source packages in /usr/src branch of rootfs (but you may copy compressed source tar file into any folder).  This document assumes /usr/src/ to be the location of source tar file .

root@techveda:/usr/src# pwd
root@techveda:/usr/src# ls 
linux-3.2.9          linux-3.5.4.tar.bz2     linux-headers-2.6.38-8-generic                             linux-source-2.6.38
linux-3.2.9.tar.bz2  linux-headers-2.6.38-8  linux-image-3.2.9.debug_3.2.9.debug-10.00.Custom_i386.deb  linux-source-2.6.38.tar.bz2

let’s Extract source from compressed tar file and change our working directory into kernel source tree.

root@techveda:/usr/src# tar xvf linux-3.5.4.tar.bz2 
root@techveda:/usr/src# cd linux-3.5.4
root@techveda:/usr/src/linux-3.5.4# pwd
root@techveda:/usr/src/linux-3.5.4# ls
arch     CREDITS        drivers   include  Kbuild   lib          mm      REPORTING-BUGS  security  usr
block    crypto         firmware  init     Kconfig  MAINTAINERS  net     samples         sound     virt
COPYING  Documentation  fs        ipc      kernel   Makefile     README  scripts         tools
root@techveda:/usr/src/linux-3.5.4# ls arch/
alpha  avr32     c6x   frv    hexagon  Kconfig  m68k        mips     openrisc  powerpc  score  sparc  um         x86
arm    blackfin  cris  h8300  ia64     m32r     microblaze  mn10300  parisc    s390     sh     tile   unicore32  xtensa

Above dump shows the contents of arch branch of kernel source.  arch contains Linux hardware abstraction layer code for various architectures that Linux kernel  already includes  support for, kernel maintains an architecture specific system call table  which holds system call addresses.  To add an new system call we need to append a new entry into system call table and store address of syscall function.

syscall table for x86 architecture can be  found in  arch/x86/syscalls branch of kernel source.

root@techveda:/usr/src/linux-3.5.4# ls arch/x86/syscalls/
Makefile  syscall_32.tbl  syscall_64.tbl

syscall_32.tbl  contains system calls for x86 32 bit and syscall_64.tbl   contains intel x86-64bit syscalls. add new entry into appropriate table as per x86 variant you are using .

root@techveda:/usr/src/linux-3.5.4# vim arch/x86/syscalls/syscall_32.tbl 
349     i386    kcmp           sys_kcmp
350     i386    lsproc         sys_lsproc

 New system call is implemented to show list of processes currently load and is given a name lsproc.  we have added a new entry into syscall tableentry  begins with offset no of the table , we added a new offset 350(already table had 249 offsets) , next we mention ABI tag as i386 for 32 bit x86,  next we mentioned name of new system call (lsproc) and at last address of the system call function sys_lsproc (fucntion name resolves to its address).  system call functions in Linux kernel  are assingned names as per  sys_syscallname naming convention, this makes it easy to identify a function as system call  while browsing kernel sources.

User Mode apps will use the offset no of the system call table while invoking system call, in our case the offset application will have to use  350 for invoking lsproc syscall.

X86 64bit

root@techveda:/usr/src/linux-3.5.4# vim arch/x86/syscalls/syscall_64.tbl 
313     64      lsproc                  sys_lsproc

313 is the offset, 64  indicates  x86 64bit  ABI and lsproc is the name of the system call and sys_lsproc is the address of function

Let us now implement our system call function with the name sys_lsproc. we will first declare fucntion prototype in the linux/syscalls.h kernel header


root@techveda:/usr/src/linux-3.5.4# vim include/linux/syscalls.h 
asmlinkage int sys_lsproc(void);

asmlinkage  is kernel tag  #defined with a magic  no to instruct compiler   that all arguments to the function must be accessed from kernel stack.  let us implement function code in a new source file lsproc.c . we will host our source file lsproc.c in lsproc directory under kernel branch of source tree.

root@techveda:/usr/src/linux-3.5.4# mkdir kernel/lsproc
root@techveda:/usr/src/linux-3.5.4# vim kernel/lsproc/lsproc.c
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/sched.h>
#include <linux/syscalls.h>

/* system call to print process information
 * prints processname  pid  state
asmlinkage int sys_lsproc(void)
        struct task_struct *p;
        for_each_process(p) {
                pr_info("%15s\t%u\t%ld", p->comm, task_pid_nr(p), p->state);
        return 0;

we need to add  Makefile and modify kernel build script Kconfig to intergrate our changes into source tree.

root@techveda:/usr/src/linux-3.5.4# vim kernel/lspoc/Makefile 
obj-$(CONFIG_LSPROC) += lsproc.o


root@techveda:/usr/src/linux-3.5.4# vim kernel/Kconfig.lsproc
config LSPROC
  bool "list current processes"
  default y
    This will list all current running process pid and their state

This completes the process of integrating new system call into kernel source tree.  Now Its time to compile kernel sources  and create a new kernel image that includes our system call. Following make commands will compile, build and install the new kernel image.


root@techveda:/usr/src/linux-3.5.4# vim Makefile 
root@techveda:/usr/src/linux-3.5.4#make menuconfig
root@techveda:/usr/src/linux-3.5.4#make modules_install 
root@techveda:/usr/src/linux-3.5.4#make install

look here for more details on kernel build

 Test Application

Application can invoke the system call using x86-32bit  trap exception  with system call offset as a parameter passed into eax accumulator register.

root@techveda:~# vim lsproc.c

#include <stdio.h>
#include <stdlib.h>

int lsproc()
        int ret;
        __asm__("movl $350, %eax");
        __asm__("int $0x80");
        __asm__("movl %eax, -4(%ebp)");
        return ret;

int main()
        int ret;
        printf("invoking system call\n");
        ret = lsproc();
        if (ret < 0)
        return 0;
root@techveda:~# gcc lsproc.c -o lsproc
root@techveda:~# ./lsproc 
invoking system call
root@techveda:~# dmesg

Dmesg would list process list extracted and printed by our system call on console.



Linux For Space Constrained Small Embedded Devices

Tuesday, September 18th, 2012

A considerable segment of embedded systems are often found in mass-market products and are therefore subjected to hard economic constraints. The basic nature of these systems mandates further constraints on physical size and power consumption. These in turn give rise to resource constraints on the computing platform level, e.g., constraints on computing speed, memory size, and communication bandwidth etc. In spite of the rapid development of computer hardware these constraints are true due to the economic overheads. In most cases it is not economically justified to use a processor with more capacity due to the overall product’s cost limits.

Now coming to the software that goes into these space crunched hardware?

A recent survey conducted on small embedded device manufacturers revealed the following facts.

Most of these vendors avoided using embedded operating systems and the response was a straightforward, “we don’t need one.” More than half of these vendors mentioned a simple lack of need as the sole reason for skipping the operating-system. It’s easy to think of many products that might not need an OS so this response isn’t surprising. Out of those about 30% said an OS will strain their system’s processor and/or memory. Some said an OS was too expensive and the remaining felt that operating systems are too complicated. However the need for an embedded OS was felt considering the fast changing demands of the consumers, who are looking for multi-purpose small embedded devices, which mandates the use of an OS

Now the question, what is the most economical and easy OS solution?

This brings up Linux as a viable option. Let’s explore the reasons behind this choice and most importantly understand how it can be used on these space constrained devices?

•Linux is open source kernel backed by a strong community of developers; members of the community provide bug fixes, enhancements, and even support old devices long after the manufacture has ceased   support.

•Linux offers a large library of functionality that can be leveraged to accelerate development. Linux is easy to customize and port even on to MMU less processors. With Linux community’s commitment to support embedded systems and various embedded hardware vendors contributing development tools and support.Linux is the best bet to migrate from hardware specific code to richer software.

Linux kernel is massive collection of sources targeting various architectures and platforms. Linux kernel currently contains around 9.2 million lines of code with a 10% increase every year. With the current version, pattern new stable kernel is available every 3 to 4 months. Linux Currently supports 25 different architectures, 60 Filesystems, 20 net protocols(L2), 50+ ATA Drivers, 300+ pci Drivers, 200+ Ethernet Drivers

“Linux supports more individual types of devices overall than any other kernel” -Greg KH

Default vanilla kernel configuration includes full network support, support for wide variety of input devices, and does not offer fixed boot time (can afford the time delays related to device discovery).

So with all these code buried within Linux, how can we fit it into these small devices?

since Linux kernel sources are available, it allows embedded engineers to customize kernel build by integrating various optimizations that help us keep check on kernel foot-print and reduce boot up-time.

Following is a list of few tweaks , that we could consider.

  1. Disabling unrequired Device drivers and file systems like Ext2, Ext3 (not required on mass storage less embedded devices)
  2. Disabling unrequired network protocols and other network services like Net filters, packet forwarding etc…
  3. Disable SMP Support when not running on dual CPU boards. This reduces the kernel image size and also results in better performance on uniprocessor machines
  4. Disabling support for other CPU variants  for the architecture we have selected would help us reduce kernel image size by stripping out unrequired start up code.
  5. Disable support for Dynamic configuration of kernel parameters through sysctl’s.
  6. Reduce the kernel log buffer to 4k (default 16k).
  7. Support for hotplug events / user space notifications are often not required in Embedded systems kernel so choosing to disable them would help reduce kernel image size
  8. Enable CONFIG_EMBEDDED which enables a group option to further trim down the kernel. Notable options enabled are the ability to select a different allocator and a selection to optimize for size (CONFIG_CC_OPTIMIZE_FOR_SIZE).
  9. Enable SLOB Allocator
    Current kernels are by default set to SLAB allocator which is more efficient on the large memory desktop machines. So enable SLOB allocator which perfectly suits smaller, resource limited devices.
  10. Disable Module Support
    Modules under Linux allow kernel code to be dynamically installed or removed. However, to accomplish this, each module is loaded onto a fresh page. This has two consequences. On architectures that support large memory pages, modules do not share the large page typically used by the primary kernel code. Consequences of this is another page table entry is needed and page table caches may get thrashed more which may be a potential performance impact. A second consequence is if a module is the page size plus one byte, the total memory consumed is 2 entire pages. This is almost 50% waste. Or in another case if the module is small, a full page has to be allocated. If the page size is 4K and the module is only 1K, this is a 300% waste. If many such modules are loaded, there can be considerable waste of memory.
  11. Consider UClinux when there are no stringent MMU requirements
  12. Other issues apart from normal selections include appropriate tagging of disposable code. Custom drivers or even existing drivers should have the initialization code tagged appropriately so memory occupied by them can be freed. This can be done by the use of the GCC __init attribute.
  13. Developers also need to be aware of performance constraints with XIP(Execute in place) NOR flashes on some custom hardware. XIP allows the kernel to run form the same place that it is stored. This requires the flash to appear as linear memory. On custom hardware (such as certain ARM based systems) the bus for interfacing to NOR flash may be smaller than the native size of the processors for ex: 16bit NOR memory interface requiring two access cycles to load a native size piece of data, 32bits wide. This would result in massive performance bottleneck. In addition, XIP prevents the use of compressed kernel images.

Benefits of Learning an Online Linux Device Drivers Course

Tuesday, September 18th, 2012

Though there are many reasons for participants to go for an online course when it comes to a high intensity courses like Linux Device Drivers, these were surveyed to be the major reasons for choosing it.

Learning System Programming should be slow and steady
By no stretch system programming is an easy and fast pill to swallow; it takes pure intention and good clean learning to seal things for you. As against normal application software learning where it is mostly restricted to functionality based learning, system programming like Linux Device Drivers involves deep understanding and analysis of issues like the Linux kernel and its subsystem. As an online course doesn’t mandate you to learn things daily, you can take it slowly and easily as you comprehend it. This makes learning Linux Device Drivers easier and more complete

More work involved
Conventional classroom based learning hammers you with daily doses of information and mostly leaves no room for putting theory to practice, this seriously dampens the learning and leaves the information undigested. As online learning can be taken at your own pace you get ample space to put theory to practice and move on once you mastered the topic.

Learn only when you are really prepared to
There are certain times when you really feel like getting to things and deal with it with maximum focus and force. Learning is no different; it’s these short spans of energy bursts which get you the best. As most online trainings are accessible 24×7 they empower you with using these times of peak interest to best use unlike conventional time bound training programs.

Redo portions which need double attention
Irrespective of how attentive you are during training sessions, some core sessions need that extra bit of attention or double the attention to grab the topic, sometimes you need to redo the topic all over again to get the complete picture. This is where an online course is truly unparalleled with its openness unlike classroom based training which can never be repeated

Manage learning easily against tight work schedules
As working professionals you may already be fighting time and even if you commit for a fixed time to learn you may often fail to keep up as scheduled. This on a continued basis may let your interest and focus deviate from learning. An online course helps you here with the freedom of time schedules and lets you learn only when you find the right time to do so. Online courses may also bring you other benefits apart from the above mentioned. Final word is that it’s always better to go for an online course for a core training program like Linux Device Drivers

How to Start Your Career in System Programming?

Tuesday, September 18th, 2012

If you are a fresher and are really looking for a personally and professionally satisfying career in embedded systems or system level programming then these are the exact skills that you need to get hold of

Skill guide for System/Embedded/Real Time Programmers

•Effectively programming using Data Structures
•Application of C for System programming purposes
•Complete understanding of any Operating System (Linux, most widely used here)
•OS customization skills, Device Drivers, Embedded Drivers
•OS building for target hardware
•Network programming
•C++ for Embedded and Real Time programming
•Understanding finer aspects of Real Time OS designs
•Programming for Multicore architectures
•Above all true passion for hard core programming

The above guide is purely indicative and is based on the current and expected future skill requirements. However these are the skills which are going to stay for a very long time. I call them “skills forever”. Most of these skills are age old and are proven effective time and again with minor modifications to meet moving requirements.

How to master these skills

Here I recommend three proven ways to master these skills

1. Self Learning

Go out and do it on your own, this is the mantra of self-learning, definitely, the most satisfying path of all the. With so much open information available self-learning is definitely recommended. A decade back may be it would have been ill advised, due to lack of information available through Internet, books and so on, but the current situation only augments the use of it.

Pros & Cons

This is indeed the most satisfying of all paths. Information is readily available and it is only left to you to explore and deduce the right information. Now students willing to walk this path should be very selective in choosing information as there are every possibility that he/she may end up with heavy unwanted or even misleading information and most importantly taking much of his time. To sum it up I would say that in this path the learning curve i.e. the time taken to learn is more compared to all other kinds of learning.

Sources for this style of learning are: internet, books, practical experimentation, technical forums, seminars and so on.

2. Mentoring/Professional Training Course

To put it in a single sentence, “Look for a mentor/trainer who can extend his expertise and help you build your skills in the smartest and shortest way”. Definitely the most widely preferred channel of learning and also the most confusing, thanks to the ever growing number of training organizations offering a wide variety of courses. I certainly am not against this trend but I would only like to caution students on being very selective while choosing these organizations for building their skills.

Let us look at some of the common training programs available for Embedded/System software development

There are a growing number of institutions offering specialized training courses on this domain. Most of these institutions offer a similar line of course contents with minor deviations. However most of them do talk about:

•Programming with C and Data Structures
•Programming with C++
•Linux/Unix Essentials
•Some part of Linux Drivers
•Micro-controllers and Programming them
•Embedded Systems Concepts

Now if you compare this with the above mentioned Skill guide, surely most of the training organizations need to drastically improve their contents to cater the exact skill requirements of the students and mind you most of these institutions are delivering these courses just at the knowledge gaining level. So be extremely selective in choosing your training courses. This is a highly recommended source of learning as there are plenty of options to choose from, obviously with a bit of scrutiny.

3. On Job Training

Definitely the most preferred method of learning. This gives the student a great opportunity to be in the company of experts and also in the most conducive learning environment. The student who enters this course would develop his/her skills in a way which is most desired by his/her employer. This could help in building both generic as well as specific skills. However there is a good chance of being very specific in the skills gained and this may hamper his overall generic outlook.

Now it’s your turn to choose

So the choices are open, rate your skills and choose any path or even try to come up with something of your own. Just remember that, our expertise can always be availed and we will be more than happy to help you.

Why Learn Linux As an Embedded OS?

Tuesday, September 18th, 2012

The deeper you go the more clearer it becomes. This is true with system level programming including programming for embedded systems. If you are used to working on proprietary software you should have a hard time in getting the pulse of hard-core programming. It’s not that you cannot but you will always find it difficult to sail through. An embedded operating system can be defined as, a bare software without cosmetics (used to spruce up end- user experience). So most times you are dealing with core programing and its implications and this is where your understanding of an OS like Linux, which is open in nature, will help you rip through the core programming issues at a much faster pace.

Linux with its unmatched flexibility and huge support base is the ideal platform for you to go deeper and understand the finer aspects of programming for the embedded world. In all these years as a trainer I have met several experienced programmers who struggle to make it when dealing with core programming. I have and still recommend developers to start with; understanding the basic architecture of an OS, its interactions with the hardware and so on. This is where the real crux is, once you get hold of these concepts it not only delivers great clarity on the internal working of an OS but also empowers you with great portability skill from one OS to another.

So if you desire to make it big as an embedded developer then get started with Linux kernel.

Ubuntu Essential Add-on’s for Programmers

Monday, September 17th, 2012

Ubuntu is a popular Linux distribution that is widely being used on desktops, servers and laptops. The Ubuntu community is built on the ideas enshrined in the Ubuntu Manifesto: that software should be available free of charge, that software tools should be usable by people in their local language and despite any disabilities, and that people should have the freedom to customize and alter their software in whatever way they see fit. “Ubuntu” is an ancient African word, meaning “humanity to others”. The Ubuntu distribution brings the spirit of Ubuntu to the software world.

Ubuntu is easy to install and upgrade, it has an active support community, with millions of software packages ready to install through Ubuntu software center. Ubuntu Installation iso images are available for download here. Default Ubuntu installation images include minimal software packages and do not include tools and libraries required while using Ubuntu systems for applications and kernel programming

Ubuntu systems provide a tool called apt-get that can be used to install requires add-on’s. Here is a partial list of add-on’s needed for kernel developers

Editors and manual  pages

$sudo apt-get install vim
$sudo apt-get install eclipse
$sudo apt-get install manpages manpages-dev manpages-posix manpages-posix-dev

Kernel source and source-code  browsers

$sudo apt-get install linux-source
$sudo apt-get install ctags
$sudo apt-get install cscope

Libraries and tools

$sudo apt-get install ncurses-dev 
$sudo apt-get install kernel-package
$sudo apt-get install bison flex texinfo
$sudo apt-get install tree
$sudo apt-get install zlibc zlib1g zlib1g-dev libglib2.0-dev
$sudo apt-get install rar unrar unzip zip
$sudo apt-get install minicom
$sudo apt-get install atftpd