Development Infrastructure: Tooling with Open Source Software

Overview

Goals

• An overview best-of-breed tools
• Application examples
• Best practices
• Productivity tips and tricks
• Turn OSS tools into a strategic advantage

Your Instructor

• Diomidis Spinellis (http://www.spinellis.gr)
• Professor of Software Engineering, AUEB (http://www.aueb.gr)
• Secretary General of Information Systems, Ministry of Finance
• IEEE Software (http://www.computer.org/software): Tools of the Trade
• Member of the FreeBSD development team
• Code Reading: The Open Source Perspective (Addison Wesley, 2003)
• Code Quality: The Open Source Perspective (Addison Wesley, 2006)
• Beautiful Architecture(O'Reilly, 2009)

The Case for Tooling

Capital expenditures in different industries

The Case for Open Source

• Open architecture: combine, mix and match
• Industrial strength solutions
• State of the art technologies
• Vendor independence
• Ability to support and enhance in-house
• Easy procural
• Cost

Ten Software Tool Sins (10-6)

• 10. Disjoined source and documentation
• 9. No automated testing
• 8. No bug tracking
• 7. Underutilizing compiler, types
• 6. Scripting ignorance

Ten Software Tool Sins (5-1)

• 5. Silencing the compiler
• 4. No version control
• 3. Manual identifier tracking
• 2. Debugging through print statements
• 1. Manual repetitive editing

Overview

• System Software
• Version Control
• Build Management
• Collaboration
• Code Manipulation
• Inspection and Testing
• Performance Measurement
• Documentation and Visualization
• Tool Building

System Software

Operating System

• Linux
• OpenSolaris
• FreeBSD
• OpenBSD

Command Line Tools

• Archivers
• Audio
• Databases
• Development
• Documentation
• Graphics
• Interpreters
• Math
• Network and web
• Security

Scripting Language

• Groovy
• Perl
• Python
• Ruby

The Unix Toolchest

For Windows:

• Cygwin (http://www.cygwin.com/)
• MinGW (http://www.mingw.org/)
• UnxUtils (http://unxutils.sourceforge.net/)
• UWin (http://www.research.att.com/sw/tools/uwin/)
• Outwit (http://www.spinellis.gr/sw/outwit) (Windows integration)

Working with Unix Tools

• Fetch or generate data
• Selection
• Processing
• Summarizing
• Program development

Data Fetching and Generation

• nm: object files
• nm, ldd: executables
• tar: archives
• ar, jar: libraries
• find: directory hierarchies
• wget: the web
• cvs or svn log or annotate: version control
• jot or dd: generate artificial data

Selection

• awk: delimited records
• cut: fixed width files
• sed: select row parts with regular expressions
• grep: select rows with regular expressions
• Combine the above in pipelines

Processing

• sort: order by multiple keys
• uniq: remove duplicates, count
• diff: compare sequential files
• comm: compare ordered lists
• tr: map between character sets
• recode: map between text representations
• tac/rev: reverse order of lines, characters
• rs: reshape arrays
• join: relational join
• Scripting languages handle more complex tasks

Summarizing

• wc: count lines, characters
• head: first elements
• tail: last elements
• fmt: format into lines
• awk with an END block

Plumbing

• Pass data through pipeline
• tee: tap output for further processing
• xargs: apply command to numerous arguments
• Pipe into a while loop for complicated processing

Example: Analyze Java Files

Example: Determine Commit Times

Example Result:Plotted Commit Times

Program Development

• Editors
• GNU Compiler Collection
• Language development tools (bison, flex)
• Tracers and debuggers
• Profilers
• Code formatting tools (cb, indent)
• Version control systems

Outwit Tools

• winclip - Access the windows clipboard
• winreg - manipulate the windows registry
• docprop - read document properties
• odbc - select data from relational databases
• readlink - resolve shell shortcuts
• readlog - access the windows event log

Outwit Examples

An Editor Checklist

• Regular expressions
• Syntax coloring
• Folding
• Error highlighting
• Indentation
• Marking of matching delimiters
• On-line help for API elements
• Code browsing
• Multiple buffers
• Macros
• Refactoring support

Version Control

Version Control

• Tracks project across time
• Collaboration, documentation, undo
• Open source systems in order of increasing capabilities
  • RCS
  • CVS
  • Subversion, Git, Mercurial, Bazaar

The Great Debate

• Centralized (CVS, Subversion)
  • Single source of truth
  • Pre-commit checks
• Distributed (Bazaar, Git, Mercurial, darcs)
  • Can work off line
  • Easier participation (no committer bit)
  • Cheap local commits

Version Control Operations

• Central repository keeps historical data
• A commit or check-in adds a change into the repository
• An update or check-out retrieves a version from the repository
• Versions are tracked with version numbers
  • Numeric: automatically assigned
  • Symbolic: assigned by humans to track milestones
• Keywords in the source files are automatically expanded, identifying version, status, author, etc.

Branching

• Development can split into different branches e.g.
  • Trunk or head
  • Stable or release
  • Feature
  • Vendor
• Branches can join again

Revision Tree Example

Life Under a VCS

• Import
• Update or synchronize
• Commit
• Label or tag

The Goodies

• Identify conflicting changes
• Versioning:
  • Annotated listings
  • Version history
  • File differences
• Branching
• Source of truth
• Metrics

Example of a Change Log

Example of a File Difference List

Example of an Annotated Listing

Extracting Metrics

• Changes per source file
• Changes per author
• Authors per source file
• Lines per day
• Changes corresponding to bug categories

Tracking Example

Best Practices

• Put everything under version control
• Use VCS on your personal projects
• Think carefully about file name and organization
• Perform one separate commit for every change
• Label all releases
• Establish and follow policies and procedures

Build Management

The Build Process

Typical Project Dependencies

Example: Apache Project Dependencies

Makefiles

• Tools like ant and make allow the automatic processing of dependencies
• Dependencies are specified in a special file
• Elements of a Makefile may include
  • Variable definitions
  • Rules for suffixes
  • Dependencies
  • Target build instructions
  • Pseudo-targets
  • Auto-generated dependencies
  • Procedural specifications

Part of an Apache Makefile

Make Problems and Solutions

Complexity

• Dry run with make -n
• Debug mode: print variables
• Before lunch: make -k

Portability

• ant/jam
• Configuration

The Ant Approach

• Build instructions written in XML
• File typically named build . xml
• Variable definitions (named properties)
• Targets and dependencies
• By default, the target specified in the default attribute of the project tag is built
• Tasks are part of ant, typically not external programs

Some ant tasks

• mkdir
• copy
• cvs
• delete
• javac
• jar

Example Ant Build File

Best Practices

• Put all the project's elements under the build system
• Minimize work through dependency analysis
• Modularize the build into several targets
• Common include files
• Explore parallel builds to reduce build times
• Provide a way to build from scratch
• Create a tinderbox build

Best Practices (Variables)

• Directories
• Tool options
• Web sites
• Version strings
• OS dependencies

Collaboration

Mailing Lists

• Easy to setup through /etc/aliases
• Create pointers to owners and files
• Create pointers to files
  mobinet: :include: /home/kfouskas/mobinet
  mobinet-request: kfouskas@aueb.gr
  owner-mobinet: kfouskas@aueb.gr
  
• The file contains the list's addresses
• For high-churn lists consider Mailman (http://www.gnu.org/software/mailman/) or Majordomo (http://www.greatcircle.com/majordomo/).

VCS Notifications

• Arrange for your VCS to notify all committers
• Committers can arrange for a special folder for these notifications
• Concept similar to ATC radio channel

Example Notification

Date: Sat, 19 Aug 2006 08:24:01 +0000 (UTC)
Subject: cvs commit: src/usr.bin/pkill Makefile
X-FreeBSD-CVS-Branch: HEAD

yar         2006-08-19 08:24:01 UTC

  FreeBSD src repository

  Modified files:
    usr.bin/pkill        Makefile
  Log:
  Install pkill(1), aka pgrep(1), to /bin so that rc scripts
  can use this small and nifty utility.  Create compatibility
  symlinks from /usr/bin for the time being to avoid breaking
  custom scripts relying on the hardcoded path to the utility.

  Idea by:        des
  Discussed with: brooks (in cvs-src and cvs-all)

  Revision  Changes    Path
  1.6       +5 -0      src/usr.bin/pkill/Makefile

Setting up CVS Notifications

• Add a line in CVSROOT/loginfo

  ^projname /home/dds/bin/cvs-log %s projname projname@domain.org
  

• Create a notification script

Documentation is Ubiquitous

• System specification
• Software requirements specification
• Design specification
• Test specification
• User documentation
  • Functional description
  • Installation instructions
  • Introductory guide, tutorial
  • Reference manual
  • Administrator manual

Wikis

• Low-overhead editing
• Always current
• Revision history
• Easily accessible

Things to put on a wiki

• Project scorecards
• Personal pages (assgined projects, tasks, planned vacations)
• Time schedule
• Setup, build, release documentation
• Specifications
• Third party documentation pointers

Wiki Best Practices

• Assign people roles
• Publish charter
• Establish naming conventions
• Personal pages
• Meeting agendas / minutes
• Create templates
• Use categories
• See also wikipatterns.com (http://www.wikipatterns.com/)

Issue Tracking

• Together with VCS the two most important process-oriented tools
• Holds bugs, problems, ideas
• Each issue gets a unique id
• Should be coordinated with VCS

Issue Reports

• area of functionality
• state
• resolution
• priority
• severity

Example: Bugzilla

Severity of a Bug

An Issue's Lifecycle

Resolution Options

• Fixed
• Invalid (not a bug)
• Won't fix
• Duplicate (of another bug)
• Works for me
• Moved (to another project)

Example: Releasing

• Triage bugs
  • Fix
  • Document
  • Hide feature
  • Next release
• Introduce a code freeze period
  • Only showstoppers ...
  • ... with no workaround

Bug Tracking Best Practices

• Use an existing issue tracking system
• No more than one bug per issue
• Include bug replication information
• Associate VCS commits with bug ids
• Minimize the number of bugs left open
• Use priorities to manage the developer workload
• Backup your issue database

Code Manipulation

Typical Tasks

• Identify the declarations
• Locate definitions
• Go through the places where an entity is used.
• List deviations from coding standards.
• Discover related code
• Find related comments
• Check for common errors.
• View the code structure.
• Understand how the code interacts with its environment

Regular Expressions

• A regular expression allows the declarative specification of complex strings.
• Regular expressions consist of:
  • letters (that represent themselves) and
  • special symbols
  • Backslash escapes special symbols

Regular Expression Symbols

Character Classes

The Editor as a Code Browser

• Regular expression searches
• Tag facility
• Outlining and folding
• Bird's eye view with a 10% zoom factor
  

Code Searching with grep

• Simple wildcard search
• Search for definitions
• Using a word's root
• Locating files: grep on the output of ls
• File list: grep -l
• Use the result of a file list vi `grep -l ...`
• Stream edit the grep result to create scripts
• Postprocess with grep (-v) to eliminate noise (malloc / xmalloc)
• Use sort -u to eliminate duplicates
• Use fgrep to search for a fixed list of strings
• Use find and xargs to obtain filename lists

Locating File Differences

• Uses:
  • Differences between versions
  • Examining code modifications
  • Verifying test results
• The diff program
• Obtaining a context diff (-c)
• Ignoring blanks (-w)

Using the Compiler

• Generate warning messages.
• Tweak the code to generate error messages.
• Generate program listings.
• Obtain the preprocessor output.
• Examine the generated symbolic (assembly) code.
• Work through the final object code

Compiler Warning Messages

• Expressions associated with portability problems
• Implicit type casts and conversions between different but compatible types
• The use of a constant in place of a conditional expression in if or while statements
• Functions that do not return a value although they should
• Variable declarations that mask other variables with the same name
• Missing enumeration members or extraneous values in a switch statement
• Unknown #pragma options
• Unreferenced variables, structure members, or labels
• Failing to initialize a const declared object

The Compiler as a Code-Reading Tool

• Change the name of a definition and read the errors
• Post-process error messages to create scripts
• Use intermediate files (preprocessing, symbolic code)
• Examine the object code using nm (Unix) or dumpbin (Windows)

Assembly Can be Useful

• To understand how an implementation-defined operation is actually performed
• To see how compiler optimizations affect the resulting code
• To convince yourself that the compiler correctly translates a piece of code, or (rarely) to find a compiler bug
• To verify that hardware-specific code written in C actually performs the operations you expect
• To learn how a particular construct can be written in symbolic code

Code Browsers

• Definitions: Locate where an entity is defined.
• References: Go through all places where an entity is used.
• A call graph: List the functions called by a given function.
• A caller graph: List the functions calling a given function.
• A file outline: List the functions, variables, types, and macros defined in a file.

Code Browsers in OO

• Where the class is defined
• The locations where the class is referenced
• The classes derived from this class
• The base classes for this class
• Its public, protected, and private methods and fields

Refactoring

• Rename and move (various)
• Change organization (classes, interfaces, inheritance)
• Local variables into class fields
• Code into method
• Change method signature
• Create getters and setters

Beautifiers

• Fix code that is written truly inconsistently without following any formatting standards
• Adopt orphaned code for maintenance
• Create a temporary version of the code to help you decipher it
• Integrate code under the common roof of a larger project

Cdecl: Decoding C Declarations

• Convert C declarations into English
• Convert English into a C declaration

void (*signal(int sig, void (*func)(int)))(int);

declare signal as function that expects (sig as int, func as pointer to
function that expects (int) returning void) returning pointer to function
that expects (int) returning void;

Inspection and Testing

Static Verification

• C/C++: gcc, lint (*BSD)
• Java: FindBugs (http://findbugs.sourceforge.net/), PMD (http://pmd.sourceforge.net/)
• HTML: W3C validator (http://validator.w3.org/)

• Detection of common errors
• Various levels of inference
• False positives and negatives

Identified Issues

• Possible bugs
• Portability problems (C/C++)
• Function return values (C/C++)
• Dead code
• Overcomplicated expressions
• Suboptimal code
• Duplicate code

Some FindBugs Issues

• An apparent infinite recursive loop.
• A container is added to itself.
• A volatile reference to an array doesn't treat the array elements as volatile
• Usage of GetResource may be unsafe if class is extended
• Creates an empty jar file entry
• Dubious catching of IllegalMonitorStateException
• Method performs math using floating point precision
• Class implements Cloneable but does not define or use clone method
• clone method does not call super.clone()
• Method might drop exception
• Method might ignore exception
• Method invokes dubious new String(String) constructor; just use the argument
• Method invokes dubious new String() constructor; just use ""

A FindBugs Run

GCC Best Practices

• Compile C/C++ with all warnings enabled (gcc -Wall)
• When warnings are enabled, alse enable optimization (gcc -O)
• Treat warnings as errors (gcc -Werror)
• Compile a debug version of C++ code defining -D_GLIBCXX_DEBUG -D_GLIBCXX_DEBUG_PEDANTIC
• Annotate printf-like functions. Example:
  extern void argp_error (__const struct argp_state *__restrict __state,
                          __const char *__restrict __fmt, ...)
       __attribute__ ((__format__ (__printf__, 2, 3)));
  
  (Other formats include scanf, wprintf, wscanf).

C Compilation Example

$ gcc t.c
$

$ gcc -O4 -Wall -Werror t.c
cc1: warnings being treated as errors
t.c:9: warning: return-type defaults to `int'
t.c:9: warning: first argument of `main' should be `int'
t.c:9: warning: `main' takes only zero or two arguments
t.c: In function `main':
t.c:13: warning: implicit declaration of function `printf'
t.c:13: warning: double format, different type arg (arg 2)
t.c:10: warning: unused variable `c'
t.c:14: warning: control reaches end of non-void function
t.c:10: warning: `b' might be used uninitialized in this function

Lint Best Practices

• Include a lint pass in your build
• Comment case statements lacking a break with FALLTHROUGH
• Comment uneachable code (exit, longmp) with NOTREACHED
• Comment functions taking format arguments with PRINTFLIKE, SCANFLIKE
• Sparingly use #ifndef lint to silence lint warnings

Tracing Tools

Using Tracing Tools

• Locate bugs without using the source code
• Explain an application's behavior
• Determine undocumented features
• Discover appropriate APIs
• Locate performance bottlenecks

[sl]trace Tips and Tricks

• Trace running processes using the -p flag
• Send output to a file using the -o flag
• Read complete strings using a large argument (e.g. 1024) to -s
• Trace forked processes with the -f flag

Example: System Call Tracing

$ strace dot </dev/null 2>&1 | fgrep .so | fgrep -v ENOENT
open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/usr/lib/libfreetype.so.6", O_RDONLY) = 3
open("/usr/lib/libpng.so.2", O_RDONLY)  = 3
open("/usr/lib/libjpeg.so.62", O_RDONLY) = 3
open("/usr/lib/libz.so.1", O_RDONLY)    = 3
open("/lib/libm.so.6", O_RDONLY)        = 3
open("/lib/libc.so.6", O_RDONLY)        = 3

Example: Library Call Tracing

$ ltrace whoami
[...]
geteuid()                                         = 1000
getpwuid(1000, 0x40013678, 0xbffffcac, 0x08048cff, 0x4012ee48) = 0x401310f0
puts("dds"dds
)                                       = 4
exit(0 <unfinished ...>

Unit Testing

• JUnit (http://www.junit.org/) and its friends is the name of the game
• Based on test classes
• Test class contains method annotated @Before
• Test methods are annotated with @Test
• Test methods call assertTrue
• main calls junit.textui.TestRunner.run(TestClass.class)

JUnit Example

Test Class and Fields

Initialization Code

A Test Method

Run the Class's Test Methods

Textual Regression Testing

• Used for batch oriented applications
• A loop goes over the input files
• Each input file creates a new output file
• Output file compared with correct file
• A "priming" mode allows the creation of the correct files

Example: Testing the Sort Program

Test Coverage in C/C++ Code

• Compile with gcc -pg -g -fprofile-arcs -ftest-coverage
• Execute program; a file name.bb will be created
• Execute program; a file name.bb will be created
• Run gcov -n name.bb
• You will get a report of how many lines were executed

 83.33% of 650 source lines executed in file hilbert.c

Performance Measurement

Program States

• Directly executing code (user time)
• The kernel executes code on behalf of the program (system time)
• Waiting for an external operation to complete (idle time)

• Real or clock time
• CPU time

Measuring Workload

• For terminating processes use the time command

  $ time make
  [...]
  real    0m9.380s
  user    0m2.580s
  sys     0m6.640s
  

• For servers use top

   22:14:03 up  5:22,  2 users,  load average: 0.00, 0.01, 0.00
  39 processes: 38 sleeping, 1 running, 0 zombie, 0 stopped
  CPU states:   0.1% user,   0.4% system,   0.0% nice,  99.5% idle
  

Workload and Tools

System Monitoring Tools

I/O

iostat (under *BSD), vmstat (Linux)

Memory

vmstat, free (Linux)

Network

netstat

$ vmstat 1
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 1  0  0      0  52296   3476  50092   0   0   340     0  189   190   0   6  94
 0  1  0      0  51736   3588  50428   0   0   448     0  214   243   0   5  95
 1  0  0      0  51072   3716  50724   0   0   424     0  210   218   0   5  95
 0  1  0      0  50596   3848  50968   0   0   376     0  196   204   0   4  96
 1  0  0      0  49952   3976  51304   0   0   464     0  220   247   0   6  94
 0  1  0      0  49556   4068  51512   0   0   300     0  177   172   2   2  96
 1  0  0      0  49024   4132  51816   0   0   368     0  196   195   0   7  93
 1  0  0      0  48228   4200  52416   0   0   668     0  270   307   0   9  91
 1  0  0      0  47668   4352  52712   0   0   448     0  215   227   0   5  95
 1  0  0      0  47048   4412  53092   0   0   440     0  213   244   1   5  94
 0  1  0      0  44912   4668  54080   0   0   432     0  211   225   0   4  96

Profiling System Calls

• The strace -c flag counts calls and reports time
• The results typically follow a Pareto distribution

$ strace -c find . -name test
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 66.06    0.367503         584       629           getdents64
 26.31    0.146389          80      1826           lstat64
  2.27    0.012621          40       313           fcntl64
  1.83    0.010168          31       326        10 open
  1.78    0.009894          16       625           chdir
  1.23    0.006846          22       316           fstat64
[...]

Function Profiling

• Statistical (prof)
• Graph-based (gprof, Java)
• Pinpoint hotspots

Profiling C/C++ Code with Gprof

• Compile with gcc -pg
• Execute program; a file gmon.out will be produced
• The gprof program produces the report

A Flat Profile

A Call Graph Profile

Profiling Java Code with EJP

• Add the tracer library to your library path
• Run program under the Java VM with -Xruntracer
• View the results with the EJP Presenter

A Java Program Profile

Basic Block Counting in C/C++ Code

• Compile with gcc -pg -g -fprofile-arcs -ftest-coverage
• Execute program; a file name.bb will be created
• Run gcov -l name.bb
• A .gcov file for each source file will be created

Architectural Inefficiencies

• Cache misses
• Branch mispredicts
• Front end stalls (failure to deliver enough instructions)
• Cache pollution through address aliasing (set associativity issues)
• Long latency instructions
• TLB misses

Architectural Profiling Tools

Processor event monitoring

Oprofile (http://oprofile.sourceforge.net/) (Linux)

Locality optimizations

SLO: Suggestions for Locality Optimizations (http://slo.sourceforge.net/)

• Clock cycles when not halted
• number of instructions retired
• number of L2 cache requests
• L2 cache demand requests from this core
• Page table walk events
• cycles divider is busy
• L1 cacheable data read operations
• Bus cycles when data is sent on the bus
• number of instruction fetch misses
• Branch predicted taken with bubble 1
• [100 more]

Memory Performance

• Difference in latency between memory types
• Large data sets
• Embedded applications
• Bandwidth requirements

Memory Profiling C/C++ Code with Valgrind

• Create a debug build (-g)
• Run the program under valgrind
• Examine valgrind's output
• Can also detect memory leaks

Example: sed under valgrind

Documentation and Visualization

Declarative Text Processing

• troff
• LaTeX
• DocBook (http://docbook.sourceforge.net/)

DocBook Details

• Input is XML
  <book id="simple_book">
    <title>Code Quality</title>
    <chapter id="perf">
      <title>Time Performance</title>
      <para>
        A number of software attributes are associated with the behavior
        of the system across time.
      </para>
    </chapter>
  </book>
  
• Output (via XSL) can be
  • HTML
  • PDF
  • Unix man pages
  • RTF
  • HTML Help

Best Practices

• Separate content from presentation
• Modularize
• Declare new elements
• Put documents under revision control
• Use build tools
• Automatically extract content from source code

Documentation from Source Code

• Javadoc
• UMLGraphDoc
• Doxygen (http://www.doxygen.org/)
• Ad hoc
  Examples:
  • Database schema
  • Error messages
  • Configuration file contents

Javadoc Example

Automation Example: C++ Code

Automation Example: Processing Script

Automation Example: Documentation Page

Table STRINGS

Strings in the code.

Table LINEPOS

Line number offsets within each file.

Table PROJECTS

Project details.

Table FCALLS

Function calls.

Graph and Diagram Types

• Metrics
• Call graphs
• Class hierarchies
• Concrete object instances
• Data structures (trees, lists, and so on)
• Bit fields and corresponding masks in bit-mapped registers
• State transition diagrams
• Strings or arrays and corresponding indices or pointers
• Entity-relationship diagrams

Declarative Drawing Tools

• GraphViz (http://www.graphviz.org) dot: directed graphs
• GNU Plot (http://www.gnuplot.info): data plotting
• UMLGraph (http://www.spinellis.gr/sw/umlgraph): UML class and sequence diagrams
• GraphViz neato: undirected graphs
• pic (http://www.gnu.org/software/plotutils/): declarative vector graphics
• GMT (http://gmt.soest.hawaii.edu): map drawing
• SVG (inkscape)/ Postscript (Ghostscript): low-level vector graphics

UMLGraph Class Diagram

Detailed Class Diagram

UMLGraph Sequence Diagram

# Define the objects object(O,"o:Toolkit"); placeholder_object(P); step(); # Activation and messages active(O); message(O,O,"callbackLoop()"); create_message(O,P,"p:Peer"); message(O,P,"handleExpose()"); active(P); return_message(P,O,""); inactive(P); destroy_message(O,P); inactive(O); # Complete the lifeline of O step(); complete(O);

Directory Hierarchies with dot

Plotting Test Coverage: GNU Plot Script

Plotting Test Coverage: Result

Memory Fragmentation Example

Stack Usage Example

Map Example

Tool Building

Implementation Options

• The Unix shell and tools (sed, awk, grep)
• Scripting language (Groovy, Perl, Python, Ruby)
• Visual Basic
• C, C++, Java

Choosing an Implementation

• A filtering pipeline (e.g. CVS logs): the Unix shell
• Record processing: awk
• Simple line-oriented string processing: sed
• Combination of tasks: scripting language
• Web front end: PHP, SWILL, GWT, Ruby on Rails
• Tight integration with Microsoft products: Visual Basic
• Extreme performance requirements: C, C++, Java

A Simple Grep Program in Java ...

... And its Equivalent in Perl

Advice

• Exploit the capabilities of modern rapid-prototyping languages.
• Start with a simple design, gradually improving it as needed.
• Use heuristics that exploit the lexical structure of the code.
• Tolerate some output noise or silence
• Use other tools to preprocess your input or postprocess your output.

Example: Signature Survey

• Contents

 Last change: Friday, January 29, 2010 11:39 am
 Unless otherwise expressly stated, all original material on this page created by Diomidis Spinellis is licensed under a Creative Commons Attribution-Share Alike 3.0 Greece License.