# otrace: An object-oriented debugger for nonlinear tracing

Contents

# 1 Introduction

otrace is an object-oriented debugger for nonlinear tracing of asynchronous or multithreaded interactive programs. It addresses some of the limitations of sequential debugging techniques which do not work well with server programs, where multiple requests are handled in parallel. For example, instrumenting web servers with print/logging statements can often result in voluminous log output with interleaved streams of messages.

otrace takes a different approach to debugging that relies less on sequential operations. Its features including taking "snapshots" of variables for tracing, "tagging" objects for tracking across different method invocations, and modifying live code ("monkey patching") to insert print statements etc.

otrace maps all the objects in the running program, as well as the "snapshot" objects, to a virtual filesystem mounted under `/osh`. It provides a shell-like interface, oshell, with commands like cd, ls etc. that can be used to browse classes, methods, and instance variables in the virtual filesystem. Tab completion and simple wildcarding are supported.

otrace may be used as:
  - a tracing tool for debugging web servers and interactive programs
  - a console or dashboard for monitoring production servers
  - a teaching tool for exploring the innards of a program
  - a code patching tool for unit testing

# 2 Installation

If you wish to install otrace without the sample programs, the `easy_install otrace` command should be sufficient (provided the `setuptools` module is installed).

The latest archive of otrace including sample programs may be downloaded from the Github tags page. The unzipped archive should contain the following files (and some more):

> `hello_trace.py ordereddict.py otrace.py README.rst setup.py ...`

All the code for the otrace module is contained in a single file, `otrace.py`. (For python 2.6 or earlier, you will also need `ordereddict.py`.) To use otrace without installing it, just ensure that these files are present in the module load path. If you wish to install otrace, use:

> `python setup.py install`

# 3 Support

This README file provides a brief introduction to otrace. A tutorial using a demo program is available in docs/GettingStarted.rst Additional documentation will be made available on the project home page, info.mindmeldr.com/code/otrace.

Report bugs and other problems using Github Issue Tracker.

# 4 Usage

otrace does not consume any resources until some tracing action is initiated. So it can be included in production code without any performance penalty. otrace works well with detached server processes (daemons) via the GNU screen utility that emulates a terminal.

otrace is meant to be used in conjunction with an event loop, which is usually present in programs that interact with users such as web servers or GUI applications. otrace takes control of the terminal, and would not work very well with programs that read user input directly from the terminal (or standard input).

To use otrace, simply `import otrace` and instantiate the class `otrace.OShell`, which provides a unix-like shell interface to interact with a running program via the terminal.

Here is a simple server example:

    import BaseHTTPServer
    from SimpleHTTPServer import SimpleHTTPRequestHandler
    from otrace import OShell, traceassert

    http_server = BaseHTTPServer.HTTPServer(("", 8888), SimpleHTTPRequestHandler)
    oshell = OShell(locals_dict=locals(), globals_dict=globals(),
                    new_thread=True, allow_unsafe=True, init_file="server.trc")
    try:
        oshell.loop()
        http_server.serve_forever()   # Main event loop
    except KeyboardInterrupt:
        oshell.shutdown()

Usage notes:

> - If you run in oshell in its own daemon thread as shown above, use the ^C sequence to abort the main thread, and call `OShell.shutdown` from main thread to cleanup terminal I/O etc.

> - If you run oshell in the main thread and the event loop in a separate thread, ^C will abort and cleanup oshell. You may need to shutdown the event loop cleanly after that.

> - Install the python `readline` module (`easy_install readline`) to enable TAB command completion.

> - To start a detached server (daemon) process, use the command:

>     `screen -d -m -S <screen_name> <executable> <argument1> ...`

>   To attach a terminal to this process, use:

>     `screen -r <screen_name>`

> - By default, otrace logs to the `logging` module. Subclass `TraceCallback`, overriding the methods `callback` and `returnback` to implement your own logging (see `DefaultCallback` for a simple example)

# 5 Implementation

otrace uses a Virtual Directory Shell Interface which maps all the objects in a a running python program to a virtual filesystem mounted in the directory `/osh` (sort of like the unix `/proc` filesystem, if you are familiar with it). Each module, class, method, function, and variable in the global namespace is mapped to a virtual file within this directory. For example, a class `TestClass` in the `globals()` dictionary can be accessed as:

    /osh/globals/TestClass

and a method `test_method` can be accessed as:

    /osh/globals/TestClass/test_method

and so on.

otrace provides a unix shell-like interface, oshell, with commands such as `cd`, `ls`, `view`, and `edit` that can be used navigate, view, and edit the virtual files. Editing a function or method "monkey patches" it, allowing the insertion of `print` statements etc. in the running program.

The `trace` command allows dynamic tracing of function or method invocations, return values, and exceptions. This is accomplished by dynamically decorating (or wrapping) the function to be traced. When a trace condition is satisfied, the function-wrapper saves context information, such as arguments and return values, in a newly created virtual directory in:

    /osh/recent/*

These trace context directories can be navigated just like `/osh/globals/*`. (If there are too many trace contexts, the oldest ones are deleted, unless they have been explicitly saved.)

oshell allows standard unix shell commands to be interspersed with oshell-specific commands. The path of the "current working directory" determines which of the these two types of commands will be executed. If the current working directory is not in `/osh/*`, the command is treated as a standard unix shell command (except for `cd`, which is always handled by oshell.)

# 6 Commands

oshell supports the following commands ([..] denotes optional parameters; | denotes alternatives):

    alias name cmd <arg\*> <arg\1>... # Define alias for command
    cd [pathname]             # change directory to "pathname", which may be omitted, "..", or "/" or a path
    cdls [pathname]           # cd to "pathname" and list "files" (cd+ls)
    del [trace_id1..]         # Delete trace context
    dn                        # Command alias to move one level down in stack frames in a trace context (to a newer frame)
    edit [-f] (filename|class[.method]) [< readfile]  # Edit/patch file/method/function
    exec python_code          # Execute python code (also !<python_code>)
    help [command|*]          # Display help information
    lock                      # Lock terminal until password is entered
    ls [-acflmtv] [-(.|..|.baseclass)] [pathname1|*]   # List pathname values (or all pathnames in current "directory")
    pr python_expression      # Print value of expression (DEFAULT COMMAND)
    pwd                       # Print current working "directory"
    quit                      # Quit shell
    repeat command            # Repeat command till new input is received
    resume [trace_id1..]      # Resume from breakpoint
    rm [-r] [pathname1..]     # Delete entities corresponding to pathnames (if supported)
    save [trace_id1..]        # Save current or specified trace context
    set [parameter [value]]   # Set (or display) parameter
    source filename           # Read input lines from file
    tag [(object|.) [tag_str]]    # Tag object for tracing
    trace [-a (break|ipdb|pdb|hold|tag)] [-c call|return|all|tag|comma_sep_arg_match_conditions] [-n +/-count] ([class.][method]|db_key|*)   # Enable tracing for class/method/key on matching condition
    unpatch class[.method]|* [> savefile]  # Unpatch method (and save patch to file)
    untag [object|.]          # untag object
    untrace ([class.][method]|*|all)  # Disable tracing for class/method
    up                        # Command alias to move one level up in stack frames in a trace context (to an older frame)
    view [-d] [-i] [class/method/file]  # Display source/doc for objects/traces/files

The default command is `pr`, which evaluates an expression. So you can simply type a python variable to print out its value. You can also insert `otrace.traceassert(<condition>,label=..,action=..)` to trace assertions.

# 7 Python 3

`otrace.py` and the demo program `hello_trace.py` work with Python 3, after porting using the `2to3` tool. Further testing remains to be done.

# 8 Caveats

> - Reliability: This software has not been subject to extensive testing. Use at your own risk.

> - Thread safety: In principle, otrace should thread-safe, but more testing is needed to confirm this in practice.

> - Memory leaks: The trace contexts saved by otrace could potentially lead to increased memory usage. Again, only experience will tell.

> - Platforms: otrace is pure-python, but with some OS-specific calls for file, shell, and terminal-related operations. It has been tested only on Linux and Mac OS X so far, although the demo program works with the Windows console as well.

> - Current limitations:
>     - Decorated methods cannot be patched.
>     - TAB command completion is a work in progress.
>     - Spaces and other special characters in command arguments need to be handled better.

# 9 Credits

otrace was developed as part of the Mindmeldr project, which is aimed at improving classroom interaction.

otrace was inspired by the following:
  - the tracing module echo.py written by Thomas Guest <tag@wordaligned.org>. This nifty little program uses decorators to trace function calls.
  - the python `dir()` function, which treats objects as directories. If objects are directories, then shouldn't we be able to inspect them using the familiar `cd` and `ls` unix shell commands?
  - the unix proc filesystem, which cleverly maps non-file data to a filesystem interface mounted at `/proc`
  - the movie Being John Malkovich (think of `/osh` as the portal to the "mind" of a running program)

# 10 License

otrace is distributed as open source under the BSD-license.