Author: John J. Lee <jjl@pobox.com>
Keywords: Python,HTML,HTTP,browser,stateful,web,client,client-side,mechanize,cookie,form,META,HTTP-EQUIV,Refresh,ClientForm,ClientCookie,pullparser,WWW::Mechanize
Date: 2010-04-22

# mechanize — Hints

Hints for debugging programs that use mechanize.

## Cookies

A common mistake is to use `mechanize.urlopen()`, and the `.extract_cookies()` and `.add_cookie_header()` methods on a cookie object themselves. If you use `mechanize.urlopen()` (or `OpenerDirector.open()`), the module handles extraction and adding of cookies by itself, so you should not call `.extract_cookies()` or `.add_cookie_header()`.

Are you sure the server is sending you any cookies in the first place? Maybe the server is keeping track of state in some other way (`HIDDEN` HTML form entries (possibly in a separate page referenced by a frame), URL-encoded session keys, IP address, HTTP `Referer` headers)? Perhaps some embedded script in the HTML is setting cookies (see below)? Turn on logging.

When you `.save()` to or `.load()`/`.revert()` from a file, single-session cookies will expire unless you explicitly request otherwise with the `ignore_discard` argument. This may be your problem if you find cookies are going away after saving and loading.

    import mechanize
    cj = mechanize.LWPCookieJar()
    opener = mechanize.build_opener(mechanize.HTTPCookieProcessor(cj))
    mechanize.install_opener(opener)
    r = mechanize.urlopen("http://foobar.com/")
    cj.save("/some/file", ignore_discard=True, ignore_expires=True)

JavaScript code can set cookies; mechanize does not support this. See the FAQ.

## General

Enable logging.

Sometimes, a server wants particular HTTP headers set to the values it expects. For example, the `User-Agent` header may need to be set to a value like that of a popular browser.

Check that the browser is able to do manually what you’re trying to achieve programatically. Make sure that what you do manually is exactly the same as what you’re trying to do from Python — you may simply be hitting a server bug that only gets revealed if you view pages in a particular order, for example.

Try comparing the headers and data that your program sends with those that a browser sends. Often this will give you the clue you need. There are browser addons available that allow you to see what the browser sends and receives even if HTTPS is in use.

If nothing is obviously wrong with the requests your program is sending and you’re out of ideas, you can reliably locate the problem by copying the headers that a browser sends, and then changing headers until your program stops working again. Temporarily switch to explicitly sending individual HTTP headers (by calling `.add_header()`, or by using `httplib` directly). Start by sending exactly the headers that Firefox or IE send. You may need to make sure that a valid session ID is sent — the one you got from your browser may no longer be valid. If that works, you can begin the tedious process of changing your headers and data until they match what your original code was sending. You should end up with a minimal set of changes. If you think that reveals a bug in mechanize, please report it.

## Logging

To enable logging to stdout:

    import sys, logging
    logger = logging.getLogger("mechanize")
    logger.addHandler(logging.StreamHandler(sys.stdout))
    logger.setLevel(logging.DEBUG)

You can reduce the amount of information shown by setting the level to `logging.INFO` instead of `logging.DEBUG`, or by only enabling logging for one of the following logger names instead of `"mechanize"`:

- `"mechanize"`: Everything.

- `"mechanize.cookies"`: Why particular cookies are accepted or rejected and why they are or are not returned. Requires logging enabled at the `DEBUG` level.

- `"mechanize.http_responses"`: HTTP response body data.

- `"mechanize.http_redirects"`: HTTP redirect information.

## HTTP headers

An example showing how to enable printing of HTTP headers to stdout, logging of HTTP response bodies, and logging of information about redirections:

    import sys, logging
    import mechanize

    logger = logging.getLogger("mechanize")
    logger.addHandler(logging.StreamHandler(sys.stdout))
    logger.setLevel(logging.DEBUG)

    browser = mechanize.Browser()
    browser.set_debug_http(True)
    browser.set_debug_responses(True)
    browser.set_debug_redirects(True)
    response = browser.open("http://python.org/")

Alternatively, you can examine request and response objects to see what’s going on. Note that requests may involve “sub-requests” in cases such as redirection, in which case you will not see everything that’s going on just by examining the original request and final response. It’s often useful to use the `.get_data()` method on responses during debugging.

### Handlers

This section is not relevant if you use `mechanize.Browser`.

An example showing how to enable printing of HTTP headers to stdout, at the `HTTPHandler` level:

    import mechanize
    hh = mechanize.HTTPHandler()  # you might want HTTPSHandler, too
    hh.set_http_debuglevel(1)
    opener = mechanize.build_opener(hh)
    response = opener.open(url)

The following handlers are available:

NOTE: as well as having these handlers in your `OpenerDirector` (for example, by passing them to `build_opener()`) you have to turn on logging at the `INFO` level or lower in order to see any output.

`HTTPRedirectDebugProcessor`: logs information about redirections

`HTTPResponseDebugProcessor`: logs HTTP response bodies (including those that are read during redirections)

I prefer questions and comments to be sent to the mailing list rather than direct to me.

John J. Lee, April 2010.