mechanize

library0.4.10pypypi✓ verified 85d ago

mechanize provides a stateful, programmatic web browsing interface, allowing for opening URLs, following links, submitting forms, and handling cookies. It simulates a web browser's behavior without a GUI or JavaScript engine. The current version is 0.4.10, released in 2023, and it follows a slow release cadence, primarily for maintenance and bug fixes.

http-networking web-framework testing

pip install mechanize

github ↗package ↗

INSTALL

IMPORT

SIG · MECHANIZE

mechanize

http-networkingpythonv0.4.10

Install

1.8s avg

Import

161ms

Disk

18MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.4.10 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.920 runs

installs and imports cleanly · install 0.0s · import 0.164s · 19.9MB

glibc

py 3.10–3.920 runs

installs and imports cleanly · install 1.8s · import 0.159s · 20MB

18MB installed

● package 18MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Browser

✓ import mechanize br = mechanize.Browser()

✗ from mechanize import Browser

While `from mechanize import Browser` works, the idiomatic and generally recommended way to import and use mechanize is `import mechanize` and then access its components via `mechanize.Browser()`.

This quickstart demonstrates how to initialize a mechanize Browser, configure it with a cookie jar and custom user-agent, and open a URL. It includes common settings like disabling robots.txt handling.

import mechanize import http.cookiejar as cookielib br = mechanize.Browser() # Cookie Jar setup (optional but recommended for stateful browsing) cj = cookielib.LWPCookieJar() br.set_cookiejar(cj) # Browser options br.set_handle_equiv(True) br.set_handle_gzip(True) br.set_handle_redirect(True) br.set_handle_robots(False) # Often set to False for scraping # User-Agent br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')] # Open a page url = "http://www.example.com/" # Use a placeholder for environment variables if authentication is needed # url = os.environ.get('MECHANIZED_TARGET_URL', 'http://www.example.com/') try: response = br.open(url) print(f"Title: {br.title()}") print(f"Status: {response.code}") # print(response.read().decode('utf-8')) except Exception as e: print(f"An error occurred: {e}")

Debug

Known issues

gotchamechanize does NOT execute JavaScript. It's a 'headless' browser in the sense it has no GUI, but it cannot render or interact with dynamic content generated by JavaScript. If a page relies on JavaScript for content loading, form submission, or navigation, mechanize will not see or interact with it.

fix

For JavaScript-heavy sites, consider a full-fledged browser automation library like Selenium or Playwright.

affects: All versions

breakingMajor breaking changes occurred during the transition from Python 2 to Python 3. Code written for mechanize on Python 2.x is likely incompatible with Python 3.x due to changes in internal modules (e.g., `_mechanize` C module removed) and string/bytes handling.

fix

Ensure you are using `mechanize` version 0.3.0 or higher for Python 3. Refactor Python 2 code, paying attention to `str`/`bytes` conversions and any reliance on internal mechanize Python 2 specifics.

affects: <0.3.0 (Python 2) vs. >=0.3.0 (Python 3)

gotchaBy default, mechanize respects `robots.txt` rules. Many scraping tasks require bypassing this, which can lead to `HTTP Error 403: Forbidden` or simply not accessing desired content.

fix

To ignore `robots.txt`, set `br.set_handle_robots(False)` immediately after initializing the browser instance.

affects: All versions

gotchaMany modern websites require specific User-Agent headers to display content correctly or to prevent blocking. Without setting a realistic User-Agent, you might receive errors or be served different content.

fix

Always set a custom User-Agent string using `br.addheaders = [('User-agent', 'YOUR_USER_AGENT_STRING')]`.

affects: All versions

Errors

Common errors & fixes

http.client.BadStatusLine: ''

The server responded with an invalid or empty HTTP status line. This often indicates a malformed server response, or the server closing the connection unexpectedly.

fix

Check the target URL in a regular browser. Ensure your User-Agent is realistic. The server might be blocking your requests due to suspicious headers or rate limiting. Try increasing timeouts or retrying.

AttributeError: 'NoneType' object has no attribute 'get_header' or 'AttributeError: 'NoneType' object has no attribute 'headers'

This error typically occurs when `br.open()` fails to retrieve a valid response object (e.g., due to a network error, DNS failure, or a very quick connection reset), and subsequent code tries to access headers or other attributes on a `None` object.

fix

Wrap `br.open()` calls in a `try-except` block to catch `mechanize.URLError` or `mechanize.HTTPError`. Verify the URL and network connectivity. Inspect `br.response()` if available for details.

mechanize._response.html.FormNotFoundError: no form matching name ... or nr ...

You are trying to select a form that doesn't exist on the current page, or your selection criteria (name, id, index `nr`) do not match any available forms.

fix

Inspect the HTML content of the page (`response.read()`) to identify the correct form attributes (name, id) or its numerical index. You can iterate `for form in br.forms(): print(form)` to see all available forms.

mechanize._urllib2_fork.HTTPError: HTTP Error 403: Forbidden

The server denied access to the resource. This could be due to not respecting `robots.txt`, an invalid or missing User-Agent, IP blocking, or other security measures.

fix

Ensure `br.set_handle_robots(False)` is set if you intend to ignore `robots.txt`. Set a realistic User-Agent string. If persistent, consider rotating IP addresses or waiting before retrying.

Upgrade

Version history

0.4.10latest on PyPI · released Apr 26, 2024

Audit

Dependencies

No dependency data recorded yet.

Agent activity

20 hits · last 30 days

node

Resources

githubhttps://github.com/python-mechanize/mechanize ↗

packagehttps://pypi.org/project/mechanize/ ↗