Install & Compatibility
Where this runs
tested against v0.2.8 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 0.000s · 123.3MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 10.8s · import 0.000s · 125MB
112MB installed
● package 112MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
Article
✓ from newspaper import Article
✗ from newspaper3k import Article
Despite the package name 'newspaper3k', the top-level import for classes is typically 'newspaper'.
build
✓ import newspaper; newspaper.build(...)
✗ import newspaper3k; newspaper3k.build(...)
The 'build' function is accessed directly from the imported 'newspaper' module.
Config
✓ from newspaper import Config
Used for advanced configurations like user agents, proxies, and caching.
This quickstart demonstrates how to extract an article's content and metadata, including NLP-generated keywords and summaries. It also includes configuration for a user agent and NLTK 'punkt' tokenizer download, which is necessary for NLP features.
import newspaper
from newspaper import Article, Config
import os
# Configure a user agent to avoid being blocked
config = Config()
config.browser_user_agent = os.environ.get('USER_AGENT', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36')
config.request_timeout = 10 # Set a timeout
# Ensure NLTK 'punkt' is downloaded for NLP features
try:
import nltk
nltk.data.find('tokenizers/punkt')
except nltk.downloader.DownloadError:
print("Downloading NLTK 'punkt' tokenizer...")
nltk.download('punkt')
print("NLTK 'punkt' tokenizer downloaded.")
url = 'https://www.reuters.com/world/europe/ukraine-braces-russian-attacks-east-civilians-flee-2022-04-08/'
article = Article(url, config=config)
article.download()
article.parse()
print(f"Title: {article.title}")
print(f"Authors: {article.authors}")
print(f"Publish Date: {article.publish_date}")
print(f"Top Image: {article.top_image}")
print(f"\nText (first 500 chars):\n{article.text[:500]}...")
article.nlp() # Run NLP for keywords and summary
print(f"\nKeywords: {article.keywords}")
print(f"Summary: {article.summary[:200]}...")
# Example for a news source
# cnn_paper = newspaper.build('http://cnn.com', config=config)
# print(f"CNN has {cnn_paper.size()} articles.")
# for article_obj in cnn_paper.articles[:3]:
# print(f" - {article_obj.url}")
Debug
Known issues
breakingThe `newspaper` package is for Python 2 and is deprecated. For Python 3, you MUST install `newspaper3k`. Using `pip install newspaper` on Python 3 might lead to issues or install an old, unmaintained version.fixAlways use `pip install newspaper3k`. Ensure your environment uses Python 3.
affects: <=0.0.9 (for `newspaper`), all versions of `newspaper3k` when incorrectly installed
deprecatedThe `newspaper3k` library has not seen a PyPI release since 2018 (version 0.2.8). While still functional, it may struggle with modern web structures or newer Python versions. Consider `newspaper4k` (a community fork) for active development and bug fixes.fixBe aware of potential parsing failures on complex or JavaScript-heavy sites. For active development and improved parsing, evaluate the `newspaper4k` library (e.g., `pip install newspaper4k`).
affects: 0.2.8 and older
gotchaWebsite HTML structures change frequently, which can break `newspaper3k`'s article extraction logic. Common issues include missing authors, incomplete text, or inability to parse specific elements.fixFor persistent issues on specific sites, inspect the website's HTML. You may need to manually extract content using `newspaper.utils.BeautifulSoup` or integrate other scraping tools like `requests` and `BeautifulSoup` for pre-processing.
affects: All
gotchaThe NLP features (like `article.nlp()` for keywords and summaries) rely on NLTK and require the `punkt` tokenizer data to be downloaded. Without it, you will encounter `LookupError`.fixRun `python -c "import nltk; nltk.download('punkt')"` once to download the necessary data after installing the library. Ensure `nltk` is installed (it's a dependency, but the data is separate). affects: All
gotchaAggressive or frequent requests without setting a proper `User-Agent` or `request_timeout` can lead to `ReadTimeout` errors or IP blocking by target websites.fixAlways configure a `Config` object with `config.browser_user_agent` set to a common browser user agent string and `config.request_timeout` to a reasonable value before using `Article` or `build`. Consider using proxies if scraping at scale.
affects: All
Upgrade
Version history
0.2.8latest on PyPI · released Sep 28, 2018
Audit
Dependencies
nltkrequiredUsed for Natural Language Processing features like keyword extraction and summarization, specifically requires the 'punkt' tokenizer.
lxmlrequiredCore dependency for efficient HTML parsing.
beautifulsoup4requiredUsed for parsing HTML, often internally, and can be used for custom extraction when core library fails.
requestsrequiredHandles HTTP requests for downloading web content.
PillowrequiredFor image processing.