Registry / http-networking / tweet-harvest

tweet-harvest

library2.7.1jsnpmunverified

Tweet Harvest is an active command-line interface (CLI) tool designed for scraping tweets from Twitter search results. Utilizing Playwright, it automates browser interactions to retrieve data based on specified keywords and date ranges, exporting the results to CSV or XLSX formats. The current stable version is 2.7.1, with frequent minor releases addressing bug fixes, performance improvements, and new export functionalities (e.g., XLSX export in v2.7.0). A key differentiator is its reliance on a valid Twitter `auth_token` cookie for authentication, as Twitter prohibits unauthenticated search. While primarily a CLI, it also exposes programmatic APIs for integration into Node.js applications, offering functions to initiate the scraping process and process tweet data. Its continuous updates ensure compatibility with Twitter's evolving interface and provide enhanced data quality features like ISO 8601 timestamps.

http-networking data devops

npm install tweet-harvest

github ↗package ↗

INSTALL

IMPORT

SIG · TWEET-HARVEST

tweet-harvest

http-networkingjavascriptv2.7.1

Install

—

Import

—

Disk

—

Pass rate

0/ 6

Env Coverage0 / 6

glibc

18–22

musl

18–22

Install & Compatibility

Where this runs

tested against v? · npm install

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

node 18–226 runs

build_error

glibc

node 18–226 runs

build_error

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

harvest

✓ import { harvest } from 'tweet-harvest';

✗ const { harvest } = require('tweet-harvest');

Primary programmatic function to initiate the tweet scraping process. Designed for ESM usage. CommonJS `require` is generally not supported without transpilation.

Options

✓ import { type Options } from 'tweet-harvest';

TypeScript type definition for the configuration object passed to the `harvest` function. Essential for type-safe programmatic usage.

cleanTweet

✓ import { cleanTweet } from 'tweet-harvest';

✗ const cleanTweet = require('tweet-harvest').cleanTweet;

Utility function for processing and cleaning individual tweet data. Primarily designed for ESM.

Demonstrates programmatic usage of `tweet-harvest` to scrape tweets with specified keywords and date ranges, requiring a Twitter authentication token.

import { harvest } from 'tweet-harvest'; import type { Options } from 'tweet-harvest'; const twitterAuthToken = process.env.TWITTER_AUTH_TOKEN ?? ''; // Get this from your browser cookies if (!twitterAuthToken) { console.error('TWITTER_AUTH_TOKEN environment variable is not set. Please provide a valid Twitter auth token from your browser cookies.'); process.exit(1); } const options: Options = { keyword: 'AI ethics', from: '2023-01-01', to: '2023-12-31', filename: 'ai-ethics-tweets', limit: 100, // Limit to 100 tweets for this example exportFormat: 'csv', auth_token: twitterAuthToken, withReplies: false, withImages: false, withVideos: false }; async function runHarvest() { console.log('Starting tweet harvest...'); try { await harvest(options); console.log(`Successfully harvested tweets to ${options.filename}.csv`); } catch (error) { console.error('Error during tweet harvest:', error); if (error instanceof Error && error.message.includes('auth_token')) { console.error('Ensure your TWITTER_AUTH_TOKEN is valid and up-to-date.'); } } } runHarvest();

tweet-harvest --version

Debug

Known issues

breakingThe short option for the `--to` flag (`-t`) was removed due to ambiguity with other short options. Users relying on `-t` for the 'to date' will need to update their scripts.

fix

Use the full `--to` flag instead of the `-t` short option, or consult CLI help for current short options.

affects: >=2.7.1

gotchaTweet Harvest requires a valid Twitter `auth_token` cookie for authentication. This token can expire or become invalid, leading to failed scrapes. Twitter actively prohibits unauthenticated search, making this token essential.

fix

Regularly obtain an updated `auth_token` by logging into Twitter in your browser and extracting the `auth_token` cookie. Pass it via the `--auth-token` CLI option or `auth_token` option in programmatic usage.

affects: >=2.0.0

breakingThe default CSV delimiter was changed from `;` to `,`. This will affect any scripts or tools parsing the output that expected the semicolon delimiter.

fix

Update CSV parsing logic to expect comma-separated values, or check for an option to explicitly set the delimiter if one is introduced in later versions.

affects: >=2.4.2

gotchaChanges in Twitter's cookie domains or internal structure can cause authentication or scraping failures. Version 2.6.1 specifically addressed a fix for 'cookie domain changes'.

fix

Ensure you are using the latest version of `tweet-harvest` to benefit from upstream fixes for Twitter's evolving platform. If issues persist, your `auth_token` might be outdated or Twitter may have implemented new anti-scraping measures.

affects: >=2.6.1

breakingConsistency of CSV headers order was fixed, and support for Gephi format was added. Existing scripts that rely on a specific, potentially inconsistent, header order might break or yield incorrect data.

fix

Review CSV output and update any downstream processing scripts to be resilient to changes in header order or to specifically match headers by name rather than by index.

affects: >=2.5.3

Errors

Common errors & fixes

Error: Playwright browser has not been installed.

The Playwright browser binaries (e.g., Chromium) required by `tweet-harvest` have not been downloaded.

fix

Run `npx playwright install` in your project directory to download the necessary browser binaries for Playwright.

Error: Auth token is not valid. Please make sure you enter a valid Twitter auth token.

The provided Twitter `auth_token` is either expired, invalid, or incorrectly formatted, preventing successful authentication with Twitter.

fix

Log into Twitter in a browser, extract a fresh `auth_token` cookie, and update your configuration or environment variable. Ensure there are no leading/trailing spaces or other characters.

Error: Cannot read properties of undefined (reading 'page')

This often indicates that Playwright failed to launch the browser or navigate to the Twitter page, potentially due to network issues, an unsupported environment, or conflicting browser processes.

fix

Check your internet connection, ensure no other processes are interfering with Playwright, and try running in a headful mode (if available via options) to debug browser launch issues. Ensure your Node.js version is compatible with Playwright.

CSV output does not match expected format / Missing columns.

Breaking changes in CSV header order (v2.5.3), delimiter (v2.4.2), or added fields can alter the structure of the output CSV.

fix

Check the `tweet-harvest` changelog for recent versions to identify changes in export format. Adjust your CSV parsing logic to account for new delimiters, header order, or additional columns.

Upgrade

Version history

2.7.1latest on npm

Audit

Dependencies

playwrightrequiredCore dependency for browser automation to scrape Twitter. Requires `playwright install` to download browser binaries.

Agent activity

19 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/helmisatria/tweet-harvest ↗

packagehttps://www.npmjs.com/package/tweet-harvest ↗