HdfsCLI provides a Python API and command-line interface for interacting with Hadoop HDFS via the WebHDFS (and HttpFS) API. It supports both secure and insecure clusters, offering Python 3 bindings for common HDFS operations. The library includes optional extensions for handling Avro files, Pandas DataFrames, and Kerberos authentication. The current version, 2.7.3, was released on October 12, 2023, indicating active maintenance.
pip install hdfsVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to establish a connection to an HDFS Namenode using `InsecureClient`, write a simple file, list directory contents, read the file back, and then delete it. It uses environment variables for the Namenode URL and user for flexibility. Ensure your HDFS cluster is running and accessible at the specified URL.
Upgrade your Python environment to 3.7 or newer. If you must use Python 2, you'll need to use an older version of the `hdfs` library (e.g., `hdfs<2.0.0`), but this is not recommended due to lack of maintenance.
When calling `client.write()`, include `overwrite=True` in the arguments if you intend to replace an existing file (e.g., `client.write(path, overwrite=True)`).
To delete a directory and its contents, use `client.delete(path, recursive=True)`. Consider `skip_trash=False` (requires Hadoop 2.9+) if you want files to go to trash instead of being permanently deleted.
Ensure you have a `~/.hdfscli.cfg` file (or `HDFSCLI_CONFIG` environment variable pointing to one) with valid alias definitions, including `url` and optional `user` or `client` (e.g., `KerberosClient`).
Install the kerberos extension (`pip install hdfs[kerberos]`). Ensure your `krb5.conf` is correctly configured and you have a valid Kerberos ticket. Refer to the HdfsCLI documentation for detailed Kerberos setup instructions.
Verify the HDFS Namenode host and port (e.g., from `core-site.xml`'s `fs.default.name` property) and ensure they are correctly specified when initializing the `hdfs.InsecureClient` or `hdfs.Client`. Ensure the HDFS service is running and accessible from the client machine.
```python
from hdfs import InsecureClient
# Replace 'your_hdfs_namenode_host' and 'your_hdfs_port' with actual values
# Example: 'http://localhost:9870' or 'http://your_namenode:50070'
client = InsecureClient('http://your_hdfs_namenode_host:your_hdfs_port', user='your_hdfs_user')
# Or, for a secure cluster with Kerberos (requires hdfs[kerberos] installed and kinit):
# from hdfs.ext.kerberos import KerberosClient
# client = KerberosClient('http://your_hdfs_namenode_host:your_hdfs_port', user='your_hdfs_user')
```Check that the HDFS Namenode and DataNode services are running, the host and port are correct and reachable from your client machine, and no firewall is blocking the WebHDFS port (typically 50070 or 9870 for the Namenode UI/WebHDFS endpoint, or 50075 for DataNode). Use `curl` to test connectivity to the WebHDFS endpoint from your client machine. ```bash # Example: Test WebHDFS API endpoint curl -i 'http://your_hdfs_namenode_host:your_hdfs_port/webhdfs/v1/?op=GETHOMEDIRECTORY' ``` If `curl` also fails, the issue is with network connectivity or the HDFS cluster setup. If `curl` succeeds, verify the `hdfs` client initialization parameters.
Ensure you are using an up-to-date version of the `hdfs` library (version 2.7.3 or newer is recommended) and use the correct import pattern for `InsecureClient` or `Client`. If upgrading, remove the old version first.
```bash
pip uninstall hdfs
pip install hdfs
```
Then, use the standard import:
```python
from hdfs import InsecureClient
# client = InsecureClient('http://namenode_host:port', user='your_user')
# For Kerberos:
# from hdfs.ext.kerberos import KerberosClient
# client = KerberosClient('http://namenode_host:port', user='your_user')
```For Kerberized clusters, ensure you have obtained a valid Kerberos ticket before running your Python application. This typically involves using the `kinit` command. Additionally, ensure the `hdfs[kerberos]` extra is installed if using Kerberos.
```bash
kinit your_user@YOUR.REALM
# Then, in your Python code:
pip install 'hdfs[kerberos]'
from hdfs.ext.kerberos import KerberosClient
client = KerberosClient('http://your_hdfs_namenode_host:your_hdfs_port')
# The 'user' parameter is often not needed with KerberosClient as it's derived from the kinit ticket.
```