Registry / ai-ml / sagemaker-huggingface-inference-toolkit

sagemaker-huggingface-inference-toolkit

JSON →
library2.6.0pypypiunverified

Open source library for serving Hugging Face models on Amazon SageMaker. It wraps the Hugging Face Transformers pipeline and provides built-in handlers for inference. Latest version 2.6.0 supports PyTorch 2.0+ and TensorFlow 2.x. Released quarterly.

ai-mldevops
pip install sagemaker-huggingface-inference-toolkit
github ↗homepage ↗
INSTALL
IMPORT
SIG · SAGEMAKER-HUGGINGF
S
sagemaker-huggingface-inference-toolkit
ai-mlpythonv2.6.0
harness data pending
Install & Compatibility
Where this runs

No compatibility data collected yet for this library.

Code
Verified usage

Verified import paths — ran on the pinned version, not inferred.

HuggingFaceModel
✓ from sagemaker_huggingface_inference_toolkit import HuggingFaceModel
✗ from sagemaker.huggingface import HuggingFaceModel

Deploy a Hugging Face model on SageMaker using the inference toolkit.

from sagemaker.huggingface import HuggingFaceModel import os hub = { 'HF_MODEL_ID':'distilbert-base-uncased-distilled-squad', 'HF_TASK':'question-answering' } huggingface_model = HuggingFaceModel( env=hub, role='SageMakerRole', transformers_version='4.26', pytorch_version='1.13', py_version='py39', ) predictor = huggingface_model.deploy( initial_instance_count=1, instance_type='ml.m5.large' ) data = { 'question': 'What is the capital of France?', 'context': 'France is a country in Europe. Its capital is Paris.' } response = predictor.predict(data) print(response)
Debug
Known issues
breakingIn version 2.0+, the default SageMaker SDK integration changed. Use `HuggingFaceModel` from `sagemaker.huggingface` instead of `sagemaker.model.Model`.
fix
Update imports to `from sagemaker.huggingface import HuggingFaceModel`.
affects: >=2.0
gotchaThe `HF_TASK` environment variable must match a valid Hugging Face pipeline task. Common mistakes: using 'text-classification' instead of 'text-classification' (correct) or misspelling.
fix
Refer to Hugging Face pipeline tasks: https://huggingface.co/docs/transformers/main_classes/pipelines
affects: all
gotchaWhen using a custom inference script, the entry point must be named `inference.py` and placed in the same directory as the model artifacts.
fix
Ensure the inference script is named `inference.py` and included in the model tar.gz file.
affects: all
Upgrade
Version history
2.6.0latest on PyPI · released Apr 23, 2025
Audit
Dependencies
transformersrequiredCore dependency for model loading and inference.
torchoptionalRequired for PyTorch models.
tensorflowoptionalRequired for TensorFlow models.
sagemakerrequiredFor deployment and endpoint interaction.
Agent activity
15 hits · last 30 days
node
12
OpenAI (training)
1
Resources
githubhttps://github.com/aws/sagemaker-huggingface-inference-toolkit ↗
homepagehttps://github.com/aws/sagemaker-huggingface-inference-toolkit ↗
sagemaker-huggingface-inference-toolkit — pip install sagemaker-huggingface-inference-toolkit · libregistry