Registry /
llm-agents / llama-index-multi-modal-llms-openai
Install & Compatibility
Where this runs
tested against v0.6.2 · pip install
no network on importno background threads
Install × environment matrix
Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux
muslpy 3.10–3.95 runs
installs and imports cleanly · install 0.0s · import 6.978s · 258.8MB
glibcpy 3.10–3.95 runs
installs and imports cleanly · install 20.2s · import 6.386s · 255MB
270MB installed
● package 270MB
Code
Verified usage
Verified import paths — ran on the pinned version, not inferred.
OpenAIMultiModal
✓ from llama_index.multi_modal_llms.openai import OpenAIMultiModal
✗ from llama_index.llms.openai import OpenAI
While 'OpenAIMultiModal' is the current class in this package, LlamaIndex is consolidating multi-modal support directly into the 'OpenAI' LLM class within 'llama_index.llms.openai' using ChatMessage content blocks for newer versions of `llama-index-llms-openai`.
load_image_urls
✓ from llama_index.core.multi_modal_llms.generic_utils import load_image_urls
Utility for loading images from URLs into ImageDocument objects.
SimpleDirectoryReader
✓ from llama_index.core import SimpleDirectoryReader
Used for loading documents, including images from local directories.
This quickstart demonstrates how to initialize the `OpenAIMultiModal` class with an OpenAI vision model (e.g., `gpt-4-vision-preview` or `gpt-4o`), load image documents from URLs, and then use the LLM to get a descriptive response based on both a text prompt and the provided images. Ensure `OPENAI_API_KEY` is set in your environment.
import os
from llama_index.multi_modal_llms.openai import OpenAIMultiModal
from llama_index.core.multi_modal_llms.generic_utils import load_image_urls
# Ensure your OpenAI API key is set as an environment variable
os.environ["OPENAI_API_KEY"] = os.environ.get("OPENAI_API_KEY", "")
# Example image URL (replace with a real URL or local path for actual use)
# For a runnable example, you'd need a publicly accessible image URL or a local image file.
# For demonstration purposes, we'll use a placeholder URL and note its purpose.
image_urls = [
"https://docs.llamaindex.ai/en/stable/_static/assets/img/llama-index-logo.png"
]
# Load image documents from URLs
image_documents = load_image_urls(image_urls)
# Initialize the OpenAI Multi-Modal LLM
openai_mm_llm = OpenAIMultiModal(
model="gpt-4-vision-preview", # Or "gpt-4o"
api_key=os.environ["OPENAI_API_KEY"],
max_new_tokens=300,
)
# Complete a prompt with image documents
response = openai_mm_llm.complete(
prompt="What is in the image? Describe it.",
image_documents=image_documents,
)
print(response.text)
Debug
Known issues
deprecatedThe `OpenAIMultiModal` class and its specific multi-modal LLM abstraction are being phased out in LlamaIndex. Future development is consolidating multi-modal support directly into the unified `OpenAI` LLM class (from `llama_index.llms.openai`) by using `ChatMessage` objects with `ImageBlock` and `TextBlock` content.fixFor new implementations, consider using `from llama_index.llms.openai import OpenAI` and construct multimodal prompts using `ChatMessage` with `ImageBlock` and `TextBlock` content instead.
affects: >=0.6.2 (future versions of llama-index-llms-openai will replace this functionality)
gotchaAn OpenAI API key is required and must be set as an environment variable (`OPENAI_API_KEY`) or passed explicitly to the `OpenAIMultiModal` constructor for authentication with OpenAI services.fixSet `export OPENAI_API_KEY='your-key'` in your terminal or `os.environ["OPENAI_API_KEY"] = "your-key"` in your code before initialization.
affects: All
gotchaOnly specific OpenAI models (e.g., `gpt-4-vision-preview`, `gpt-4o`) support multi-modal (text + image) inputs. Using a text-only model with `OpenAIMultiModal` or multimodal prompts will result in an error or unexpected behavior.fixEnsure you specify a multi-modal capable model like `model="gpt-4-vision-preview"` or `model="gpt-4o"` when initializing `OpenAIMultiModal`.
affects: All
Errors
Common errors & fixes
ModuleNotFoundError: No module named 'llama_index.multi_modal_llms'
The import path for `OpenAIMultiModal` and other multi-modal LLM components has changed in newer versions of LlamaIndex, moving from a top-level `llama_index.multi_modal_llms` to `llama_index.llms.openai` or within `llama_index.core`.
fixUpdate your import statement from `from llama_index.multi_modal_llms.openai import OpenAIMultiModal` to `from llama_index.llms.openai import OpenAI` for general OpenAI LLM usage, or `from llama_index.core.multi_modal_llms.openai import OpenAIMultiModal` if you are using an older structure that still explicitly uses `OpenAIMultiModal` but needs the core path.
openai.error.InvalidRequestError: The API deployment for this resource does not exist. If you created the deployment within the last 5 minutes, please wait a moment and try again.
This error typically occurs when using Azure OpenAI and the specified deployment name for the model (e.g., `text-davinci-003`, `gpt-4o`) does not match an actual deployment in your Azure OpenAI service, or the API key is incorrect/missing.
fixEnsure that your `OPENAI_API_KEY`, `OPENAI_API_BASE` (for Azure), `OPENAI_API_VERSION`, and the `model` or `deployment_name` parameters passed to `OpenAIMultiModal` correctly match your Azure OpenAI deployment settings and that the API key is valid. Verify that the specified model has been deployed in your Azure resource. For regular OpenAI, ensure `OPENAI_API_KEY` is set and the `model` parameter is a valid OpenAI model name.
openai.error.InvalidRequestError: No API key found for OpenAI. Please set the OPENAI_API_KEY environment variable or pass `api_key` to the `OpenAI` constructor.
The LlamaIndex OpenAI integration requires an OpenAI API key to authenticate requests, and this error indicates that the API key was not found either in the environment variables or provided directly to the LLM constructor.
fixSet your OpenAI API key as an environment variable named `OPENAI_API_KEY` (e.g., `os.environ["OPENAI_API_KEY"] = "sk-..."`) before initializing `OpenAIMultiModal`, or pass it directly using the `api_key` parameter: `OpenAIMultiModal(model="gpt-4o", api_key="your_openai_api_key")`.
openai.error.InvalidRequestError: This model's maximum context length is X tokens. However, your messages resulted in Y tokens. Please reduce the length of the messages.
The total length of the input (prompt, chat history, and image data) provided to the OpenAI multi-modal model exceeds the maximum token limit allowed for that specific model.
fixReduce the size of your input. This might involve shortening the text prompt, summarizing chat history, or optimizing the image data (e.g., using lower `image_detail` if supported, or fewer images). Check the model's documentation for its specific context length limits.
AttributeError: 'OpenAIMultiModal' object has no attribute '_update_tool_calls'
This `AttributeError` often occurs when attempting to use advanced features like function/tool calling with `OpenAIMultiModal` where the internal implementation for handling tool calls has changed or is not supported in the specific version or configuration being used.
fixEnsure your `llama-index` and `llama-index-multi-modal-llms-openai` packages are up-to-date. If the issue persists, review the LlamaIndex documentation for the correct way to implement tool calling with multi-modal models, as the API for handling tools might have been refactored or moved to a different class/method.
Upgrade
Version history
0.6.2latest on PyPI · released Sep 25, 2025
Audit
Dependencies
llama-index-corerequiredCore LlamaIndex framework components are required for integration.
openairequiredProvides the underlying OpenAI API client for model interaction.