Using Downsizing with LlamaIndex

Prerequisites

Before configuring LlamaIndex, make sure your Downsizing account can make requests — pay-per-use or a connected subscription both work. LlamaIndex ships its LLM integrations as separate packages; install the one for the API you want to use:

pip install llama-index-llms-anthropic

pip install llama-index-llms-openai-like

Option A: Anthropic-compatible endpoint (recommended)

LlamaIndex's Anthropic class accepts a base_url parameter, so it can be pointed at Downsizing's Anthropic-compatible endpoint directly:

from llama_index.llms.anthropic import Anthropic
from llama_index.core import Settings

llm = Anthropic(
    model="claude-opus-4-8",
    base_url="http://localhost:3001/YOUR_INFERENCE_ID/anthropic",
    api_key="dws_YOUR_API_KEY",
)

# Use it globally...
Settings.llm = llm

# ...or pass it directly to a single component
# query_engine = index.as_query_engine(llm=llm)

Assign the resulting llm to Settings.llm to use it as the default across your application, or pass it directly to a specific index, query engine, or agent via its llm= argument.

Option B: OpenAI-compatible endpoint

For OpenAI-style models, use OpenAILike (from llama-index-llms-openai-like), which is built for pointing at third-party OpenAI-compatible APIs. Set api_baseto Downsizing's OpenAI-compatible endpoint and set is_chat_model=True:

from llama_index.llms.openai_like import OpenAILike
from llama_index.core import Settings

llm = OpenAILike(
    model="gpt-5",
    api_base="http://localhost:3001/YOUR_INFERENCE_ID/openai/v1",
    api_key="dws_YOUR_API_KEY",
    is_chat_model=True,
    context_window=128000,
)

Settings.llm = llm

Configuration

Find your Inference ID and API key on the Inference API page. Replace YOUR_INFERENCE_ID with your workspace UUID and dws_YOUR_API_KEY with your Downsizing API key in either snippet above.

Verify

Call the configured llm directly to confirm traffic is routing through Downsizing:

resp = llm.complete("Say hello in one short sentence.")
print(resp)

If you get a response back, Downsizing is set up correctly. You can also check your savings dashboard to see the request appear in your usage metrics.