Using Downsizing with Haystack

Prerequisites

Before configuring Haystack, make sure your Downsizing account can make requests — pay-per-use or a connected subscription both work. Haystack ships its core generators in haystack-ai; the Anthropic generator lives in a separate integration package:

pip install haystack-ai anthropic-haystack

pip install haystack-ai

Option A: Anthropic-compatible endpoint (recommended)

Haystack's AnthropicChatGenerator (from anthropic-haystack) doesn't expose a base_url constructor argument, but it builds its client from the standard Anthropic Python SDK, which reads the ANTHROPIC_BASE_URL environment variable whenever no explicit base URL is passed. Set that variable before constructing the generator to route requests through Downsizing:

import os
from haystack.utils import Secret
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.anthropic import AnthropicChatGenerator

# AnthropicChatGenerator doesn't expose a base_url parameter directly, but the
# underlying Anthropic SDK reads ANTHROPIC_BASE_URL when no base_url is passed.
os.environ["ANTHROPIC_BASE_URL"] = "http://localhost:3001/YOUR_INFERENCE_ID/anthropic"

generator = AnthropicChatGenerator(
    api_key=Secret.from_token("dws_YOUR_API_KEY"),
    model="claude-opus-4-8",
    generation_kwargs={"max_tokens": 4096},  # required by Anthropic models
)

Pass the resulting generator to any Haystack pipeline component that accepts a chat generator (e.g. a ChatPromptBuilder → generator pipeline, or an agent component).

Option B: OpenAI-compatible endpoint

For OpenAI-style models, Haystack's built-in OpenAIChatGenerator accepts an api_base_urlparameter directly, so it can be pointed at Downsizing's OpenAI-compatible endpoint without any environment variable workaround:

from haystack.utils import Secret
from haystack.dataclasses import ChatMessage
from haystack.components.generators.chat import OpenAIChatGenerator

generator = OpenAIChatGenerator(
    api_key=Secret.from_token("dws_YOUR_API_KEY"),
    model="gpt-5",
    api_base_url="http://localhost:3001/YOUR_INFERENCE_ID/openai/v1",
)

Configuration

Find your Inference ID and API key on the Inference API page. Replace YOUR_INFERENCE_ID with your workspace UUID and dws_YOUR_API_KEY with your Downsizing API key in either snippet above.

Verify

Call the configured generator directly to confirm traffic is routing through Downsizing:

result = generator.run(messages=[ChatMessage.from_user("Say hello in one short sentence.")])
print(result["replies"][0].text)

If you get a response back, Downsizing is set up correctly. You can also check your savings dashboard to see the request appear in your usage metrics.