Run batch jobs

​

Enable batch

​

Batch endpoints

• OpenAI: /v1/files, /v1/batches, and /v1/batches/{id}, including cancel and long-poll with ?wait=true.
• Anthropic: /anthropic/v1/messages/batches and its retrieve, results, cancel, and delete routes.

​

How the gateway runs a batch

​

Submit a batch

from openai import OpenAI

client = OpenAI(base_url="https://<your-domino-host>/apps/<gateway-app-id>/v1", api_key=token)
f = client.files.create(file=open("prompts.jsonl", "rb"), purpose="batch")
batch = client.batches.create(input_file_id=f.id, endpoint="/v1/chat/completions",
                              completion_window="24h")

b = client.batches.retrieve(batch.id)
if b.output_file_id:
    print(client.files.content(b.output_file_id).text)  # successful lines
if b.error_file_id:
    print(client.files.content(b.error_file_id).text)   # failed lines, by custom_id

​

Limits

• Streaming and fallback chains don’t apply inside a batch.
• One bring your own token (BYOT) key applies to the whole batch.
• Bedrock and Vertex batches don’t get the AWS or Google discount.
• The gateway keeps input files for 7 days and output files for 30 days by default.

​

Related

• LLM Gateway 2.0: get a token and point the SDK at the gateway.
• Configure guardrails and fallback routing: the rules applied to each batch line.