Observability for together.ai with New Relic

OpenLLMetry lets you trace any model running on together.ai platform. With 5 minutes of work you can get complete view of your system directly into New Relic. See how below.

Step 1

Install the Traceloop SDK and initialize it. It will automatically log all calls to models served from together.ai, with prompts and completions as separate spans.

from together import Together
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow

Traceloop.init(app_name="joke_generation_service")

client = Together(api_key=os.environ.get("TOGETHER_API_KEY"))

@workflow(name="joke_creation")
def create_joke():
    response = client.chat.completions.create(
        model="mistralai/Mixtral-8x7B-Instruct-v0.1",
        messages=[{"role": "user", "content": "Tell me a joke about OpenTelemetry."}],
    )

    return response.choices[0].message.content

Step 2

Get an access key from New Relic dashboard and set the following environment variables.

TRACELOOP_BASE_URL=https://otlp.nr-data.net:443
TRACELOOP_HEADERS="api-key=<YOUR_NEWRELIC_LICENSE_KEY>"

Docs

Observability for together.ai with New Relic

Step 1

Step 2

Discover use cases

More things you can connect to {platform}

LlamaIndex

LangChain

Watsonx

Ollama

Bedrock

Anthropic

OpenAI

Mistral AI

Cohere

Gemini

Chroma