Observability for Watsonx with Splunk

Step 1

Install the Traceloop SDK and initialize it. It will automatically log all calls to Watsonx, with prompts and completions as separate spans.

from ibm_watson_machine_learning.foundation_models import Model

from traceloop.sdk import Traceloop

Traceloop.init(app_name="watsonx_example")


def get_credentials(api_key):
    return {
        "url": "https://us-south.ml.cloud.ibm.com",
        "apikey": api_key,
    }


iam_api_key = os.environ["IAM_API_KEY"]
project_id = os.environ["PROJECT_ID"]

prompt_input = """Calculate result

Input:
what is the capital of China.

Output:
"""

model_id = "meta-llama/llama-2-70b-chat"
parameters = {
    "decoding_method": "sample",
    "max_new_tokens": 60,
    "min_new_tokens": 10,
    "random_seed": 111,
    "temperature": 0.9,
    "top_k": 50,
    "top_p": 1,
    "repetition_penalty": 2
}

model = Model(
    model_id=model_id,
    params=parameters,
    credentials=get_credentials(iam_api_key),
    project_id=project_id
    )

prompt_input = "What is the the captical of China"
generated_response = model.generate(prompt=prompt_input)

Step 2

Configure your collector to collect traces and send them to Splunk Observability Cloud; point OpenLLMetry to send traces to your Splunk collector

# In your Splunk collector configuration
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"
exporters:
  # Traces
  sapm:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    endpoint: "https://ingest.${SPLUNK_REALM}.signalfx.com/v2/trace"
    sending_queue:
      num_consumers: 32
 pipelines:
  traces:
    receivers: [jaeger, otlp, sapm, zipkin]
    processors:
    - memory_limiter
    - batch
    #- resource/add_environment
    exporters: [sapm]
        
# In the environment of your application
TRACELOOP_BASE_URL=http://splunk-otel-collector:4318

Docs

Observability for Watsonx with Splunk

Step 1

Step 2

Discover use cases

Trace prompts and completions in Watsonx

Trace your RAG retrieval pipeline

More things you can connect to {platform}

Bedrock

LlamaIndex

OpenAI

together.ai

Anthropic

Chroma

Mistral AI

Cohere

Gemini

LangChain

Ollama