Observability for Bedrock with Splunk

Step 1

Install the Traceloop SDK and initialize it. It will automatically log all calls to Bedrock using boto3 - both prompts and completions.

import boto3
import json

from traceloop.sdk import Traceloop

Traceloop.init(app_name="joke_generation_service")
brt = boto3.client(service_name='bedrock-runtime')

body = json.dumps({
    "prompt": "Tell me a joke about opentelemetry",
    "max_tokens": 200,
    "temperature": 0.5,
    "p": 0.5,
})

response = brt.invoke_model(
    body=body,
    modelId='cohere.command-text-v14',
    accept='application/json',
    contentType='application/json'
)

response_body = json.loads(response.get('body').read())

response = response_body.get('generations')[0].get('text')
print(response)

import * as traceloop from "@traceloop/node-server-sdk";
import {
  BedrockRuntimeClient,
  InvokeModelCommand,
} from "@aws-sdk/client-bedrock-runtime";

traceloop.initialize({
  appName: "sample_bedrock_ai21",
  apiKey: process.env.TRACELOOP_API_KEY,
  disableBatch: true,
});

// Create a BedrockRuntimeClient with your configuration
const client = new BedrockRuntimeClient();

const prompt = "How do you say Good Morning in French?";

const input = {
  modelId: "ai21.j2-ultra-v1",
  contentType: "application/json",
  accept: "application/json",
  body: JSON.stringify({
    prompt,
    maxTokens: 200,
    temperature: 0.7,
    topP: 1,
    stopSequences: [],
    presencePenalty: { scale: 0 },
    frequencyPenalty: { scale: 0 },
  }),
};

async function generateTextContent() {
  return await traceloop.withWorkflow("sample_completion", {}, async () => {
    // Create an InvokeModelCommand with the input parameters
    const command = new InvokeModelCommand(input);

    // Send the command to invoke the model and await the response
    client.send(command).then((response) => {
      // Save the raw response
      const rawRes = response.body;

      // Convert it to a JSON String
      const jsonString = new TextDecoder().decode(rawRes);

      // Parse the JSON string
      const parsedResponse = JSON.parse(jsonString);

      console.log(">>> non-stream", parsedResponse);
    });
  });
}

generateTextContent();

Step 2

Configure your collector to collect traces and send them to Splunk Observability Cloud; point OpenLLMetry to send traces to your Splunk collector

# In your Splunk collector configuration
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"
exporters:
  # Traces
  sapm:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    endpoint: "https://ingest.${SPLUNK_REALM}.signalfx.com/v2/trace"
    sending_queue:
      num_consumers: 32
 pipelines:
  traces:
    receivers: [jaeger, otlp, sapm, zipkin]
    processors:
    - memory_limiter
    - batch
    #- resource/add_environment
    exporters: [sapm]
        
# In the environment of your application
TRACELOOP_BASE_URL=http://splunk-otel-collector:4318

Docs

Observability for Bedrock with Splunk

Step 1

Step 2

Discover use cases

Trace prompts and completions in Bedrock

More things you can connect to {platform}

LlamaIndex

OpenAI

Watsonx

together.ai

Anthropic

Chroma

Mistral AI

Cohere

Gemini

LangChain

Ollama