Observability for Bedrock with Datadog

Step 1

Install the Traceloop SDK and initialize it. It will automatically log all calls to Bedrock using boto3 - both prompts and completions.

import boto3
import json

from traceloop.sdk import Traceloop

Traceloop.init(app_name="joke_generation_service")
brt = boto3.client(service_name='bedrock-runtime')

body = json.dumps({
    "prompt": "Tell me a joke about opentelemetry",
    "max_tokens": 200,
    "temperature": 0.5,
    "p": 0.5,
})

response = brt.invoke_model(
    body=body,
    modelId='cohere.command-text-v14',
    accept='application/json',
    contentType='application/json'
)

response_body = json.loads(response.get('body').read())

response = response_body.get('generations')[0].get('text')
print(response)

import * as traceloop from "@traceloop/node-server-sdk";
import {
  BedrockRuntimeClient,
  InvokeModelCommand,
} from "@aws-sdk/client-bedrock-runtime";

traceloop.initialize({
  appName: "sample_bedrock_ai21",
  apiKey: process.env.TRACELOOP_API_KEY,
  disableBatch: true,
});

// Create a BedrockRuntimeClient with your configuration
const client = new BedrockRuntimeClient();

const prompt = "How do you say Good Morning in French?";

const input = {
  modelId: "ai21.j2-ultra-v1",
  contentType: "application/json",
  accept: "application/json",
  body: JSON.stringify({
    prompt,
    maxTokens: 200,
    temperature: 0.7,
    topP: 1,
    stopSequences: [],
    presencePenalty: { scale: 0 },
    frequencyPenalty: { scale: 0 },
  }),
};

async function generateTextContent() {
  return await traceloop.withWorkflow("sample_completion", {}, async () => {
    // Create an InvokeModelCommand with the input parameters
    const command = new InvokeModelCommand(input);

    // Send the command to invoke the model and await the response
    client.send(command).then((response) => {
      // Save the raw response
      const rawRes = response.body;

      // Convert it to a JSON String
      const jsonString = new TextDecoder().decode(rawRes);

      // Parse the JSON string
      const parsedResponse = JSON.parse(jsonString);

      console.log(">>> non-stream", parsedResponse);
    });
  });
}

generateTextContent();

Step 2

Configure your Datadog agent to receive traces from OpenLLMetry

# In your values.yaml
otlp:
  receiver:
    protocols:
      http:
        enabled: true
        
# In the environment of your application
TRACELOOP_BASE_URL=http://datadog-agent-hostname:4318

Docs

Observability for Bedrock with Datadog

Step 1

Step 2

Discover use cases

Trace prompts and completions in Bedrock

More things you can connect to {platform}

Mistral AI

Gemini

Chroma

Anthropic

together.ai

Ollama

LlamaIndex

OpenAI

Watsonx

Cohere

LangChain