When building LLM applications, it quickly becomes highly useful and important to track user feedback on the result of your LLM workflow.

Doing that with OpenLLMetry is easy. First, make sure you associate your LLM workflow with unique identifiers.

Then, create an Annotation Task within Traceloop to collect user feedback as annotations. The annotation task should include:

  • The entity you want to collect feedback for (e.g., chat_id)
  • Tags you want to track (e.g., score, feedback_text)

You can log user feedback by calling our Python SDK or TypeScript SDK. All feedback must follow the structure defined in your annotation task.

For example, to implement thumbs-up/thumbs-down feedback, create an annotation task with a tag named is_helpful that accepts the values thumbs-up and thumbs-down.

The entity you report feedback for must match the one defined in your annotation task and association property.

Was this page helpful?

Suggest edits
Associating Entities with TracesManually reporting calls to LLMs and Vector DBs