Create a new custom evaluator at the organization level without project or environment bindings. The evaluator uses LLM-as-a-judge to evaluate inputs based on the provided configuration. The response format is automatically generated from the output schema.
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Custom evaluator creation input
Schemas
LLM Message configuration
LLM parameters (from BaseLLMConfig)
Basic fields
Provider configuration
OK
AGENT_GOAL_ACCURACY, CHAR_COUNT, CHAR_COUNT_RATIO, ANSWER_COMPLETENESS, FAITHFULNESS, JSON_VALIDATOR, PERPLEXITY, PII_DETECTOR, PLACEHOLDER_REGEX, PROFANITY_DETECTOR, SEXISM_DETECTOR, PROMPT_INJECTION, TONE_DETECTION, PROMPT_PERPLEXITY, TOXICITY_DETECTOR, ANSWER_RELEVANCY, ANSWER_CORRECTNESS, REGEX_VALIDATOR, SECRETS_DETECTOR, SQL_VALIDATOR, SEMANTIC_SIMILARITY, TOPIC_ADHERENCE, UNCERTAINTY_DETECTOR, WORD_COUNT, WORD_COUNT_RATIO, INSTRUCTION_ADHERENCE, AGENT_FLOW_QUALITY, AGENT_EFFICIENCY, AGENT_GOAL_COMPLETENESS, CONVERSATION_QUALITY, INTENT_CHANGE, CONTEXT_RELEVANCE, AGENT_TOOL_ERROR_DETECTOR, AGENT_TOOL_TRAJECTORY, HTML_COMPARISON, LLM_AS_A_JUDGE