이 페이지는 아직 한국어로 제공되지 않습니다. 번역 작업 중입니다. 현재 번역 프로젝트에 대한 질문이나 피드백이 있으신 경우 언제든지 연락주시기 바랍니다.
Overview
In the context of LLM applications, it’s important to track user feedback and evaluate the quality of your LLM application’s responses. While LLM Observability provides a few out-of-the-box evaluations for your traces, you can submit your own evaluations to LLM Observability in two ways: with Datadog’s SDK, or with the LLM Observability API. See Naming custom metrics for guidelines on how to choose an appropriate label for your evaluations.
Evaluation labels must be unique for a given LLM application (ml_app) and organization.
Submitting external evaluations with the SDK
The LLM Observability SDK provides the methods LLMObs.submit_evaluation() and LLMObs.export_span() to help your traced LLM application submit external evaluations to LLM Observability. See the Python or Node.js SDK documentation for more details.
Example
fromddtrace.llmobsimportLLMObsfromddtrace.llmobs.decoratorsimportllmdefmy_harmfulness_eval(input:Any)->float:score=...# custom harmfulness evaluation logicreturnscore@llm(model_name="claude",name="invoke_llm",model_provider="anthropic")defllm_call():completion=...# user application logic to invoke LLM# joining an evaluation to a span via span ID and trace IDspan_context=LLMObs.export_span(span=None)LLMObs.submit_evaluation(span=span_context,ml_app="chatbot",label="harmfulness",metric_type="score",# can be score or categoricalvalue=my_harmfulness_eval(completion),tags={"reasoning":"it makes sense","type":"custom"},)
Submitting external evaluations with the API
You can use the evaluations API provided by LLM Observability to send evaluations associated with spans to Datadog. See the Evaluations API for more details on the API specifications.
