LangSmith to RAGAS: Converting an Exported Dataset

LangSmith nests every example under inputs and outputs; RAGAS wants named fields on a flat record. Here is the mapping, and what it drops.

Paste your dataset JSON
Target:
Converted output


        

Field changes

  • —

Validation warnings

  • —

What you can evaluate

  • —

Short answer

LangSmith stores each example as a record with an inputs object and an outputs object, both free-form. RAGAS SingleTurnSample wants named fields on a flat record: user_input, retrieved_contexts, response and reference. Converting lifts inputs.question to user_input, and outputs.contexts, outputs.answer, outputs.reference to retrieved_contexts, response and reference.

What a LangSmith export looks like

A dataset export is an array of example records. LangSmith does not validate the inner shape of inputs or outputs, so the key names are whatever your pipeline logged:

[
  {
    "id": "0b7f6a2e-1c4d-4c1a-9f0e-2b6d5a8c3e11",
    "inputs": { "question": "What is RAG?" },
    "outputs": {
      "contexts": ["RAG combines retrieval with generation."],
      "answer": "RAG retrieves documents, then generates an answer.",
      "reference": "Retrieval-augmented generation retrieves documents and uses them to generate answers."
    },
    "created_at": "2026-01-14T09:12:03Z"
  }
]

The field mapping

LangSmithRAGAS modernRAGAS currentRequired
inputs.questionuser_inputquestionYes
outputs.contextsretrieved_contextscontextsFor retrieval metrics
outputs.answerresponseanswerFor answer metrics
outputs.referencereferencereferenceFor reference-based metrics

Two structural changes happen along the way: the inputs / outputs nesting is flattened, and the records are regrouped into columnar parallel arrays. RAGAS reads columns, not rows — one array entry per sample, all arrays the same length.

What gets dropped

RAGAS has no equivalent for LangSmith's record identity or run metadata. Converting drops id, created_at and any other top-level key, plus any outputs key that is not answer, contexts or reference. This converter lists each one as an unmapped field rather than removing it silently, so you can decide whether the loss matters before you run anything.

Converting without the tool

The manual route is short if your inner keys already match. Export, then build the samples directly:

from langsmith import Client
from ragas import EvaluationDataset, SingleTurnSample

client = Client()
examples = client.list_examples(dataset_name="my-eval-set")

samples = [
    SingleTurnSample(
        user_input=ex.inputs["question"],
        retrieved_contexts=ex.outputs["contexts"],
        response=ex.outputs["answer"],
        reference=ex.outputs.get("reference"),
    )
    for ex in examples
]

dataset = EvaluationDataset(samples=samples)

This breaks as soon as one example is missing a key, which is the usual reason to run the JSON through a converter first and read the validation report.

Why the question column can come out empty

This converter reads inputs.question. LangSmith does not enforce that name — a pipeline that logged {"query": ...} or {"input": ...} still exports JSON that looks valid, and the converted user_input column comes out empty for every record. Rename the key in the export, or rename the column after conversion. Empty questions are also flagged with their record index in the validation panel.

Why the reference field decides which metrics you get

If your LangSmith examples never logged outputs.reference, the converted dataset has no reference column, and the reference-based metrics cannot run: Context Recall, Context Precision, Answer Correctness, Factual Correctness and Semantic Similarity. Faithfulness, Response Relevancy and the rubric-based metrics still work. The full split is in RAGAS without ground truth, and the field-by-field reference for every format is in RAGAS dataset format.

Primary sources

Field names and metric definitions on this page follow the upstream projects:

Related