Skip to main content

​
POST /v1/eval

Run an evaluation by providing a saved evaluation spec and a dataset of input/output pairs.

​
Headers

  • Authorization: Api-Key: API key for authentication

​
Request body

  • eval_spec_id (string): Identifier of the evaluation spec to use
  • dataset (array): List of objects with:
    • input (string): Prompt or expected input
    • output (string): Model output to grade
  • model (string, optional): Override the default model used for evaluation

​
Example request

curl -X POST https://api.example.com/v1/eval \
  -H "Content-Type: application/json" \
  -H "Authorization: Api-Key YOUR_API_KEY" \
  -d '{
        "eval_spec_id": "00000000-0000-0000-0000-000000000000",
        "dataset": [
          { "input": "What is 2+2?", "output": "4" }
        ],
        "model": "gpt-4o-mini"
      }'

​
Example response

{
  "results": [
    {
      "earned_points": 1,
      "possible_points": 1,
      "percentage": 1,
      "report": {}
    }
  ],
  "summary": {
    "total_earned_points": 1,
    "total_possible_points": 1,
    "overall_percentage": 1
  }
}

​
Status codes

  • 200 OK – Evaluation completed
  • 400 Bad Request – Missing or empty dataset
  • 403 Forbidden – API key not authorized for the evaluation spec
  • 404 Not Found – Evaluation spec or version not found

​
GET /v1/eval//documents

Retrieve the processed documents that belong to a dataset, including presigned URLs for downloading the original files.

​
Path parameters

  • dataset_id (string): Identifier of the dataset whose documents you want to download

​
Headers

  • Authorization: Api-Key: API key for authentication

​
Example request

curl -X GET "https://api.example.com/v1/eval/00000000-0000-0000-0000-000000000000/documents" \
  -H "Authorization: Api-Key YOUR_API_KEY"

​
Example response

{
  "dataset_id": "00000000-0000-0000-0000-000000000000",
  "documents": [
    {
      "filename": "policies.pdf",
      "mime": "application/pdf",
      "size_bytes": 283719,
      "extracted_md": "# Company Policies\n...",
      "download_url": "https://signed-s3-url"
    }
  ]
}

​
Status codes

  • 200 OK – Documents were found and returned successfully
  • 403 Forbidden – API key is not authorized for the dataset’s organization
  • 404 Not Found – Dataset does not exist or it has no documents
⌘I