Test Datasets

What are Test Datasets?#

Test datasets are collections of input-output pairs used to systematically evaluate prompt quality. They form the foundation of professional prompt testing in PromptReports, enabling reproducible evaluations, regression detection, and data-driven prompt optimization.

A well-designed test dataset provides:

Creating Datasets#

PromptReports offers multiple ways to create test datasets, each suited to different workflows:

Dataset Structure#

Each dataset consists of rows (test cases) with the following fields:

{
  "inputVars": {
    "customer_message": "I need to return this product, it's been 3 weeks",
    "customer_type": "premium",
    "product_category": "electronics",
    "order_id": "ORD-2024-98765"
  },
  "expectedOutput": "I understand you'd like to return your electronics item. As a premium customer, you have a 90-day return window, so you're still well within the return period. I'll process this return for order ORD-2024-98765 right away...",
  "category": "returns",
  "tags": ["premium", "within-policy", "electronics"],
  "difficulty": "easy",
  "priority": 3,
  "notes": "Standard return request, should reference return policy timeframe"
}

Importing Data#

Import datasets from external files for bulk test case creation. PromptReports supports multiple formats:

customer_message,customer_type,product_category,expected_output,category,difficulty
"I need a refund",standard,clothing,"We can process your refund within 3-5 business days...",refunds,easy
"When will my order arrive?",premium,electronics,"Let me check your order status. As a premium member...",shipping,easy
"This item is broken and I want to speak to a manager",standard,furniture,"I'm so sorry about the damage. I'll connect you with...",complaints,hard
"Can I change my delivery address?",standard,clothing,"Yes, I can update your delivery address...",modifications,medium

Column names should match your prompt variables. The system will automatically map them during import. You can also manually adjust mappings if column names don't exactly match your variable names.

AI-Generated Datasets#

PromptReports can automatically generate diverse test cases using AI. This is especially useful when you need to quickly build out test coverage or explore edge cases you might not have considered.

{
  "promptDescription": "Customer support chatbot for e-commerce returns and refunds",
  "categories": {
    "typical": 20,
    "edge_cases": 10,
    "adversarial": 5,
    "multi_language": 5
  },
  "constraints": [
    "Include various customer sentiment levels",
    "Mix premium and standard customers",
    "Cover all product categories",
    "Include some requests that should be declined"
  ]
}

From Execution History#

Build test datasets from real production data by sampling from your prompt's execution history. This ensures your test cases reflect actual usage patterns.

To create a dataset from history:

• Navigate to your prompt's Evaluation tab
• Click New Dataset and select From History
• Choose a date range and sampling strategy (random, stratified by category, or focused on specific outcomes)
• Review the selected executions and optionally mark outputs as expected results
• Save the dataset with a descriptive name

Dataset Versioning#

Like prompts, datasets are versioned to maintain reproducibility and enable historical comparisons. When you modify a dataset, PromptReports creates a new version while preserving the original.

Managing Datasets#

PromptReports provides comprehensive tools for organizing and maintaining your test datasets:

Each dataset can be associated with one or more prompts. You can also create "shared" datasets that apply across multiple prompts—useful when testing prompts that should handle similar types of inputs consistently.

Best Practices#

Follow these guidelines for effective test datasets:

With well-structured datasets in place, you're ready to start running evaluations to measure and improve prompt quality systematically.