Working through this module?
Module 14 · Page 15 of 20 · 10 sections
Evaluation and Testing
How to measure an AI system: evaluation types, LLM and RAG metrics, an evaluation workflow, testing types and example test cases.
1. Introduction
- Evaluation and testing are essential to ensure that AI systems provide accurate, reliable, fair and safe responses.
- For LLM and RAG systems, we need to evaluate both the model's capability and the overall application performance.
- Proper evaluation helps in improving the system, reducing errors and building trust with users.
2. Why Evaluation is Important?
- 1Measure how good the model / system is.
- 2Identify strengths and weaknesses.
- 3Compare different models and approaches.
- 4Ensure reliability and consistency.
- 5Improve prompts, retrieval and generation.
- 6Detect bias, hallucination and harmful content.
- 7Validate performance in real-world scenarios.
- 8Essential for production readiness.
3. Types of Evaluation
Evaluation
Automatic Evaluation
- ROUGE
- BLEU
- BERTScore
- Cosine similarity
- Semantic similarity
Human Evaluation
- Relevance
- Correctness
- Helpfulness
- Coherence
- Safety
- User satisfaction
Task-specific Evaluation
- RAG evaluation
- Chatbot evaluation
- Summarization
- Question answering
- Translation
- Classification
System Evaluation
- Latency
- Throughput
- Error rate
- Scalability
- Cost
- Availability
4. Evaluation Metrics for LLMs
- Perplexity
- Measures how well the model predicts text.
- BLEU
- Commonly used for machine translation.
- ROUGE
- Used for text summarization.
- BERTScore
- Measures semantic similarity.
- Accuracy
- Correctness of the answer.
- Relevance
- How relevant the response is to the query.
- Faithfulness
- Whether the answer is grounded in the source.
- Coherence
- Logical flow and readability of the response.
1
2
3
4
5
6
7
8
5. Metrics for RAG Systems
- Context Precision
- How much retrieved context is relevant.
- Context Recall
- How much relevant context is retrieved.
- Answer Relevance
- Is the answer relevant to the question?
- Faithfulness
- Is the answer supported by retrieved context?
- Hallucination Rate
- Percentage of unsupported statements.
- Source Attribution
- Correct citation of source documents.
- Response Quality
- Helpfulness, completeness, clarity.
- End-to-end Latency
- Total time (retrieval + generation).
1
2
3
4
5
6
7
8
Related07RAG System Design
6. Evaluation Workflow (RAG Example)
Test QueriesQuestion set
RAG SystemRetrieval + generation
EvaluationRelevance, faithfulness, correctness, context precision, human review
ResultsMetrics, analysis, improvement areas
7. Example Evaluation Techniques
- LLM-as-a-Judge
- Use another LLM to evaluate responses.
- Reference-based Evaluation
- Compare with ground truth answers.
- Semantic Similarity
- Use embeddings to compare meaning.
- Pairwise Comparison
- Compare responses from two models.
- Human Evaluation
- Manual review by domain experts.
- RAGAS
- Popular framework for RAG evaluation.
- LangSmith
- Track and evaluate LLM applications.
- Custom Scoring
- Task-specific evaluation criteria.
1
2
3
4
5
6
7
8
8. Testing Types for AI Systems
- Unit Testing
- Test individual components (e.g. prompts, functions).
- Integration Testing
- Test retrieval + generation together.
- Functional Testing
- Verify expected behavior.
- Regression Testing
- Ensure new changes don't break existing flow.
- A/B Testing
- Compare two models or prompts.
- Load Testing
- Test the system under high traffic.
- Adversarial Testing
- Test with tricky or malicious inputs.
- Safety Testing
- Check for harmful, biased or inappropriate content.
1
2
3
4
5
6
7
8
9. Example Test Cases (RAG)
| Test case | Example input | Expected output |
|---|---|---|
| Relevant query | What is Spring Boot? | Correct and relevant answer with source citation |
| Irrelevant query | Who won the 2022 World Cup? | Response: not in context, or an appropriate message |
| Ambiguous query | Explain this | Ask for clarification or give the best possible answer |
| Incorrect context | (wrong document) | Detect and avoid hallucination |
| Multi-turn query | Tell me about Kafka. What are its use cases? | Maintain context and give an accurate answer |
10. Best Practices
- 1Use a diverse and representative test dataset.
- 2Evaluate with both automatic metrics and human feedback.
- 3Continuously monitor model performance in production.
- 4Track and analyze failure cases.
- 5Set clear evaluation criteria for your use case.
- 6Use tools like RAGAS, LangSmith and OpenAI Evals.
- 7Test for bias, safety and adversarial inputs.
- 8Keep improving prompts, retrieval and system design based on evaluation results.