Velamuvi AIAcademy
Generative AI

LLM Evaluation

Measure whether your AI feature actually works. Test sets, automatic metrics, LLM-as-judge and regression testing.

  • Advanced
  • About 8 hours
  • 3 modules
  • 10 lessons
  • Free

This course is not released yet. We publish courses one at a time, each one complete. The full syllabus is below.

What you will learn

  • Why and WhatWhy Evaluation Is Hard, Building an Evaluation Set, Defining Success Criteria
  • MethodsExact and Semantic Metrics, LLM-as-Judge, Human Review and more
  • ProcessRegression Testing Prompts, Evaluation in CI, Online Evaluation and Feedback

Course syllabus

Not released yet · full syllabus below

  1. 1

    Why and What

    3 lessons
    1. 1Why Evaluation Is HardSoon
    2. 2Building an Evaluation SetSoon
    3. 3Defining Success CriteriaSoon
  2. 2

    Methods

    4 lessons
    1. 4Exact and Semantic MetricsSoon
    2. 5LLM-as-JudgeSoon
    3. 6Human ReviewSoon
    4. 7Evaluating RAG and AgentsSoon
  3. 3

    Process

    3 lessons
    1. 8Regression Testing PromptsSoon
    2. 9Evaluation in CISoon
    3. 10Online Evaluation and FeedbackSoon

Recommended next