github-featured / vibrantlabsai--ragas

发布

0 发布 90 Git 标签

RSS 订阅源
  • v0.3.7 f31c365e9d
    比较

    docs: Add how-to guide for aligning LLM-as-Judge (#2348)

    frostbyte_neo 发布于 2025-10-14 15:45:47 +00:00 | 133 次提交 在此版本后已推送到 main

    Issue Link / Problem Description

    Adds documentation and examples for aligning LLM-as-Judge evaluators
    with human expert judgments. This addresses a common challenge where LLM
    judges may not align well with human evaluations, leading to unreliable
    automated assessments.

    Changes Made

    • Add comprehensive how-to guide:
      docs/howtos/applications/align-llm-as-judge.md
    • Step-by-step instructions for measuring and improving judge alignment
    • Add complete evaluation example:
      examples/ragas_examples/judge_alignment/
    • evals.py: Baseline and improved judge metrics with alignment
      measurement
      • __init__.py: Module initialization with main entry points
    • Documentation covers:
      • Why judge alignment matters
      • Dataset structure and loading
      • Defining judge and alignment metrics
      • Running baseline evaluations
      • Iterating on judge prompts to improve alignment

    Testing

    How to Test

    • Manual testing steps:
    1. Run baseline evaluation: uv run python -m ragas_examples.judge_alignment
    2. Verify alignment metrics are calculated correctly
    3. Build docs locally: make serve-docs and navigate to the new guide
    下载附件