Skip to content

Evals

Data structures for fed_rag.evals

BenchmarkExample

Bases: BaseModel

Benchmark example data class.

Source code in src/fed_rag/data_structures/evals.py 
 8
 9
10
11
12
13
class BenchmarkExample(BaseModel):
    """Benchmark example data class."""

    query: str
    response: str
    context: str | None = None

BenchmarkResult

Bases: BaseModel

Benchmark result data class.

Source code in src/fed_rag/data_structures/evals.py 
16
17
18
19
20
21
22
class BenchmarkResult(BaseModel):
    """Benchmark result data class."""

    score: float
    metric_name: str
    num_examples_used: int
    num_total_examples: int

AggregationMode

Bases: str, Enum

Mode for aggregating evaluation scores.

Source code in src/fed_rag/data_structures/evals.py 
25
26
27
28
29
30
31
class AggregationMode(str, Enum):
    """Mode for aggregating evaluation scores."""

    AVG = "avg"
    SUM = "sum"
    MAX = "max"
    MIN = "min"