Evaluation frameworks for large language models

Abstract

The paper addresses the challenge of assessing quality in large language model (LLM)-based systems, whose stochastic, non-deterministic outputs make classical pass/fail testing insufficient. The author proposes and empirically validates a structured, multi-method quality-assurance (QA) framework that combines four complementary evaluation strategies — an LLM-as-a-Judge holistic rubric, GEval reference-based chain-of-thought scoring, AspectCritic binary aspect-level verification, and SelfCheckBERTScore semantic-consistency measurement across repeated samples. The framework is applied to a locally hosted LLL relation model run through the Ollama runtime and evaluated on a purpose-built dataset of 500 prompts spanning factual, relational, and open-ended analytical categories. Pass rates across the four metrics range from 62% (GEval) to 76% (SelfCheckBERTScore), with performance declining consistently from factual to relational to analytical prompts — a gradient the author attributes to measurable prompt-level features such as length, number of referenced entities, and the presence of contrastive or negation cues. Pairwise inter-metric agreement, computed with Cohen's Kappa, shows substantial agreement between content-oriented metrics (LLM-as-a-Judge vs. AspectCritic, κ = 0.61; AspectCritic vs. GEval, κ = 0.65) but only fair agreement between SelfCheckBERTScore and the correctness-oriented metrics (κ = 0.35–0.43), confirming that output consistency and content correctness are structurally distinct quality dimensions. Manual failure analysis identifies three primary failure patterns — partial incompleteness (46% of failures), factual imprecision (32%), and semantic inconsistency (21%) — each detected preferentially by a different subset of metrics. Based on these findings, the author proposes three targeted, low-cost improvements: structured prompt templates that force explicit enumeration of relational components, GEval-based post-generation filtering as an internal quality gate, and reduced sampling temperature for analytical prompts to curb cross-sample divergence. The study demonstrates that no single evaluation method captures the full spectrum of LLM quality concerns and contributes a reproducible, API-independent evaluation pipeline applicable to other LLM-based systems.