About The Project
LLM Evaluation Platform continuously validates classifier performance against a curated golden dataset before prompt deployments reach production. The system compares prompt versions, measures classification accuracy, detects performance drift, generates detailed HTML reports, and automatically alerts teams through Slack when regressions exceed configurable thresholds. Built for production-grade AI operations, it helps teams confidently ship prompt updates while maintaining quality and reliability.
Key Features & Architecture
Golden Dataset Evaluation
Runs prompt versions against 100+ manually verified historical test cases
Prompt Regression Detection
Compares baseline and candidate prompts to identify accuracy drops before deployment
Drift Monitoring Engine
Tracks long-term performance degradation across evaluation windows
Automated Slack Alerting
Sends severity-based notifications with regression summaries and report links
HTML Reporting System
Generates detailed category-level accuracy breakdowns and evaluation insights
GitHub Actions Integration
Executes evaluations automatically within CI/CD pipelines
Dockerized Deployment
Production-ready containerized execution with environment-based configuration
LLM Summary Quality Scoring
Uses AI judges to assess summary quality beyond binary classification accuracy