LLM Evaluation Platform

About This Project

LLM Evaluation Platform continuously validates classifier performance against a curated golden dataset before prompt deployments reach production. The system compares prompt versions, measures classification accuracy, detects performance drift, generates detailed HTML reports, and automatically alerts teams through Slack when regressions exceed configurable thresholds. Built for production-grade AI operations, it helps teams confidently ship prompt updates while maintaining quality and reliability.

What's Included

• •
  Golden Dataset Evaluation — Runs prompt versions against 100+ manually verified historical test cases
• •
  Prompt Regression Detection — Compares baseline and candidate prompts to identify accuracy drops before deployment
• •
  Drift Monitoring Engine — Tracks long-term performance degradation across evaluation windows
• •
  Automated Slack Alerting — Sends severity-based notifications with regression summaries and report links
• •
  HTML Reporting System — Generates detailed category-level accuracy breakdowns and evaluation insights
• •
  GitHub Actions Integration — Executes evaluations automatically within CI/CD pipelines
• •
  Dockerized Deployment — Production-ready containerized execution with environment-based configuration
• •
  LLM Summary Quality Scoring — Uses AI judges to assess summary quality beyond binary classification accuracy

Project Impact

• •Prevented prompt regressions from reaching production environments
• •Reduced manual evaluation effort through fully automated benchmark testing
• •Enabled rapid experimentation with versioned prompts and configurable thresholds
• •Provided early-warning drift detection for long-term model quality monitoring
• •Integrated AI quality assurance directly into CI/CD workflows
• •Delivered actionable Slack alerts and HTML reports for faster incident response