Agent♥︎Age
Catalog

RAGScore

Official

by HZYAI · Python

Generate QA datasets & evaluate RAG systems with failure diagnosis. Any LLM.

io.github.HZYAI/ragscore — Model Context Protocol (MCP) Server

This MCP server generates QA datasets and evaluates RAG systems, including failure diagnosis. It supports “Any LLM” for evaluation, with capabilities related to synthetic-data and dataset-generation. The repository is focused on RAG evaluation, LLM-as-a-judge workflows, and LLMOps use in local environments.

🛠️ Key Features

  • QA dataset generation
  • RAG system evaluation
  • Failure diagnosis for RAG
  • “Any LLM” evaluation backends
  • Synthetic-data and fine-tuning dataset-generation support

🚀 Use Cases

  • Evaluate RAG pipelines and identify failure modes
  • Generate synthetic QA datasets for testing
  • Run RAG evaluation locally (e.g., with Ollama)
  • Use in Colab/Jupyter-based RAG workflows
  • Support dataset generation for LLM fine-tuning

⚡ Developer Benefits

  • Topics align with ai-evaluation, rag-evaluation, and llm-as-a-judge
  • Designed for llmops-style RAG evaluation
  • Developer-facing focus on evaluation and diagnostics

⚠️ Limitations

  • Source data does not specify available MCP tools, interfaces, or toolCount details

Topics

evaluationollamaprivacyqa-generationragsynthetic-datadataset-generationfine-tuningllmlocal-llmcolabjupytermcprag-evaluationai-evaluationllmopsllm-as-a-judge