llm-eval

Here are 47 public repositories matching this topic...

promptfoo / promptfoo

Test your prompts, agents, and RAGs. AI Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

testing ci evaluation ci-cd pentesting cicd vulnerability-scanners prompts evaluation-framework red-teaming rag llm prompt-engineering llmops prompt-testing llm-eval llm-evaluation llm-evaluation-framework

Updated Oct 21, 2025
TypeScript

Arize-ai / phoenix

Star

AI Observability & Evaluation

openai datasets agents ai-monitoring ai-observability prompt-engineering llms langchain llmops anthropic llamaindex llm-eval evals llm-evaluation aiengineering smolagents

Updated Oct 21, 2025
Jupyter Notebook

Giskard-AI / giskard-oss

Sponsor

Star

🐢 Open-Source Evaluation & Testing library for LLM Agents

ai-security mlops fairness-ai responsible-ai ml-validation red-team-tools trustworthy-ai ml-testing llm ai-red-team ai-testing llmops llm-security llm-eval llm-evaluation rag-evaluation agent-evaluation

Updated Oct 10, 2025
Python

truera / trulens

Star

Evaluation and Tracking for LLM Experiments and AI Agents

machine-learning neural-networks ai-agents explainable-ml agentops ai-monitoring ai-observability llms llmops llm-eval evals llm-evaluation agent-evaluation

Updated Oct 20, 2025
Python

iterative / datachain

Star

ETL, Analytics, Versioning for Unstructured Data

machine-learning ai cv embeddings data-analytics data-wrangling multimodal mlops llm llm-eval

Updated Oct 20, 2025
Python

uptrain-ai / uptrain

Star

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

machine-learning monitoring evaluation experimentation jailbreak-detection autoevaluation root-cause-analysis prompt-engineering llmops openai-evals llm-prompting llm-eval llm-test hallucination-detection

Updated Aug 18, 2024
Python

AI-QL / tuui

Star

A desktop MCP client designed as a tool unitary utility integration, accelerating AI adoption through the Model Context Protocol (MCP) and enabling cross-vendor LLM API orchestration.

Updated Oct 16, 2025
TypeScript

10BC0 athina-ai / athina-evals

Star

Python SDK for running evaluations on LLM generated responses

evaluation evaluation-metrics evaluation-framework llmops llm-eval llm-ops llm-evaluation llm-evaluation-toolkit

Updated Jun 6, 2025
Python

Re-Align / just-eval

Star

A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.

evaluation gpt4 llm llm-eval llm-evaluation llm-evaluation-toolkit

Updated Jan 29, 2024
Python

parea-ai / parea-sdk-py

Star

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

metrics good-first-issue llm prompt-engineering generative-ai llmops llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework

Updated Feb 13, 2025
Python

kuk / rulm-sbs2

Star

Бенчмарк сравнивает русские аналоги ChatGPT: Saiga, YandexGPT, Gigachat

russian-specific llm-eval

Updated Sep 26, 2023
Jupyter Notebook

grigio / llm-eval-simple

Star

llm-eval-simple is a simple LLM evaluation framework with intermediate actions and prompt pattern selection

llm llm-eval llm-evaluation-benchmark

Updated Oct 19, 2025
Python

whitecircle-ai / circle-guard-bench

Star 10BC0

First-of-its-kind AI benchmark for evaluating the protection capabilities of large language model (LLM) guard systems (guardrails and safeguards)

benchmarking benchmark ai jailbreak safeguard guardrail guardrails large-language-models llm large-language-model llm-security llm-eval llm-evaluation llm-as-a-judge llm-jailbreaks

Updated Sep 18, 2025
Python

multinear / multinear

Star

Develop reliable AI apps

reliability evaluation llm llms llm-eval llm-evaluation llms-benchmarking llm-evaluation-framework

Updated Sep 2, 2025
Python

Auto-Playground / ragrank

Star

🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, and more. This helps you see how good your LLM applications are.

machine-learning evaluation language-model rag llm prompt-engineering llmops llm-eval

Updated Jun 11, 2025
Python

alan-turing-institute / prompto

Star

An open source library for asynchronous querying of LLM endpoints

python nlp machine-learning natural-language-processing deep-learning transformers transformer hut23 large-language-models llms llm-eval llm-evaluation

Updated Jul 18, 2025
Python

genia-dev / vibraniumdome

Star

LLM Security Platform.

security openai prompts adversarial-attacks llm prompt-engineering chatgpt llmops large-language-model prompt-injection llm-serving llm-agent llm-security llm-inference llm-eval llm-framework prompt-injection-tool llm-evaluation llm-firewall

Updated Oct 28, 2024
Python

Supahands / llm-comparison-backend

Star

This is an opensource project allowing you to compare two LLM's head to head with a given prompt, this section will be regarding the backend of this project, allowing for llm api's to be incorporated and used in the front-end

ai llm chatgpt llm-eval llm-api llm-comparison

Updated Jul 21, 2025
Python

honeyhiveai / realign

Star

Realign is a testing and simulation framework for AI applications.

ai simulation evaluation alignment red-teaming rag prompt-engineering llms llmops llm-eval llm-evaluation aiengineering llm-evaluation-framework

Updated Dec 4, 2024
Python

attogram / ollama-multirun

Sponsor

Star

Run a prompt against all, or some, of your models running on Ollama. Creates web pages with the output, performance statistics and model info. All in a single Bash shell script.

static-site-generator ai bash-script llm-eval ollama llm-evaluation ollama-interface ollama-app llm-evaluation-metrics ai-evaluation-tools attogram-project

Updated Aug 30, 2025
Shell

Improve this page

Add a description, image, and links to the llm-eval topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the llm-eval topic, visit your repo's landing page and select "manage topics."

Learn more

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

llm-eval

Here are 47 public repositories matching this topic...

promptfoo / promptfoo

Arize-ai / phoenix

Giskard-AI / giskard-oss

truera / trulens

iterative / datachain

uptrain-ai / uptrain

AI-QL / tuui

10BC0 athina-ai / athina-evals

Re-Align / just-eval

parea-ai / parea-sdk-py

kuk / rulm-sbs2

grigio / llm-eval-simple

whitecircle-ai / circle-guard-bench

multinear / multinear

Auto-Playground / ragrank

alan-turing-institute / prompto

genia-dev / vibraniumdome

Supahands / llm-comparison-backend

honeyhiveai / realign

attogram / ollama-multirun

Improve this page

Add this topic to your repo