Langtrace 🔍 is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.. Integrate using Typescript, Python. 🚀💻📊

Test Generation for Prompts

The TypeScript LLM Evaluation Library

Python client library for improving your LLM app accuracy

Code Repository for: AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models

The fastest java expression compiler/evaluator

A framework to build scenario simulation projects where human and LLM based agents can participant in, with a user-friendly web UI to visualize simulation, support automatically evaluation on agent action level.

IELTS listening, speaking, reading and writing modules practice and evaluation with IELTS band calculation based on speech and text analysis and evaluation.

This workshop teaches systematic approaches to evaluating Generative AI workloads for production use. You'll learn to build evaluation frameworks that go beyond basic metrics to ensure reliable model performance while optimizing cost and performance.

This library implements various metrics (including Kaggle Competition, Medicine) for evaluating ML, DL, AI models, and algorithms. 📐📊📈📉📏

NLP tool for wide-range model reliability evaluations

CSH Evals, the modern way.

3cb: Catastrophic Cyber Capabilities Benchmarking of Large Language Models

ARGAEL is an open-source Java desktop application designed to maximize the experience and efficiency of the process of annotating and evaluating arguments in large text corpora.

Galileo Documentation

The official repository for the paper "Paraphrase Detection: Human vs. Machine Content".

EntRAG - Enterprise RAG Benchmark

Moodle plugin for evaluations with Moodle. This is the evaluation activity plugin.

Official Node.js client for the Mandoline API

Code for ICML 2025 How Do Large Language Monkeys Get Their Power (Laws)?