Comet tool for logging and evaluating LLM traces
Project description
Open-source end-to-end LLM Development Platform
Confidently evaluate, test and monitor LLM applications.
Website • Slack community • Twitter • Documentation
🚀 What is Opik?
Opik is an open-source platform for evaluating, testing and monitoring LLM applications. Built by Comet.
You can use Opik for:
-
Development:
- Tracing: Track all LLM calls and traces during development and production (Quickstart, Integrations)
- Annotations: Annotate your LLM calls by logging feedback scores using the Python SDK or the UI.
-
Evaluation: Automate the evaluation process of your LLM application:
-
Datasets and Experiments: Store test cases and run experiments (Datasets, Evaluate your LLM Application)
-
LLM as a judge metrics: Use Opik's LLM as a judge metric for complex issues like hallucination detection, moderation and RAG evaluation (Answer Relevance, Context Precision and Answer Relevance)
-
CI/CD integration: Run evaluations as part of your CI/CD pipeline using our PyTest integration
-
-
Production Monitoring: Monitor your LLM application in production and easily close the feedback loop by adding error traces to your evaluation datasets.
🛠️ Installation
The easiest way to get started with Opik is by creating a free Comet account at comet.com.
If you'd like to self-host Opik, you create a simple local version of Opik using::
pip install opik-installer
opik-server install
For more information about the different deployment options, please see our deployment guides:
| Installation methods | Docs link |
|---|---|
| Local instance | |
| Kubernetes |
🏁 Get Started
If you are logging traces to the Cloud Opik platform, you will need to get your API key from the user menu and set it as the OPIK_API_KEY environment variable:
export OPIK_API_KEY=<Your API key>
export OPIK_WORKSPACE=<You workspace, often the same as your username>
If you are using a local Opik instance, you don't need to set the OPIK_API_KEY or OPIK_WORKSPACE environment variable and isntead set the environment variable OPIK_BASE_URL to point to your local Opik instance:
export OPIK_BASE_URL=http://localhost:5173
You are now ready to start logging traces using the Python SDK.
📝 Logging Traces
The easiest way to get started is to use one of our integrations. Opik supports:
| Integration | Description | Documentation | Try in Colab |
|---|---|---|---|
| OpenAI | Log traces for all OpenAI LLM calls | Documentation | |
| LangChain | Log traces for all LangChain LLM calls | Documentation | |
| LlamaIndex | Log traces for all LlamaIndex LLM calls | Documentation |
[!TIP]
If the framework you are using is not listed above, feel free to open an issue or submit a PR with the integration.
If you are not using any of the frameworks above, you can also using the track function decorator to log traces:
from opik import track
@track
def my_llm_function(user_question: str) -> str:
# Your LLM code here
return "Hello"
[!TIP]
The track decorator can be used in conjunction with any of our integrations and can also be used to track nested function calls.
🧑⚖️ LLM as a Judge metrics
The Python Opik SDK includes a number of LLM as a judge metrics to help you evaluate your LLM application. Learn more about it in the metrics documentation.
To use them, simply import the relevant metric and use the score function:
from opik.evaluation.metrics import Hallucination
metric = Hallucination()
score = metric.score(
input="What is the capital of France?",
output="Paris",
context=["France is a country in Europe."]
)
print(score)
Opik also includes a number of pre-built heuristic metrics as well as the ability to create your own. Learn more about it in the metrics documentation.
🔍 Evaluating your LLM Application
Opik allows you to evaluate your LLM application during development through Datasets and Experiments.
You can also run evaluations as part of your CI/CD pipeline using our PyTest integration.
🤝 Contributing
There are many ways to contribute to Opik:
- Submit bug reports and feature requests
- Review the documentation and submit Pull Requests to improve it
- Speaking or writing about Opik and letting us know
- Upvoting popular feature requests to show your support
To learn more about how to contribute to Opik, please see our contributing guidelines.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file opik-0.1.7.tar.gz.
File metadata
- Download URL: opik-0.1.7.tar.gz
- Upload date:
- Size: 83.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.0 CPython/3.12.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | cf0c0fd42147d3c17c882e6a5cbbf92edf36a066c907279e06ca9078fcb5eb67 |
|
| MD5 | 9b8351d221fac8969165881ed69881f1 |
|
| BLAKE2b-256 | 5d5c94f03e4e7ebe77efaea348164b03eb0bb87f2012613a34ea96a89787f93f |
File details
Details for the file opik-0.1.7-py3-none-any.whl.
File metadata
- Download URL: opik-0.1.7-py3-none-any.whl
- Upload date:
- Size: 172.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.0 CPython/3.12.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 847f5a31917f388909da08fde1a47e43d4c96480cacd81f96570ba072a47a948 |
|
| MD5 | 57f7f94d1dc054335da01b0ba7f95c16 |
|
| BLAKE2b-256 | ecdf8465ba842d44c60fc1bc517fdb8abc1625b32a76b2723d588309bdf46bc1 |