Test your prompts, agents, and RAGs. AI Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

LLM Reasoning and Generation Benchmark. Evaluate LLMs in complex scenarios systematically.

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

Test, compare, and optimize your AI prompts in minutes

The prompt engineering, prompt management, and prompt evaluation tool for TypeScript, JavaScript, and NodeJS.

EvalWise is a developer-friendly platform for LLM evaluation and red teaming that helps test AI models for safety, compliance, and performance issues

LLM Prompt Test helps you test Large Language Models (LLMs) prompts to ensure they consistently meet your expectations.

Community Plugin for Genkit to use Promptfoo

prompt-evaluator is an open-source toolkit for evaluating, testing, and comparing LLM prompts. It provides a GUI-driven workflow for running prompt tests, tracking token usage, visualizing results, and ensuring reliability across models like OpenAI, Claude, and Gemini.

Test Claude Projects without copy-pasting. Local workbench for prompt engineering, agent testing, and workflow iteration. Direct Claude.ai access via cookie auth, 20+ prompt templates, web fetch/search tools, file uploads. Stop switching tabs to test your prompts.

An open-source AI prompt engineering playground with live code execution. Test OpenAI & Claude prompts, execute JavaScript, and iterate in real-time.

A Beautiful, Cinematic Prompt Engineering Studio

Sample project demonstrates how to use Promptfoo, a test framework for evaluating the output of generative AI models

A pytest-based framework for testing multi AI agents systems. It provides a flexible and extensible platform for complex multi-agent simulations. Supports many integrations like LiteLLM, CrewAI, LangChain etc.

A collection of prompts that I use on a day-to-day basis for work and leisure.

Run 1,000 LLM evaluations in 10 minutes. Test prompts across Claude, GPT-4, and Gemini with parallel execution, real-time cost tracking, and beautiful visualizations. Open source.

AI RAG evaluation project using Ragas. Includes RAG metrics (precision, recall, faithfulness), retrieval diagnostics, and prompt testing examples for fintech/banking LLM systems. Designed as an AI QA Specialist portfolio project.

Visual prompt engineering platform for creating, testing, and versioning LLM prompts across multiple providers (OpenAI, Anthropic, Mistral, Gemini).

Quickstart guide for using PromptFoo to evaluate LLM prompts via CLI or Colab.