orbench: benchmark LLMs on your own tasks
You can pick a model on a gut feeling, orbenchmark it.
orbench is a desktop app that lets you benchmark LLMs on your own tasks before you switch models.
A public leaderboard ranks models on someone else's prompts. Production evals tell you the truth, but only after you've done all the work to implement it. orbench is the step in between: bring a CSV of your real tasks (or a Hugging Face dataset), pick any models on OpenRouter with their real prices, and let your own rubric grade every answer with an AI judge, a blind human review, or both.
One report per run: quality score per model, p50/p95 latency, cache hit rate, and the real cost per answered task.
Runs on your machine, with your own OpenRouter key.
Windows, macOS, and Linux apps available.
Built with
