Local LLM Model Testing CLI

A CLI tool that helps developers systematically test and compare local Ollama models across different prompts, temperatures, and run counts. Saves test results in organized Markdown/JSON format. Solves the problem of manually evaluating which model works best for specific tasks on local hardware.

Ollama's rapid adoption has created a real workflow gap — developers are running dozens of local models but have no structured way to benchmark them against each other, and that friction compounds as model releases accelerate. No clear incumbent owns this space; most developers are currently hacking together shell scripts or using general-purpose benchmarking tools that weren't built for prompt-level LLM comparison. The $500–2k/mo revenue band is honest for a CLI tool — charging via one-time purchase or a small annual license is plausible, but subscription fatigue and the "just open source it" instinct in this audience will cap ceiling hard. The biggest risk is that Ollama itself, or a well-resourced devtools company, ships this as a native feature within six months, which given the pace of the ecosystem is a genuine possibility rather than a remote one.