Which open model should your agent use?
We held tryouts. Same agent, same search, same tools, every model on Nebius Token Factory against Claude Opus 5.
Easy questions say: any of them.
On single facts every model scores 94 to 100%. Qwen3 30B ties Claude at 80× lower cost. Most teams would switch here.
In plain wordsOn simple facts almost every model scores close to 100%. Stop here and you would just pick the cheapest.
Right answers
Cost per question
50 verified facts in German and English. Every model answers from the same search results.
Hard questions say: not that one.
On research that chains 2 to 5 facts, Qwen3 30B drops from 98% to 50%, and 35% of its answers are confidently wrong. Claude: 88%.
In plain wordsWhen an answer needs several facts chained together, the cheap pick gets half of them wrong and sounds sure about it.
Right answers
Confidently wrong
40 FRAMES questions. Each model plans its own searches, up to three rounds, with the same 16k output budget. Nemotron Lightning runs with thinking on; with it off it scores 38%, see the NVIDIA section.
Two open models hold up.
Reflection Beam is right 80% of the time at 13× lower cost. DeepSeek V4 Flash: 70% at 30× lower cost, with the fewest wrong answers of any open model (8%).
In plain wordsTwo open models stay close to Claude for a small fraction of the price.
Cost per question
Live Token Factory prices, every call priced by LiteLLM. Shorter is cheaper.
Nemotron on hard research.
Super: 77% at 11× lower cost.
Turning thinking on lifts Lightning from 38% to 65%. Super solves 12 of 14 tool tasks. Ultra's tool calls come back from Token Factory as raw text, so it solves none: a server setting, not the model.
Hard questions, right answers
Tool tasks solved
Agents use tools. So we gave them real ones.
Every model drives the same MCP servers: Tavily, semble code search and a real Chrome for pages that need JavaScript. DeepSeek V4 Flash solves 14 of 14 at 51× lower cost. Claude solves 14.
In plain wordsThe AI decides by itself which tool to use: search, read a page, look at code, or open a real browser.
One of the 14 tasks: DeepSeek V4 Flash opens Chrome by itself and reads a page that only exists after JavaScript runs.
Tasks solved, out of 14
14 tasks with answers checked against the source on 18 October 2026.
On our own logs, the best open model wins.
Questions written from the research our agents ran last month. DeepSeek V4 Flash is right 75% of the time, Claude 69%. What still trips every model: questions with a false premise.
In plain wordsThese are questions our own agents really researched last month. This is the test that matters for us.
Right answers
Confidently wrong
16 questions. Every answer is a quote checked against the page; some rest on a false premise, so the right answer is "unknown".
Watch one answer.
Nemotron 3.5 Super VL in deep mode, on a real question from our logs: What is the name of the brewery featured at the Berlin community event on AI agents and physical AI that takes place on October 17, 2026 and does not require a GTC pass?
Then we fixed what we could.
Same questions from our logs, same models. Full pages instead of snippets, the best passages picked by Searchify's reranker, one search per fact, and an answer only with quotes that are really in the sources. It lifts the strong models, Nemotron 3.5 Super VL to 75–94% across two machines, and trips up the small ones.
In plain wordsReading whole pages, keeping only the best passages and checking every quote helps the big models. The small ones get confused by the extra rules.
Right answers
Confidently wrong
16 questions, so read these as direction, not decimals. A second machine (Sami's laptop) got 75% for Nemotron 3.5 Super VL where ours got 94%: 3 answers flipped. Reranker: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1, top 12 passages. Every setting is stored with each result, see REPRODUCE.md.
Run it twice.
Same questions, same prompts, a second time. Qwen3 30B goes from 49% to 41%: 9 of 39 answers flip. One run is not a decision.
In plain wordsAsk the same thing twice and cheap models can change their answer. Test twice before you trust a winner.
Reflection Beam
Nemotron 3 Super
Qwen3 30B
Why they miss.
We rebuilt exactly what every model read. When the cheap models are wrong on hard questions, the answer was usually not in anything they found, and they answered anyway instead of searching again.
- Our own plumbing first. Rate-limited searches came back empty and made some models search blind. Fixed and rerun; check yours too.
- They stop too early. One search round, then a confident guess.
- They don't say "I don't know". On false-premise questions they make up an answer.
- Settings hurt them. Thinking turned off, output cut off, a tool parser set wrong on the server.
How we make them better.
- Free: fix the settings. Thinking on, longer output, the right tool parser for Nemotron Ultra.
- Runtime: answer only with a quoted sentence, otherwise search again or say "unknown".
- Swap: Tongyi DeepResearch, the same Qwen3 30B base trained for research. Its makers report 90.6 on FRAMES.
- Train: LoRA on Qwen3 30B on Token Factory, from our own verified research logs, with Tryouts as the test.
Training data comes from open models and checked answers only. Every step is measured with the same tryouts, before and after.
So which one?
- Tool-heavy agents: DeepSeek V4 Flash.
- Multi-hop research: Reflection Beam.
- Single facts: Qwen3 30B.
- Questions like our logs: Nemotron 3.5 Super VL in deep mode, DeepSeek V4 Flash without it.
- Lowest cost that still holds up: Nemotron Lightning, thinking on.
That's our workload. Yours will be different, which is why you should run your own tryouts.
Two editions. Pick one.
Searchify runs on its own, with no key. Searchify for Tavily lets Tavily search and read the web with your key, while Searchify ranks every passage on your machine and takes over any call where Tavily is slow.
In plain wordsTavily brings the reach, Searchify picks the best sentences. Together they answer more questions than Searchify alone.
- Searches the web itself
- Reads pages itself, including JavaScript pages
- Ranks passages on your machine
- Tavily searches the web
- Tavily reads the pages; Searchify reads what Tavily can't
- Searchify ranks passages on your machine
- Tavily slower than 3 s? Searchify takes over that call
Answer in the top 5 passages
Search and reading, seconds per question (median)
Came back with nothing
50 single-fact questions with checked answers, 18 October 2026, one question at a time. No model involved: this measures what reaches the model. Ranking runs on the same machine in both editions, so it is left out of the times. Searchify took over from Tavily on 16 of 50 questions.
Set it up your way.
Plugin, slash command, skill, MCP server, command line, or your own open models. Every way comes in both editions.
Searchify, keyless
/plugin marketplace add Malek1414/tryouts/plugin install searchify@searchifySearchify for Tavily
export TAVILY_API_KEY=tvly-... # your key, in the shell that starts Claude Code/plugin install searchify-tavily@searchifySearchify, keyless
/searchify:research How much memory does the DGX Spark have?Searchify for Tavily
/searchify-tavily:research How much memory does the DGX Spark have?Searchify, keyless
npx skills add Malek1414/tryouts --skill searchifySearchify for Tavily
npx skills add Malek1414/tryouts --skill searchify-tavilySearchify, keyless
claude mcp add searchify -e SEARCHIFY_MODE=local -- uvx --from "searchify[mcp,rerank] @ git+https://github.com/Malek1414/tryouts#subdirectory=searchify-tavily-edition" searchify-mcpSearchify for Tavily
claude mcp add searchify-tavily -e TAVILY_API_KEY=$TAVILY_API_KEY -- uvx --from "searchify[mcp,rerank] @ git+https://github.com/Malek1414/tryouts#subdirectory=searchify-tavily-edition" searchify-mcpSearchify, keyless
uv tool install "searchify[mcp,rerank] @ git+https://github.com/Malek1414/tryouts#subdirectory=searchify-tavily-edition"SEARCHIFY_MODE=local searchify research "your question"Searchify for Tavily
searchify research "your question" # with TAVILY_API_KEY setAny model you run (Ollama, LM Studio, llama.cpp, vLLM)
searchify chat --preset ollama:qwen3searchify chat --base-url http://localhost:1234/v1 --model qwen/qwen3-4bsearchify models # every model we tested, as a presetThe models we tested, on Nebius Token Factory
export NEBIUS_API_KEY=...searchify ask "your question" --preset deepseek-v4-flashsearchify chat --preset nemotron-3.5-super-vlOpen WebUI hosts any of these models as a chatbot
uvx mcpo --port 8000 -- searchify-mcp# then in Open WebUI: Settings, Tools, add http://localhost:8000LM Studio, Jan, Continue, Cline, Goose
{ "mcpServers": { "searchify": { "command": "searchify-mcp", "env": { "SEARCHIFY_MODE": "local" } } } }Full guide with every app: docs/use-with-any-model.md in the repo.
See it work.
Prompt by Daniel
Run it on your logs.
Get it on GitHub Browse every question and every answer, model by model Every question and every run on Hugging Face1 Turn your agent's logs into questions with checked answers.
python scripts/build_hard_set.py --runs your-runs/ --out data/mine.jsonl
2 Hold the tryouts: python -m tryouts.run --eval data/mine.jsonl --mode agentic
Models served by Nebius Token Factory. Search by Tavily. Sami and Malek, Berlin, 18 October 2026.