Breaking down: "What is a useful comparison for understanding open models for tool-using AI agents, based on available evidence?"

Identified 6 research target(s) to investigate; these are not established facts

Deep mode: up to 6 paid/cached/public reads plus one bounded gap-expansion pass when needed.

Web search: 7/7 planned queries attempted, 0 succeeded, 0 public page previews/leads, 7 unavailable queries. Previews and supplied URLs are discovery only. Public reads spend no USDC; model and service operating costs remain separate.

Discovered 2 verified creator source(s) and 30 free public reference(s)

Claim-aware portfolio (exhaustive; bounded selection, not a claim of global optimality) selected 5/5 positive proposal(s): 5 free/cache selections + 0 paid fresh selections, predicting 6/6 claim(s) above the evidence floor with $0.000000/$0.025000 fetch USDC reserved.

Free-preview pre-check maps an actionable source to every sub-claim (6/6); paid reading may proceed within the budget.

Read Vicki Boykis - Running local models is good now - free public feed reference, no creator payment - S1

Read Microsoft Research - Agent Lightning v1.0: A 3,500-Line Lightweight Agentic RL Framework for Training Agents with Real Harnesses - free public feed reference, no creator payment - S2

Read Docker - Manufacturing Trust for AI Agents | Docker’s WeAreDevelopers Keynote - free public feed reference, no creator payment - S3

Read Chip Huyen - What I learned from looking at 900 most popular open source AI tools - free public feed reference, no creator payment - S4

Read Netlify - Open models are having a moment. We’re all in. - free public feed reference, no creator payment - S5

Sub-claim "What open models for tool-using AI agents have been evaluate…": 30% covered by S1, S2 — S1 mentions using Mistral 7B, Gemma 3, OpenAI OSS-20B, Qwen 3 MOE, Qwen 2.5 Coder, and Gemma 4 for agentic coding, but does not systematically evaluate them for tool use. S2 mentions Qwen3.5-9B in a coding agent pipeline, but only as a training example, not a comparative evaluation. No source provides a list of open models specifically evaluated for tool-using agents.

Sub-claim "What dimensions are commonly used to compare open models for…": 20% covered by S1 — S1 uses a personal 'vibe metric' of whether the model needs double-checking against an API model, and mentions accuracy/speed relative to frontier models, but no systematic comparison dimensions are provided. Other sources do not discuss comparison dimensions for open tool-using models.

Sub-claim "What evidence exists about the performance of open models on…": 40% covered by S2 — S2 provides a specific benchmark result: Qwen3.5-9B improved from 41.8% to 56.4% Pass@1 on SWE-bench Verified after RL training. However, this is a single model and a single benchmark, not a broad comparison of open models on tool-use benchmarks.

Sub-claim "What evidence exists about the limitations of open models fo…": 30% covered by S1 — S1 notes that early local models were slow, hard to use, and not accurate for programming tasks, and that agentic coding loops work at about 75% accuracy/speed of frontier models. This is anecdotal and limited to one user's experience.

Sub-claim "What evidence exists about the training data or fine-tuning …": 40% covered by S2 — S2 describes a training pipeline using SWE-smith, mini-SWE-agent, and Qwen3.5-9B with about 6,000 training samples and RL training, including data cleaning and reward-hacking safeguards. However, it does not detail the training data composition or fine-tuning methods for open models in general.

Sub-claim "What evidence exists about the safety or reliability of open…": 30% covered by S3 — S3 discusses the need for trust in AI agents, mentioning that agents can install dependencies, access networks, use credentials, and the importance of guardrails and deterministic boundaries. However, it does not specifically evaluate the safety or reliability of open models for tool-using agents.

Coverage is low across all sub-claims, with no source providing a comprehensive comparison of open models for tool-using AI agents. The two affordable skipped sources (Keryx release notes and Keryx engineering) are unlikely to directly address the research question, but they are the only options within the remaining budget of 0.025. However, they are not relevant to open models for tool-using agents, so buying them may not improve coverage. The recommendation is made because coverage is below 0.5 for all claims and there are affordable sources, but their relevance is questionable.

Filling gap — buying Keryx owner release notes — first-party validation — Keryx v0.27.63 ($0.005)…

Paid $0.005 to Keryx owner release notes — first-party validation — Keryx v0.27.63 (settled 7a58b582-9…) — S6

Attention budget reached 6 source(s); stopping gap expansion.

Final check — "What open models for tool-using AI agents have been evaluate…": 80% assessed by S1, S2, S3

Final check — "What dimensions are commonly used to compare open models for…": 40% assessed by S1, S2, S4

Final check — "What evidence exists about the performance of open models on…": 70% assessed by S1, S2

Final check — "What evidence exists about the limitations of open models fo…": 50% assessed by S1, S3

Final check — "What evidence exists about the training data or fine-tuning …": 60% assessed by S2

Final check — "What evidence exists about the safety or reliability of open…": 40% assessed by S2, S3

Final coverage assessment — The gathered sources provide limited direct evidence about open models for tool-using AI agents. S1 (Vicki Boykis) describes personal experience with local models like Mistral 7B, Gemma 3, OpenAI OSS-20B, Qwen 3 MOE, and Qwen 2.5 Coder, and mentions agentic coding with Gemma 4 at ~75% accuracy/speed of frontier models. S2 (Microsoft Research) reports a training pipeline using Qwen3.5-9B with mini-SWE-agent, improving SWE-bench Verified Pass@1 from 41.8% to 56.4%. S3 (Docker) discusses trust and packaging for agents, mentioning Hermes running in Docker Sandboxes. S4 (Chip Huyen) provides a taxonomy of open source AI tools but no specific model evaluations. S5 and S6 are abstracts with no relevant content. No source systematically compares multiple open models on tool-use benchmarks or discusses training data, fine-tuning methods, safety, or reliability in depth. The assessment does not establish a complete supported answer for every requested part.

Synthesizing a grounded answer from 6 source(s)…

Relevance review returned; only checked excerpts can retain support, and review cannot raise it.

Delivering 11 summary sentence(s), each tied to one checked verbatim excerpt; complete synthesis remains unverified.

Source-matched public excerpt (no creator reward) — S1, research target 1, proposed support 80% (estimate, not entailment): “I ve so far been using gemma-4-26b-a4b LM Studio implementation as my default local model.”

Source-matched public excerpt (no creator reward) — S1, research target 1, proposed support 70% (estimate, not entailment): “Gemma-4-12b-qat just came out but I ve already also really been impressed with its performance relative to its size.”

Source-matched public excerpt (no creator reward) — S2, research target 2, proposed support 90% (estimate, not entailment): “Coding agents such as mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex each bring their own context management, tool protocols, e…”

Source-matched public excerpt (no creator reward) — S2, research target 3, proposed support 90% (estimate, not entailment): “Data-efficient training recipe: an end-to-end coding agent pipeline raised Qwen3.5-9B from 41.8% to 56.4% Pass@1 on SWE-bench Verified, a 14…”

Source-matched public excerpt (no creator reward) — S1, research target 3, proposed support 80% (estimate, not entailment): “But with the most recent releases from Google in the Gemma 4 , family, I ve finally been able to do agentic coding locally and have loops wo…”

Source-matched public excerpt (no creator reward) — S1, research target 4, proposed support 80% (estimate, not entailment): “Early on, models were slow, hard to use, and just not that accurate for most programming tasks.”

Source-matched public excerpt (no creator reward) — S1, research target 4, proposed support 70% (estimate, not entailment): “But, the larger story for me is that these kinds of tasks, even as simple as they are, used to be impossible for local models as recently as…”

Source-matched public excerpt (no creator reward) — S2, research target 5, proposed support 90% (estimate, not entailment): “Data-efficient training recipe: an end-to-end coding agent pipeline raised Qwen3.5-9B from 41.8% to 56.4% Pass@1 on SWE-bench Verified, a 14…”

Source-matched public excerpt (no creator reward) — S2, research target 5, proposed support 90% (estimate, not entailment): “The training set holds about 6,000 samples and needs no large-scale compute.”

Source-matched public excerpt (no creator reward) — S3, research target 6, proposed support 80% (estimate, not entailment): “The challenge is building enough trust into the system to give agents more freedom without giving them access to everything.”

Source-matched public excerpt (no creator reward) — S3, research target 6, proposed support 70% (estimate, not entailment): “Sandboxes provide a deterministic boundary.”

Prepared a sentence-cited summary from 3 source(s)

Confidence: Low — Each summary sentence is tied to a verbatim excerpt and model-checked; completeness and independent factual correctness remain unverified. Evidence assessment: the final assessment does not establish a complete supported answer for every requested part.

Vicki Boykis contributed 40% - free public reference; reward share withheld

Microsoft Research contributed 40% - free public reference; reward share withheld

Docker contributed 20% - free public reference; reward share withheld

Keryx operating fee withheld: reviewed authority, unclaimed-source proof or funding capacity is unavailable. The answer remains available.

Done. Spent $0.005 across 1 confirmed/simulated payment(s) to creators.