Joint Optimization of Tool Creation and Use for Large Language Model Agents

SMITH: Schema-grounded Multi-task Iterative Tool Honing

SMITH is a multi-task RL framework, trained with DAPO (a clip-higher variant of GRPO), that mixes two rollout types into every batch: build and use. Both are optimized inside the same policy, so gradients from tool creation and tool consumption update the same weights every step.

def solve(question: str) -> str:
    n = int(re.search(r"\d+", question).group())
    return str(bin(n).count("1"))

{
  "name": "solve",
  "description": "Counts the 1-bits in the
    binary form of the integer named in
    the question.",
  "parameters": {
    "type": "object",
    "properties": {
      "question": { "type": "string" }
    },
    "required": ["question"]
  }
}

Three reward axes, kept separate on purpose

Rather than collapsing everything into one score, SMITH keeps format, execution, and judge feedback as independent axes fed to DAPO, so each failure mode gets its own gradient instead of being averaged away.

Use-task correctness, with an efficiency penalty

Let \(c \in \{0,1\}\) mark whether the final answer matches ground truth. The reward is scaled by an efficiency multiplier \(\eta(\rho)\) that decays as the turn fraction \(\rho = \min(n/T, 1)\) grows, with a floor \(\eta_{\min}=0.3\) so the policy is never indifferent to correctness even at the turn limit (\(\eta_{\mathrm{mid}} = 0.7\)):

$$ r^{\mathrm{correct}} = 2c\,\eta(\rho), \qquad \eta(\rho) = \begin{cases} 1 - 2(1-\eta_{\mathrm{mid}})\rho & \rho \le 0.5 \\[2pt] \max\!\bigl(\eta_{\min},\ \eta_{\mathrm{mid}}\,(1-2(\rho-0.5))^2\bigr) & \rho > 0.5 \end{cases} $$

Every training batch is split evenly, \(\mathcal{B} = \mathcal{B}_{\mathrm{build}} \sqcup \mathcal{B}_{\mathrm{use}}\) with \(|\mathcal{B}_{\mathrm{build}}| = |\mathcal{B}_{\mathrm{use}}| = B/2\), and DAPO accumulates both losses in a single backward pass: \(\mathcal{L} = \mathcal{L}_{\mathrm{DAPO}}(\mathcal{B}_{\mathrm{build}}) + \mathcal{L}_{\mathrm{DAPO}} (\mathcal{B}_{\mathrm{use}})\). Build and use gradients therefore update the same parameters \(\theta\) every step. Any tool with \(r^{\mathrm{eval}} > 0\) is pushed into a shared Tool Pool for reuse by future use-task rollouts. This is the "iterative honing" in SMITH's name.

Results

Main results on Reasoning-Gym

SMITH is the only method that leads on genuinely held-out tasks while also using the fewest tokens. Against distillation, a 4B model trained with SMITH's RL objective generalizes more reliably than 4B models distilled from far larger oracles: ReTool leads on seen tasks (92.2) but drops nearly 30 points on unseen ones, a sign of overfitting to the demonstrator's distribution rather than learning a transferable build-and-use policy. Against more elaborate inference-time scaffolds (CRAFT, TroVE, KTCE) and a bigger tool-writer (Qwen3-30B-A3B), SMITH still wins on unseen tasks, using roughly 32× fewer output tokens than standard chain-of-thought.

Tools transfer across model scale

Do SMITH's tools encode a genuinely reusable solution, or a private convention only the writer understands? We test both directions: pairing the RL-trained 4B writer with a much smaller consumer, and with a much larger one.

Pairing tools from the RL-trained 4B writer with a frozen LFM2.5-350M consumer lifts its held-out accuracy from 11.6 to 42.9, matching a tool-writer eighty times larger (Qwen3-30B-A3B, untrained, at 41.5).

RG (Unseen) accuracy of LFM2.5-350M as the tool-writing source changes; the writer itself is never fine-tuned on LFM2.5's outputs.

The reverse direction also holds. Pairing SMITH's 4B-written tools with a much stronger Qwen3-30B-A3B consumer beats that same 30B model writing tools for itself (LATM*) on every task group, most sharply on TabMWP-Hard (0.7 → 38.8), lifting the task-weighted overall score from 70.2 to 76.6. A bigger tool user doesn't make its own self-written tool preferable. SMITH's 4B writer remains the better source of tools either way.

Out-of-domain generalization

Neither TabMWP-Hard (tabular reasoning) nor GQA (visual question answering) appears anywhere in SMITH's training data. SMITH leads TabMWP-Hard outright and is the only 4B, non-distilled method to top either column. The GQA gap to GPT-4.1-distilled LATM is the acknowledged cost of not distilling visual primitives from a stronger oracle.

Scaling across backbones

The same RL recipe, applied unmodified, improves every backbone tested, including Granite-3.3-8B, which starts from a much weaker base. A Self-Judge variant (Qwen3-8B judging its own rollouts instead of an external 30B-A3B judge) pushes held-out RG even higher (85.9) but trades off OOD GQA, suggesting a smaller self-judge is weaker but less biased on in-distribution data.

Generalization to external tool-calling

SMITH is never trained on BFCL v4's schemas, multi-turn traces, or judges, so a gain here isolates a learned tool-use prior rather than benchmark-specific fitting. It lifts BFCL overall accuracy on both Qwen backbones, most sharply on Qwen3-8B.