Jiahao Wu, Zhongwen Xu, Qiang Fu, and Wei Yang

Tencent · TEG · AIPD

December 2025

TL;DR

• Using the **rLLM [1] training framework + Qwen3-8B [2] base model + offline Wikipedia / BrowseComp-Plus [3] corpora + the synthetic multi-turn data from ASearcher [4]**, you can cost-effectively train a search agent that reliably performs 10+ retrieval turns.
• SFT is not strictly necessary: starting from the base model and doing GRPO-based [5] RL is enough to build stable multi-turn search capability. SFT can provide a better starting point for long-horizon training, but it tends to lock the model into a particular pattern and makes it overuse multi-turn search even for simple questions.
• Multi-turn search training dataset [4] is critical: standard single-hop / two-hop datasets (HotpotQA [6], etc.) do not train long-horizon multi-turn search; you need a dedicated synthetic multi-turn dataset.
• Training stability is mostly an engineering problem: the key issues are train/inference mismatch, strict token-in-token-out alignment, and handling “abnormal trajectories”.
• Summarizing retrieved documents with an auxiliary LLM [7] improves training stability, supports longer horizons, and generalizes to setups that directly return raw search results.

News

• We have uploaded the model weights to huggingface
  • https://huggingface.co/aidenjhwu/SearchAgent-8B-hq : The Qwen3-8B model trained w/ outlier suppression training strategy (introduced in Section 5.4.1)
  • https://huggingface.co/aidenjhwu/SearchAgent-8B : The Qwen3-8B model trained w/o outlier suppression training strategy
  • https://huggingface.co/aidenjhwu/SearchAgent-A3B : The Qwen3-30B-A3B model trained w/o outlier suppression training strategy

1. The Research Bottleneck for Multi-Turn Search Agents

Recently, LLM reasoning and tool-use capabilities have improved rapidly: on the one hand, the open-source community has released many strong models [2,8]; on the other hand, when moving to multi-turn, long-horizon search-and-verify settings (multiple retrievals, iterative disambiguation, evidence aggregation, and converging to a final answer), there is still a clear gap, where smaller open-source models often lag far behind closed-source commercial models in both effective use of search turns and final accuracy.

To narrow this gap, the community has proposed many effective approaches [4,7,9,10,11,12]. Most follow a pipeline of synthetic data construction → SFT alignment → RL optimization to gradually push smaller models toward more robust long-horizon search strategies. However, for individual researchers, it is still hard to use these methods as a starting point: many are not fully open-sourced, key training details are not fully disclosed, and training stability is highly sensitive to engineering details (data synthesis and cleaning, tool environment stability, concurrency and timeouts, truncation and abnormal trajectory handling, train/inference alignment, etc.). In addition, multi-stage training is harder to reproduce, and paid Web search APIs further raise iteration cost.

This post provides a researcher-oriented tutorial for deploying and training a multi-turn search agent baseline. We systematize the key engineering choices and stability tricks we found in practice, so you can train your first multi-turn search agent at lower cost and with better reproducibility on classic multi-turn search benchmarks, increasing average search turns from ~2 to 15+ and accuracy from single digits to ~30%.