Back

ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

Yujia QinShi LiangYining YeKunlun ZhuLan YanYaxi LuYankai LinXin CongXiangru TangBill Qian

article2023ICLR2,246 citations

Introduces an instruction-tuning framework and a benchmark of over 16,000 real-world APIs that enables open-source language models to execute complex multi-tool tasks and achieve tool-use performance competitive with proprietary systems like ChatGPT.

Listen

Open-source large language models have advanced rapidly in basic language tasks but remain deficient in tool-use capabilities, such as interacting with external software tools and application programming interfaces (APIs) to solve complex, multi-step human instructions. While leading proprietary models possess strong tool-use abilities, their closed-source nature restricts community innovation and broader technological democratization. Addressing this capability gap is increasingly critical as software automation and artificial intelligence integration demand open, adaptable systems that can reliably connect with external applications.

The article introduces and evaluates ToolLLM, a comprehensive open-source framework designed to teach models how to master thousands of real-world APIs across single-tool and complex multi-tool scenarios. To achieve this, the researchers gathered 16,464 RESTful APIs across 49 categories from the RapidAPI platform and used automated prompting methods to construct ToolBench, a dataset containing over 126,000 instruction-solution pairs. To overcome the planning limitations of conventional linear reasoning, the authors developed a novel depth-first search-based decision tree algorithm that allows models to explore multiple reasoning paths and backtrack from faulty steps. They fine-tuned an open-source model into ToolLLaMA and integrated a neural API retriever to automatically recommend relevant tools for any given user query.

Key findings show that ToolLLaMA, when powered by the decision tree algorithm, achieves a 66.7% average pass rate and a 60.0% win rate against proprietary baseline methods, outperforming prominent models such as Claude-2 and Text-Davinci-003 while performing on par with ChatGPT. The depth-first search decision tree substantially outperformed standard linear reasoning, raising average ChatGPT pass rates from 35.3% to 63.8% and proving especially vital for difficult, multi-tool instructions. Furthermore, when coupled with the neural API retriever, ToolLLaMA slightly improved overall pass rates to 67.3% by identifying superior alternative tools from the 16,000-plus pool. Finally, the model exhibited robust out-of-distribution generalization on the independent APIBench benchmark, matching or exceeding models specifically trained on those domains.

These findings demonstrate that open-source models can achieve state-of-the-art tool-use competencies without expensive human data annotation, significantly lowering technical and operational barriers to deploying autonomous agent systems. The success of the retriever and decision-tree architecture shows that practical tool use requires both effective discovery across massive software libraries and flexible, multi-step error recovery rather than rigid, single-path reasoning.

Organizations aiming to implement tool-augmented AI agents should adopt tree-based search strategies and neural retrieval pipelines rather than relying exclusively on proprietary closed-source models. Future work should focus on refining automatic evaluation frameworks for complex reasoning paths and expanding API validation mechanisms to handle real-world API latency, instability, and dynamic environment changes.

ModelBest Inc.Renmin University of ChinaTencentTsinghua UniversityYale UniversityZhihularge language modelsreasoningdecision-tree algorithmAPIAPIsName
arXiv: 2307.16789OpenBMB/ToolBench
Cover for ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

Abstract

Despite the advancements of open-source large language models (LLMs), e.g., LLaMA, they remain significantly limited in tool-use capabilities, i.e., using external tools (APIs) to fulfill human instructions. The reason is that current instruction tuning largely focuses on basic language tasks but ignores the tool-use domain. This is in contrast to the excellent tool-use capabilities of state-of-the-art (SOTA) closed-source LLMs, e.g., ChatGPT. To bridge this gap, we introduce ToolLLM, a general tool-use framework encompassing data construction, model training, and evaluation. We first present ToolBench, an instruction-tuning dataset for tool use, which is constructed automatically using ChatGPT. Specifically, the construction can be divided into three stages: (i) API collection: we collect 16,464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub; (ii) instruction generation: we prompt ChatGPT to generate diverse instructions involving these APIs, covering both single-tool and multi-tool scenarios; (iii) solution path annotation: we use ChatGPT to search for a valid solution path (chain of API calls) for each instruction. To enhance the reasoning capabilities of LLMs, we develop a novel depth-first search-based decision tree algorithm. It enables LLMs to evaluate multiple reasoning traces and expand the search space. Moreover, to evaluate the tool-use capabilities of LLMs, we develop an automatic evaluator: ToolEval. Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction. Experiments show that ToolLLaMA demonstrates a remarkable ability to execute complex instructions and generalize to unseen APIs, and exhibits comparable performance to ChatGPT. Our ToolLLaMA also demonstrates strong zero-shot generalization ability in an out-of-distribution tool-use dataset: APIBench.

Topics

large language modelsreasoningdecision-tree algorithmautomatic evaluationtuning methodcognitive processsearch processstatistical language models

Keywords

APIAPIsNameToolInstructionInstructionsChatGPTSolution PathDFSDTLanguage

Table of Contents

  • 1 Introduction
  • 2 Dataset Construction
  • 2.1 API Collection
  • 2.2 Instruction Generation
  • 2.3 Solution Path Annotation
  • 3 Experiments
  • 3.1 Preliminary Experiments
  • 3.2 Main Experiments
  • 3.3 Out-of-Distribution (OOD) Generalization to APIBench (Patil et al., 2023)
  • 4 Related Work
  • 5 Conclusion
  • References
  • A Implementation Details
  • A.1 Details for Filtering RapidAPI
  • A.2 API Response Compression
  • A.3 Details for Training ToolLLaMA
  • A.4 Details for DFSDT
  • A.5 Details for ToolEval
  • A.6 Details for Experiments on APIBench
  • A.7 Prompts for Instruction Generation
  • A.8 Prompts for Solution Path Annotation

Knowls

  1. Knowl 1 — ToolBench Instruction-Tuning Dataset and Three-Stage Construction Framework

    model/method

    ToolBench is an instruction-tuning dataset designed for tool learning and API execution in large language models (LLMs). It encompasses 16,464 RESTful APIs spanning 49 categories and 3,451 unique tools collected from RapidAPI Hub. The automated dataset construction pipeline consists of three sequential phases:

    1. API Collection and Filtering: API metadata is scraped from RapidAPI, including functionality descriptions, host URLs, HTTP methods, required and optional parameters, request bodies, and code execution examples. Non-functional, unstable, slow-responding, or error-returning APIs (e.g., returning HTTP 404 or raw HTML error pages) are pruned to retain 3,451 functional tools.

    2. Instruction Generation: ChatGPT (gpt-3.5-turbo-16k) is prompted with sampled API documentation and human-written in-context seed examples to generate diverse, multi-step queries that require API calls without explicitly mentioning API names. Three scenarios are sampled to avoid tool sparsity:

      • Single-tool instructions (I1I_1): Prompts generated using multiple endpoints within a single tool (87,413 instances).
      • Intra-category multi-tool instructions (I2I_2): Prompts requiring 2 to 5 tools randomly selected from the same high-level RapidAPI category, sampling at most 3 APIs per tool (84,815 instances).
      • Intra-collection multi-tool instructions (I3I_3): Prompts requiring 2 to 5 tools from the same fine-grained RapidAPI collection (25,251 instances). Instructions containing hallucinated APIs not in the sampled candidate subset are filtered out.

    3. Solution Path Annotation: ChatGPT is prompted to interact with the real API execution environment via a Depth-First Search-based Decision Tree (DFSDT) strategy. API responses exceeding 1,024 tokens are compressed using ChatGPT key-selection schemas or truncated to 1,024 tokens. Only successful solution paths are retained, yielding 126,486 valid (instruction, solution path) pairs encompassing 469,585 real API calls.

  2. Knowl 2 — Depth-First Search-Based Decision Tree Algorithm for Tool Reasoning

    algorithm

    The Depth-First Search-based Decision Tree (DFSDT) algorithm is a planning and decision-making strategy for tool-augmented language models. Standard reasoning methods such as ReACT (Reasoning + Acting) follow a single linear trajectory where early errors or API failures trigger cascading hallucinations. DFSDT expands the reasoning search space into a tree structure, allowing the model to evaluate paths, backtrack from failed branches, and explore alternative API calls.

    To eliminate the O(nlog⁡n)O(n \log n) LLM scoring complexity required by classical DFS node sorting, DFSDT implements a pre-order traversal variant. The model executes actions greedily until a branch fails, at which point it invokes a specialized pseudo-action Finish: give_up_and_restart to backtrack. When expanding an alternative child node from an existing state, a diversity prompt containing the history of previously attempted and failed actions at that state is fed to the model to force exploration of distinct actions.

    Input: User instruction InstInst, Set of candidate API documentations SAPIsubS_{API}^{sub}, Maximum search depth DmaxD_{max}
    Output: Valid execution trajectory P=[(a1,r1),(a2,r2),… ]P = [(a_1, r_1), (a_2, r_2), \dots] or failure indication
    Initialize search tree root node v0v_0 with state containing (Inst,SAPIsub)(Inst, S_{API}^{sub})
    Initialize traversal stack Stack←[v0]Stack \leftarrow [v_0]
    while StackStack is not empty do
        v←pop(Stack)v \leftarrow \text{pop}(Stack)
        if depth(v)≥Dmax\text{depth}(v) \ge D_{max} then
            continue
        end if
        Retrieve previous failed child actions AprevA_{prev} generated at node vv
        Prompt LLM with context history of vv and diversity constraint against AprevA_{prev}
        a←LLM.generate_action(v,Aprev)a \leftarrow \text{LLM.generate\_action}(v, A_{prev})
        Record action aa in AprevA_{prev} for node vv
        if aa is "Finish: give_answer" then
            return Path from root v0v_0 to vv including final response
        else if aa is "Finish: give_up_and_restart" then
            Mark current branch as abandoned (backtrack)
        else
            r←Execute_API_Call(a)r \leftarrow \text{Execute\_API\_Call}(a)
            Create child node vchildv_{child} with state updated by action aa and response rr
            Push vv to StackStack (retaining branch point for subsequent backtracking)
            Push vchildv_{child} to StackStack
        end if
    end while
    return Failure (no valid solution path found)

  3. Knowl 3 — ToolEval Automatic Evaluation System for Tool Learning

    model/method

    ToolEval is an automated evaluation framework backed by ChatGPT (gpt-3.5-turbo-16k) designed to assess the tool-use capabilities of large language models across dynamic, non-deterministic RESTful API environments without requiring static ground-truth answer strings. ToolEval defines two primary metrics:

    1. Pass Rate: Measures the model's ability to successfully resolve an instruction within execution budgets. Instructions are partitioned by the evaluator into solvable (at least one provided tool can address the task) or unsolvable (tools are irrelevant or required user input parameters are invalid):

      • For solvable instructions, the trajectory passes if the final answer completely resolves the query or if the model gives up/refuses only after exhaustively testing the APIs and discovering they lack required data. It fails if the model gives up prematurely without sufficient API exploration, hallucinates answers without API support, or refuses despite valid API data being available.
      • For unsolvable instructions, the trajectory passes if the model correctly refuses or gives up, and fails if the model hallucinates an answer. Pass status is decided by majority vote across ≥4\ge 4 independent evaluations.

    2. Win Rate: Evaluates the relative quality and utility between two candidate solution paths for the same instruction. It compares paths annotated with identical pass status (both passed or both failed) across six criteria:

      • Information richness: Completeness of information answering the prompt.
      • Factuality: Correct description of executed steps and observed errors.
      • Reasoning: Clarity and precision of failure explanations when unresolved.
      • Milestones: Number of intermediate task objectives achieved.
      • Exploration: Breadth of potentially useful APIs attempted.
      • Cost: Minimization of redundant or duplicate API calls.

    ToolEval scores achieve an 87.1% agreement rate in Pass Rate and an 80.3% agreement rate in Win Rate compared with human expert annotations.

  4. Knowl 4 — Dense Neural API Retriever for Large-Scale Tool Selection

    model/method

    The Neural API Retriever is a dense dual-encoder model designed to recommend relevant APIs from a corpus of over 16,000 candidates given an arbitrary natural language user instruction qq.

    The retriever utilizes a BERT-Base backbone trained with a Sentence-BERT siamese architecture. The instruction qq and each API document dd (containing tool description, endpoint name, description, and parameter specifications) are mapped into dense vectors eq,ed∈R768e_q, e_d \in \mathbb{R}^{768}. The relevance score between instruction and API is computed via cosine similarity: Score(q,d)=eq⋅ed∥eq∥∥ed∥\text{Score}(q, d) = \frac{e_q \cdot e_d}{\|e_q\| \|e_d\|}

    The model is trained with contrastive learning using the generated instructions from ToolBench as queries, their assigned ground-truth relevant APIs as positive keys, and in-batch / sampled unrelated APIs as negative keys. At inference time, the retriever returns the top-kk (e.g., k=5k=5) highest-scoring candidate APIs to be placed into the context window of the language model for tool execution.

  5. Knowl 5 — ToolLLaMA Model Architecture and Fine-Tuning Setup

    experimental setup

    ToolLLaMA is a 7B-parameter open-source tool-augmented language model initialized from LLaMA-2 7B and fine-tuned on the 126,486 execution trajectories of ToolBench.

    To accommodate verbose API documentation and long REST HTTP payload responses, the context window of LLaMA-2 7B is expanded from 4,096 tokens to 8,192 tokens using positional interpolation with a scaling ratio of 2.

    The training data is organized as multi-round conversational sequences where candidate API documentation is structured as function definitions concatenated directly into the system prompt. Training hyperparameters are:

    • Base model: LLaMA-2 7B
    • Maximum sequence length: 8,192
    • Learning rate: 5×10−55 \times 10^{-5}
    • Learning rate warmup ratio: 0.040.04
    • Total batch size: 64
    • Training epochs: 2

    The final model checkpoint is selected based on task performance on a held-out development set.

  6. Knowl 6 — ToolBench Main Evaluation: Pass Rate and Win Rate Across Generalization Levels

    data/table

    Performance of various open-source and proprietary LLMs evaluated on ToolBench test splits across three instruction formats: single-tool (I1I_1), intra-category multi-tool (I2I_2), and intra-collection multi-tool (I3I_3). Models are tested across three generalization levels: unseen instructions for seen tools (Inst.), unseen tools within seen categories (Tool), and unseen tools within unseen categories (Cat.). Win rates are computed pairwise against ChatGPT-ReACT, where tie fractions are split equally between win and lose.

    Model Method I1-Inst. I1-Tool I1-Cat. I2-Inst. I2-Cat. I3-Inst. Average
    Pass Win Pass Win Pass Win Pass Win Pass Win Pass Win Pass Win
    ChatGPT ReACT 41.5 - 44.0 - 44.5 - 42.5 - 46.5 - 22.0 - 40.2 -
    ChatGPT DFSDT 54.5 60.5 65.0 62.0 60.5 57.3 75.0 72.0 71.5 64.8 62.0 69.0 64.8 64.3
    Claude-2 ReACT 5.5 31.0 3.5 27.8 5.5 33.8 6.0 35.0 6.0 31.5 14.0 47.5 6.8 34.4
    Claude-2 DFSDT 20.5 38.0 31.0 44.3 18.5 43.3 17.0 36.8 20.5 33.5 28.0 65.0 22.6 43.5
    Text-Davinci-003 ReACT 12.0 28.5 20.0 35.3 20.0 31.0 8.5 29.8 14.5 29.8 24.0 45.0 16.5 33.2
    Text-Davinci-003 DFSDT 43.5 40.3 44.0 43.8 46.0 46.8 37.0 40.5 42.0 43.3 46.0 63.0 43.1 46.3
    GPT-4 ReACT 53.5 60.0 50.0 58.8 53.5 63.5 67.0 65.8 72.0 60.3 47.0 78.0 57.2 64.4
    GPT-4 DFSDT 60.0 67.5 71.5 67.8 67.0 66.5 79.5 73.3 77.5 63.3 71.0 84.0 71.1 70.4
    Vicuna ReACT DFSDT 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    Alpaca ReACT DFSDT 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    ToolLLaMA ReACT 25.0 45.0 29.0 42.0 33.0 47.5 30.5 50.8 31.5 41.8 25.0 55.0 29.0 47.0
    ToolLLaMA DFSDT 57.0 55.0 61.0 55.3 62.0 54.5 77.0 68.5 77.0 58.0 66.0 69.0 66.7 60.0
    ToolLLaMA DFSDT-Retriever 64.0 62.3 64.0 59.0 60.5 55.0 81.5 68.5 68.5 60.8 65.0 73.0 67.3 63.1

    Key takeaways:

    • Standard open-source instruction-tuned models (Vicuna, Alpaca) completely fail to execute tool operations (0.0% pass rate).
    • DFSDT consistently outperforms ReACT across all model families, boosting ChatGPT's pass rate from 40.2% to 64.8% and ToolLLaMA's from 29.0% to 66.7%.
    • ToolLLaMA-DFSDT surpasses Claude-2 and Text-Davinci-003, performing on par with ChatGPT (66.7% vs. 64.8% pass rate).
    • Equipping ToolLLaMA with the Neural API Retriever (ToolLLaMA-DFSDT-Retriever) achieves a higher overall pass rate (67.3%) and win rate (63.1%) than using the oracle API set, as the retriever identifies functional substitute APIs with higher utility.

  7. Knowl 7 — Dense API Retriever Evaluation Results

    data/table

    Evaluation of candidate API retrieval methods across three instruction subsets of ToolBench: single-tool (I1I_1), intra-category multi-tool (I2I_2), and intra-collection multi-tool (I3I_3). Performance is evaluated using Normalized Discounted Cumulative Gain at ranks 1 and 5 (NDCG@1 and NDCG@5).

    Method I1 I2 I3 Average
    NDCG@1 NDCG@5 NDCG@1 NDCG@5 NDCG@1 NDCG@5 NDCG@1 NDCG@5
    BM25 18.4 19.7 12.0 11.0 25.2 20.4 18.5 17.0
    Ada 57.5 58.8 36.8 30.7 54.6 46.8 49.6 45.4
    Ours (Retriever) 84.2 89.7 68.2 77.9 81.7 87.1 78.0 84.9

    The trained BERT-Base dense retriever substantially outperforms lexical retrieval (BM25) and dense embeddings from OpenAI's text-embedding-ada-002 (Ada), achieving an average NDCG@5 of 84.9% compared to 45.4% for Ada and 17.0% for BM25 across 16,464 candidate APIs.

  8. Knowl 8 — Pass Rate Improvement of DFSDT Over ReACT

    data/table

    Comparison of pass rates achieved by ChatGPT under different reasoning and search strategies across single-tool (I1I_1), intra-category multi-tool (I2I_2), and intra-collection multi-tool (I3I_3) instructions. ReACT@N conducts multiple independent ReACT runs until the total number of consumed API calls matches the computational cost of DFSDT, recording a pass if any attempt succeeds.

    Method I1 I2 I3 Average
    ReACT 37.8% 40.6% 27.6% 35.3%
    ReACT@N 49.4% 49.4% 34.6% 44.5%
    DFSDT 58.0% 70.6% 62.8% 63.8%

    DFSDT achieves an average pass rate of 63.8%, substantially outperforming both single-trajectory ReACT (35.3%) and cost-matched repeated sampling ReACT@N (44.5%). The performance margin is largest on complex multi-tool tasks (I2I_2 and I3I_3), demonstrating that structured tree search with backtracking and state-conditioned action diversification discovers valid solutions that cannot be reached by repeated linear execution.

  9. Knowl 9 — Zero-Shot Out-of-Distribution Generalization on APIBench

    data/table

    Zero-shot out-of-distribution (OOD) performance of ToolLLaMA evaluated on APIBench across three machine learning tool ecosystems: HuggingFace, TorchHub, and TensorHub. ToolLLaMA is evaluated without any fine-tuning on APIBench, and is compared against Gorilla (LLaMA-7B fine-tuned directly on APIBench data) under zero-shot (ZS) and retrieval-aware (RS) training setups using BM25 or Oracle retrievers. Metrics reported are Hallucination Rate (Hallu., ↓\downarrow) and Abstract Syntax Tree accuracy (AST, ↑\uparrow).

    Method HuggingFace TorchHub TensorHub
    Hallu. () AST () Hallu. () AST () Hallu. () AST ()
    ToolLLaMA + Our Retriever 10.60% 16.77% 15.70% 51.16% 6.48% 40.59%
    Gorilla-ZS + BM25 46.90% 10.51% 17.20% 44.62% 20.58% 34.31%
    Gorilla-RS + BM25 6.42% 15.71% 5.91% 50.00% 2.77% 41.90%
    ToolLLaMA + Oracle 8.66% 88.80% 14.12% 85.88% 7.44% 88.62%
    Gorilla-ZS + Oracle 52.88% 44.36% 39.25% 59.14% 12.99% 83.21%
    Gorilla-RS + Oracle 6.97% 89.27% 6.99% 93.01% 2.04% 94.16%

    Despite never seeing APIBench tools or formats during fine-tuning, ToolLLaMA paired with its neural API retriever outperforms Gorilla-ZS+BM25 and Gorilla-RS+BM25 on HuggingFace and TorchHub AST accuracy. Under oracle retrieval, ToolLLaMA achieves 88.80% (HuggingFace), 85.88% (TorchHub), and 88.62% (TensorHub) AST accuracy, performing on par with domain-specifically trained Gorilla-RS.

  10. Knowl 10 — Evaluation Ambiguity from Solution Path Multiplicity

    limitation

    A fundamental limitation in evaluating tool-augmented language models is the multiplicity of valid execution paths for complex instructions. Unlike standard natural language generation where ground-truth references can be compared directly, tool-use tasks often admit infinitely many valid execution trajectories (e.g., executing a concise sequence using minimal API calls versus performing extensive calls across redundant APIs for cross-validation).

    Human experts frequently disagree on which path is qualitatively superior, which limits inter-annotator win rate agreement (observed at 80.3%) and constrains the ceiling of automated LLM-based preference evaluation.

Coverage note — None was omitted; all key contributions including dataset construction, DFSDT algorithm, ToolEval, neural API retriever, main benchmark results, and OOD generalization are covered.

References

  1. 1.Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. ArXiv preprint, abs/2204.01691, 2022.
  2. 2.Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. Promptsource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 93–104, 2022.
  3. 3.Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  4. 4.Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023.
  5. 5.Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
  6. 6.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.
  7. 7.Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023.
  8. 8.Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306.08640, 2023.
  9. 9.Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953–14962, 2023.
  10. 10.Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arXiv:2305.11554, 2023.
  11. 11.Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118–9147. PMLR, 2022a.
  12. 12.Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. ArXiv preprint, abs/2207.05608, 2022b.
  13. 13.Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446, 2002.
  14. 14.Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. Genegpt: Augmenting large language models with domain tools for improved access to biomedical information. ArXiv, 2023.
  15. 15.Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244, 2023a.
  16. 16.Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023b.
  17. 17.Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470–3487, 2022.
  18. 18.Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. ArXiv preprint, abs/2112.09332, 2021.
  19. 19.OpenAI. OpenAI: Introducing ChatGPT, 2022. URL https://openai.com/blog/chatgpt.
  20. 20.OpenAI. Gpt-4 technical report, 2023.
  21. 21.Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.
  22. 22.Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023.
  23. 23.Cheng Qian, Chi Han, Yi R Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. Creator: Disentangling abstract and concrete reasonings of large language models through tool creation. arXiv preprint arXiv:2305.14318, 2023.
  24. 24.Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, et al. Webcpm: Interactive web search for chinese long-form question answering. arXiv preprint arXiv:2305.06849, 2023a.
  25. 25.Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint arXiv:2304.08354, 2023b.
  26. 26.Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.
  27. 27.Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009.
  28. 28.Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. ArXiv preprint, abs/2302.04761, 2023.
  29. 29.Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface, 2023.
  30. 30.Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
  31. 31.Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Restgpt: Connecting large language models with real-world applications via restful apis. arXiv preprint arXiv:2306.06624, 2023.
  32. 32.Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023.
  33. 33.Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
  34. 34.Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  35. 35.Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  36. 36.Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Technical Report MSR-TR-2023-8, Microsoft, February 2023.
  37. 37.Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
  38. 38.Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
  39. 39.Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
  40. 40.Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. ArXiv preprint, abs/2303.04671, 2023.
  41. 41.Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023a.
  42. 42.Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504, 2023b.
  43. 43.Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. Chatgpt is not enough: Enhancing large language models with knowledge graphs for fact-aware language modeling. arXiv preprint arXiv:2306.11489, 2023.
  44. 44.Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv preprint, abs/2210.03629, 2022.
  45. 45.Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023.
  46. 46.Yining Ye, Xin Cong, Yujia Qin, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Large language model as autonomous decision maker. arXiv preprint arXiv:2308.12519, 2023.
  47. 47.Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools. arXiv preprint arXiv:2306.13304, 2023.

Citation

MLA
Qin, Y., et al. “ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs”. arXiv, 2023, http://arxiv.org/abs/2307.16789v2.
APA
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Hong, L., Tian, R., Xie, R., Zhou, J., Gerstein, M., Li, D., Liu, Z., & Sun, M. (2023). ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv. http://arxiv.org/abs/2307.16789v2
Chicago
Qin, Y., S. Liang, Y. Ye, et al. 2023. “ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs”. arXiv. http://arxiv.org/abs/2307.16789v2.
Harvard
Qin, Y. et al. (2023) “ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2307.16789v2.
Vancouver
1. Qin Y, Liang S, Ye Y, et al (2023) ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv

BibTeX

@article{qin2023toolllm,
  title = {ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs},
  author = {Qin, Yujia and Liang, Shihao and Ye, Yining and Zhu, Kunlun and Yan, Lan and Lu, Yaxi and Lin, Yankai and Cong, Xin and Tang, Xiangru and Qian, Bill and Zhao, Sihan and Hong, Lauren and Tian, Runchu and Xie, Ruobing and Zhou, Jie and Gerstein, Mark and Li, Dahai and Liu, Zhiyuan and Sun, Maosong},
  year = {2023},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2307.16789v2},
  eprint = {2307.16789}
}
Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors