Fixing Tool-Call Accuracy in vLLM

Tool call accuracy is an important part of serving open-weight models. When your tool calls aren’t accurate, the model builds up on nonsensical context. This leads to low quality results, particularly for agentic workloads where multi-turn sessions are common.

Tool-call Schemas

You can enable an LLM to call an external tool in two steps: (1) inform the AI model about the existence of externally available tool calls and (2) provide the model with instructions on how to output structured tool-calls. For example, when connecting OpenCode to Kimi, OpenCode sends Kimi a JSON array describing available tools. One of the tools in this list is webfetch, which fetches Web page contents:

[{
  "type": "function",
  "function": {
    "name": "webfetch",
    "description": "- Fetches content from a specified URL\n- Takes a URL and optional format as input\n- Fetches the URL content, converts to requested format (markdown by default)\n- Returns the content in the specified format\n- Use this tool when you need to retrieve and analyze web content\n\nUsage notes:\n  [content redacted for brevity]\n",
    "parameters": {
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "type": "object",
      "properties": {
        "url": {
          "type": "string",
          "description": "The URL to fetch content from"
        },
        "format": {
          "type": "string",
          "enum": [
            "text",
            "markdown",
            "html"
          ],
          "description": "The format to return the content in (text, markdown, or html). Defaults to markdown.",
          "default": "markdown"
        },
        "timeout": {
          "type": "number",
          "description": "Optional timeout in seconds (max 120)"
        }
      },
      "required": [
        "url"
      ]
    }
  }
}, ...]

The inference engine then wraps this JSON description in LLM-specific tokens before passing it into the LLM, to match how the LLM was trained to recognize tool declarations. When serving Kimi, this looks something like:

<|im_system|>tool_declare<|im_middle|>[{"type":"function","function":{...}, ...]<|im_end|>

The JSON object describing the tool (or function, in the JSON above) includes the tool’s parameters, their types, which of the parameters are required, etc. This is known as the tool’s schema. For an agent harness (e.g., OpenCode) to successfully execute a tool call generated by the LLM, the call must follow the rules specified in its schema.

I'll fetch that page. <|tool_calls_section_begin|><|tool_call_begin|>functions.webfetch:0<|tool_call_argument_begin|>{"url": "https://ubicloud.com", "format": "markdown"}<|tool_call_end|><|tool_calls_section_end|>

These tokens are parsed by the inference server’s parser, extracting the LLM-specific tokens and turning the call into a JSON object such as:

{
  "role": "assistant",
  "content": "I'll fetch that page.",
  "tool_calls": [
    {
      "id": "functions.webfetch:0",
      "type": "function",
      "function": {
        "name": "webfetch",
        "arguments": "{\"url\": \"https://example.com\", \"format\": \"markdown\"}"
      }
    }
  ]
}

This is then sent back to your harness (OpenCode, in our example), which runs its webfetch tool and sends its output back to the model.

The Importance of Schema Accuracy

For a tool-call to succeed, the call must adhere to the tool’s schema.

Constrained Decoding

# Kimi K2.6 tool call syntax:
# <|tool_call_begin|>{function.name}:{index}
# <|tool_call_argument_begin|>{arguments}
# <|tool_call_end|>
TagFormat(
    begin=f"{TOOL_CALL_BEGIN_PREFIX}{name}{TOOL_CALL_SUFFIX}",
    content=SequenceFormat(
        elements=[
            RegexFormat(pattern=r"\d+"),
            ConstStringFormat(value=TOOL_CALL_ARGUMENT_BEGIN),
            JSONSchemaFormat(
                json_schema=parameters,
                any_order=any_order,
                max_whitespace_cnt=max_whitespace_cnt,
            ),
        ]
    ),
    end=TOOL_CALL_END,
)

Differences in Constrained-Decoding Behavior

Conclusion

When using LLMs, the quality you observe isn’t based on the model alone. The inference engine also plays an important role. Change the inference engine and its settings, and you change the model’s output quality.