Suppose you ask an LLM agent to create an issue. The response states, “Issue created,” but there is no new issue in the project. When you ask again, the agent claims it used the tool.

Before changing the model or adding prompts like “Do not lie,” check the following. How far did the execution process actually progress?

You cannot answer this from the agent's sentences alone. You must separate whether a tool call request was generated, whether the runtime handled the request, and whether results remain in the external service.

1. “Called” conflates different events

The workflow of a tool-using agent roughly follows these steps:

  1. The model selects the tool and arguments.
  2. The runtime interprets the call request and checks permissions.
  3. The executor calls the actual function or external API.
  4. The tool result is passed to the model.
  5. It determines whether the state requested by the user has been reached.

Each step can fail independently.

Even if the model generates a proper call request, an exception may occur in the executor. The API may have accepted the request, but the asynchronous task might still be running. The execution might be complete, but the issue might have been created in the wrong project.

Therefore, when investigating failures, it is better not to cover all steps with a single “tool call successful” status.

2. Distinguish plain text from call events

Consider an example of custom function calls in the OpenAI Responses API. The following is a simplified example of the key fields in a call item.

json
{
  "type": "function_call",
  "call_id": "call_42",
  "name": "create_ticket",
  "arguments": "{\"title\":\"로그인 오류 점검\"}"
}

The application receives this item, executes the corresponding function, and connects the result to the same call_id.

json
{
  "type": "function_call_output",
  "call_id": "call_42",
  "output": "{\"status\":\"created\",\"ticket_id\":\"T-731\"}"
}

Here, status and ticket_id are result formats defined by the sample application. They are not standard fields guaranteeing issue creation from OpenAI.

The difference between this structure and the following sentence in a typical response is clear.

I created the issue by calling create_ticket.

This sentence is text generated by the model. Even if you list tool names or call formats inside code blocks, execution will not occur unless the runtime is implemented to process them as call requests.

While applications execute custom functions, server-side tools operated by providers may execute in different locations. Therefore, you must first verify where the tools you use are executed and what events they leave. OpenAI function calling documentation and Claude tool use documentation explain these execution boundaries.

3. Symptoms may be identical, but causes differ

Completion statement generated without a call request

If the model output contains only a plain message without a call event, check if the model claimed completion without execution evidence.

Possible causes include the tool not being exposed to the request, ambiguous tool descriptions, or the model not choosing to call. You must first compare the tool definitions included in the actual request with the model's raw output.

However, do not definitively conclude non-execution solely because there is no record in the trace screen. You must also check for collection misses, sampling, or separate execution paths.

Call request exists, but the executor missed it

In this case, you need to examine the application's control flow rather than the model.

  • Did the application read only the text from the response and discard the call item?
  • Did it process only the first of multiple calls?
  • Do the tool name and the actual handler name match?
  • Did the execution loop end while waiting for approval?

Streaming should also be checked. If argument fragments are treated as complete requests or call completion events are missed, execution may be omitted. response.function_call_arguments.done indicates that argument generation has finished, not that the external function execution has finished.

Execution failure converted to success before delivery

Exception handling code sometimes erases failures.

python
try:
    result = execute_tool(arguments)
except Exception:
    result = {"message": "처리했습니다"}

This code removes clues necessary to investigate the cause. The result passed to the model must still retain the failure and its scope.

In MCP, receiving a response and the success of the tool operation are distinct. For example, the following response indicates a tool error.

json
{
  "jsonrpc": "2.0",
  "id": 17,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "프로젝트에 이슈를 생성할 권한이 없습니다."
      }
    ],
    "isError": true
  }
}

If the adapter passes only content and discards the error indicator, important information is lost. MCP Tools specification handles protocol errors and tool execution errors separately.

Interpreting request acceptance as task completion

Be particularly cautious in tasks involving multiple stages, such as deployments, file conversions, or email sending.

If a deployment API returns a job ID, it may only mean “deployment request accepted.” The mail service accepting a message does not confirm it has arrived in the recipient's inbox.

You must define what state constitutes “completion” for each tool.

4. Separate the scope addressed by prompts and schemas

The instruction “Do not say it is done before receiving the execution result” may help. However, it cannot recover from executor exceptions or broken task flows.

Configurations also have limited scope.

  • tool_choice: "required" requires tool calls.
  • strict: true helps generate arguments conforming to supported schema constraints.

Neither guarantees the success of external tasks. You must separately check if a project ID with the correct format actually exists and if the user has permission to write to that project.

Configurations that enforce calls should be applied to necessary stages and able to terminate at stages describing results. Unconditionally enforcing calls on every turn may lead to unnecessary repetitions.

5. Derive completion states from execution records

Practically, it is useful to manage task states separately from the model's response. For example, you can distinguish the following states:

  • requested: Call request received.
  • waiting_for_approval: Waiting for execution approval.
  • running: Execution started.
  • succeeded: Tool returned a success result.
  • verified: Defined completion conditions verified.
  • failed: Failure confirmed.
  • unknown: Execution result cannot yet be determined.

These names are design examples. The important point is that the executor and validation code, not the model's arbitrary declaration, update states based on actual events.

Completion conditions for issue creation can be defined as follows:

  1. The creation API returned an issue ID.
  2. The issue retrieved by that ID exists in the requested project.
  3. Required fields, such as the title, match the request.

Additional retrieval is not necessary for every tool. Determine this based on what the original response guarantees and the risk and cost of the task. If a service has delays when retrieving immediately after saving, its characteristics must also be considered.

For important UI elements like completion badges, it is safer to render directly from these states. The model should describe verified results, but not be the sole entity determining completion status.

6. Verify before retrying upon timeout

Assume an external API request ends with a timeout. The server may have already created the issue, but only the response was lost.

Treating this state immediately as a failure and re-creating will cause duplicate issues. First, record it as unknown, and establish a path to verify results through retrieval or the provider's request identifier.

If the service supports idempotency keys, use the same key for retries of the same logical task. Follow the service documentation for actual guarantee scopes and retention periods. Stripe's idempotent request documentation provides a concrete example.

Distinguish the model's call_id from the idempotency key for business tasks. If the model re-requests the same task, a new call ID may be generated; therefore, designs preventing duplicate execution solely by call ID may be insufficient.

7. Attach execution evidence to response evaluation

In agent tests, you must verify response sentences alongside actual states.

At minimum, create scenarios for:

  • Does it refrain from claiming completion when no tool was called?
  • Does it explain permission denials as successes?
  • Are tool errors passed to the model verbatim?
  • Does it indicate asynchronous tasks as running?
  • Does it avoid duplicate execution after timeouts without verifying results?
  • Does it link results of multiple calls to the correct requests?
  • Does it refrain from acknowledging results created for incorrect targets as completed?

Traces should contain identifiers linking model calls, tool requests, execution start and end, errors, and validation results. You can utilize features that collect these, such as tracing in the OpenAI Agents SDK.

However, secret values or personal information should not be unconditionally logged in production logs. Prioritize recording identifiers and states necessary for troubleshooting, and apply masking and access control to arguments and results.

Conclusion

When an agent says “Called,” check the following three things:

  1. Was there an actual call request?
  2. What result did the executor record?
  3. Does that result meet the completion conditions requested by the user?

Distinguishing these three questions allows for more accurate diagnosis of the model's erroneous claims and runtime execution failures.

Reliable agents are not built with fluent completion sentences alone. You must also design the execution evidence and validation paths that support those sentences.