To prevent grounded Gemini calls from silently truncating output when internal thinking tokens exhaust token limits, inspect candidate.finish_reason on every call path and retry with doubled token limits up to a hard ceiling. In the Google Gen AI API, max_output_tokens bounds reasoning thoughts and visible text together. Catching MAX_TOKENS ensures partial research is rejected and audit logs record cumulative token consumption across all retry attempts.
Before you start
- Python 3.10 or later installed with the
google-genaiSDK (pip install google-genai). - A Gemini API key set in your environment as
GEMINI_API_KEY. - An agent workflow where web-grounded research is passed to downstream reasoning nodes.
How it works
flowchart LR
A[Execute grounded Gemini call] --> B[Record attempt token costs]
B --> C{Check candidate finish reason}
C -- STOP --> D[Return complete text and usage]
C -- MAX_TOKENS --> E{Attempts under limit and ceiling?}
E -- Yes --> F[Double max output tokens and retry]
F --> A
E -- No --> G[Raise TruncatedOutputError]
Step 1: Define a truncation error class
Create a custom exception that preserves cumulative token telemetry and partial text when retries fail. This prevents broken output from contaminating downstream systems.
class TruncatedOutputError(RuntimeError):
"""Raised when an LLM call hits MAX_TOKENS and exhausts retries."""
def __init__(self, message: str, cumulative_usage: dict, partial_text: str):
super().__init__(message)
self.cumulative_usage = cumulative_usage
self.partial_text = partial_text
Step 2: Implement the grounded caller with token doubling
Wrap the client call to enforce Google Search grounding, accumulate usage across attempts, inspect candidate.finish_reason, and back off with doubled output limits when hit by MAX_TOKENS.
from google import genai
from google.genai import types
def call_gemini_grounded_safe(
prompt: str,
client: genai.Client | None = None,
model: str = "gemini-2.5-flash",
initial_max_tokens: int = 4096,
max_token_ceiling: int = 32768,
backoff_factor: int = 2,
max_attempts: int = 3,
) -> dict:
if client is None:
client = genai.Client()
current_max_tokens = initial_max_tokens
attempt = 0
cumulative_cost = {
"prompt_tokens": 0,
"candidates_tokens": 0,
"thoughts_tokens": 0,
"total_tokens": 0,
"attempts": 0,
}
while attempt < max_attempts:
attempt += 1
cumulative_cost["attempts"] = attempt
config = types.GenerateContentConfig(
max_output_tokens=current_max_tokens,
tools=[{"google_search": {}}],
temperature=0.2,
)
response = client.models.generate_content(
model=model,
contents=prompt,
config=config,
)
if response.usage_metadata:
meta = response.usage_metadata
cumulative_cost["prompt_tokens"] += meta.prompt_token_count or 0
cumulative_cost["candidates_tokens"] += meta.candidates_token_count or 0
if hasattr(meta, "candidates_tokens_details") and meta.candidates_tokens_details:
for detail in meta.candidates_tokens_details:
cumulative_cost["thoughts_tokens"] += getattr(detail, "thought_tokens", 0) or 0
cumulative_cost["total_tokens"] = (
cumulative_cost["prompt_tokens"] + cumulative_cost["candidates_tokens"]
)
candidate = response.candidates[0] if response.candidates else None
if not candidate:
raise RuntimeError("Model returned no candidate responses.")
finish_reason = candidate.finish_reason
is_truncated = finish_reason in (
types.FinishReason.MAX_TOKENS,
"MAX_TOKENS",
"LENGTH",
)
if not is_truncated:
return {
"text": response.text,
"finish_reason": str(finish_reason),
"truncated": False,
"attempts_needed": attempt,
"final_max_output_tokens": current_max_tokens,
"usage": cumulative_cost,
}
next_max_tokens = current_max_tokens * backoff_factor
if next_max_tokens > max_token_ceiling or attempt >= max_attempts:
raise TruncatedOutputError(
f"Call halted mid-output due to MAX_TOKENS after {attempt} attempts.",
cumulative_usage=cumulative_cost,
partial_text=response.text or "",
)
current_max_tokens = min(next_max_tokens, max_token_ceiling)
raise TruncatedOutputError(
"Max retry attempts exhausted.",
cumulative_usage=cumulative_cost,
partial_text="",
)
Step 3: Run the caller and handle failures
Invoke the safe caller inside your pipeline. Catch TruncatedOutputError to stop truncated summaries from entering downstream tasks as verified research.
if __name__ == "__main__":
prompt = "Search for recent Gemini architecture updates and summarize key details."
try:
result = call_gemini_grounded_safe(
prompt=prompt,
initial_max_tokens=4096,
max_token_ceiling=16384,
)
print("Completed generation:", result["text"][:120], "...")
print("Total tokens consumed:", result["usage"]["total_tokens"])
except TruncatedOutputError as err:
print(f"Call failed safely: {err}")
print("Total audited token waste:", err.cumulative_usage)
Common mistakes
- Checking HTTP status codes instead of
finish_reason. The SDK returns HTTP 200 OK even when generation stops mid-sentence. Inspectresponse.candidates[0].finish_reasondirectly on every code path. - Dropping discarded attempt tokens from billing audits. Discarding a failed attempt without tracking its tokens hides real operational costs. Accumulate prompt, thinking, and candidate tokens across every attempt.
- Assuming output caps apply only to visible text. Thinking models consume
max_output_tokensfor internal reasoning and visible output combined. A call spending ~3,900 tokens thinking leaves only 196 tokens under a 4,096 cap. - Prompting the model to continue from truncated text. Appending cut-off assistant text into conversational history corrupts internal grounding state. Retry the clean prompt with an increased token limit instead.
Frequently asked questions
Why does Gemini return HTTP 200 when an answer is cut off?
The API protocol treats reaching max_output_tokens as a normal completion event rather than a server error. The server returns HTTP 200 with finish_reason="MAX_TOKENS" on the candidate. The SDK does not throw an exception, so your application code must inspect candidate.finish_reason explicitly.
Why do search-grounded calls consume thinking tokens so quickly?
When Google Search is active, reasoning models generate thinking tokens to construct queries, analyze search results, and evaluate citations. For complex queries, this synthesis step can consume thousands of thinking tokens before visible generation starts, exhausting small output limits prematurely.
Should I retry a truncated call with the same token cap?
No. Repeating an identical grounded query with the same max_output_tokens will repeat the reasoning cycle and fail at the same boundary. You will pay double the token costs without receiving a complete answer. Always scale up the token limit or abort.
Related
- Track and Cap: A Workflow for Managing Agent Coding Costs covers how to budget tokens across multi-step agent executions.
- Large Context Window Prompting: 2M Token Guide explains how token budgets and context growth behave in reasoning models.
- AI Agent Oversight: Why Passive Approval Fails Systems discusses validation boundaries before passing agent outputs downstream.