[bot] Anthropic `usage.output_tokens_details.thinking_tokens` metric not extracted

[braintrust-bot]

Summary

The Anthropic Messages API returns usage.output_tokens_details.thinking_tokens when extended thinking or adaptive thinking is active. This sub-metric counts the output tokens consumed by internal reasoning (always <= output_tokens). The current Anthropic instrumentation does not extract this field; it is silently ignored and never appears in span metrics.

What instrumentation is missing

parseMetricsFromUsage() in js/src/instrumentation/plugins/anthropic-plugin.ts (lines 209–239) maps four fields from AnthropicUsage:

• input_tokens → prompt_tokens
• output_tokens → completion_tokens
• cache_read_input_tokens → prompt_cached_tokens
• cache_creation_input_tokens → prompt_cache_creation_tokens

It also iterates over usage.server_tool_use sub-fields generically (lines 230–236). However, output_tokens_details is never read — its thinking_tokens sub-field is not extracted and not mapped to a span metric.

The AnthropicUsage interface (js/src/vendor-sdk-types/anthropic.ts, lines 155–162) does not declare output_tokens_details:

export interface AnthropicUsage {
  input_tokens: number;
  output_tokens: number;
  cache_read_input_tokens?: number;
  cache_creation_input_tokens?: number;
  server_tool_use?: AnthropicServerToolUseUsage;
  [key: string]: unknown;  // output_tokens_details falls through here, never read
}

The missing metric is: usage.output_tokens_details.thinking_tokens → should map to completion_thinking_tokens (following the existing pattern where completion_tokens_details.reasoning_tokens → completion_reasoning_tokens in the OpenAI plugin).

Why this matters

This is a direct billing-transparency parity gap. OpenAI already exposes completion_tokens_details.reasoning_tokens and this repo extracts it as completion_reasoning_tokens. For Anthropic Claude 3.7+, Claude 4, and similar models used with thinking: { type: "enabled" } or thinking: { type: "adaptive" }, users cannot currently see what fraction of their billed completion_tokens was spent on reasoning vs. response text.

Braintrust docs status

not_found — The Braintrust Anthropic integration page does not mention thinking_tokens or output_tokens_details metrics. The extended thinking content gap was previously tracked in #1621 (closed as completed), but that issue was about content capture, not this token-count metric.

Upstream sources

• Anthropic Messages API usage reference: https://docs.anthropic.com/en/api/messages
• output_tokens_details.thinking_tokens — returned in usage when extended or adaptive thinking is active; counts tokens consumed by reasoning including delimiter tokens; always <= output_tokens
• Beta header required to see this field: thinking-token-count-2026-05-13 (or this feature may be GA by now)

Braintrust docs sources

• https://www.braintrust.dev/docs/guides/tracing
• https://www.braintrust.dev/docs/integrations/ai-providers/anthropic

Local repo files inspected

• js/src/instrumentation/plugins/anthropic-plugin.ts — parseMetricsFromUsage() lines 209–239: no output_tokens_details handling
• js/src/vendor-sdk-types/anthropic.ts — AnthropicUsage interface lines 155–162: output_tokens_details not declared
• e2e/scenarios/anthropic-instrumentation/ — no test cases checking thinking_tokens in span metrics

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If this issue is still relevant, please leave a comment or remove the stale label. Thank you for your contributions!