feat: enabling output_schema and tools to coexist#1062
Open
copybara-service[bot] wants to merge 1 commit intomainfrom
Open
feat: enabling output_schema and tools to coexist#1062copybara-service[bot] wants to merge 1 commit intomainfrom
copybara-service[bot] wants to merge 1 commit intomainfrom
Conversation
This CL enables the simultaneous use of `output_schema` (structured output) and `tools` for models that do not natively support both features at once (specifically Gemini 1.x and 2.x on Vertex AI). ### Core Logic The CL implements a workaround for models with this limitation: 1. **Synthetic Tooling**: Instead of passing the `output_schema` directly to the model's configuration, it introduces a synthetic tool called `set_model_response`. 2. **Schema Injection**: The parameters of this tool are set to the requested `output_schema`. 3. **Instruction Prompting**: System instructions are appended, directing the model to provide its final response using this specific tool in the required format. 4. **Response Interception**: The `BaseLlmFlow` is updated to check if `set_model_response` was called. If so, it extracts the JSON arguments and converts them into a standard model response event. ### Key Changes * **`OutputSchema.java` (New)**: A new `RequestProcessor` that detects when the workaround is needed, adds the `SetModelResponseTool`, and provides utilities for extracting the structured response. * **`SetModelResponseTool.java` (New)**: A marker tool that simply returns its input arguments, used to "capture" the structured output from the model. * **`ModelNameUtils.java`**: Added logic to identify Gemini 1.x and 2.x models and determine if they can handle native `output_schema` alongside tools. * **`BaseLlmFlow.java`**: Updated the flow logic to detect the synthetic tool response and generate the final output event. * **`Basic.java`**: Updated to prevent native `outputSchema` configuration when the workaround is active. * **`SingleFlow.java`**: Registered the new `OutputSchema` processor. PiperOrigin-RevId: 886133734
cd728ac to
7649d70
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: enabling output_schema and tools to coexist
This CL enables the simultaneous use of
output_schema(structured output) andtoolsfor models that do not natively support both features at once (specifically Gemini 1.x and 2.x on Vertex AI).Core Logic
The CL implements a workaround for models with this limitation:
output_schemadirectly to the model's configuration, it introduces a synthetic tool calledset_model_response.output_schema.BaseLlmFlowis updated to check ifset_model_responsewas called. If so, it extracts the JSON arguments and converts them into a standard model response event.Key Changes
OutputSchema.java(New): A newRequestProcessorthat detects when the workaround is needed, adds theSetModelResponseTool, and provides utilities for extracting the structured response.SetModelResponseTool.java(New): A marker tool that simply returns its input arguments, used to "capture" the structured output from the model.ModelNameUtils.java: Added logic to identify Gemini 1.x and 2.x models and determine if they can handle nativeoutput_schemaalongside tools.BaseLlmFlow.java: Updated the flow logic to detect the synthetic tool response and generate the final output event.Basic.java: Updated to prevent nativeoutputSchemaconfiguration when the workaround is active.SingleFlow.java: Registered the newOutputSchemaprocessor.