google.generativeai: tools don't work with JSON mode · Issue #393 ...
We read every piece of feedback, and take your input very seriously. To see all available qualifiers, see our documentation.
Error Description
If I configure the model to use response_mime_type=application/json and provide some tools to the model, I always receive the following error:
Expected behavior - model calls tool and returns some JSON as output
Actual:
function_calling_config {
mode: AUTO
}
Traceback (most recent call last):
File "/home/bva/src/thdevelop/src/thebot/tt.py", line 35, in resp = developer_chat.send_message(prompt.format(bu="Please sum 42 and 42 and then please insert to recipes list recipe_name=OLOOL42"), request_options=retry_policy) File "/home/bva/src/thdevelop/src/venv/lib/python3.12/site-packages/google/generativeai/generative_models.py", line 578, in send_message response = self.model.generate_content( File "/home/bva/src/thdevelop/src/venv/lib/python3.12/site-packages/google/generativeai/generative_models.py", line 331, in generate_content response = self._client.generate_content( File "/home/bva/src/thdevelop/src/venv/lib/python3.12/site-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py", line 830, in generate_content response = rpc( File "/home/bva/src/thdevelop/src/venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in call return wrapped_func(*args, **kwargs) File "/home/bva/src/thdevelop/src/venv/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func return retry_target( File "/home/bva/src/thdevelop/src/venv/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target _retry_error_helper( File "/home/bva/src/thdevelop/src/venv/lib/python3.12/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper raise final_exc from source_exc File "/home/bva/src/thdevelop/src/venv/lib/python3.12/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target result = target() File "/home/bva/src/thdevelop/src/venv/lib/python3.12/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout return func(*args, **kwargs) File "/home/bva/src/thdevelop/src/venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable raise exceptions.from_grpc_error(exc) from exc google.api_core.exceptions.InvalidArgument: 400 For controlled generation of only function calls (forced function calling), please set 'tool_config.function_calling_config.mode' field to ANY instead of populating 'response_mime_type' and 'response_schema' fields. For more details, see: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#tool-config
According to the error documentation (which is linked to Cloud Vertex AI and not to generativeai documentation), tool_config should be properly configured. I tried to configure it in two different ways and always received the same error.
First version via ToolConfig and FunctionCallingConfig:
Second version via simple dict:
In both cases, print(f"Model tool config: {model._tool_config}") shows the proper config. It looks like a bug or am I doing something wrong?
