ep 4: Multi-Provider Support - OpenAI and Google Gemini

Introduction

In this episode, 01:20 we continued from episode 3 where we hit roadblocks integrating with ChatGPT and Gemini. After some debugging, we identified the core issues and successfully completed integration with both providers.

tl;dr:

• Fixed OpenAI tool use/tool result conversation patterns
• Completed Google Gemini integration with thought signatures
• Added read_file tool to enable actual file reading
• Learned about provider-specific quirks and requirements

The Problems with OpenAI Integration

Bug 1: How to send back Assistant’s tool call in the conversation history?

03:15 When we ran the agent with OpenAI, the second inference call failed. The issue was in how we constructed the conversation history.

Understanding OpenAI’s Message Types

09:00 I drew a diagram to understand OpenAI’s type system:

Key insight: OpenAI has different message types for:

• User messages qualifies by term “param”
• The Output messages are not qualified with any term, which creates little confusion.
• Assistant messages (LLM responses, including tool calls) output must be converted to their corresponding param types before sending them back for inference.
• Tool messages (results of tool executions)

The Fix is to use correct Param object to send back tool call.

in llm/openai.go

func transformToOpenAIMessages(messages []Message) []openai.ChatCompletionMessageParamUnion {
    openAIMessages := make([]openai.ChatCompletionMessageParamUnion, len(messages))
        for i, msg := range messages {
            switch msg.Type {
                    case MessageTypeText:
						//...
                    case MessageTypeToolUse:
                        openAIMessages[i] = openai.ChatCompletionMessageParamUnion{
                            OfAssistant: &openai.ChatCompletionAssistantMessageParam{
                                ToolCalls: []openai.ChatCompletionMessageToolCallUnionParam{
                                    {
                                        OfFunction: &openai.ChatCompletionMessageFunctionToolCallParam{
                                            ID: msg.ToolUse.ID,
                                            Function: openai.ChatCompletionMessageFunctionToolCallFunctionParam{
                                                Arguments: string(msg.ToolUse.Input),
                                                Name:      msg.ToolUse.Name,
                                            },
                                        },
                                    },
                                },
                            },
                        }
                    case MessageTypeToolResult:
                    //...
            }
        }
    return openAIMessages

}

**Bug 2: Multiple tools in single inference **

When OpenAI return multiple tool calls to be made in the same inference turn, the order in which we send the tool result is critical it seems.

Our current implementation in main.go is:

// WRONG: Adding all tool results at the end
for _, tr := range toolResult {
    inputMessages = append(inputMessages, llm.Message{
        Type:       llm.MessageTypeToolResult,
        ToolResult: &tr,
    })
}

OpenAI requires tool calls and tool results to be interlaced - each tool call from the assistant must be immediately followed by its result, not batched at the end.

This makes sense. This rule makes it easy for the llm to manage it’s attention efficiently when the tool calls and tool responses are one after the other.

The Fix: Interlaced Message Pattern

27:00 We refactored the agent loop to add messages one by one:

for _, block := range respMessage {
    switch block.Type {
    case llm.MessageTypeText:
        // Add text response
        inputMessages = append(inputMessages, block)

    case llm.MessageTypeToolUse:
        hasToolUse = true
        // Add the tool call
        inputMessages = append(inputMessages, block)

        // Execute tool immediately
        toolResp, toolErr := tool.ExecuteTool(block.ToolUse.Name, block.ToolUse.Input)

        // Add result immediately after
        var toolResult llm.ToolResult
        if toolErr != nil {
            toolResult = llm.ToolResult{
                ToolName: block.ToolUse.Name,
                ID:       block.ToolUse.ID,
                IsError:  true,
                Content:  toolErr.Error(),
            }
        } else {
            toolResult = llm.ToolResult{
                ToolName: block.ToolUse.Name,
                ID:       block.ToolUse.ID,
                IsError:  false,
                Content:  toolResp,
            }
        }

        inputMessages = append(inputMessages, llm.Message{
            Type:       llm.MessageTypeToolResult,
            ToolResult: &toolResult,
        })
    }
}

Google Gemini Integration

43:00 After fixing OpenAI, we moved to Google Gemini integration.

Installing the SDK:

go get google.golang.org/genai

Creating the client:

func NewGoogleClient() (*googleClient, error) {
    key := os.Getenv("GOOGLE_API_KEY")
    client, err := genai.NewClient(context.Background(), &genai.ClientConfig{
        APIKey:  key,
        Backend: genai.BackendGeminiAPI,
    })
    if err != nil {
        return nil, err
    }
    return &googleClient{client}, nil
}

Google’s Tool Response Format

51:00 Google has a unique way of handling tool results:

// Google expects tool results in this format
response := map[string]any{}
if message.ToolResult.IsError {
    response["error"] = message.ToolResult.Content
} else {
    response["output"] = message.ToolResult.Content
}

googleMessages = append(googleMessages, &genai.Content{
    Role: "user",
    Parts: []*genai.Part{
        {
            FunctionResponse: &genai.FunctionResponse{
                Name:     message.ToolResult.ToolName,
                ID:       message.ToolResult.ID,
                Response: response,
            },
        },
    },
})

Key difference: Google uses output and error keys in the response map, unlike OpenAI which just takes a string.

The Thought Signature Mystery

1:19:00 When testing with Gemini 3.0 Pro, we encountered an error about missing “thought signatures.”

What are thought signatures? According to Google’s docs, they’re “encrypted representations of the model’s internal thought process” used in reasoning models. You MUST pass them back during function responses, or the model will fail.

type ToolUse struct {
    ID               string
    Name             string
    Input            json.RawMessage
    ThoughtSignature []byte  // Required by Google
}

Adding thought signature support:

// When receiving from Google
messages = append(messages, Message{
    Type: MessageTypeToolUse,
    ToolUse: &ToolUse{
        ID:               part.FunctionCall.ID,
        Name:             part.FunctionCall.Name,
        Input:            toolArgs,
        ThoughtSignature: part.ThoughtSignature,
    },
})

// When sending back to Google
googleMessages = append(googleMessages, &genai.Content{
    Role: "model",
    Parts: []*genai.Part{
        {
            FunctionCall: &genai.FunctionCall{
                Name: message.ToolUse.Name,
                ID:   message.ToolUse.ID,
                Args: args,
            },
            ThoughtSignature: message.ToolUse.ThoughtSignature,
        },
    },
})

1:24:00 This design choice is understandable - they want to prevent distillation attacks where competitors could use the reasoning chains to train smaller models. But also is frustrating - why can’t they store the signature on their end?

Adding the Read File Tool

36:00 We discovered the read_file tool wasn’t hooked up! We had defined it but forgot to add it to the tool map:

var ToolMap = map[string]llm.ToolDefinition{
    "list_files": ListFilesToolDefinition,
    "read_file":  ReadFileToolDefinition,  // ← This was missing!
}

Improved Console Output

We enhanced the console output to better track what’s happening:

fmt.Println(fmt.Sprintf("Assistant: ToolUse: ID: %s, Name: %s, Input: %s",
    message.ToolUse.ID,
    message.ToolUse.Name,
    string(inputJson)))

fmt.Println(fmt.Sprintf("User: ToolResult of ID: %s, of length %d",
    toolResult.ID,
    len(toolResult.Content)))

Testing with All Three Providers

OpenAI (ChatGPT):

go run main.go -goal "explain this project" -provider openai

Output was verbose but thorough, exploring the codebase systematically.

Anthropic (Claude):

go run main.go -goal "explain this project" -provider anthropic

Most concise summary, focused on key architectural points.

Google (Gemini):

go run main.go -goal "explain this project" -provider google

1:27:00 Gemini 3.0 Pro had surprising behavior - it tried to read files outside the project directory, including attempting to read /etc/passwd! This was concerning and showed the importance of implementing proper sandboxing.

Key Learnings

Provider Differences:

Architecture Insights:

• Anthropic’s SDK has the cleanest abstraction
• OpenAI’s type inflation (separate types for input/output) makes it verbose
• Google’s thought signatures add coupling that breaks abstraction

Security Concern: 1:28:00 Gemini attempting to read SSH keys highlights the need for:

• Path restrictions (stay within project directory)
• File type filtering (block sensitive files)
• Tool permission system

Common Issues

“Questions empty” error with Google: This happens if you don’t provide parts in the content. Make sure each message has at least one part:

googleMessages = append(googleMessages, &genai.Content{
    Role: transformToGoogleRole(message.Role),
    Parts: []*genai.Part{
        {
            Text: message.Text,
        },
    },
})

Tool name required by Google: We had to add ToolName to our ToolResult struct because Google requires it in the function response.

Prompt tuning matters: 40:00 Changing list_files description from “Returns a list of files in the current directory” to “Returns a list of files in the given directory” fixed issues where the model wasn’t calling the tool correctly.

Next Steps

With multi-provider support working, we can now:

• Add more tools (search, write files)
• Implement proper sandboxing
• Build evaluation systems to compare providers

Full Code

You can find the complete code from this stream at: https://github.com/agentengineering-dev/agent-framework/tree/ep-004

Key commits:

• Fixed OpenAI tool use pattern
• Added Google Gemini integration
• Implemented thought signature support
• Added read_file tool to tool map