How to use tool calling with Generate API?

Follow this quick guide to learn how to use tool calling with the Generate API in Langbase.

We will not use stream mode for this example.

Step #0Create a Pipe

Create a new generate type Pipe or open an existing generate Pipe in your Langbase account. Go ahead and turn off the stream mode for this example and deploy the Pipe.

Alternatively, you can fork this tool call generate Pipe Pipe and skip to step 3.

Step #1Select OpenAI model

Tool calling is available with OpenAI models. So select any of the available OpenAI models in your Pipe.

Step #2Add a tool to the Pipe

Let's add a tool to get the current weather of a given location. Click on the Add button in the Tools section to add a new tool.

This will open a modal where you can define the tool. The tool we are defining will take two arguments:

location
unit.

The location argument is required and the unit argument is optional.

The tool definition will look something like the following.

{
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather of a given location",
        "parameters": {
            "type": "object",
            "required": [
                "location"
            ],
            "properties": {
                "unit": {
                    "enum": [
                        "celsius",
                        "fahrenheit"
                    ],
                    "type": "string"
                },
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA"
                }
            }
        }
    }
}

Go ahead and deploy the Pipe to production.

Note

Playground is disabled

If a Pipe has tools, the playground will be disabled. You can only test tool calling with our Generate and Chat API.

Step #3User prompt to call the tool

Go ahead and copy your Pipe API key from the Pipe API page. You will need this key to call the Generate API.

Now let's create an index.js file where we will define get_current_weather function and also call the Pipe.

const get_current_weather = ({ location, unit }) => {
	// get weather for the location and return the temperature
};

const tools = {
	get_current_weather
};

(async () => {
	const messages = [
		{
			role: 'user',
			content: 'Whats the weather in San Francisco?'
		}
	];

	// replace this with your Pipe API key
	const pipeApiKey = ``;

	const res = await fetch('https://api.langbase.com/beta/generate', {
		method: 'POST',
		headers: {
			'Content-Type': 'application/json',
			Authorization: `Bearer ${pipeApiKey}`
		},
		body: JSON.stringify({
			messages
		})
	});
})();

Because the user prompt requires the current weather of San Francisco, the model will respond with a tool call like the following:

{
	"role": "assistant",
	"content": null,
	"tool_calls": [
		{
			"id": "call_u28sPmmCAWkop0OdgDYDJ9OG",
			"type": "function",
			"function": {
				"name": "get_current_weather",
				"arguments": "{\"location\": \"San Francisco\"}"
			}
		}
	]
}

To check if the model has called the tool, you can check the tool_calls array in the model's response. If it exists, call the functions specified in the tool_calls array and send the response back to Langbase.

Note

Push assistant response in Generate API

In Generate API, it's required that you push the assistant response (above one) that contained tool_calls before pushing tool responses into the messages array.

(async () => {
	const messages = [
		{
			role: 'user',
			content: 'Whats the weather in Lahore and saad?'
		}
	];

	// add your Pipe API key here
	const pipeApiKey = ``;

	const res = await fetch('https://api.langbase.com/beta/generate', {
		method: 'POST',
		headers: {
			'Content-Type': 'application/json',
			Authorization: `Bearer ${pipeApiKey}`
		},
		body: JSON.stringify({
			messages
		})
	});

	const data = await res.json();

	const { raw } = data;

	// get the response message from the model
	const responseMessage = raw.choices[0].message;

	// get the tool calls from the response message
	const toolCalls = responseMessage.tool_calls;

	if (toolCalls) {
		// push the assistant response to the messages array to send back to the API
		messages.push(responseMessage);

		// Call all the functions in the tool_calls array
		toolCalls.forEach(toolCall => {
			const toolName = toolCall.function.name;
			const toolParameters = JSON.parse(toolCall.function.arguments);
			const toolFunction = tools[toolName];
			const toolResponse = toolFunction(toolParameters);

			// push the tool response to the messages array to send back to the API
			messages.push({
				tool_call_id: toolCall.id, // required: id of the tool call
				role: 'tool', // required: role of the message
				name: toolName, // required: name of the tool
				content: JSON.stringify(toolResponse) // required: response of the tool
			});
		});

		// send the messages array back to the API
		const res = await fetch('https://api.langbase.com/beta/generate', {
			method: 'POST',
			headers: {
				'Content-Type': 'application/json',
				Authorization: `Bearer ${pipeApiKey}`
			},
			body: JSON.stringify({
				messages
			})
		});

		const data = await res.json();
	}
})();

This is what a typical model response will look like after calling the tool:

{
	"completion": "The current temperature in San Francisco, CA is 25°C.",
	"raw": {
		"id": "chatcmpl-9hQG8k2pD1A6JoFKQ0O6BKKvJzogS",
		"object": "chat.completion",
		"created": 1720136072,
		"model": "gpt-4o-2024-05-13",
		"choices": [
			{
				"index": 0,
				"message": {
					"role": "assistant",
					"content": "The current temperature in San Francisco, CA is 25°C."
				},
				"logprobs": null,
				"finish_reason": "stop"
			}
		],
		"usage": {
			"prompt_tokens": 121,
			"completion_tokens": 14,
			"total_tokens": 135
		},
		"system_fingerprint": "fp_ce0793330f"
	}
}

And that's it! You have successfully used tool calling with the Generate API in Langbase.

Tool calling