Custom fine-tuned gemini model using Google Vertex

Help
1

Hi,

I am trying to understand the correct way to use the AI SDK Core and AI SDK UI to interact with a custom fine tuned model that I deployed in Google Cloud using their Vertex service. In particular, I am looking at this page: Foundations: Providers and Models

I was able to get thing working using a simple setup where I used useChat. I simply used createVertex passing it a keyFilename file (code shown below) and then a very simple POST function (code shown below). However, when I try to use a more complicated POST function with a createDataStreamResponse I am unable to get the code working.

This code worked:

import { openai } from '@ai-sdk/openai';
import { vertex } from '@ai-sdk/google-vertex';
import { createVertex } from '@ai-sdk/google-vertex';
import { streamText, convertToCoreMessages } from 'ai';
const {GoogleAuth} = require('google-auth-library');

// Allow streaming responses up to 30 seconds
export const maxDuration = 30;

const euVertex = createVertex({
  project: PROJECT_ID,
  location: 'us-central1', // optional
  googleAuthOptions: {
  keyFilename:'edubot-439916-0950981fb9bc.json'
  }
});

const model = euVertex('projects/' + PROJECT_ID + '/locations/us-central1/endpoints/' + ENDPOINT)

export async function POST(req: Request) {
  console.error("In POST")

  const { messages } = await req.json();
  console.log(messages)
  const result = await streamText({
    model: model,
    messages: convertToCoreMessages(messages),
  });

  return result.toDataStreamResponse();
}

However, when I try using this code, I do not seem to get a response from the Google Server.:

return createDataStreamResponse({
    execute: (dataStream) => {
      console.log('In execute...')
      dataStream.writeData({
        type: 'user-message-id',
        content: userMessageId,
      });

      const result = streamText({
        //model: customModel(model.apiIdentifier),
        model: customFineTunedModel,
        system: systemPrompt,
        messages: coreMessages,
        maxSteps: 5,
...

Here custonFineTunedModel is a model I fined tuned in cloud and is created as follows (as per the original code snippet):

const euVertex = createVertex({
  project: PROJECT_ID,
  location: 'us-central1', // optional
  googleAuthOptions: {
  keyFilename:'edubot-439916-0950981fb9bc.json'
  }
});

const model = euVertex('projects/' + PROJECT_ID + '/locations/us-central1/endpoints/' + ENDPOINT)

I am unclear when I use useChat whether I need to provide the endpoint for a custom model and what the id should be set to?

Also, do I need to set anything specifically in vertex for getting the streaming working for the model?

In general, it would be greatly helpful to understand more explicitly how to use custom fine tuned models within the AI SDK ecosystem and specifically with the LLM Specification approach described in the following page.

Any code examples would be helpful.

Thanks in advance.

2

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.