Supported models
The following table lists the models that support audio understanding:
Model | Audio modality details |
---|---|
Gemini 1.5 Flash Go to the Gemini 1.5 Flash model card |
Maximum audio length per prompt: ~8.4 hours or up to 1 million tokens. Speech can be understood for audio summarization, transcription, and translation. |
Gemini 1.5 Pro Go to the Gemini 1.5 Pro model card |
Maximum audio length per prompt: ~8.4 hours or up to 1 million tokens. Speech can be understood for audio summarization, transcription, and translation. |
For a list of languages supported by Gemini models, see model information Google models. To learn more about how to design multimodal prompts, see Design multimodal prompts. If you're looking for a way to use Gemini directly from your mobile and web apps, see the Vertex AI in Firebase SDKs for Android, Swift, web, and Flutter apps.
Add audio to a request
You can add audio files in your requests to Gemini.
Single audio
The following shows you how to use an audio file to summarize a podcast.
Python
To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Vertex AI SDK for Python API reference documentation.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the stream
parameter in
generate_content
.
response = model.generate_content(contents=[...], stream = True)
For a non-streaming response, remove the parameter, or set the parameter to
False
.
Sample code
Java
Before trying this sample, follow the Java setup instructions in the Vertex AI quickstart. For more information, see the Vertex AI Java SDK for Gemini reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
generateContentStream
method.
public ResponseStream<GenerateContentResponse> generateContentStream(Content content)
For a non-streaming response, use the
generateContent
method.
public GenerateContentResponse generateContent(Content content)
Sample code
Node.js
Before trying this sample, follow the Node.js setup instructions in the Generative AI quickstart using the Node.js SDK. For more information, see the Node.js SDK for Gemini reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
generateContentStream
method.
const streamingResp = await generativeModel.generateContentStream(request);
For a non-streaming response, use the
generateContent
method.
const streamingResp = await generativeModel.generateContent(request);
Sample code
Go
Before trying this sample, follow the Go setup instructions in the Vertex AI quickstart. For more information, see the Vertex AI Go SDK for Gemini reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
GenerateContentStream
method.
iter := model.GenerateContentStream(ctx, genai.Text("Tell me a story about a lumberjack and his giant ox. Keep it very short."))
For a non-streaming response, use the GenerateContent
method.
resp, err := model.GenerateContent(ctx, genai.Text("What is the average size of a swallow?"))
Sample code
C#
Before trying this sample, follow the C# setup instructions in the Vertex AI quickstart. For more information, see the Vertex AI C# reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
StreamGenerateContent
method.
public virtual PredictionServiceClient.StreamGenerateContentStream StreamGenerateContent(GenerateContentRequest request)
For a non-streaming response, use the
GenerateContentAsync
method.
public virtual Task<GenerateContentResponse> GenerateContentAsync(GenerateContentRequest request)
For more information on how the server can stream responses, see Streaming RPCs.
Sample code
REST
After you set up your environment, you can use REST to test a text prompt. The following sample sends a request to the publisher model endpoint.
Before using any of the request data, make the following replacements:
LOCATION
: The region to process the request. Enter a supported region. For the full list of supported regions, see Available locations.Click to expand a partial list of available regions
us-central1
us-west4
northamerica-northeast1
us-east4
us-west1
asia-northeast3
asia-southeast1
asia-northeast1
PROJECT_ID
: Your project ID.FILE_URI
: The Cloud Storage URI of the file to include in the prompt. The bucket object must either be publicly readable or reside in the same Google Cloud project that's sending the request. You must also specify the media type (mimeType
) of the file.If you don't have an audio file in Cloud Storage, then you can use the following publicly available file:
gs://cloud-samples-data/generative-ai/audio/pixel.mp3
with a mime type ofaudio/mp3
. To listen to this audio, open the sample MP3 file.MIME_TYPE
: The media type of the file specified in thedata
orfileUri
fields. Acceptable values include the following:Click to expand MIME types
application/pdf
audio/mpeg
audio/mp3
audio/wav
image/png
image/jpeg
text/plain
video/mov
video/mpeg
video/mp4
video/mpg
video/avi
video/wmv
video/mpegps
video/flv
TEXT
The text instructions to include in the prompt. For example,Please provide a summary for the audio. Provide chapter titles, be concise and short, no need to provide chapter summaries. Do not make up any information that is not part of the audio and do not be verbose.
To send your request, choose one of these options:
curl
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
cat > request.json << 'EOF' { "contents": { "role": "USER", "parts": [ { "fileData": { "fileUri": "FILE_URI", "mimeType": "MIME_TYPE" } }, { "text": "TEXT" } ] } } EOF
Then execute the following command to send your REST request:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/gemini-1.5-flash:generateContent"
PowerShell
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
@' { "contents": { "role": "USER", "parts": [ { "fileData": { "fileUri": "FILE_URI", "mimeType": "MIME_TYPE" } }, { "text": "TEXT" } ] } } '@ | Out-File -FilePath request.json -Encoding utf8
Then execute the following command to send your REST request:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/gemini-1.5-flash:generateContent" | Select-Object -Expand Content
You should receive a JSON response similar to the following.
Note the following in the URL for this sample:- Use the
generateContent
method to request that the response is returned after it's fully generated. To reduce the perception of latency to a human audience, stream the response as it's being generated by using thestreamGenerateContent
method. - The multimodal model ID is located at the end of the URL before the method
(for example,
gemini-1.5-flash
orgemini-1.0-pro-vision
). This sample may support other models as well.
Console
To send a multimodal prompt by using the Google Cloud console, do the following:
- In the Vertex AI section of the Google Cloud console, go to the Vertex AI Studio page.
- In Prompt design (single turn), click Open.
Optional: Configure the model and parameters:
- Model: Select a model.
- Region: Select the region that you want to use.
Temperature: Use the slider or textbox to enter a value for temperature.
The temperature is used for sampling during response generation, which occurs whentopP
andtopK
are applied. Temperature controls the degree of randomness in token selection. Lower temperatures are good for prompts that require a less open-ended or creative response, while higher temperatures can lead to more diverse or creative results. A temperature of0
means that the highest probability tokens are always selected. In this case, responses for a given prompt are mostly deterministic, but a small amount of variation is still possible.If the model returns a response that's too generic, too short, or the model gives a fallback response, try increasing the temperature.
Output token limit: Use the slider or textbox to enter a value for the max output limit.
Maximum number of tokens that can be generated in the response. A token is approximately four characters. 100 tokens correspond to roughly 60-80 words.Specify a lower value for shorter responses and a higher value for potentially longer responses.
- Add stop sequence: Optional. Enter a stop sequence, which is a series of characters that includes spaces. If the model encounters a stop sequence, the response generation stops. The stop sequence isn't included in the response, and you can add up to five stop sequences.
- Optional: To configure advanced parameters, click Advanced and configure as follows:
Top-K: Use the slider or textbox to enter a value for top-K (not supported for Gemini 1.5).
Top-K changes how the model selects tokens for output. A top-K of1
means the next selected token is the most probable among all tokens in the model's vocabulary (also called greedy decoding), while a top-K of3
means that the next token is selected from among the three most probable tokens by using temperature.For each token selection step, the top-K tokens with the highest probabilities are sampled. Then tokens are further filtered based on top-P with the final token selected using temperature sampling.
Specify a lower value for less random responses and a higher value for more random responses.
- Top-P: Use the slider or textbox to enter a value for top-P.
Tokens are selected from most probable to the least until the sum of their
probabilities equals the value of top-P. For the least variable results,
set top-P to
0
. - Enable Grounding: Grounding isn't supported for multimodal prompts.
- To upload media, such as MP3 and WAV files, do the following:
- Click Insert Media, and select a source.
If you choose Google Drive as your source, you must choose an account and give consent to Vertex AI Studio to access your account the first time you select this option. You can upload multiple media files that have a total size of up to 10 MB. A single file can't exceed 7 MB.
- Click the file that you want to add.
- Click Select.
The file thumbnail displays in the Prompt pane. The total number of tokens also displays. If your prompt data exceeds the token limit, the tokens are truncated and aren't included in processing your data.
- Optional: To display the number of tokens calculated in your audio files and the sum of all tokens, click View tokens.
It can take up to 15 seconds to calculate the token count for media files. The Token ID to text and Token ID views don't display any valuable output, because media tokens aren't supported.
To close the tokenizer tool pane, click X, or click outside of the pane.
- Enter your text prompt in the Prompt pane.
- Optional: To display the number of tokens calculated in your audio files, the number of text tokens, and the sum of all tokens, click View tokens. You can view the tokens or token IDs of your text prompt.
- To view the tokens in the text prompt that are highlighted with different colors marking the boundary of each token ID, click Token ID to text. Media tokens aren't supported.
- To view the token IDs, click Token ID.
To close the tokenizer tool pane, click X, or click outside of the pane.
- Click Submit.
- Optional: To save your prompt to My prompts, click Save.
- Optional: To get the Python code or a curl command for your prompt, click Get code.
Click to expand advanced configurations
Audio transcription
The following shows you how to use an audio file to transcribe an interview.
Python
To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Vertex AI SDK for Python API reference documentation.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the stream
parameter in
generate_content
.
response = model.generate_content(contents=[...], stream = True)
For a non-streaming response, remove the parameter, or set the parameter to
False
.
Sample code
Java
Before trying this sample, follow the Java setup instructions in the Vertex AI quickstart. For more information, see the Vertex AI Java SDK for Gemini reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
generateContentStream
method.
public ResponseStream<GenerateContentResponse> generateContentStream(Content content)
For a non-streaming response, use the
generateContent
method.
public GenerateContentResponse generateContent(Content content)
Sample code
Node.js
Before trying this sample, follow the Node.js setup instructions in the Generative AI quickstart using the Node.js SDK. For more information, see the Node.js SDK for Gemini reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
generateContentStream
method.
const streamingResp = await generativeModel.generateContentStream(request);
For a non-streaming response, use the
generateContent
method.
const streamingResp = await generativeModel.generateContent(request);
Sample code
Go
Before trying this sample, follow the Go setup instructions in the Vertex AI quickstart. For more information, see the Vertex AI Go SDK for Gemini reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
GenerateContentStream
method.
iter := model.GenerateContentStream(ctx, genai.Text("Tell me a story about a lumberjack and his giant ox. Keep it very short."))
For a non-streaming response, use the GenerateContent
method.
resp, err := model.GenerateContent(ctx, genai.Text("What is the average size of a swallow?"))
Sample code
C#
Before trying this sample, follow the C# setup instructions in the Vertex AI quickstart. For more information, see the Vertex AI C# reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
StreamGenerateContent
method.
public virtual PredictionServiceClient.StreamGenerateContentStream StreamGenerateContent(GenerateContentRequest request)
For a non-streaming response, use the
GenerateContentAsync
method.
public virtual Task<GenerateContentResponse> GenerateContentAsync(GenerateContentRequest request)
For more information on how the server can stream responses, see Streaming RPCs.
Sample code
REST
After you set up your environment, you can use REST to test a text prompt. The following sample sends a request to the publisher model endpoint.
Before using any of the request data, make the following replacements:
LOCATION
: The region to process the request. Enter a supported region. For the full list of supported regions, see Available locations.Click to expand a partial list of available regions
us-central1
us-west4
northamerica-northeast1
us-east4
us-west1
asia-northeast3
asia-southeast1
asia-northeast1
PROJECT_ID
: Your project ID.FILE_URI
: The Cloud Storage URI of the file to include in the prompt. The bucket object must either be publicly readable or reside in the same Google Cloud project that's sending the request. You must also specify the media type (mimeType
) of the file.If you don't have an audio file in Cloud Storage, then you can use the following publicly available file:
gs://cloud-samples-data/generative-ai/audio/pixel.mp3
with a mime type ofaudio/mp3
. To listen to this audio, open the sample MP3 file.MIME_TYPE
: The media type of the file specified in thedata
orfileUri
fields. Acceptable values include the following:Click to expand MIME types
application/pdf
audio/mpeg
audio/mp3
audio/wav
image/png
image/jpeg
text/plain
video/mov
video/mpeg
video/mp4
video/mpg
video/avi
video/wmv
video/mpegps
video/flv
TEXT
The text instructions to include in the prompt. For example,Can you transcribe this interview, in the format of timecode, speaker, caption. Use speaker A, speaker B, etc. to identify speakers.
To send your request, choose one of these options:
curl
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
cat > request.json << 'EOF' { "contents": { "role": "USER", "parts": [ { "fileData": { "fileUri": "FILE_URI", "mimeType": "MIME_TYPE" } }, { "text": "TEXT" } ] } } EOF
Then execute the following command to send your REST request:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/gemini-1.5-flash:generateContent"
PowerShell
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
@' { "contents": { "role": "USER", "parts": [ { "fileData": { "fileUri": "FILE_URI", "mimeType": "MIME_TYPE" } }, { "text": "TEXT" } ] } } '@ | Out-File -FilePath request.json -Encoding utf8
Then execute the following command to send your REST request:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/gemini-1.5-flash:generateContent" | Select-Object -Expand Content
You should receive a JSON response similar to the following.
Note the following in the URL for this sample:- Use the
generateContent
method to request that the response is returned after it's fully generated. To reduce the perception of latency to a human audience, stream the response as it's being generated by using thestreamGenerateContent
method. - The multimodal model ID is located at the end of the URL before the method
(for example,
gemini-1.5-flash
orgemini-1.0-pro-vision
). This sample may support other models as well.
Console
To send a multimodal prompt by using the Google Cloud console, do the following:
- In the Vertex AI section of the Google Cloud console, go to the Vertex AI Studio page.
- In Prompt design (single turn), click Open.
Optional: Configure the model and parameters:
- Model: Select a model.
- Region: Select the region that you want to use.
Temperature: Use the slider or textbox to enter a value for temperature.
The temperature is used for sampling during response generation, which occurs whentopP
andtopK
are applied. Temperature controls the degree of randomness in token selection. Lower temperatures are good for prompts that require a less open-ended or creative response, while higher temperatures can lead to more diverse or creative results. A temperature of0
means that the highest probability tokens are always selected. In this case, responses for a given prompt are mostly deterministic, but a small amount of variation is still possible.If the model returns a response that's too generic, too short, or the model gives a fallback response, try increasing the temperature.
Output token limit: Use the slider or textbox to enter a value for the max output limit.
Maximum number of tokens that can be generated in the response. A token is approximately four characters. 100 tokens correspond to roughly 60-80 words.Specify a lower value for shorter responses and a higher value for potentially longer responses.
- Add stop sequence: Optional. Enter a stop sequence, which is a series of characters that includes spaces. If the model encounters a stop sequence, the response generation stops. The stop sequence isn't included in the response, and you can add up to five stop sequences.
- Optional: To configure advanced parameters, click Advanced and configure as follows:
Top-K: Use the slider or textbox to enter a value for top-K (not supported for Gemini 1.5).
Top-K changes how the model selects tokens for output. A top-K of1
means the next selected token is the most probable among all tokens in the model's vocabulary (also called greedy decoding), while a top-K of3
means that the next token is selected from among the three most probable tokens by using temperature.For each token selection step, the top-K tokens with the highest probabilities are sampled. Then tokens are further filtered based on top-P with the final token selected using temperature sampling.
Specify a lower value for less random responses and a higher value for more random responses.
- Top-P: Use the slider or textbox to enter a value for top-P.
Tokens are selected from most probable to the least until the sum of their
probabilities equals the value of top-P. For the least variable results,
set top-P to
0
. - Enable Grounding: Grounding isn't supported for multimodal prompts.
- To upload media, such as MP3 and WAV files, do the following:
- Click Insert Media, and select a source.
If you choose Google Drive as your source, you must choose an account and give consent to Vertex AI Studio to access your account the first time you select this option. You can upload multiple media files that have a total size of up to 10 MB. A single file can't exceed 7 MB.
- Click the file that you want to add.
- Click Select.
The file thumbnail displays in the Prompt pane. The total number of tokens also displays. If your prompt data exceeds the token limit, the tokens are truncated and aren't included in processing your data.
- Optional: To display the number of tokens calculated in your audio files and the sum of all tokens, click View tokens.
It can take up to 15 seconds to calculate the token count for media files. The Token ID to text and Token ID views don't display any valuable output, because media tokens aren't supported.
To close the tokenizer tool pane, click X, or click outside of the pane.
- Enter your text prompt in the Prompt pane.
- Optional: To display the number of tokens calculated in your audio files, the number of text tokens, and the sum of all tokens, click View tokens. You can view the tokens or token IDs of your text prompt.
- To view the tokens in the text prompt that are highlighted with different colors marking the boundary of each token ID, click Token ID to text. Media tokens aren't supported.
- To view the token IDs, click Token ID.
To close the tokenizer tool pane, click X, or click outside of the pane.
- Click Submit.
- Optional: To save your prompt to My prompts, click Save.
- Optional: To get the Python code or a curl command for your prompt, click Get code.
Click to expand advanced configurations
Set model parameters
The following model parameters can be set on multimodal models:
Top-P
Top-P changes how the model selects tokens for output. Tokens are selected
from the most (see top-K) to least probable until the sum of their probabilities
equals the top-P value. For example, if tokens A, B, and C have a probability of
0.3, 0.2, and 0.1 and the top-P value is 0.5
, then the model will
select either A or B as the next token by using temperature and excludes C as a
candidate.
Specify a lower value for less random responses and a higher value for more random responses.
Temperature
The temperature is used for sampling during response generation, which occurs when topP
and topK
are applied. Temperature controls the degree of randomness in token selection.
Lower temperatures are good for prompts that require a less open-ended or creative response, while
higher temperatures can lead to more diverse or creative results. A temperature of 0
means that the highest probability tokens are always selected. In this case, responses for a given
prompt are mostly deterministic, but a small amount of variation is still possible.
If the model returns a response that's too generic, too short, or the model gives a fallback response, try increasing the temperature.
Valid parameter values
Parameter | Gemini 1.5 Pro | Gemini 1.5 Flash |
---|---|---|
Top-P | 0 - 1.0 (default 0.95) | 0 - 1.0 (default 0.95) |
Temperature | 0 - 2.0 (default 1.0) | 0 - 2.0 (default 1.0) |
Audio requirements
Gemini multimodal models support the following audio MIME types:
Audio MIME type | Gemini 1.5 Flash | Gemini 1.5 Pro |
---|---|---|
AAC - audio/aac |
||
FLAC - audio/flac |
||
MP3 - audio/mp3 |
||
MPA - audio/m4a |
||
MPEG - audio/mpeg |
||
MPGA - audio/mpga |
||
MP4 - audio/mp4 |
||
OPUS - audio/opus |
||
PCM - audio/pcm |
||
WAV - audio/wav |
||
WEBM - audio/webm |
You can include a maximum of
Limitations
While Gemini multimodal models are powerful in many multimodal use cases, it's important to understand the limitations of the models:
- Non-speech sound recognition: The models that support audio might make mistakes recognizing sound that's not speech.
- Audio-only timestamps: The models that support audio can't accurately generate timestamps for requests with audio files. This includes segmentation and temporal localization timestamps. Timestamps can be generated accurately for input that includes a video that contains audio.
- Transcription punctuation: (if using Gemini 1.5 Flash) The models might return transcriptions that don't include punctuation.
What's next
- Start building with Gemini multimodal models - new customers get $300 in free Google Cloud credits to explore what they can do with Gemini.
- Learn how to send chat prompt requests.
- Learn about responsible AI best practices and Vertex AI's safety filters.