Supported models
The following table lists the models that support document understanding:
Model | PDF modality details |
---|---|
Gemini 1.5 Flash Go to the Gemini 1.5 Flash model card |
Maximum pages per PDF: 1000 Maximum PDF file size: 30 MB |
Gemini 1.5 Pro Go to the Gemini 1.5 Pro model card |
Maximum pages per PDF: 1000 Maximum PDF file size: 30 MB |
Gemini 1.0 Pro Vision Go to the Gemini 1.0 Pro Vision model card |
Maximum pages per prompt: 16 Maximum PDF file size: 30 MB |
For a list of languages supported by Gemini models, see model information Google models. To learn more about how to design multimodal prompts, see Design multimodal prompts. If you're looking for a way to use Gemini directly from your mobile and web apps, see the Vertex AI in Firebase SDKs for Android, Swift, web, and Flutter apps.
Add documents to a request
The following code sample shows you how to include a PDF in a prompt request. This PDF sample works with all Gemini multimodal models.
Python
To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Vertex AI SDK for Python API reference documentation.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the stream
parameter in
generate_content
.
response = model.generate_content(contents=[...], stream = True)
For a non-streaming response, remove the parameter, or set the parameter to
False
.
Sample code
Java
Before trying this sample, follow the Java setup instructions in the Vertex AI quickstart. For more information, see the Vertex AI Java SDK for Gemini reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
generateContentStream
method.
public ResponseStream<GenerateContentResponse> generateContentStream(Content content)
For a non-streaming response, use the
generateContent
method.
public GenerateContentResponse generateContent(Content content)
Sample code
Node.js
Before trying this sample, follow the Node.js setup instructions in the Generative AI quickstart using the Node.js SDK. For more information, see the Node.js SDK for Gemini reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
generateContentStream
method.
const streamingResp = await generativeModel.generateContentStream(request);
For a non-streaming response, use the
generateContent
method.
const streamingResp = await generativeModel.generateContent(request);
Sample code
Go
Before trying this sample, follow the Go setup instructions in the Vertex AI quickstart. For more information, see the Vertex AI Go SDK for Gemini reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
GenerateContentStream
method.
iter := model.GenerateContentStream(ctx, genai.Text("Tell me a story about a lumberjack and his giant ox. Keep it very short."))
For a non-streaming response, use the GenerateContent
method.
resp, err := model.GenerateContent(ctx, genai.Text("What is the average size of a swallow?"))
Sample code
C#
Before trying this sample, follow the C# setup instructions in the Vertex AI quickstart. For more information, see the Vertex AI C# reference documentation.
To authenticate to Vertex AI, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Streaming and non-streaming responses
You can choose whether the model generates streaming responses or non-streaming responses. For streaming responses, you receive each response as soon as its output token is generated. For non-streaming responses, you receive all responses after all of the output tokens are generated.
For a streaming response, use the
StreamGenerateContent
method.
public virtual PredictionServiceClient.StreamGenerateContentStream StreamGenerateContent(GenerateContentRequest request)
For a non-streaming response, use the
GenerateContentAsync
method.
public virtual Task<GenerateContentResponse> GenerateContentAsync(GenerateContentRequest request)
For more information on how the server can stream responses, see Streaming RPCs.
Sample code
REST
After you set up your environment, you can use REST to test a text prompt. The following sample sends a request to the publisher model endpoint.
Before using any of the request data, make the following replacements:
LOCATION
: The region to process the request. Enter a supported region. For the full list of supported regions, see Available locations.Click to expand a partial list of available regions
us-central1
us-west4
northamerica-northeast1
us-east4
us-west1
asia-northeast3
asia-southeast1
asia-northeast1
PROJECT_ID
: Your project ID.FILE_URI
: The Cloud Storage URI of the file to include in the prompt. The bucket object must either be publicly readable or reside in the same Google Cloud project that's sending the request. You must also specify the media type (mimeType
) of the file.If you don't have a PDF file in Cloud Storage, then you can use the following publicly available file:
gs://cloud-samples-data/generative-ai/pdf/2403.05530.pdf
with a mime type ofapplication/pdf
. To view this PDF, open the sample PDF file.MIME_TYPE
: The media type of the file specified in thedata
orfileUri
fields. Acceptable values include the following:Click to expand MIME types
application/pdf
audio/mpeg
audio/mp3
audio/wav
image/png
image/jpeg
text/plain
video/mov
video/mpeg
video/mp4
video/mpg
video/avi
video/wmv
video/mpegps
video/flv
TEXT
: The text instructions to include in the prompt. For example,You are a very professional document summarization specialist. Please summarize the given document.
To send your request, choose one of these options:
curl
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
cat > request.json << 'EOF' { "contents": { "role": "USER", "parts": [ { "fileData": { "fileUri": "FILE_URI", "mimeType": "MIME_TYPE" } }, { "text": "TEXT" } ] } } EOF
Then execute the following command to send your REST request:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/gemini-1.5-flash:generateContent"
PowerShell
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
@' { "contents": { "role": "USER", "parts": [ { "fileData": { "fileUri": "FILE_URI", "mimeType": "MIME_TYPE" } }, { "text": "TEXT" } ] } } '@ | Out-File -FilePath request.json -Encoding utf8
Then execute the following command to send your REST request:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/gemini-1.5-flash:generateContent" | Select-Object -Expand Content
You should receive a JSON response similar to the following.
Note the following in the URL for this sample:- Use the
generateContent
method to request that the response is returned after it's fully generated. To reduce the perception of latency to a human audience, stream the response as it's being generated by using thestreamGenerateContent
method. - The multimodal model ID is located at the end of the URL before the method
(for example,
gemini-1.5-flash
orgemini-1.0-pro-vision
). This sample may support other models as well.
Console
To send a multimodal prompt by using the Google Cloud console, do the following:
- In the Vertex AI section of the Google Cloud console, go to the Vertex AI Studio page.
- In Prompt design (single turn), click Open.
Optional: Configure the model and parameters:
- Model: Select a model.
- Region: Select the region that you want to use.
Temperature: Use the slider or textbox to enter a value for temperature.
The temperature is used for sampling during response generation, which occurs whentopP
andtopK
are applied. Temperature controls the degree of randomness in token selection. Lower temperatures are good for prompts that require a less open-ended or creative response, while higher temperatures can lead to more diverse or creative results. A temperature of0
means that the highest probability tokens are always selected. In this case, responses for a given prompt are mostly deterministic, but a small amount of variation is still possible.If the model returns a response that's too generic, too short, or the model gives a fallback response, try increasing the temperature.
Output token limit: Use the slider or textbox to enter a value for the max output limit.
Maximum number of tokens that can be generated in the response. A token is approximately four characters. 100 tokens correspond to roughly 60-80 words.Specify a lower value for shorter responses and a higher value for potentially longer responses.
- Add stop sequence: Optional. Enter a stop sequence, which is a series of characters that includes spaces. If the model encounters a stop sequence, the response generation stops. The stop sequence isn't included in the response, and you can add up to five stop sequences.
- Optional: To configure advanced parameters, click Advanced and configure as follows:
Top-K: Use the slider or textbox to enter a value for top-K. (not supported for Gemini 1.5).
Top-K changes how the model selects tokens for output. A top-K of1
means the next selected token is the most probable among all tokens in the model's vocabulary (also called greedy decoding), while a top-K of3
means that the next token is selected from among the three most probable tokens by using temperature.For each token selection step, the top-K tokens with the highest probabilities are sampled. Then tokens are further filtered based on top-P with the final token selected using temperature sampling.
Specify a lower value for less random responses and a higher value for more random responses.
- Top-P: Use the slider or textbox to enter a value for top-P.
Tokens are selected from most probable to the least until the sum of their
probabilities equals the value of top-P. For the least variable results,
set top-P to
0
. - Enable Grounding: Grounding isn't supported for multimodal prompts.
- To upload media, such as a PDF file, do the following:
- Click Insert Media, and select a source.
If you choose Google Drive as your source, you must choose an account and give consent to Vertex AI Studio to access your account the first time you select this option. You can upload multiple media files that have a total size of up to 10 MB. A single file can't exceed 7 MB.
- Click the file that you want to add.
- Click Select.
The file thumbnail displays in the Prompt pane. The total number of tokens also displays. If your prompt data exceeds the token limit, the tokens are truncated and aren't included in processing your data.
- Optional: To display the number of tokens calculated in your document files and the sum of all tokens, click View tokens.
The Token ID to text and Token ID views don't display any valuable output, because media tokens aren't supported.
To close the tokenizer tool pane, click X, or click outside of the pane.
- Enter your text prompt in the Prompt pane. The model uses previous messages as context for new responses.
- Optional: To display the number of tokens calculated in your documents, the number of text tokens, and the sum of all tokens, click View tokens. You can view the tokens or token IDs of your text prompt.
- To view the tokens in the text prompt that are highlighted with different colors marking the boundary of each token ID, click Token ID to text. Media tokens aren't supported.
- To view the token IDs, click Token ID.
To close the tokenizer tool pane, click X, or click outside of the pane.
- Click Submit.
- Optional: To save your prompt to My prompts, click Save.
- Optional: To get the Python code or a curl command for your prompt, click Get code.
- Optional: To clear all previous messages, click Clear conversation.
Click to expand advanced configurations
Set model parameters
The following model parameters can be set on multimodal models:
Top-P
Top-P changes how the model selects tokens for output. Tokens are selected
from the most (see top-K) to least probable until the sum of their probabilities
equals the top-P value. For example, if tokens A, B, and C have a probability of
0.3, 0.2, and 0.1 and the top-P value is 0.5
, then the model will
select either A or B as the next token by using temperature and excludes C as a
candidate.
Specify a lower value for less random responses and a higher value for more random responses.
Top-K
Top-K changes how the model selects tokens for output. A top-K of
1
means the next selected token is the most probable among all
tokens in the model's vocabulary (also called greedy decoding), while a top-K of
3
means that the next token is selected from among the three most
probable tokens by using temperature.
For each token selection step, the top-K tokens with the highest probabilities are sampled. Then tokens are further filtered based on top-P with the final token selected using temperature sampling.
Specify a lower value for less random responses and a higher value for more random responses.
Temperature
The temperature is used for sampling during response generation, which occurs when topP
and topK
are applied. Temperature controls the degree of randomness in token selection.
Lower temperatures are good for prompts that require a less open-ended or creative response, while
higher temperatures can lead to more diverse or creative results. A temperature of 0
means that the highest probability tokens are always selected. In this case, responses for a given
prompt are mostly deterministic, but a small amount of variation is still possible.
If the model returns a response that's too generic, too short, or the model gives a fallback response, try increasing the temperature.
Valid parameter values
Parameter | Gemini 1.0 Pro Vision | Gemini 1.5 Pro | Gemini 1.5 Flash |
---|---|---|---|
Top-K | 1 - 40 (default 32) | Not supported | Not supported |
Top-P | 0 - 1.0 (default 1.0) | 0 - 1.0 (default 0.95) | 0 - 1.0 (default 0.95) |
Temperature | 0 - 1.0 (default 0.4) | 0 - 2.0 (default 1.0) | 0 - 2.0 (default 1.0) |
Document requirements
Gemini multimodal models support the following document MIME types:
Document MIME type | Gemini 1.5 Flash | Gemini 1.5 Pro | Gemini 1.0 Pro Vision |
---|---|---|---|
PDF - application/pdf |
PDFs are treated as images, so a single page of a PDF is treated as one image. The number of pages allowed in a prompt is limited to the number of images the model can support:
- Gemini 1.0 Pro Vision: 16 pages
- Gemini 1.5 Pro and Gemini 1.5 Flash: 300 pages
PDFs are treated as images, so each page of a PDF is tokenized in the same way as an image.
Also, the cost for PDFs follows Gemini image pricing. For example, if you include a two-page PDF in a Gemini API call, you incur an input fee of processing two images.
PDF best practices
When using PDFs, use the following best practices and information for the best results:
- If your prompt contains a single PDF, place the PDF before the text prompt in your request.
- If you have a long document, consider splitting it into multiple PDFs to process it.
- Use PDFs created with text rendered as text instead of using text in scanned images. This format ensures text is machine-readable so that it's easier for the model to edit, search, and manipulate compared to scanned image PDFs. This practice provides optimal results when working with text-heavy documents like contracts.
Limitations
While Gemini multimodal models are powerful in many multimodal use cases, it's important to understand the limitations of the models:
- Spatial reasoning: The models aren't precise at locating text or objects in PDFs. They might only return the approximated counts of objects.
- Accuracy: The models might hallucinate when interpreting handwritten text in PDF documents.
What's next
- Start building with Gemini multimodal models - new customers get $300 in free Google Cloud credits to explore what they can do with Gemini.
- Learn how to send chat prompt requests.
- Learn about responsible AI best practices and Vertex AI's safety filters.