1. Before you begin
In this codelab, you'll learn to use Firebase Genkit for integrating generative AI into your app. Firebase Genkit is an open source framework that helps you build, deploy, and monitor production-ready AI-powered apps.
Genkit is designed for app developers, to help you easily integrate powerful AI capabilities into your apps with familiar patterns and paradigms. It's built by the Firebase team, leveraging our experience in building tools used by millions of developers worldwide.
Prerequisites
- Familiarity with Firestore, Node.js, and TypeScript.
What you'll learn
- How to build smarter apps with Firestore's advanced vector similarity search capabilities.
- How to practically implement generative AI in your apps using Firebase Genkit.
- Get your solution ready for deployment and integration.
What you'll need
- A browser of your choice, such as Google Chrome
- A development environment with a code editor and terminal
- A Google Account for the creation and management of your Firebase project
2. Review the web app and cloud services used
In this section, you'll review the web app that you'll build with this codelab, as well learn about the cloud services that you'll use.
Web app
In this codelab, you'll work in the codebase of an app called Compass — a vacation planning app. Users can pick a destination, look through activities at the destination, and create an itinerary for their trip.
In this codelab, you'll implement two new features that are intended to improve user engagement with the app's homepage. Both features are powered by generative AI:
- The app currently displays a static list of destinations. You'll change it to be dynamic!
- You'll implement an auto-populated itinerary to hopefully increase stickiness.
Services used
In this codelab, you'll use many Firebase and Cloud services and features, and most of the starter code for them is provided for you. The following table contains the services that you'll use and the reasons for using them.
Service | Reason for use |
You use Genkit to bring generative AI into a Node.js / Next.js application. | |
You store data in Cloud Firestore, which is then used for vector similarity search. | |
You use foundational models from Vertex AI (like Gemini) to power your AI features. | |
You can optionally use the new streamlined Firebase App Hosting to serve your dynamic Next.js web app (connected to a GitHub repo). |
3. Set up your development environment
Verify your Node.js version
- In your terminal, verify that you have Node.js version 20.0.0 or higher installed:
node -v
- If you don't have Node.js version 20.0.0 or higher, download the latest LTS version and install it.
Get the source code for the codelab
If you have a GitHub account:
- Create a new repository using our template from github.com/FirebaseExtended/codelab-ai-genkit-rag
- Create a local clone of the codelab's GitHub repository you just created:
git clone https://github.com/<your-github-handle>/codelab-ai-genkit-rag
If you don't have git installed or prefer to not create a new repo:
Download the GitHub repository as a zip file.
Review the folder structure
On your local machine, find the cloned repository and review the folder structure:
Folder | Description |
| Backend Genkit code |
| Helper command line tool to quickly pre-populate your Firestore collection |
*everything else | Next.js web app code |
The root folder includes a README.md
file that offers a quick start to run the web app, using streamlined instructions. However, if you're a first-time learner, you should complete this codelab (instead of the quickstart) because the codelab contains the most comprehensive set of instructions.
If you're unsure of whether you correctly applied code as instructed throughout this codelab, you can find the solution code in the end
git branch.
Install the Firebase CLI
- Verify that you have the Firebase CLI installed and that it's version 13.6 or higher:
firebase --version
- If you have the Firebase CLI installed, but it's not version 13.6 or higher, update it:
npm update -g firebase-tools
- If you don't have the Firebase CLI installed, install it:
npm install -g firebase-tools
If you're unable to update or install the Firebase CLI because of permission errors, see the npm documentation or use another installation option.
Log into Firebase
- In your terminal, log in to Firebase:
If your terminal says that you're already logged in to Firebase, you can skip to the Set up your Firebase project section of this codelab.firebase login
- In your terminal, depending on whether you want Firebase to collect data, enter
Y
orN
. (either option works for this codelab) - In your browser, select your Google Account and click Allow.
Install Google Cloud's gcloud CLI
- Install the gcloud CLI.
- In your terminal, log into Google Cloud:
gcloud auth login
4. Set up your Firebase project
In this section, you'll set up a Firebase project and register a Firebase Web App in it. You'll also enable a few services that are used by the sample web app later in this codelab.
All of the steps in this section are performed in the Firebase console.
Create a Firebase project
- Sign into the Firebase console using the same Google Account you used in the previous step.
- Click Create a project, and then enter a project name (for example,
Compass Codelab
).
Remember the auto-assigned project ID for your Firebase project (or click the Edit icon to set your preferred project ID). You'll need this ID later to identify your Firebase project in the Firebase CLI. If you forget your ID, you can always find it later in the Project Settings. - Click Continue.
- If prompted, review and accept the Firebase terms, and then click Continue.
- For this codelab, you do not need Google Analytics, so toggle off the Google Analytics option.
- Click Create project, wait for your project to provision, and then click Continue.
Add a web app to your Firebase project
- Navigate to the Project Overview screen in your Firebase project, and then click
Web.
- In the App nickname text box, enter a memorable app nickname, such as
My Compass Codelab App
. You can leave the check box for setting up Firebase Hosting unchecked, because you will be optionally setting up a hosting service in the last step of this codelab. - Click Register app > Continue to console.
Great! You have now registered a web app in your new Firebase project.
Upgrade your Firebase billing plan
To use Firebase Genkit and Vertex AI (and their underlying cloud services), you need to upgrade your Firebase project to enable billing.
To upgrade your project's billing plan, follow these steps:
- Navigate to Firebase Billing Plans within your Firebase project.
- In the Firebase billing plans dialog, select the Blaze plan and purchase it.
Enable Cloud Firestore
- Navigate to Build > Firestore Database using the left-navigation pane.
- Click Create database.
- Leave the Database ID set to
(default)
. - Select your preferred Cloud Firestore location (or just leave it as the default).
- Click Next.
- Select Start in test mode.
- Click Create.
Enable Vertex AI
Use the gcloud
CLI to set up Vertex AI. For all the commands on this page, make sure to replace YOUR_PROJECT_ID
with the ID of your Firebase project.
- In your terminal, set the default project for the Google Cloud SDK:
gcloud config set project YOUR_PROJECT_ID
- If you see a warning message saying "WARNING: Your active project does not match the quota project in your local Application Default Credentials file. This might result in unexpected quota issues.", then run the following command to set the quota project:
gcloud auth application-default set-quota-project YOUR_PROJECT_ID
- Enable the Vertex AI service in your project:
gcloud services enable aiplatform.googleapis.com
- Grant permissions to your service account:
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ --member "serviceAccount:firebase-app-hosting-compute@YOUR_PROJECT_ID.iam.gserviceaccount.com" \ --role "roles/aiplatform.user"
5. Set up the web app
To run the web app, you'll need to run commands in your terminal and add code within your code editor. For all the commands on this page, make sure to replace YOUR_PROJECT_ID
with the ID of your Firebase project.
Configure the Firebase CLI to target your project
- In your terminal, navigate to the root directory of your codelab project.
- To make the Firebase CLI execute all commands against your Firebase project, run the following command:
firebase use YOUR_PROJECT_ID
Import sample data into Firestore
So that you can get started quickly, this codelab provides you with pre-generated sample data for Firestore.
- To allow the local codebase to run code that would normally use a service account, run the following command in your terminal:
This will open a new tab in your browser. Log in using the same Google Account you used in the previous steps.gcloud auth application-default login
- To import the sample Firestore data, run the following commands:
cd load-firestore-data npm ci node index.js YOUR_PROJECT_ID cd ..
- Verify that the data was successfully added to your database by going to the Firestore section of your Firebase project in the Firebase console.You should see the imported data schema and its contents displayed.
Connect your web app to your Firebase project
Your web app's codebase needs to be associated with the correct Firebase project to utilize its services, such as the database. To accomplish this, you need to add your Firebase configuration to your app's codebase. This configuration includes essential values like your project ID, your app's API key and app ID, as well as other values that enable your app to interact with Firebase.
- Obtain your app's Firebase configuration:
- In the Firebase console, navigate to your Firebase project.
- In the left-side panel, click the gear icon next to Project Overview and select Project settings.
- In the "Your apps" card, select your web app.
- Under the "SDK setup and configuration" section, select the Config option.
- Copy the snippet. It begins with
const firebaseConfig ...
.
- Add your Firebase configuration to your web app's codebase:
- In your code editor, open the
src/lib/genkit/genkit.config.ts
file. - Replace the relevant section with the code that you copied.
- Save the file.
- In your code editor, open the
Preview the web app in your browser
- In your terminal, install dependencies and then run the web app:
npm install npm run dev
- In your browser, navigate to the locally hosted Hosting URL to view the web app. For example, in most cases, the URL is http://localhost:3000/ or something similar.
Compass is a Next.js app using React Server Components, and this is the homepage.
Click Find my dream trip. You can see it currently displays some hard-coded data for some fixed destinations:
Feel free to explore. When you're ready to continue, click the home button (in the top-right corner).
6. Dive into generative AI with Genkit
Now you're ready to leverage the power of generative AI in your application! This section of the codelab will guide you through implementing a feature that suggests destinations based on user-provided inspiration. You'll use Firebase Genkit and Google Cloud's Vertex AI as the provider for the generative model (you will be using Gemini).
Genkit can store trace and flow state (which allows you to inspect the result of executing Genkit flows). In this codelab, you'll be using Firestore to store these traces.
As a final step in this codelab, you'll deploy your Genkit app to Firebase App Hosting.
Connect your Genkit app to your Firebase project
Before you can launch Genkit, your codebase needs to be associated with the correct Firebase project to utilize its services, such as the database. To accomplish this, you need to add your Firebase configuration to your Genkit app's codebase. This configuration includes essential values like your project ID, your app's API key and app ID, as well as other values that enable your app to interact with Firebase.
- Obtain your app's Firebase configuration:
- In the Firebase console, navigate to your Firebase project.
- In the left-side panel, click the gear icon next to Project Overview and select Project settings.
- In the "Your apps" card, select your web app.
- Under the "SDK setup and configuration" section, select the Config option.
- Copy the snippet. It begins with
const firebaseConfig ...
.
- Add your app's Firebase configuration to your Genkit app's codebase:
- In your code editor, open the
genkit-functions/src/lib/genkit.config.ts
file. - Replace the relevant section with the code that you copied.
- Save the file.
- In your code editor, open the
Launch the Genkit Developer UI
Genkit comes with a web-based UI that allows you to interact with LLMs, Genkit flows, retrievers, and other AI components.
- Launch the Genkit Developer UI:
- Open a new terminal window.
- Navigate to the root of your
genkit-functions
directory. - Run the following command to start the Genkit developer UI:
cd genkit-functions npx genkit start
- In your browser, navigate to the locally hosted Genkit URL. In most cases, it's http://localhost:4000/.
Interact with Gemini
You can now use Genkit's Developer UI to interact with any of the supported models or any of the other AI components, such as the prompts, retrievers, and Genkit flows.
For example, try asking Gemini to suggest a holiday vacation. Note how you can use system instructions to steer the behaviour of the model based on your specific needs. This also works for models that don't natively support system instructions.
Manage prompts
Firebase Genkit introduces Dotprompt, a plugin and text format designed to streamline the creation and management of your generative AI prompts. The core idea behind Dotprompt is treating prompts as code, allowing you to write, maintain, and version control them alongside your application code.
To use Dotprompt, start with a hello-world:
- In your code editor, open the
genkit-functions/prompts/1-hello-world.prompt
file. - In the Genkit Developer UI, open
dotprompt/1-hello-world
. - Use any language name or code that you're familiar with, or leave it as an empty string.
- Click Run.
- Try a few different values. Large language models are good at understanding abbreviated, misspelled, or incomplete prompts in simple queries like this one.
Enrich your output with structured data
Beyond generating plain text, Genkit empowers you to structure your output for enhanced presentation and integration within your app's UI. By defining a schema, you can instruct the LLM to produce structured data that aligns with your desired format.
Exploring output schemas
You can also specify the output schema of an LLM call.
- In your code editor, examine the prompt file:
- Open the
dotprompt/2-simple-itinerary
file. - Examine the defined input and output schemas.
- Open the
- Interact with the UI:
- In the Genkit Developer UI, navigate to the
dotprompt/2-simple-itinerary
section. - Provide input by populating the
place
andinterests
fields with sample data:{ "interests": [ "Museums" ], "place": "Paris" }
- Click Run.
- In the Genkit Developer UI, navigate to the
Understanding schema-driven output
Notice how the generated output conforms to the defined schema. By specifying the desired structure, you've instructed the LLM to produce data that's easily parsed and integrated into your application. Genkit automatically validates the output against the schema, ensuring data integrity.
Moreover, you can configure Genkit to retry or attempt to repair the output if it doesn't match the schema.
Key advantages of output schemas
- Simplified integration: Easily incorporate structured data into your app's UI elements.
- Data validation: Ensure the accuracy and consistency of generated output.
- Error handling: Implement mechanisms to address schema mismatches.
Leveraging output schemas elevates your Genkit experience, enabling you to create tailored, structured data for richer and more dynamic user experiences.
Utilize multimodal input
Imagine your app suggesting personalized vacation destinations based on images your users find inspiring. Genkit, combined with a multimodal generative model, empowers you to bring this vision to life.
- In your code editor, examine the prompt file:
- Open the
genkit-functions/prompts/imgDescription.prompt
file. - Note the inclusion of
{{media url=this}}
, which is a Handlebars syntax element that facilitates incorporating images into your prompt.
- Open the
- Interact with the UI:
- In the Genkit Developer UI, open the
dotprompt/imgDescription
prompt. - Input an Image URL in the
imageUrls
field by pasting a URL of an image. For example, you can use a thumbnail image from Wikipedia showcasing the Eiffel Tower:{ "imageUrls": [ "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4b/La_Tour_Eiffel_vue_de_la_Tour_Saint-Jacques%2C_Paris_ao%C3%BBt_2014_%282%29.jpg/556px-La_Tour_Eiffel_vue_de_la_Tour_Saint-Jacques%2C_Paris_ao%C3%BBt_2014_%282%29.jpg" ] }
- Click Run.
- In the Genkit Developer UI, open the
7. Implement retrieval with vector similarity search
While generating creative content with AI models is impressive, practical applications often require grounding the output in a specific context.
In the case of this codelab, you have a database of destinations (places and activities) and aim to ensure that the Gemini model's generated suggestions exclusively reference entries within this database.
To bridge the gap between unstructured queries and relevant content, you'll harness the power of vector similarity search on generated embeddings.
Understanding embeddings and vector similarity
- Vectors: Vectors are numerical representations of data points, often used to model complex information like text or images. Each dimension in a vector corresponds to a specific feature of the data.
- Embedding Models: These specialized AI models transform input data, such as text, into high-dimensional vectors. The fascinating aspect is that similar inputs are mapped to vectors that are close to each other in this high-dimensional space.
- Vector Similarity Search: This technique leverages the proximity of embedding vectors to identify relevant data points. Given an input query, it finds entries in the database with similar embedding vectors, indicating semantic relatedness.
Understanding how the retrieval process works
- Embedding the query: Your user's input (for example, "romantic dinner in Paris") is passed through an embedding model, generating a query vector.
- Database embeddings: Ideally, you've pre-processed your database of destinations, creating embedding vectors for each entry.
- Similarity calculation: The query vector is compared to each embedding vector in the database using a similarity metric (for example, cosine similarity).
- Retrieval: The most similar entries from the database, based on their proximity to the query vector, are retrieved as relevant suggestions.
By incorporating this retrieval mechanism into your application, you leverage the Gemini model to generate suggestions that are not only creative but also firmly rooted in your specific dataset. This approach ensures that the generated output remains contextually relevant and aligned with the information available in your database.
Enable vector similarity search in Firestore
In a previous step of this codelab, you populated your Firestore database with sample places and activities. Each place entry includes a knownFor
text field describing its notable attributes, along with a corresponding embedding
field containing the vector representation of this description.
To leverage the power of vector similarity search on these embeddings, you'll need to create a Firestore index. This index enables efficient retrieval of places based on the similarity of their embedding vectors to a given query.
- In your terminal, run the following command to install the latest alpha component. You need version
2024.05.03
or later:gcloud components install alpha
- Create the index, making sure to replace
YOUR_PROJECT_ID
with your project's ID.gcloud alpha firestore indexes composite create --project=YOUR_PROJECT_ID --collection-group=places --query-scope=COLLECTION --field-config=vector-config='{"dimension":"768","flat": "{}"}',field-path=embedding
- In the Genkit Developer UI, open the
placesRetriever
. - Click Run. Observe the scaffolded object with placeholder text, indicating where you'll implement the retriever logic.
- In your code editor, open the
genkit-functions/src/lib/placesRetriever.ts
file. - Scroll all the way down and replace the placeholder
placesRetriever
with the following:export const placesRetriever = defineFirestoreRetriever({ name: 'placesRetriever', firestore, collection: 'places', contentField: 'knownFor', vectorField: 'embedding', embedder: textEmbeddingGecko, distanceMeasure: 'COSINE', });
Test the retriever
- In the Genkit Developer UI, open the
placesRetriever
retriever. - Provide the following Query:
{ "content": [ { "text": "UNESCO" } ] }
- You can also provide Options. For example, here's how to specify how many documents the retriever should return:
{ "limit": 4 }
- Click Run.
You can do additional filtering on the data beyond similarity by adding where
clauses to the Options.
8. Retrieval augmented generation (RAG) with Genkit
In previous sections, you've built individual prompts capable of handling text, JSON, and images, generating vacation destinations and other engaging content for your users. You've also implemented a prompt that retrieves relevant destinations from your Firestore database. Now, it's time to orchestrate these components into a cohesive retrieval augmented generation (RAG) flow.
This section introduces an important Genkit concept called flows. Flows are strongly typed, streamable functions that can be invoked both locally and remotely, with full observability. You can mange and invoke flows both from Genkit's CLI and the Genkit Developer UI.
- In your code editor, examine the itinerary prompt:
- Open the
genkit-functions/prompts/itineraryGen.prompt
file. - Note how the prompt has been expanded to accept additional inputs, specifically the activities data sourced from the retriever.
- Open the
- In the Genkit Developer UI, view a Genkit flow in the
genkit-functions/src/lib/itineraryFlow.ts
file.
Tip: To streamline debugging, consider breaking down lengthy flows into smaller, manageable steps. - Enhance the flow by integrating an "image description" step:
- Locate the
TODO: 2
comment (around line 70). This marks the spot where you'll enhance your flow. - Replace the empty
imgDescription
placeholder with the output generated by theimgDescription
prompt call.
- Locate the
- Test the flow:
- Navigate to the
itineraryFlow
. - Use the following input to test successful execution of
itineraryFlow
with your newly added step:{ "request": "Sightseeing in Paris", "imageUrls": [ "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4b/La_Tour_Eiffel_vue_de_la_Tour_Saint-Jacques%2C_Paris_ao%C3%BBt_2014_%282%29.jpg/556px-La_Tour_Eiffel_vue_de_la_Tour_Saint-Jacques%2C_Paris_ao%C3%BBt_2014_%282%29.jpg" ] }
- Click Run.
- Observe the generated output, which should incorporate the image description into the itinerary suggestion.
- Navigate to the
- If you encounter any errors or unexpected behaviors, check the Inspect tab for details. You can also use this tab to review the history of executions from the Trace Store.
RAG for your web application
- Make sure the web app is still running by visiting http://localhost:3000/ in your browser.
- If the web app is no longer running, run these commands in your terminal:
npm install npm run dev
- Check out the Dream Your Vacation web app page (http://localhost:3000/gemini).
- View its source code (
src/app/gemini/page.tsx
) for a Next.js integration example.
9. Deploy your application with Firebase App Hosting
The final step in this journey is deploying your web app. You'll leverage Firebase App Hosting, a framework-aware hosting solution designed to simplify the deployment of Next.js and Angular apps to a serverless backend.
- Commit your changes to your local git repository and then push to GitHub.
- In the Firebase console, navigate to App Hosting within your Firebase project.
- Click Get started > Connect to GitHub.
- Select your GitHub account and Repository. Click Next.
- In Deployment setting > Root directory, keep the default value.
- For the Live branch, select the main branch of your GitHub repo. Click Next.
- Enter an ID for your backend (for example,
compass
). - When asked whether to Create or associate a Firebase web app, choose Select an existing Firebase web app, and pick the app you created in an earlier step in this codelab.
- Click Finish and Deploy.
Monitoring deployment status
The deployment process will take a few minutes. You can track the progress in the App Hosting section of the Firebase console. Once completed, your web app will be accessible to users.
Automatic redeployment
Firebase App Hosting streamlines future updates. Whenever you push changes to the main branch of your GitHub repository, Firebase App Hosting will automatically rebuild and redeploy your app, ensuring your users always experience the latest version.
10. Conclusion
Congratulations on completing this comprehensive codelab!
You've successfully harnessed the power of Firebase Genkit, Firestore, and Vertex AI to create a sophisticated "flow" that generates personalized vacation recommendations based on user preferences and inspiration, all while grounding the suggestions in your application's data.
Throughout this journey, you've gained hands-on experience with essential software engineering patterns crucial for building robust generative AI applications. These patterns include:
- Prompt management: Organizing and maintaining prompts as code for better collaboration and version control.
- Multimodal content: Integrating diverse data types, such as images and text, to enhance AI interactions.
- Input/Output Schemas: Structuring data for seamless integration and validation in your application.
- Vector Stores: Leveraging vector embeddings for efficient similarity search and retrieval of relevant information.
- Data Retrieval: Implementing mechanisms to fetch and incorporate data from databases into AI-generated content.
- Retrieval Augmented Generation (RAG): Combining retrieval techniques with generative AI for contextually relevant and accurate outputs.
- Flow Instrumentation: Building and orchestrating complex AI workflows for seamless and observable execution.
By mastering these concepts and applying them within the Firebase ecosystem, you're well-equipped to embark on your own genAI adventures. Explore the vast possibilities, create innovative applications, and continue pushing the boundaries of what's achievable with generative AI.
Exploring alternative deployment options
Genkit offers a variety of deployment options to suit your specific needs, including:
- Cloud Functions for Firebase
- Cloud Run
- Next.js
- Any Node.js environment
- Genkit also supports Go
Just choose the one that's best for you by running the following command inside your (package.json
) node folder:
npx genkit init
Next steps
- Experiment with prompts and take advantage of the large context windows in Google AI Studio or Vertex AI Studio.
- Learn more about AI retrieval augmented generation (RAG) search.
- Check out the official docs for Firebase Genkit.
- Learn more about similarity search capabilities in Firestore and Cloud SQL for PostgreSQL.
- Go deeper into genAI flows with function calling.