Migrate to multi-cluster networking with Google Kubernetes Engine (GKE) Autopilot and Standard clusters

Prepare the environment

In this guide, you use Cloud Shell to enter commands. Cloud Shell gives you access to the command line in the Google Cloud console and includes Google Cloud SDK and other tools, such as the Google Cloud CLI. Cloud Shell appears as a window at the bottom of the Google Cloud console. It can take several minutes to initialize, but the window appears immediately.

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

2. In Cloud Shell, define environment variables used in this guide. Replace PROJECT_ID with your own project ID:

   export PROJECT=PROJECT_ID
   gcloud config set project ${PROJECT}
   

3. Enable the services required for the steps on this page:

   gcloud services enable \
       gkehub.googleapis.com \
       multiclusteringress.googleapis.com \
       dns.googleapis.com \
       trafficdirector.googleapis.com \
       cloudresourcemanager.googleapis.com \
       multiclusterservicediscovery.googleapis.com \
       container.googleapis.com
   
   gcloud container fleet multi-cluster-services enable
   

   Multi-cluster Services manages Google Cloud components like Cloud DNS, firewall rules, and Cloud Service Mesh, so these APIs must also be enabled. For more information, see Cloud Service Mesh overview.

   The output is similar to the following example:

   Operation "operations/acf.p2-822685001869-ee4ebe78-6dd8-465e-b0fd-3b0e5f964bad"
   finished successfully.
   
   Waiting for Feature Multi-cluster Services to be created...done.
   

4. Verify that multi-cluster Services shows the ACTIVE state:

   gcloud container fleet multi-cluster-services describe
   

   The output is similar to the following example:

   createTime: '2021-11-30T21:59:25.245190894Z'
   name: projects/PROJECT_ID/locations/global/features/multiclusterservicediscovery
   resourceState:
     state: ACTIVE
   spec: {}
   updateTime: '2021-11-30T21:59:27.459063070Z'
   

   If the value of state is not ACTIVE, see the multi-cluster services troubleshooting details.

Create and configure GKE clusters

To demonstrate the multi-cluster pattern in this guide, you use three application clusters in three separate cloud regions, and one cluster to host the configuration for Gateway resources. You register all clusters with the fleet associated with your project. A Google Cloud project can only have a single fleet associated with it. This project is known as the fleet host project.

1. Create Standard and Autopilot GKE clusters:

   gcloud container clusters create std-west \
       --zone us-west1-a \
       --num-nodes=6 \
       --enable-ip-alias \
       --release-channel regular \
       --workload-pool=${PROJECT}.svc.id.goog \
       --async
   
   gcloud container clusters create-auto auto-east \
       --region us-east1 \
       --release-channel regular \
       --async
   
   gcloud container clusters create-auto auto-central \
       --region us-central1 \
       --release-channel regular \
       --async
   
   gcloud container clusters create config-central \
       --region us-central1 \
       --num-nodes=1 \
       --enable-ip-alias \
       --release-channel regular \
       --workload-pool=${PROJECT}.svc.id.goog \
       --async
   

   Workload Identity Federation for GKE is enabled by default on GKE Autopilot clusters so you don't have to use the --workload-pool flag when you create those clusters like you do with the GKE Standard clusters.

2. Wait for the clusters' STATUS to change from PROVISIONING to RUNNING. This process can take up to 10 minutes. You can monitor the progress by using a watch loop:

   watch -n 20 --difference=permanent "gcloud container clusters list"
   

   The output is similar to the following example:

   NAME: auto-central
   LOCATION: us-central1
   MASTER_VERSION: 1.21.5-gke.1802
   MASTER_IP: 107.178.213.138
   MACHINE_TYPE: e2-medium
   NODE_VERSION: 1.21.5-gke.1802
   NUM_NODES: 3
   STATUS: PROVISIONING
   
   NAME: config-central
   LOCATION: us-central1
   MASTER_VERSION: 1.21.5-gke.1802
   MASTER_IP:
   MACHINE_TYPE: e2-medium
   NODE_VERSION: 1.21.5-gke.1802
   NUM_NODES: 9
   STATUS: PROVISIONING
   
   NAME: auto-east
   LOCATION: us-east1
   MASTER_VERSION: 1.21.5-gke.1802
   MASTER_IP: 35.229.88.209
   MACHINE_TYPE: e2-medium
   NODE_VERSION: 1.21.5-gke.1802
   NUM_NODES: 3
   STATUS: PROVISIONING
   
   NAME: std-west
   LOCATION: us-west1-a
   MASTER_VERSION: 1.21.5-gke.1802
   MASTER_IP: 35.197.93.113
   MACHINE_TYPE: e2-medium
   NODE_VERSION: 1.21.5-gke.1802
   NUM_NODES: 6
   STATUS: PROVISIONING
   

3. After all the clusters are in the RUNNING state, press CTRL-C to interrupt the command.

4. Add an Identity and Access Management (IAM) policy binding granting the fleet host project MCS service account the Network User role for its own project:

   gcloud projects add-iam-policy-binding ${PROJECT} \
       --member "serviceAccount:${PROJECT}.svc.id.goog[gke-mcs/gke-mcs-importer]" \
       --role "roles/compute.networkViewer"
   

   You use Workload Identity Federation for GKE to grant the MCS service read access to your project VPC network configuration. As a result, the fleet host project's MCS Importer GKE service account needs this role.

   The output is similar to the following example:

   - members:
     - serviceAccount:PROJECT_ID.svc.id.goog[gke-mcs/gke-mcs-importer]
     role: roles/compute.networkViewer
   [...]
   

5. Register the GKE Standard and Autopilot clusters to your project's fleet. See Registering a cluster for more details. This step can take up to 5 minutes:

   gcloud container fleet memberships register std-west \
       --gke-cluster us-west1-a/std-west \
       --enable-workload-identity \
       --project=${PROJECT}
   
   gcloud container fleet memberships register auto-east \
       --gke-cluster us-east1/auto-east \
       --enable-workload-identity \
       --project=${PROJECT}
   
   gcloud container fleet memberships register auto-central \
       --gke-cluster us-central1/auto-central \
       --enable-workload-identity \
       --project=${PROJECT}
   
   gcloud container fleet memberships register config-central \
       --gke-cluster us-central1/config-central \
       --enable-workload-identity \
       --project=${PROJECT}
   

   For each command, the output is similar to the following example:

   Waiting for membership to be created...done.
   Created a new membership [projects/PROJECT_ID/locations/global/memberships/std-west] for the cluster [std-west]
   Generating the Connect Agent manifest...
   Deploying the Connect Agent on cluster [std-west] in namespace [gke-connect]...
   Deployed the Connect Agent on cluster [std-west] in namespace [gke-connect].
   Finished registering the cluster [std-west] with the Hub.
   

6. Connect to the clusters and generate kubeconfig entries:

   gcloud container clusters get-credentials std-west \
       --zone us-west1-a --project $PROJECT
   
   gcloud container clusters get-credentials auto-east \
       --region us-east1 --project $PROJECT
   
   gcloud container clusters get-credentials auto-central \
       --region us-central1 --project $PROJECT
   
   gcloud container clusters get-credentials config-central \
       --region us-central1 --project $PROJECT
   

   For each command, the output is similar to the following example:

   Fetching cluster endpoint and auth data.
   kubeconfig entry generated for std-west.
   

7. Rename contexts for clusters so that they are easier to work with in the rest of this page:

   kubectl config rename-context \
       gke_${PROJECT}_us-west1-a_std-west \
       std-west
   
   kubectl config rename-context \
       gke_${PROJECT}_us-east1_auto-east \
       auto-east
   
   kubectl config rename-context \
       gke_${PROJECT}_us-central1_auto-central \
       auto-central
   
   kubectl config rename-context \
       gke_${PROJECT}_us-central1_config-central \
       config-central
   

   In this guide, the contexts are named based on their location. Although you can provide alternate names, the remaining steps in this guide use the names used in this step.

Deploy Online Boutique on GKE Standard

In the first step of the demonstration deployment, you deploy the full set of Online Boutique application services to the single GKE Standard cluster std-west in us-west1.

1. Create the namespace onlineboutique on std-west:

   kubectl create namespace onlineboutique --context std-west
   

   The output is similar to the following example:

   namespace/onlineboutique created
   

2. Clone the Online Boutique GitHub repository and set up a WORKDIR variable:

   cd ~
   
   git clone --branch release/v0.4.1 \
       https://github.com/GoogleCloudPlatform/microservices-demo.git
   
   cd microservices-demo/release && export WORKDIR=`pwd`
   

3. Deploy Online Boutique on std-west. This process creates Deployments and Services for all of Online Boutique's microservices, and includes a LoadBalancer type Service which externally exposes Online Boutique's frontend service:

   cd $WORKDIR
   
   kubectl apply -f kubernetes-manifests.yaml \
       -n onlineboutique --context=std-west
   

4. Wait for the LoadBalancer Service to get an external IP:

   watch -n 20 --difference=permanent \
        "kubectl get svc frontend-external -n onlineboutique --context=std-west"
   

   Initially, the output is similar to the following example:

   NAME                TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
   frontend-external   LoadBalancer   10.60.5.62   <pending>     80:30359/TCP   43s
   

   When the Service is ready the EXTERNAL-IP column displays the public IP address of the load balancer.

5. After the Service is ready, get the external IP address of the load balancer and use curl to verify that the frontend is ready. If this curl command returns an error, wait a few moments before you try again:

     curl $(kubectl get svc frontend-external \
         -n onlineboutique --context=std-west \
         -o=jsonpath="{.status.loadBalancer.ingress[0].ip}") | \
           grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>std-west<br/>
   <b>Zone: </b>us-west1-a<br/>
   <b>Pod: </b>frontend-b7bddcc97-wdjsk
   

You now have a single-zone version of Online Boutique running in us-west1-a. You can also use a web browser to navigate to the external IP assigned to the frontend-external LoadBalancer Service to access the application and observe its behavior. This initial single deployment is shown in the following diagram:

Export cartservice as a multi-cluster Service

In this section, you start to add elements of high availability to the application. You export the backend cartservice as a multi-cluster Service to the GKE Autopilot clusters.

1. Create the namespace onlineboutique on the remaining clusters:

   kubectl create namespace onlineboutique --context auto-east
   
   kubectl create namespace onlineboutique --context auto-central
   
   kubectl create namespace onlineboutique --context config-central
   

   For each command, the output is similar to the following example:

   namespace/onlineboutique created
   

2. Export cartservice from the std-west cluster to all other clusters in the ClusterSet. The ServiceExport object registers the cartservice Service, with GKE multi-cluster Services, for export to all clusters in the fleet that have the namespace onlineboutique present. For more details, see registering a service for export.

   cat <<EOF>> $WORKDIR/cartservice-export.yaml
   kind: ServiceExport
   apiVersion: net.gke.io/v1
   metadata:
    namespace: onlineboutique
    name: cartservice
   
   EOF
   
   kubectl apply -f $WORKDIR/cartservice-export.yaml \
       -n onlineboutique --context=std-west
   

Apply application manifests for multi-cluster pattern

In this section you apply two curated manifests to deploy the multi-cluster pattern. These manifests contain selected portions of the kubernetes-manifests.yaml that you previously applied to the std-west cluster:

• The first manifest is used for the frontend Deployment, Service, and ServiceExport.
• The second manifest is used to deploy the middleware Services (emailservice, checkoutservice, recommendationservice, paymentservice, productcatalogservice, currencyservice, shippingservice, and adservice) to all regions that have a frontend running. By keeping a request local to a region for as long as possible, you avoid unnecessary inter-region network traffic charges.

A Pod that runs in any cluster in the fleet can access an exported Service by sending a request to the ClusterSet URI for that service in the format SERVICE_NAME.NAMESPACE.svc.clusterset.local. For example, the frontend Deployments in all three example clusters are able to consume the cartservice, in the onlineboutique namespace, by making a request to cartservice.onlineboutique.svc.clusterset.local.

For this reason, in each manifest, the hostname for the cartservice has been updated to its ClusterSet URI. This step is critical. If this service hostname is not updated, the frontend Service would ask kube-dns for cartservice instead of cartservice.onlineboutique.svc.clusterset.local. This behavior would result in HTTP Status 500 errors on clusters where a local version of cartservice is not available, and cause the frontend pods to be unhealthy.

1. Set an environment varianble for the GitHub repository that contains the manifests:

   export MANIFEST_REPO_PATH=https://raw.githubusercontent.com/GoogleCloudPlatform/gke-networking-recipes/master/gateway/docs/cluster-migration
   

2. Apply the manifests to deploy the frontend layer to all three workload clusters:

   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-frontend-manifests.yaml \
       -n onlineboutique --context=std-west
   
   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-frontend-manifests.yaml \
       -n onlineboutique --context=auto-east
   
   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-frontend-manifests.yaml \
       -n onlineboutique --context=auto-central
   

3. Apply the manifests to deploy the middleware layer to all three workload clusters:

   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-middleware-manifests.yaml \
       -n onlineboutique --context=std-west
   
   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-middleware-manifests.yaml \
       -n onlineboutique --context=auto-east
   
   kubectl apply -f ${MANIFEST_REPO_PATH}/onlineboutique-middleware-manifests.yaml \
       -n onlineboutique --context=auto-central
   

You now have the frontend Deployment, Service, and ServiceExport active in clusters std-west, auto-east, and auto-central. You also have locally running Online Boutique middleware services in each cluster. However, external traffic is still routed only to the Service that runs in the initial cluster in us-west1, as shown in the following diagram:

Enable and configure multi-cluster Gateways

In this section, you route traffic to and load balance external traffic across frontends in all three clusters. To achieve this configuration, you use multi-cluster Gateways (MCG). These steps to set up MCG follow the guidance described in more detail on Enabling multi-cluster Gateways.

In these steps, you use the config-central cluster to host the configuration for Gateway resources.

1. Confirm that all clusters have successfully registered to the fleet:

   gcloud container fleet memberships list --project=$PROJECT
   

   The following example output shows that all the clusters are successfully registered:

   NAME: auto-central
   EXTERNAL_ID: 21537493-32ea-4a41-990d-02be2c1b319f
   
   NAME: config-central
   EXTERNAL_ID: 4369423e-ea7b-482d-a0eb-93b560e67b98
   
   NAME: std-west
   EXTERNAL_ID: 7fcb048b-c796-476b-9698-001a00f91ab3
   
   NAME: auto-east
   EXTERNAL_ID: aae2d2ff-b861-4a38-bcaf-612f14810012
   

2. Install the Gateway API custom resource definition on the config-central cluster:

   kubectl --context=config-central kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.5.0" \
       | kubectl apply -f -
   

   This step installs the Gateway API custom resource definitions including the GatewayClass, Gateway, and HTTPRoute resources. The custom resource definitions are maintained by the Kubernetes Network Special Interest Group. Once installed, you can use the GKE Gateway controller.

3. Enable Multi Cluster Ingress for your fleet if you haven't done so already. Enabling this feature also enables the multi-cluster Gateway controller.

   gcloud container fleet ingress enable \
       --config-membership=config-central \
       --project=$PROJECT
   
   gcloud container fleet ingress describe --project=$PROJECT
   

   The output is similar to the following example:

   createTime: '2021-12-08T23:10:52.505888854Z'
   name: projects/PROJECT_ID/locations/global/features/multiclusteringress
   resourceState:
     state: ACTIVE
   spec:
     multiclusteringress:
       configMembership: projects/zl-mcs-expf61cbd13/locations/global/memberships/config-central
   state:
     state:
       code: OK
       description: Ready to use
       updateTime: '2021-12-08T23:11:37.994971649Z'
   updateTime: '2021-12-08T23:11:38.098244178Z'
   

   If the value of state is not ACTIVE, see Troubleshooting and operations for Multi Cluster Ingress.

4. Confirm that GatewayClasses are available on the config-central cluster:

   kubectl get gatewayclasses --context=config-central
   

   The output is similar to the following example:

   NAME                                  CONTROLLER                  AGE
   gke-l7-global-external-managed        networking.gke.io/gateway   18s
   gke-l7-global-external-managed-mc     networking.gke.io/gateway   19s
   gke-l7-regional-external-managed      networking.gke.io/gateway   18s
   gke-l7-regional-external-managed-mc   networking.gke.io/gateway   19s
   gke-l7-gxlb                           networking.gke.io/gateway   74s
   gke-l7-gxlb-mc                        networking.gke.io/gateway   16s
   gke-l7-rilb                           networking.gke.io/gateway   74s
   gke-l7-rilb-mc                        networking.gke.io/gateway   16s
   

   Different GatewayClass resources have different capabilities. For more information on when to use which type, see GatewayClass capabilities.

5. Deploy the external-http Gateway resource to config-central:

   cat <<EOF>> $WORKDIR/external-http-gateway.yaml
   kind: Gateway
   apiVersion: gateway.networking.k8s.io/v1beta1
   metadata:
     name: external-http
     namespace: onlineboutique
   spec:
     gatewayClassName: gke-l7-global-external-managed-mc
     listeners:
     - protocol: HTTP
       port: 80
       name: http
   EOF
   
   kubectl apply -f external-http-gateway.yaml \
       -n onlineboutique --context=config-central
   

   As indicated by the gatewayClassName field, this resource is of GatewayClass gke-l7-global-external-managed-mc which manages Layer 7 external Cloud Load Balancing and exposes the multi-cluster application

6. Deploy the HTTPRoute named public-frontend-route on config-central:

   cat <<EOF>> $WORKDIR/public-frontend-route.yaml
   kind: HTTPRoute
   apiVersion: gateway.networking.k8s.io/v1beta1
   metadata:
     name: public-frontend-route
     namespace: onlineboutique
   spec:
     parentRefs:
     - name: "external-http"
     hostnames:
     - "store.example.com"
     rules:
     - matches:
       - path:
           type: PathPrefix
           value: /
       backendRefs:
       - name: frontend
         group: net.gke.io
         kind: ServiceImport
         port: 80
   EOF
   
   kubectl apply -f public-frontend-route.yaml \
       -n onlineboutique --context=config-central
   

   When you deploy the HTTPRoute resource it creates an external Layer 7 Cloud Load Balancing resource and exposes the frontend ServiceImport backed by the frontend services that run in the std-west, auto-east, and auto-central clusters.

   The following diagram shows how, after the multi-cluster Gateway is deployed, traffic can be routed to any of the frontend multi-cluster Services on any of the three application clusters:

   

7. Wait for the load balancer to be ready with a provisioned external IP address before proceeding to the next step. It can take up to 10 minutes for the IP address to be assigned. You can monitor progress by using a watch loop. The load balancer has a name in the pattern like gkemcg-onlineboutique-external-http-k09mfhk74gop:

   watch -n 20 --difference=permanent \
       "gcloud compute forwarding-rules list \
           | grep -A 5 NAME..*external-http"
   

   The output is similar to the following example:

   NAME: gkemcg-onlineboutique-external-http-k09mfhk74gop
   REGION:
   IP_ADDRESS: 34.149.29.176
   IP_PROTOCOL: TCP
   TARGET: gkemcg-onlineboutique-external-http-k09mfhk74gop
   

8. Once the load balancer is ready, run the following command in Cloud Shell to export the external IP address of the load balancer created through the application of external-http-gateway.yaml and public-frontend-route.yaml manifests:

   export EXTERNAL_LB_IP=$(kubectl --context=config-central \
                               -n onlineboutique get gateway external-http \
                               -o=jsonpath='{.status.addresses[0].value}')
   

9. When you send a request to the load balancer with the appropriate headers, it returns the HTML contents served by the frontend service. For example, since you configured the HTTPRoute resource to map the store.example.com hostname to the frontend ServiceImport, you must supply the HOST header when you make the HTTP request. If the following curl example returns an error, wait a few minutes and try again:

   curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP | \
       grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>auto-central<br/>
   <b>Zone: </b>us-central1-f<br/>
   <b>Pod: </b>frontend-7c7d596ddc-jdh8f
   

Test application multi-region routing behavior

One of the powerful features obtained by using multi-cluster Services and multi-cluster Gateways is that external requests are routed to the geographically closest cluster.

To test application multi-region behavior, generate traffic that originates from the various regions where you have clusters deployed. Create three small pods, one in each of the serving clusters (std-west, auto-east, and auto-central), that you can use to send HTTP requests to the load balancer endpoint. The results let you see which frontend Pod responds.

1. Create the client pods:

   kubectl run --context=std-west \
               --image=radial/busyboxplus:curl client-west \
               -- sh -c 'while sleep 3600; do :; done'
   
   kubectl run --context=auto-east \
               --image=radial/busyboxplus:curl client-east \
               -- sh -c 'while sleep 3600; do :; done'
   
   kubectl run --context=auto-central \
               --image=radial/busyboxplus:curl client-central \
               -- sh -c 'while sleep 3600; do :; done'
   

2. After the pods are running, use a curl command to send a request to the load balancer endpoint from the client Pod in the std-west cluster and review the response:

   kubectl exec -it --context=std-west client-west \
       -- curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP | \
              grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>std-west<br/>
   <b>Zone: </b>us-west1-a<br/>
   <b>Pod: </b>frontend-7cf48b79cf-trzc4
   

3. Run the same curl request from the client Pod in the auto-east cluster, and look at the response:

   kubectl exec -it --context=auto-east client-east \
       -- curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP | \
              grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>auto-east<br/>
   <b>Zone: </b>us-east1-d<br/>
   <b>Pod: </b>frontend-6784b6df98-scdws
   

   As this is an Autopilot cluster, the cluster may need to provision additional resources to schedule the Pod. If you see output similar to the following example, wait a moment and try again:

    Error from server (BadRequest): pod client-east does not have a host assigned
   

4. Run the curl from the client Pod in the auto-central cluster and check the response:

   kubectl exec -it --context=auto-central client-central \
       -- curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP | \
           grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>auto-central<br/>
   <b>Zone: </b>us-central1-b<br/>
   <b>Pod: </b>frontend-6784b6df98-x2fv4
   

   These results confirm that traffic routes to the corresponding pods in the locations closest to request origin.

Test application multi-region resiliency

In addition to efficient traffic routing, running your services in multiple regions provides resiliency in the rare, yet still possible, case of infrastructure failure.

Test the behavior by deleting the frontend Deployments in specific clusters and then retry the curl command from the client Pod in those regions. Observe that the application is still available, and look at the location of the Pod that responds to the request.

1. Run the curl command from the client-west Pod in the std-west cluster, and see that the result comes from the frontend in us-west1:

   kubectl exec -it --context=std-west client-west \
       -- curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP  | \
              grep -e Cluster -e Zone -e Pod
   

   The successful output of the curl command is similar to the following example:

   <b>Cluster: </b>std-west<br/>
   <b>Zone: </b>us-west1-a<br/>
   <b>Pod: </b>frontend-7cf48b79cf-trzc4
   

2. Delete the frontend Deployment in the std-west cluster:

   kubectl delete deploy frontend \
       -n onlineboutique --context=std-west
   

   The output is similar to the following example:

   deployment.apps "frontend" deleted
   

3. Send another request from the client-west Pod in the std-west cluster. You should see a response from one of the remaining frontend Deployments located in the auto-east or auto-central clusters:

   kubectl exec -it --context=std-west client-west \
       -- curl -H 'HOST: store.example.com' $EXTERNAL_LB_IP | \
              grep -e Cluster -e Zone -e Pod
   

   Output similar to the following example indicates the location of the healthy Pod that responds to this request:

   <b>Cluster: </b>auto-central<br/>
   <b>Zone: </b>us-central1-b<br/>
   <b>Pod: </b>frontend-6784b6df98-x2fv4
   

   or

   <b>Cluster: </b>auto-east<br/>
   <b>Zone: </b>us-east1-d<br/>
   <b>Pod: </b>frontend-6784b6df98-scdws
   

   Run the command several times to see alternating results.

With this demonstration deployment, you have added elements of resiliency and geographic distribution to the Online Boutique application using multi-cluster Services and multi-cluster Gateways. Requests are routed to the closest geographical region and even if the frontend or middleware services in a region experience issues, the end user is still able to successfully use the application.