Write OTLP metrics by using an OpenTelemetry sidecar

Setting up gcloud defaults

To configure gcloud with defaults for your Cloud Run service:

1. Set your default project:

   gcloud config set project PROJECT_ID

   Replace PROJECT_ID with the name of the project you created for this tutorial.

2. Configure gcloud for your chosen region:

   gcloud config set run/region REGION

   Replace REGION with the supported Cloud Run region of your choice.

Creating a Artifact Registry image repository

Create an Artifact Registry Docker repository to host the sample service image:

gcloud artifacts repositories create run-otel \
    --repository-format=docker \
    --location=REGION \
    --project=PROJECT_ID

Replace the following:

• PROJECT_ID with the name of the project you created for this tutorial.
• REGION REGION with the supported Cloud Run region of your choice.

Retrieving the code sample

To retrieve the code sample for use:

1. Clone the sample app repository to your local machine:

   Go

   git clone https://github.com/GoogleCloudPlatform/golang-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

2. Change to the directory that contains the Cloud Run sample code:

   Go

   cd golang-samples/run/custom-metrics/

Reviewing the code

The code for this tutorial consists of the following:

• A server that handles incoming requests and generates a metric named sidecar_sample_counter.

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"os"

	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
	"go.opentelemetry.io/otel/metric"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
	semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

var counter metric.Int64Counter

func main() {
	ctx := context.Background()
	shutdown := setupCounter(ctx)
	defer shutdown(ctx)

	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
		log.Printf("defaulting to port %s", port)
	}

	http.HandleFunc("/", handler)
	log.Fatal(http.ListenAndServe(":"+port, nil))
}

func handler(w http.ResponseWriter, r *http.Request) {
	counter.Add(context.Background(), 100)
	fmt.Fprintln(w, "Incremented sidecar_sample_counter metric!")
}

func setupCounter(ctx context.Context) func(context.Context) error {
	serviceName := os.Getenv("K_SERVICE")
	if serviceName == "" {
		serviceName = "sample-cloud-run-app"
	}
	r, err := resource.Merge(
		resource.Default(),
		resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName(serviceName),
		),
	)
	if err != nil {
		log.Fatalf("Error creating resource: %v", err)
	}

	exporter, err := otlpmetricgrpc.New(ctx,
		otlpmetricgrpc.WithInsecure(),
	)
	if err != nil {
		log.Fatalf("Error creating exporter: %s", err)
	}
	provider := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exporter)),
		sdkmetric.WithResource(r),
	)

	meter := provider.Meter("example.com/metrics")
	counter, err = meter.Int64Counter("sidecar-sample-counter")
	if err != nil {
		log.Fatalf("Error creating counter: %s", err)
	}
	return provider.Shutdown
}

• A Dockerfile that defines the operating environment for the service.

FROM golang:1.21 as builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o sample-app

FROM alpine:3
RUN apk add --no-cache ca-certificates
COPY --from=builder /app/sample-app /sample-app
CMD ["/sample-app"]

The sample also includes files under the collector subdirectory for building a custom OpenTelemetry Collector:

• A config file for the OpenTelemetry Collector.

  receivers:
    otlp:
      protocols:
        grpc:
        http:
  
  processors:
    batch:
      # batch metrics before sending to reduce API usage
      send_batch_max_size: 200
      send_batch_size: 200
      timeout: 5s
  
    memory_limiter:
      # drop metrics if memory usage gets too high
      check_interval: 1s
      limit_percentage: 65
      spike_limit_percentage: 20
  
    # automatically detect Cloud Run resource metadata                                                                                                                                               
    resourcedetection:
      detectors: [env, gcp]
      timeout: 2s
      override: false
  
    resource:
      attributes:
        # add instance_id as a resource attribute                                                                                                                                                    
      - key: service.instance.id
        from_attribute: faas.id
        action: upsert
        # parse service name from K_SERVICE Cloud Run variable                                                                                                                                       
      - key: service.name
        value: ${env:K_SERVICE}
        action: insert
  
  exporters:
    googlemanagedprometheus: # Note: this is intentionally left blank   
  
  extensions:
    health_check:
  
  service:
    extensions: [health_check]
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [batch, memory_limiter, resourcedetection, resource]
        exporters: [googlemanagedprometheus]

• A Dockerfile that bundles the provided config into an upstream Collector image.

  FROM otel/opentelemetry-collector-contrib:0.101.0
  
  COPY collector-config.yaml /etc/otelcol-contrib/config.yaml

Shipping the code

Shipping code consists of three steps: building a container image with Cloud Build, uploading the container image to Artifact Registry, and deploying the container image to Cloud Run.

To ship your code:

1. Build your sample service container and publish on Artifact Registry:

   gcloud builds submit --tag REGION-docker.pkg.dev/PROJECT_ID/run-otel/sample-metrics-app

   Upon success, you should see a SUCCESS message containing the ID, creation time, and image name. The image is stored in Artifact Registry and can be re-used if desired.

2. Build your Collector container and publish on Artifact Registry:

   gcloud builds submit collector --tag REGION-docker.pkg.dev/PROJECT_ID/run-otel/otel-collector-metrics

   Upon success, you should see a SUCCESS message containing the ID, creation time, and image name. The image is stored in Artifact Registry and can be re-used if desired.

3. Deploy your application:

   YAML

   1. Create a new file called service.yaml with the following:

      apiVersion: serving.knative.dev/v1
      kind: Service
      metadata:
        name: SERVICE-NAME
        annotations:
          run.googleapis.com/launch-stage: BETA
      spec:
        template:
          metadata:
            annotations:
              run.googleapis.com/container-dependencies: "{app:[collector]}"
          spec:
            containers:
            - image: REGION-docker.pkg.dev/PROJECT_ID/run-otel/sample-metrics-app
              name: app
              ports:
              - containerPort: CONTAINER_PORT
              env:
              - name: "OTEL_EXPORTER_OTLP_ENDPOINT"
                value: "http://localhost:4317"
            - image: REGION-docker.pkg.dev/PROJECT_ID/run-otel/otel-collector-metrics
              name: collector
              startupProbe:
                httpGet:
                  path: /
                  port: 13133
      

   2. Replace the following:
      • CONTAINER_PORT with the port your service's ingress container listens on. By default, this is 8080.
      • SERVICE-NAME with any name for your service, such as custom-metrics-sample-service.

4. Create the new service with the following command:

   gcloud run services replace service.yaml

   This command returns a service URL. Use this URL to try out the sample application in Trying it out.

Trying it out

Using the URL from the gcloud run command in Shipping the code, connect to the service to generate some sample metrics (you can run this command several times to generate more interesting data):

curl -H \
"Authorization: Bearer $(gcloud auth print-identity-token)" \
SERVICE_URL

Replace SERVICE_URL with the URL of your service.

Next, navigate to the Metrics Explorer within the Cloud Monitoring section of the Google Cloud console and select the sidecar_sample_counter metric.

You can also query the metrics with PromQL. For example, the query below will filter metrics based on the Cloud Run instance ID:

sidecar_sample_counter{instance="INSTANCE_ID"}

Replace INSTANCE_ID with the ID of any instance for your service (available in the instance logs or from the metadata server).

This query produces a chart like the one below: