Migrating to Google Cloud Observability Without Losing Visibility
How dual-shipping with OpenTelemetry can preserve telemetry while teams validate a new monitoring backend.In the era of cloud-native microservices, observability isn’t just a luxury; it’s an operational necessity. As infrastructure scales, OpenTelemetry (OTel) has emerged as the clear industry…
How dual-shipping with OpenTelemetry can preserve telemetry while teams validate a new monitoring backend.In the era of cloud-native microservices, observability isn’t just a luxury; it’s an operational necessity. As infrastructure scales, OpenTelemetry (OTel) has emerged as the clear industry standard for vendor-neutral telemetry collection. It frees you from vendor lock-in and gives you total control over your data.However, as organizations grow their footprint on Google Cloud Platform (GCP), migrating from expensive third-party tools or fragmented open-source setups to Google Cloud Observability (Cloud Monitoring, Logging, and Trace) becomes an incredibly attractive proposition. The drivers are clear: massive cost reduction and tight, native platform integration based on open standards.But a massive question remains for SRE and DevOps teams:How do you migrate gigabytes of telemetry data across hundreds of production services without losing visibility?In this guide, we will walk through a robust Zero-Downtime Migration strategy using the OpenTelemetry Collector Contrib version (recent release with a step-by-step demo) and Dual-Shipping.Furthermore, we will explore how the historical friction of onboarding engineers to a new monitoring backend is being eliminated by Agentic AI Frameworks in complete no code fashion using the Antigravity 2.0 desktop app that allows anyone to query complex infrastructure using simple, natural language.The Strategy and Safety Net: Dual-Shipping with OpenTelemetry ContribFixing an engine while flying the plane is a recipe for disaster. You cannot simply flip a switch from your legacy provider to Google Cloud.Dual-shipping is the practice of configuring your OpenTelemetry Collector infrastructure to ingest telemetry once, but export it to both your legacy backend and your new Google Cloud backend simultaneously.This dual-pipeline state acts as your safety net. It allows your teams to validate dashboards, recreate alerts, and verify data parity in GCP for days or weeks without disrupting your existing, trusted monitoring system.Sample architecture for GCP onlyThe Toolkit: OTel ContribTo achieve dual-shipping, the bare-bones OpenTelemetry Core Collector won’t cut it. The core distribution only includes basic components. Instead, you need the OpenTelemetry Collector Contrib version. The latest is V0.154.0, which is demonstrated subsequently.The Contrib distribution contains the extended library of components maintained by the community. Crucially, it bundles specialized non-core exporters, like the Datadog exporter alongside the Google Cloud exporter, making it the ultimate bridge for seamless migration.Prerequisites:Before changing your architecture, ensure your environment meets these baselines:Instrumented Applications: Applications must already be emitting OTLP data over either gRPC (port 4317) or HTTP (port 4318).GCP Project Infrastructure: The target Google Cloud APIs must be actively enabled (monitoring.googleapis.com, logging.googleapis.com, and cloudtrace.googleapis.com).IAM Permissions: The IAM Service Account or Workload Identity used by your OTel Collector needs the following roles attached:roles/monitoring.metricWriterroles/logging.logWriterroles/cloudtrace.agentFor this demonstration, it is primarily focused on shipping telemetry data to GCP and not Datadog.For this demo, you may feel free to use the sample telemetry demo app from here.Sample dual-shipping pipeline, i.e., OTel YAML file# =========================================================================# OpenTelemetry Collector Contrib Configuration# Dual Shipping: Google Cloud Observability + Legacy Backend (Datadog)# =========================================================================receivers: otlp: protocols: grpc: endpoint: "0.0.0.0:4317" http: endpoint: "0.0.0.0:4318"processors: # Batching minimizes API call volume, improves network efficiency, and reduces costs batch: timeout: 5s send_batch_size: 200 send_batch_max_size: 200 # CRITICAL: Automatically enriches metrics/logs/traces with GCP resource metadata # (e.g., GKE cluster name, zone, project, instance ID) resourcedetection/gcp: detectors: [gcp] timeout: 2sexporters: # ----------------------------------------------------------------------- # Destination 1: Google Cloud Observability # ----------------------------------------------------------------------- googlecloud: project: "${GCP_PROJECT_ID}" metric: prefix: "custom.googleapis.com/otel/" log: default_log_name: "otel_application_logs" # ----------------------------------------------------------------------- # Destination 2: Legacy Backend (e.g., Datadog) # ----------------------------------------------------------------------- datadog: api: key: "${DD_API_KEY}" site: "${DD_SITE}" # e.g., datadoghq.comservice: pipelines: # Duplicating data to both exporters simultaneously ensures zero blind spots metrics: receivers: [otlp] processors: [resourcedetection/gcp, batch] exporters: [googlecloud, datadog] traces: receivers: [otlp] processors: [resourcedetection/gcp, batch] exporters: [googlecloud, datadog] logs: receivers: [otlp] processors: [resourcedetection/gcp, batch] exporters: [googlecloud, datadog]The Step-by-Step Deployment Sequence1. Initialize Environment Variables, Configure scope fields.Define your working variables. The script automatically reads your active terminal’s project ID. Replace YOUR_GCP_REGION with your intended deployment zone (e.g., us-central1).export PROJECT_ID=$(gcloud config get-value project)export CLUSTER_NAME="telemetrydemo"export REGION="YOUR_GCP_REGION"export NAMESPACE="otel-demo"2. Create a GKE Autopilot Cluster.Provision your managed Autopilot cluster. GKE Autopilot clusters are secure by default and have Workload Identity fully enabled automatically.gcloud container clusters create-auto $CLUSTER_NAME \--region=$REGION \--project=$PROJECT_IDgcloud container clusters get-credentials $CLUSTER_NAME \--region=$REGION3. Configure Workload Identity for GCP Access.# 1. Create the Namespacekubectl create namespace $NAMESPACE# 2. Create the GSAgcloud iam service-accounts create otel-gsa# 3. Attach standard Google Observability write permissionsfor role in monitoring.metricWriter logging.logWriter cloudtrace.agent; do gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:otel-gsa@$PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/$role"done# 4. Link GKE Service Account to the GCP GSAgcloud iam service-accounts add-iam-policy-binding \ otel-gsa@$PROJECT_ID.iam.gserviceaccount.com \ --role="roles/iam.workloadIdentityUser" \ --member="serviceAccount:$PROJECT_ID.svc.id.goog[$NAMESPACE/my-otel-collector]"4. Generate Custom OTel Configuration File.Defines OTel Contrib v0.154.0 & Google Cloud Pipelinescat < gcp-values.yamlopentelemetry-collector:image:repository: otel/opentelemetry-collector-contribtag: 0.154.0serviceAccount:name: "my-otel-collector"annotations: iam.gke.io/gcp-service-account: "otel-gsa@${PROJECT_ID}.iam.gserviceaccount.com"config:processors: resourcedetection/gcp: detectors: [gcp] timeout: 2sexporters: googlecloud: project: "${PROJECT_ID}"service: pipelines: traces: processors: [memory_limiter, resourcedetection/gcp, batch] exporters: [googlecloud] metrics: processors: [memory_limiter, resourcedetection/gcp, batch] exporters: [googlecloud] logs: processors: [memory_limiter, resourcedetection/gcp, batch] exporters: [googlecloud]EOF5. Deploy the OpenTelemetry Demo App, Using Helm Overrides.Pull the chart records from the repository and install the application with your direct Google Cloud Observability configuration layer active.helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-chartshelm repo updatehelm install my-otel-demo open-telemetry/opentelemetry-demo \ --namespace $NAMESPACE \ --values gcp-values.yamlOutput - Navigate to Trace -> Trace Explorer in the Cloud Console. Navigate to Monitoring -> Metrics Explorer. Navigate to the Logs Explorer. Run the following query SQL resource.type="gke_container" log_name="projects/YOUR_PROJECT_ID/logs/opentelemetry-demo-logs"Screenshot of fully functional appNo-Code Agent Blueprint in Antigravity 2.0Instead of writing a Python script, you configure the agent’s identity and its data connections visually within the standalone Antigravity 2.0 desktop application.Prompt - Create an Observability Agent that can query performance metrics, fetch production logs, and analyze traces. It should connect to our observability backend (e.g., BigQuery or Cloud Logging) to help debug system issues and summarize historical baselines. It shoud connect to my GCP project id = "xxxxxxx-xxxxx" using abc@gmail.comThe following output must be shown:Files Created:requirements.txt: Specifies backend server dependencies (Flask, CORS, Google Client SDKs, and Gemini AI SDK).server.py: Python Flask gateway connecting to Google BigQuery, Cloud Logging, and Gemini. Features a high-fidelity SRE Simulation Engine fallback if GCP credentials are not active locally.index.html: Clean, semantic dashboard UI featuring grids, terminal displays, a trace timeline workspace, a BigQuery editor, and the AI Copilot chat dock.style.css: Custom responsive stylesheet containing styling tokens, HSL colors, animated radial gauges, glassmorphic filters, and console layouts.app.js: Core script managing rolling real-time HTML5 Canvas graphs, log explorers, interactive traces (with relative timing offsets), BigQuery execution outputs, and streaming AI agent dialogues.How to Activate the Actual Live GCP ConnectionTo connect the dashboard to your actual live Google Cloud APIs for project genaidemos-414006 (pulling live logs, monitoring metrics, and querying real BigQuery tables), choose one of the two standard options:Option A: Use a Service Account Key (I went ahead with this option)Download a Service Account JSON key from the Google Cloud Console for project genaidemos-414006 (located under IAM & Admin > Service Accounts).In your terminal, set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to that JSON key file, and start the server:You must get output as below:Click the following Google OAuth link to authenticate: 👉 Google Cloud SDK Authentication LinkSign in with your corporate credentials:Grant the requested permissions, copy the resulting authorization code, and paste it here in the chat.Then grant the authorization code. The output must be as below, and the URL for the dashboard plus natural language queriesConnection Established Successfully! 🎉We have successfully authenticated the local Google Cloud SDK and connected your AI SRE Observability Agent and Dashboard to your live GCP Project!Final URL: http://127.0.0.1:5001/ (localhost)Dashboard built using AntigravityChat InterfaceTry this queryI noticed a system anomaly is active. The dashboard highlights an ‘order-db’ latency deadlock issue. Can you analyze the root cause and provide structural fixes?Do let me know the output. See you in the cloud.— PS.This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!Migrating to Google Cloud Observability Without Losing Visibility was originally published in Generative AI on Medium, where people are continuing the conversation by highlighting and responding to this story.Source: Generative AI Pub — Published — Category: Image AI