Skip to main content

Observability runbooks

Use these runbooks when a Switera launch alert fires. Start with the dashboard named in the alert, then use the checks below to decide whether the issue is user-facing, deployment-related, or a service dependency problem.

Dashboard pack​

The launch dashboard pack contains:

DashboardUse it for
Switera API and GatewayAPI target health, request rate, 5xx rate, route latency, gateway latency, and API/gateway error logs.
Switera Product RuntimeManaged Email queue depth, Email issue events, Webhook provider submissions, worker readiness, and runtime issue logs.
Switera Data PlaneDatabase, cache, object storage, secrets service, storage utilization, data-service restarts, and data/secrets logs.

First response​

  1. Confirm the active deployment finished successfully in the deployment pipeline.
  2. Open the related health dashboard.
  3. Check whether the alert is isolated to one app route, one workload, or one dependency.
  4. Check recent logs for request_id, app_id, tenant_id, or the affected route.
  5. Run the public health endpoint before making changes.
  6. If a deploy is in progress, wait for rollout or rollback before restarting workloads manually.

Deployment unavailable​

Triggered when a deployment has unavailable instances for more than 10 minutes.

Check:

  • deployment events
  • workload readiness and image pull status
  • secret injection status
  • recent rollout logs

Action:

  • If the newest rollout is bad, rollback through the normal repo-controlled deployment path.
  • If the workload is blocked by missing secrets, fix secret rendering first.
  • If only one instance is unhealthy after a dependency restart, inspect logs before restarting.

Deployment rollout stuck​

Triggered when the runtime has not observed the latest deployment generation.

Check:

  • rollout status for the named deployment
  • admission or policy errors
  • image pull errors
  • pending workloads caused by scheduling or quota

Action:

  • Do not push another unrelated change first.
  • Fix the rollout blocker, then let the deployment pipeline redeploy the same source state.

Workload crash loop​

Triggered when a container restarts more than five times in 15 minutes.

Check:

  • previous container logs
  • readiness/liveness probe failures
  • rendered environment file availability
  • database, cache, and provider endpoints needed by that service

Action:

  • Fix missing configuration or dependency readiness.
  • Restart only after the root cause is understood.

Workload not ready​

Triggered when a workload remains not ready for more than five minutes.

Check:

  • readiness probe output
  • service dependency readiness
  • workload events
  • resource pressure

Action:

  • If the workload is waiting on a dependency, fix the dependency first.
  • If the readiness probe is stale after a successful deploy, restart through a repo-controlled deployment change when practical.

High memory usage​

Triggered when a container exceeds 90% of its memory limit.

Check:

  • whether usage rose after a deploy
  • whether the workload is handling a batch job, import, export, or queue spike
  • recent out-of-memory restarts

Action:

  • Reduce workload pressure if possible.
  • Increase requests/limits only with a matching deployment note.
  • Add product-level limits if a tenant or route caused the pressure.

High CPU usage​

Triggered when a container exceeds 90% of its CPU limit for 10 minutes.

Check:

  • route traffic and API latency
  • worker queue activity
  • search/indexing jobs
  • billing or email delivery loops

Action:

  • Confirm whether this is expected batch work.
  • If user-facing latency is high, scale the deployment or disable the noisy workflow.

Database service down​

Triggered when the primary database service is not healthy.

Check:

  • service readiness
  • storage attachment
  • database logs
  • API health checks

Action:

  • Treat as user-facing if /api/v1/health reports database failure.
  • Avoid destructive cleanup until a backup status is confirmed.

Cache service down​

Triggered when the cache and queue backend is not healthy.

Check:

  • service readiness
  • storage attachment
  • queue and session symptoms
  • API and worker health checks

Action:

  • Expect degraded sessions, queues, rate limits, and Email runtime.
  • Restore the cache service before retrying failed workflows.

Object storage down​

Triggered when object storage is not healthy.

Check:

  • service readiness
  • storage attachment
  • import/export and asset upload errors

Action:

  • Expect object storage, imports, exports, or attached assets to degrade.
  • Restore service health before retrying user uploads or exports.

Secrets service down​

Triggered when the secrets service is not healthy.

Check:

  • service readiness
  • secret injection errors
  • workloads waiting for credentials

Action:

  • Treat as critical because customer secrets, encryption, and runtime credentials may be unavailable.
  • Fix secret service readiness before restarting dependent workloads.

Storage volume nearly full​

Triggered when a storage volume is above the launch threshold.

Check:

  • volume name
  • owning workload
  • growth rate
  • recent jobs or imports that increased usage

Action:

  • Add capacity before deleting data.
  • For database volumes, take or confirm a backup before cleanup work.

API target down​

Triggered when the metrics scraper cannot reach the API metrics endpoint.

Check:

  • https://switera.com/health
  • API readiness
  • metrics target labels
  • the internal /metrics route from the runtime network

Action:

  • If public health is down too, treat as API outage.
  • If only metrics are down, fix target labels or metrics route exposure.

API high error rate​

Triggered when API 5xx responses exceed 5% for five minutes.

Check:

  • Switera API and Gateway dashboard
  • top failing route by status code
  • logs for request_id
  • database, cache, secrets, and provider errors

Action:

  • Roll back if the spike started immediately after deployment.
  • Fix dependency credentials or readiness if failures share one dependency.
  • Create a follow-up regression test for the failing route.

API high latency​

Triggered when API p95 latency exceeds two seconds for five minutes.

Check:

  • top slow route
  • database and cache health
  • worker and queue load
  • gateway latency versus API latency

Action:

  • If gateway latency is low but API latency is high, inspect handlers and dependencies.
  • If both are high, inspect gateway and runtime resource pressure.

Gateway high latency​

Triggered when gateway p99 latency exceeds five seconds.

Check:

  • upstream health
  • gateway events
  • gateway CPU/memory
  • public route latency

Action:

  • Verify whether latency is in the gateway or upstream.
  • Restart the gateway only after confirming it is the bottleneck.

Email queue backlog​

Triggered when the managed Email pending queue depth is above 100 for 10 minutes.

Check:

  • Product Runtime dashboard
  • Email runtime page for affected apps
  • sender-domain health
  • delivery provider errors in API and worker logs

Action:

  • Pause non-critical workflow sends if a provider is failing.
  • Fix sender-domain or provider configuration before retrying suppressed/failing sends.

Email failure spike​

Triggered when failed, retried, bounced, or complained managed Email events exceed the launch threshold.

Check:

  • Email runtime alerts in the app console
  • sender-domain verification state
  • bounce and complaint reasons
  • recent template or route changes

Action:

  • For complaints, review consent and suppressions before retrying.
  • For provider failures, fix provider readiness and then run a workflow test send.

Webhook provider failure spike​

Triggered when webhook provider submissions fail before entering delivery history.

Check:

  • Webhooks page for endpoint state
  • provider credentials and service readiness
  • API logs around webhook event submission
  • recent Billing, Auth, or Email events that generate webhooks

Action:

  • Fix provider configuration first.
  • Retry user-visible events only when idempotency is confirmed.

Certificate expiring soon​

Triggered when a certificate expires in less than seven days.

Check:

  • certificate status
  • renewal challenge status
  • DNS records and host names

Action:

  • Fix DNS or issuer problems before manually deleting certificates.
  • Confirm renewal by checking the certificate expiration timestamp after renewal reconciles.

Job failed​

Triggered when a deployment job reports failure.

Check:

  • job logs
  • workload events
  • migration or init script output
  • rendered environment files

Action:

  • Do not delete failed jobs until logs are captured.
  • Fix the failed job input, then redeploy through the normal deployment path.

Related pages: