99th percentile successful search request duration over 5m
**Descriptions** - warning frontend: 20s+ 99th percentile successful search request duration over 5m **Next steps** - **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 20,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details. - **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/admin/repositories?filter=needs-index (it should show few or no results.) - **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization. - **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-99th_percentile_search_request_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_99th_percentile_search_request_duration" ] ``` *Managed by the Sourcegraph Code Understanding team.*90th percentile successful search request duration over 5m
**Descriptions** - warning frontend: 15s+ 90th percentile successful search request duration over 5m **Next steps** - **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 15,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details. - **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/admin/repositories?filter=needs-index (it should show few or no results.) - **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization. - **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-90th_percentile_search_request_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_90th_percentile_search_request_duration" ] ``` *Managed by the Sourcegraph Code Understanding team.*timeout search responses every 5m
**Descriptions** - warning frontend: 2%+ timeout search responses every 5m for 15m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-timeout_search_responses). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_timeout_search_responses" ] ``` *Managed by the Sourcegraph Code Understanding team.*hard error search responses every 5m
**Descriptions** - warning frontend: 2%+ hard error search responses every 5m for 15m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-hard_error_search_responses). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_hard_error_search_responses" ] ``` *Managed by the Sourcegraph Code Understanding team.*searches with no results every 5m
**Descriptions** - warning frontend: 5%+ searches with no results every 5m for 15m0s **Next steps** - A sudden increase in this metric could indicate a problem with search indexing, or a shift in search behavior that are causing fewer users to find the results they`re looking for. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-search_no_results). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_search_no_results" ] ``` *Managed by the Sourcegraph Code Understanding team.*search alert user suggestions shown every 5m
**Descriptions** - warning frontend: 5%+ search alert user suggestions shown every 5m for 15m0s **Next steps** - This indicates your user`s are making syntax errors or similar user errors. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-search_alert_user_suggestions). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_search_alert_user_suggestions" ] ``` *Managed by the Sourcegraph Code Understanding team.*90th percentile page load latency over all routes over 10m
**Descriptions** - warning frontend: 2s+ 90th percentile page load latency over all routes over 10m **Next steps** - Confirm that the Sourcegraph frontend has enough CPU/memory using the provisioning panels. - Investigate potential sources of latency by selecting Explore and modifying the `sum by(le)` section to include additional labels: for example, `sum by(le, job)` or `sum by (le, instance)`. - Trace a request to see what the slowest part is: https://sourcegraph.com/docs/admin/observability/tracing - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-page_load_latency). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_page_load_latency" ] ``` *Managed by the Sourcegraph Services team.*99th percentile code-intel successful search request duration over 5m
**Descriptions** - warning frontend: 20s+ 99th percentile code-intel successful search request duration over 5m **Next steps** - **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 20,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details. - **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/admin/repositories?filter=needs-index (it should show few or no results.) - **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization. - **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization. - This alert may indicate that your instance is struggling to process symbols queries on a monorepo, [learn more here](https://sourcegraph.com/docs/admin/monorepo). - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-99th_percentile_search_codeintel_request_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_99th_percentile_search_codeintel_request_duration" ] ``` *Managed by the Sourcegraph Code Understanding team.*90th percentile code-intel successful search request duration over 5m
**Descriptions** - warning frontend: 15s+ 90th percentile code-intel successful search request duration over 5m **Next steps** - **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 15,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details. - **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/admin/repositories?filter=needs-index (it should show few or no results.) - **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization. - **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization. - This alert may indicate that your instance is struggling to process symbols queries on a monorepo, [learn more here](https://sourcegraph.com/docs/admin/monorepo). - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-90th_percentile_search_codeintel_request_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_90th_percentile_search_codeintel_request_duration" ] ``` *Managed by the Sourcegraph Code Understanding team.*hard timeout search code-intel responses every 5m
**Descriptions** - warning frontend: 2%+ hard timeout search code-intel responses every 5m for 15m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-hard_timeout_search_codeintel_responses). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_hard_timeout_search_codeintel_responses" ] ``` *Managed by the Sourcegraph Code Understanding team.*hard error search code-intel responses every 5m
**Descriptions** - warning frontend: 2%+ hard error search code-intel responses every 5m for 15m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-hard_error_search_codeintel_responses). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_hard_error_search_codeintel_responses" ] ``` *Managed by the Sourcegraph Code Understanding team.*partial timeout search code-intel responses every 5m
**Descriptions** - warning frontend: 5%+ partial timeout search code-intel responses every 5m for 15m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-partial_timeout_search_codeintel_responses). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_partial_timeout_search_codeintel_responses" ] ``` *Managed by the Sourcegraph Code Understanding team.*search code-intel alert user suggestions shown every 5m
**Descriptions** - warning frontend: 5%+ search code-intel alert user suggestions shown every 5m for 15m0s **Next steps** - This indicates a bug in Sourcegraph, please [open an issue](https://github.com/sourcegraph/sourcegraph/issues/new/choose). - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-search_codeintel_alert_user_suggestions). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_search_codeintel_alert_user_suggestions" ] ``` *Managed by the Sourcegraph Code Understanding team.*99th percentile successful search API request duration over 5m
**Descriptions** - warning frontend: 50s+ 99th percentile successful search API request duration over 5m **Next steps** - **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 20,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details. - **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/admin/repositories?filter=needs-index (it should show few or no results.) - **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization. - **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-99th_percentile_search_api_request_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_99th_percentile_search_api_request_duration" ] ``` *Managed by the Sourcegraph Code Understanding team.*90th percentile successful search API request duration over 5m
**Descriptions** - warning frontend: 40s+ 90th percentile successful search API request duration over 5m **Next steps** - **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 15,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details. - **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/admin/repositories?filter=needs-index (it should show few or no results.) - **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization. - **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-90th_percentile_search_api_request_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_90th_percentile_search_api_request_duration" ] ``` *Managed by the Sourcegraph Code Understanding team.*hard error search API responses every 5m
**Descriptions** - warning frontend: 2%+ hard error search API responses every 5m for 15m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-hard_error_search_api_responses). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_hard_error_search_api_responses" ] ``` *Managed by the Sourcegraph Code Understanding team.*partial timeout search API responses every 5m
**Descriptions** - warning frontend: 5%+ partial timeout search API responses every 5m for 15m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-partial_timeout_search_api_responses). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_partial_timeout_search_api_responses" ] ``` *Managed by the Sourcegraph Code Understanding team.*search API alert user suggestions shown every 5m
**Descriptions** - warning frontend: 5%+ search API alert user suggestions shown every 5m **Next steps** - This indicates your user`s search API requests have syntax errors or a similar user error. Check the responses the API sends back for an explanation. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-search_api_alert_user_suggestions). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_search_api_alert_user_suggestions" ] ``` *Managed by the Sourcegraph Code Understanding team.*maximum duration since last successful site configuration update (all "frontend" instances)
**Descriptions** - critical frontend: 300s+ maximum duration since last successful site configuration update (all "frontend" instances) **Next steps** - This indicates that one or more "frontend" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself. - Check for relevant errors in the "frontend" logs, as well as frontend`s logs. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-frontend_site_configuration_duration_since_last_successful_update_by_instance). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_frontend_frontend_site_configuration_duration_since_last_successful_update_by_instance" ] ``` *Managed by the Sourcegraph Platform team.*internal indexed search error responses every 5m
**Descriptions** - warning frontend: 5%+ internal indexed search error responses every 5m for 15m0s **Next steps** - Check the Zoekt Web Server dashboard for indications it might be unhealthy. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-internal_indexed_search_error_responses). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_internal_indexed_search_error_responses" ] ``` *Managed by the Sourcegraph Code Understanding team.*internal unindexed search error responses every 5m
**Descriptions** - warning frontend: 5%+ internal unindexed search error responses every 5m for 15m0s **Next steps** - Check the Searcher dashboard for indications it might be unhealthy. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-internal_unindexed_search_error_responses). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_internal_unindexed_search_error_responses" ] ``` *Managed by the Sourcegraph Code Understanding team.*99th percentile successful gitserver query duration over 5m
**Descriptions** - warning frontend: 20s+ 99th percentile successful gitserver query duration over 5m **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-99th_percentile_gitserver_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_99th_percentile_gitserver_duration" ] ``` *Managed by the Sourcegraph Services team.*gitserver error responses every 5m
**Descriptions** - warning frontend: 5%+ gitserver error responses every 5m for 15m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-gitserver_error_responses). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_gitserver_error_responses" ] ``` *Managed by the Sourcegraph Services team.*warning test alert metric
**Descriptions** - warning frontend: 1+ warning test alert metric **Next steps** - This alert is triggered via the `triggerObservabilityTestAlert` GraphQL endpoint, and will automatically resolve itself. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-observability_test_alert_warning). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_observability_test_alert_warning" ] ``` *Managed by the Sourcegraph Platform team.*critical test alert metric
**Descriptions** - critical frontend: 1+ critical test alert metric **Next steps** - This alert is triggered via the `triggerObservabilityTestAlert` GraphQL endpoint, and will automatically resolve itself. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-observability_test_alert_critical). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_frontend_observability_test_alert_critical" ] ``` *Managed by the Sourcegraph Platform team.*cryptographic requests to Cloud KMS every 1m
**Descriptions** - warning frontend: 15000+ cryptographic requests to Cloud KMS every 1m for 5m0s - critical frontend: 30000+ cryptographic requests to Cloud KMS every 1m for 5m0s **Next steps** - Revert recent commits that cause extensive listing from "external_services" and/or "user_external_accounts" tables. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-cloudkms_cryptographic_requests). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_cloudkms_cryptographic_requests", "critical_frontend_cloudkms_cryptographic_requests" ] ``` *Managed by the Sourcegraph Services team.*error rate for periodic goroutine executions
**Descriptions** - warning frontend: 0.01reqps+ error rate for periodic goroutine executions for 15m0s **Next steps** - Check service logs for error details related to the failing periodic routine - Check if the routine depends on external services that may be unavailable - Look for recent changes to the routine`s code or configuration - More help interpreting this metric is available in the [dashboards reference](dashboards#frontend-goroutine_error_rate). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_goroutine_error_rate" ] ``` *Managed by the Sourcegraph Services team.*percentage of periodic goroutine executions that result in errors
**Descriptions** - warning frontend: 5%+ percentage of periodic goroutine executions that result in errors **Next steps** - Check service logs for error details related to the failing periodic routine - Check if the routine depends on external services that may be unavailable - Consider temporarily disabling the routine if it`s non-critical and causing cascading issues - More help interpreting this metric is available in the [dashboards reference](dashboards#frontend-goroutine_error_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_goroutine_error_percentage" ] ``` *Managed by the Sourcegraph Services team.*mean blocked seconds per conn request
**Descriptions** - warning frontend: 0.1s+ mean blocked seconds per conn request for 10m0s - critical frontend: 0.5s+ mean blocked seconds per conn request for 10m0s **Next steps** - Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed - Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf) - If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-mean_blocked_seconds_per_conn_request). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_mean_blocked_seconds_per_conn_request", "critical_frontend_mean_blocked_seconds_per_conn_request" ] ``` *Managed by the Sourcegraph Services team.*CPU usage
**Descriptions** - warning frontend: 95%+ CPU usage for 10m0s **Next steps** - Consider increasing CPU limits or scaling out. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-cpu_usage_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_cpu_usage_percentage" ] ``` *Managed by the Sourcegraph Code Understanding team.*memory (RSS)
**Descriptions** - warning frontend: 90%+ memory (RSS) for 10m0s **Next steps** - Consider increasing memory limits or scaling out. - More help interpreting this metric is available in the [dashboards reference](dashboards#frontend-memory_rss). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_memory_rss" ] ``` *Managed by the Sourcegraph Code Understanding team.*container cpu usage total (1m average) across all cores by instance
**Descriptions** - warning frontend: 99%+ container cpu usage total (1m average) across all cores by instance **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the (frontend|sourcegraph-frontend) container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-container_cpu_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_container_cpu_usage" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage by instance
**Descriptions** - warning frontend: 99%+ container memory usage by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of (frontend|sourcegraph-frontend) container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-container_memory_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_container_memory_usage" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (90th percentile over 1d) across all cores by instance
**Descriptions** - warning frontend: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the (frontend|sourcegraph-frontend) service. - **Docker Compose:** Consider increasing `cpus:` of the (frontend|sourcegraph-frontend) container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-provisioning_container_cpu_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_provisioning_container_cpu_usage_long_term" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage (1d maximum) by instance
**Descriptions** - warning frontend: 80%+ container memory usage (1d maximum) by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the (frontend|sourcegraph-frontend) service. - **Docker Compose:** Consider increasing `memory:` of the (frontend|sourcegraph-frontend) container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-provisioning_container_memory_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_provisioning_container_memory_usage_long_term" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (5m maximum) across all cores by instance
**Descriptions** - warning frontend: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the (frontend|sourcegraph-frontend) container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-provisioning_container_cpu_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_provisioning_container_cpu_usage_short_term" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage (5m maximum) by instance
**Descriptions** - warning frontend: 90%+ container memory usage (5m maximum) by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of (frontend|sourcegraph-frontend) container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-provisioning_container_memory_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_provisioning_container_memory_usage_short_term" ] ``` *Managed by the Sourcegraph Platform team.*container OOMKILL events total by instance
**Descriptions** - warning frontend: 1+ container OOMKILL events total by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of (frontend|sourcegraph-frontend) container in `docker-compose.yml`. - More help interpreting this metric is available in the [dashboards reference](dashboards#frontend-container_oomkill_events_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_container_oomkill_events_total" ] ``` *Managed by the Sourcegraph Platform team.*maximum active goroutines
**Descriptions** - warning frontend: 10000+ maximum active goroutines for 10m0s **Next steps** - More help interpreting this metric is available in the [dashboards reference](dashboards#frontend-go_goroutines). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_go_goroutines" ] ``` *Managed by the Sourcegraph Services team.*maximum go garbage collection duration
**Descriptions** - warning frontend: 2s+ maximum go garbage collection duration **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-go_gc_duration_seconds). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_go_gc_duration_seconds" ] ``` *Managed by the Sourcegraph Services team.*percentage pods available
**Descriptions** - critical frontend: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod (frontend\|sourcegraph-frontend)` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p (frontend\|sourcegraph-frontend)`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_frontend_pods_available_percentage" ] ``` *Managed by the Sourcegraph Services team.*email delivery failure rate over 30 minutes
**Descriptions** - warning frontend: 0%+ email delivery failure rate over 30 minutes - critical frontend: 10%+ email delivery failure rate over 30 minutes **Next steps** - Check your SMTP configuration in site configuration. - Check `sourcegraph-frontend` logs for more detailed error messages. - Check your SMTP provider for more detailed error messages. - Use `sum(increase(src_email_send{success="false"}[30m]))` to check the raw count of delivery failures. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#frontend-email_delivery_failures). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_frontend_email_delivery_failures", "critical_frontend_email_delivery_failures" ] ``` *Managed by the Sourcegraph Platform team.*disk space remaining
**Descriptions** - warning gitserver: less than 15% disk space remaining - critical gitserver: less than 10% disk space remaining for 10m0s **Next steps** - On a warning alert, you may want to provision more disk space: Disk pressure may result in decreased performance, users having to wait for repositories to clone, etc. - On a critical alert, you need to provision more disk space. Running out of disk space will result in decreased performance, or complete service outage. - More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-disk_space_remaining). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_disk_space_remaining", "critical_gitserver_disk_space_remaining" ] ``` *Managed by the Sourcegraph Services team.*container CPU throttling time %
**Descriptions** - warning gitserver: 75%+ container CPU throttling time % for 2m0s **Next steps** - - Consider increasing the CPU limit for the container. - More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-cpu_throttling_time). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_cpu_throttling_time" ] ``` *Managed by the Sourcegraph Services team.*echo test command duration
**Descriptions** - warning gitserver: 0.02s+ echo test command duration for 30s **Next steps** - **Single container deployments (removed in 7.0.0):** The single-container deployment mode has been sunset. [Migrate to Docker Compose](../deploy/docker-compose/migrate). - **Kubernetes and Docker Compose:** Check that you are running a similar number of git server replicas and that their CPU/memory limits are allocated according to what is shown in the [Sourcegraph resource estimator](https://sourcegraph.com/docs/self-hosted/deploy/resource-estimator). - If your persistent volume is slow, you may want to provision more IOPS, usually by increasing the volume size. - More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-echo_command_duration_test). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_echo_command_duration_test" ] ``` *Managed by the Sourcegraph Services team.*number of times a repo corruption has been identified
**Descriptions** - critical gitserver: 0+ number of times a repo corruption has been identified **Next steps** - Check the corruption logs for details. gitserver_repos.corruption_logs contains more information. - More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-repo_corrupted). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_gitserver_repo_corrupted" ] ``` *Managed by the Sourcegraph Services team.*repository clone queue size
**Descriptions** - warning gitserver: 25+ repository clone queue size **Next steps** - **If you just added several repositories**, the warning may be expected. - **Check which repositories need cloning**, by visiting e.g. https://sourcegraph.example.com/admin/repositories?filter=not-cloned - Learn more about the related dashboard panel in the [dashboards reference](dashboards#gitserver-repository_clone_queue_size). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_repository_clone_queue_size" ] ``` *Managed by the Sourcegraph Services team.*rate of git command corruption retry attempts over 5m
**Descriptions** - warning gitserver: 0.1reqps+ rate of git command corruption retry attempts over 5m for 5m0s **Next steps** - Investigate the underlying cause of corruption errors in git commands. - Check disk health and I/O performance. - Monitor for patterns in specific git operations that trigger retries. - Consider adjusting retry configuration if retries are too frequent. - More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-git_command_retry_attempts_rate). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_git_command_retry_attempts_rate" ] ``` *Managed by the Sourcegraph Services team.*error rate for periodic goroutine executions
**Descriptions** - warning gitserver: 0.01reqps+ error rate for periodic goroutine executions for 15m0s **Next steps** - Check service logs for error details related to the failing periodic routine - Check if the routine depends on external services that may be unavailable - Look for recent changes to the routine`s code or configuration - More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-goroutine_error_rate). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_goroutine_error_rate" ] ``` *Managed by the Sourcegraph Services team.*percentage of periodic goroutine executions that result in errors
**Descriptions** - warning gitserver: 5%+ percentage of periodic goroutine executions that result in errors **Next steps** - Check service logs for error details related to the failing periodic routine - Check if the routine depends on external services that may be unavailable - Consider temporarily disabling the routine if it`s non-critical and causing cascading issues - More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-goroutine_error_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_goroutine_error_percentage" ] ``` *Managed by the Sourcegraph Services team.*CPU usage
**Descriptions** - warning gitserver: 95%+ CPU usage for 10m0s **Next steps** - Consider increasing CPU limits or scaling out. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#gitserver-cpu_usage_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_cpu_usage_percentage" ] ``` *Managed by the Sourcegraph Services team.*memory (RSS)
**Descriptions** - warning gitserver: 90%+ memory (RSS) for 10m0s **Next steps** - Consider increasing memory limits or scaling out. - More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-memory_rss). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_memory_rss" ] ``` *Managed by the Sourcegraph Services team.*maximum duration since last successful site configuration update (all "gitserver" instances)
**Descriptions** - critical gitserver: 300s+ maximum duration since last successful site configuration update (all "gitserver" instances) **Next steps** - This indicates that one or more "gitserver" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself. - Check for relevant errors in the "gitserver" logs, as well as frontend`s logs. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#gitserver-gitserver_site_configuration_duration_since_last_successful_update_by_instance). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_gitserver_gitserver_site_configuration_duration_since_last_successful_update_by_instance" ] ``` *Managed by the Sourcegraph Platform team.*mean blocked seconds per conn request
**Descriptions** - warning gitserver: 0.1s+ mean blocked seconds per conn request for 10m0s - critical gitserver: 0.5s+ mean blocked seconds per conn request for 10m0s **Next steps** - Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed - Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf) - If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights - Learn more about the related dashboard panel in the [dashboards reference](dashboards#gitserver-mean_blocked_seconds_per_conn_request). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_mean_blocked_seconds_per_conn_request", "critical_gitserver_mean_blocked_seconds_per_conn_request" ] ``` *Managed by the Sourcegraph Services team.*container cpu usage total (1m average) across all cores by instance
**Descriptions** - warning gitserver: 99%+ container cpu usage total (1m average) across all cores by instance **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the gitserver container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#gitserver-container_cpu_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_container_cpu_usage" ] ``` *Managed by the Sourcegraph Services team.*container memory usage by instance
**Descriptions** - warning gitserver: 99%+ container memory usage by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of gitserver container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#gitserver-container_memory_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_container_memory_usage" ] ``` *Managed by the Sourcegraph Services team.*container cpu usage total (90th percentile over 1d) across all cores by instance
**Descriptions** - warning gitserver: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the gitserver service. - **Docker Compose:** Consider increasing `cpus:` of the gitserver container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#gitserver-provisioning_container_cpu_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_provisioning_container_cpu_usage_long_term" ] ``` *Managed by the Sourcegraph Services team.*container cpu usage total (5m maximum) across all cores by instance
**Descriptions** - warning gitserver: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the gitserver container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#gitserver-provisioning_container_cpu_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_provisioning_container_cpu_usage_short_term" ] ``` *Managed by the Sourcegraph Services team.*container OOMKILL events total by instance
**Descriptions** - warning gitserver: 1+ container OOMKILL events total by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of gitserver container in `docker-compose.yml`. - More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-container_oomkill_events_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_container_oomkill_events_total" ] ``` *Managed by the Sourcegraph Services team.*maximum active goroutines
**Descriptions** - warning gitserver: 10000+ maximum active goroutines for 10m0s **Next steps** - More help interpreting this metric is available in the [dashboards reference](dashboards#gitserver-go_goroutines). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_go_goroutines" ] ``` *Managed by the Sourcegraph Services team.*maximum go garbage collection duration
**Descriptions** - warning gitserver: 2s+ maximum go garbage collection duration **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#gitserver-go_gc_duration_seconds). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_gitserver_go_gc_duration_seconds" ] ``` *Managed by the Sourcegraph Services team.*percentage pods available
**Descriptions** - critical gitserver: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod gitserver` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p gitserver`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#gitserver-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_gitserver_pods_available_percentage" ] ``` *Managed by the Sourcegraph Services team.*active connections
**Descriptions** - warning postgres: less than 5 active connections for 5m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#postgres-connections). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_connections" ] ``` *Managed by the Sourcegraph Platform team.*connection in use
**Descriptions** - warning postgres: 80%+ connection in use for 5m0s - critical postgres: 100%+ connection in use for 5m0s **Next steps** - Consider increasing [max_connections](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-MAX-CONNECTIONS) of the database instance, [learn more](https://sourcegraph.com/docs/admin/config/postgres-conf) - Learn more about the related dashboard panel in the [dashboards reference](dashboards#postgres-usage_connections_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_usage_connections_percentage", "critical_postgres_usage_connections_percentage" ] ``` *Managed by the Sourcegraph Platform team.*maximum transaction durations
**Descriptions** - warning postgres: 0.3s+ maximum transaction durations for 5m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#postgres-transaction_durations). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_transaction_durations" ] ``` *Managed by the Sourcegraph Platform team.*database availability
**Descriptions** - critical postgres: less than 0 database availability for 5m0s **Next steps** - **Kubernetes:** - Determine if the pod was OOM killed using `kubectl describe pod (pgsql\|codeintel-db\|codeinsights)` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p (pgsql\|codeintel-db\|codeinsights)`. - Check if there is any OOMKILL event using the provisioning panels - Check kernel logs using `dmesg` for OOMKILL events on worker nodes - **Docker Compose:** - Determine if the pod was OOM killed using `docker inspect -f '\{\{json .State\}\}' (pgsql\|codeintel-db\|codeinsights)` (look for `"OOMKilled":true`) and, if so, consider increasing the memory limit of the (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `docker logs (pgsql\|codeintel-db\|codeinsights)` (note this will include logs from the previous and currently running container). - Check if there is any OOMKILL event using the provisioning panels - Check kernel logs using `dmesg` for OOMKILL events - More help interpreting this metric is available in the [dashboards reference](dashboards#postgres-postgres_up). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_postgres_postgres_up" ] ``` *Managed by the Sourcegraph Platform team.*invalid indexes (unusable by the query planner)
**Descriptions** - warning postgres: 1+ invalid indexes (unusable by the query planner) **Next steps** - Drop and re-create the invalid trigger - please contact Sourcegraph to supply the trigger definition. - More help interpreting this metric is available in the [dashboards reference](dashboards#postgres-invalid_indexes). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_invalid_indexes" ] ``` *Managed by the Sourcegraph Platform team.*errors scraping postgres exporter
**Descriptions** - warning postgres: 1+ errors scraping postgres exporter for 5m0s **Next steps** - Ensure the Postgres exporter can access the Postgres database. Also, check the Postgres exporter logs for errors. - More help interpreting this metric is available in the [dashboards reference](dashboards#postgres-pg_exporter_err). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_pg_exporter_err" ] ``` *Managed by the Sourcegraph Platform team.*active schema migration
**Descriptions** - warning postgres: 1+ active schema migration for 5m0s **Next steps** - The database migration has been in progress for 5 or more minutes - please contact Sourcegraph if this persists. - More help interpreting this metric is available in the [dashboards reference](dashboards#postgres-migration_in_progress). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_migration_in_progress" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (90th percentile over 1d) across all cores by instance
**Descriptions** - warning postgres: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the (pgsql|codeintel-db|codeinsights) service. - **Docker Compose:** Consider increasing `cpus:` of the (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#postgres-provisioning_container_cpu_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_provisioning_container_cpu_usage_long_term" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage (1d maximum) by instance
**Descriptions** - warning postgres: 80%+ container memory usage (1d maximum) by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the (pgsql|codeintel-db|codeinsights) service. - **Docker Compose:** Consider increasing `memory:` of the (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#postgres-provisioning_container_memory_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_provisioning_container_memory_usage_long_term" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (5m maximum) across all cores by instance
**Descriptions** - warning postgres: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#postgres-provisioning_container_cpu_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_provisioning_container_cpu_usage_short_term" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage (5m maximum) by instance
**Descriptions** - warning postgres: 90%+ container memory usage (5m maximum) by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#postgres-provisioning_container_memory_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_provisioning_container_memory_usage_short_term" ] ``` *Managed by the Sourcegraph Platform team.*container OOMKILL events total by instance
**Descriptions** - warning postgres: 1+ container OOMKILL events total by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`. - More help interpreting this metric is available in the [dashboards reference](dashboards#postgres-container_oomkill_events_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_postgres_container_oomkill_events_total" ] ``` *Managed by the Sourcegraph Platform team.*percentage pods available
**Descriptions** - critical postgres: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod (pgsql\|codeintel-db\|codeinsights)` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p (pgsql\|codeintel-db\|codeinsights)`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#postgres-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_postgres_pods_available_percentage" ] ``` *Managed by the Sourcegraph Platform team.*mean blocked seconds per conn request
**Descriptions** - warning precise-code-intel-worker: 0.1s+ mean blocked seconds per conn request for 10m0s - critical precise-code-intel-worker: 0.5s+ mean blocked seconds per conn request for 10m0s **Next steps** - Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed - Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf) - If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights - Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-mean_blocked_seconds_per_conn_request). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_precise-code-intel-worker_mean_blocked_seconds_per_conn_request", "critical_precise-code-intel-worker_mean_blocked_seconds_per_conn_request" ] ``` *Managed by the Sourcegraph Platform team.*percentage pods available
**Descriptions** - critical precise-code-intel-worker: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod precise-code-intel-worker` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p precise-code-intel-worker`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#precise-code-intel-worker-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_precise-code-intel-worker_pods_available_percentage" ] ``` *Managed by the Sourcegraph Code Understanding team.*mean blocked seconds per conn request
**Descriptions** - warning syntactic-indexing: 0.1s+ mean blocked seconds per conn request for 10m0s - critical syntactic-indexing: 0.5s+ mean blocked seconds per conn request for 10m0s **Next steps** - Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed - Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf) - If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights - Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-mean_blocked_seconds_per_conn_request). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_syntactic-indexing_mean_blocked_seconds_per_conn_request", "critical_syntactic-indexing_mean_blocked_seconds_per_conn_request" ] ``` *Managed by the Sourcegraph Platform team.*percentage pods available
**Descriptions** - critical syntactic-indexing: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod syntactic-code-intel-worker` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p syntactic-code-intel-worker`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntactic-indexing-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_syntactic-indexing_pods_available_percentage" ] ``` *Managed by the Sourcegraph Code Understanding team.*redis-store availability
**Descriptions** - critical redis: less than 1 redis-store availability for 10s **Next steps** - Ensure redis-store is running - More help interpreting this metric is available in the [dashboards reference](dashboards#redis-redis-store_up). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_redis_redis-store_up" ] ``` *Managed by the Sourcegraph Platform team.*redis-cache availability
**Descriptions** - critical redis: less than 1 redis-cache availability for 10s **Next steps** - Ensure redis-cache is running - More help interpreting this metric is available in the [dashboards reference](dashboards#redis-redis-cache_up). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_redis_redis-cache_up" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (90th percentile over 1d) across all cores by instance
**Descriptions** - warning redis: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the redis-cache service. - **Docker Compose:** Consider increasing `cpus:` of the redis-cache container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#redis-provisioning_container_cpu_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_redis_provisioning_container_cpu_usage_long_term" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage (1d maximum) by instance
**Descriptions** - warning redis: 80%+ container memory usage (1d maximum) by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the redis-cache service. - **Docker Compose:** Consider increasing `memory:` of the redis-cache container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#redis-provisioning_container_memory_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_redis_provisioning_container_memory_usage_long_term" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (5m maximum) across all cores by instance
**Descriptions** - warning redis: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the redis-cache container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#redis-provisioning_container_cpu_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_redis_provisioning_container_cpu_usage_short_term" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage (5m maximum) by instance
**Descriptions** - warning redis: 90%+ container memory usage (5m maximum) by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of redis-cache container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#redis-provisioning_container_memory_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_redis_provisioning_container_memory_usage_short_term" ] ``` *Managed by the Sourcegraph Platform team.*container OOMKILL events total by instance
**Descriptions** - warning redis: 1+ container OOMKILL events total by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of redis-cache container in `docker-compose.yml`. - More help interpreting this metric is available in the [dashboards reference](dashboards#redis-container_oomkill_events_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_redis_container_oomkill_events_total" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (90th percentile over 1d) across all cores by instance
**Descriptions** - warning redis: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the redis-store service. - **Docker Compose:** Consider increasing `cpus:` of the redis-store container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#redis-provisioning_container_cpu_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_redis_provisioning_container_cpu_usage_long_term" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage (1d maximum) by instance
**Descriptions** - warning redis: 80%+ container memory usage (1d maximum) by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the redis-store service. - **Docker Compose:** Consider increasing `memory:` of the redis-store container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#redis-provisioning_container_memory_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_redis_provisioning_container_memory_usage_long_term" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (5m maximum) across all cores by instance
**Descriptions** - warning redis: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the redis-store container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#redis-provisioning_container_cpu_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_redis_provisioning_container_cpu_usage_short_term" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage (5m maximum) by instance
**Descriptions** - warning redis: 90%+ container memory usage (5m maximum) by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of redis-store container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#redis-provisioning_container_memory_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_redis_provisioning_container_memory_usage_short_term" ] ``` *Managed by the Sourcegraph Platform team.*container OOMKILL events total by instance
**Descriptions** - warning redis: 1+ container OOMKILL events total by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of redis-store container in `docker-compose.yml`. - More help interpreting this metric is available in the [dashboards reference](dashboards#redis-container_oomkill_events_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_redis_container_oomkill_events_total" ] ``` *Managed by the Sourcegraph Platform team.*percentage pods available
**Descriptions** - critical redis: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod redis-cache` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p redis-cache`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#redis-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_redis_pods_available_percentage" ] ``` *Managed by the Sourcegraph Platform team.*percentage pods available
**Descriptions** - critical redis: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod redis-store` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p redis-store`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#redis-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_redis_pods_available_percentage" ] ``` *Managed by the Sourcegraph Platform team.*number of worker instances running the codeintel-upload-janitor job
**Descriptions** - warning worker: less than 1 number of worker instances running the codeintel-upload-janitor job for 1m0s - critical worker: less than 1 number of worker instances running the codeintel-upload-janitor job for 5m0s **Next steps** - Ensure your instance defines a worker container such that: - `WORKER_JOB_ALLOWLIST` contains "codeintel-upload-janitor" (or "all"), and - `WORKER_JOB_BLOCKLIST` does not contain "codeintel-upload-janitor" - Ensure that such a container is not failing to start or stay active - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-worker_job_codeintel-upload-janitor_count). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_worker_job_codeintel-upload-janitor_count", "critical_worker_worker_job_codeintel-upload-janitor_count" ] ``` *Managed by the Sourcegraph Code Understanding team.*number of worker instances running the codeintel-commitgraph-updater job
**Descriptions** - warning worker: less than 1 number of worker instances running the codeintel-commitgraph-updater job for 1m0s - critical worker: less than 1 number of worker instances running the codeintel-commitgraph-updater job for 5m0s **Next steps** - Ensure your instance defines a worker container such that: - `WORKER_JOB_ALLOWLIST` contains "codeintel-commitgraph-updater" (or "all"), and - `WORKER_JOB_BLOCKLIST` does not contain "codeintel-commitgraph-updater" - Ensure that such a container is not failing to start or stay active - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-worker_job_codeintel-commitgraph-updater_count). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_worker_job_codeintel-commitgraph-updater_count", "critical_worker_worker_job_codeintel-commitgraph-updater_count" ] ``` *Managed by the Sourcegraph Code Understanding team.*number of worker instances running the codeintel-autoindexing-scheduler job
**Descriptions** - warning worker: less than 1 number of worker instances running the codeintel-autoindexing-scheduler job for 1m0s - critical worker: less than 1 number of worker instances running the codeintel-autoindexing-scheduler job for 5m0s **Next steps** - Ensure your instance defines a worker container such that: - `WORKER_JOB_ALLOWLIST` contains "codeintel-autoindexing-scheduler" (or "all"), and - `WORKER_JOB_BLOCKLIST` does not contain "codeintel-autoindexing-scheduler" - Ensure that such a container is not failing to start or stay active - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-worker_job_codeintel-autoindexing-scheduler_count). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_worker_job_codeintel-autoindexing-scheduler_count", "critical_worker_worker_job_codeintel-autoindexing-scheduler_count" ] ``` *Managed by the Sourcegraph Code Understanding team.*time since oldest sync
**Descriptions** - critical worker: 32400s+ time since oldest sync for 10m0s **Next steps** - An alert here indicates that no code host connections have synced in at least 9h0m0s. This indicates that there could be a configuration issue with your code hosts connections or networking issues affecting communication with your code hosts. - Check the code host status indicator (cloud icon in top right of Sourcegraph homepage) for errors. - Make sure external services do not have invalid tokens by navigating to them in the web UI and clicking save. If there are no errors, they are valid. - Check the worker logs for errors about syncing. - Confirm that outbound network connections are allowed where worker is deployed. - Check back in an hour to see if the issue has resolved itself. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-src_repoupdater_max_sync_backoff). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_worker_src_repoupdater_max_sync_backoff" ] ``` *Managed by the Sourcegraph Services team.*site level external service sync error rate
**Descriptions** - warning worker: 0.5+ site level external service sync error rate for 10m0s - critical worker: 1+ site level external service sync error rate for 10m0s **Next steps** - An alert here indicates errors syncing site level repo metadata with code hosts. This indicates that there could be a configuration issue with your code hosts connections or networking issues affecting communication with your code hosts. - Check the code host status indicator (cloud icon in top right of Sourcegraph homepage) for errors. - Make sure external services do not have invalid tokens by navigating to them in the web UI and clicking save. If there are no errors, they are valid. - Check the worker logs for errors about syncing. - Confirm that outbound network connections are allowed where worker is deployed. - Check back in an hour to see if the issue has resolved itself. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-src_repoupdater_syncer_sync_errors_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_src_repoupdater_syncer_sync_errors_total", "critical_worker_src_repoupdater_syncer_sync_errors_total" ] ``` *Managed by the Sourcegraph Services team.*repo metadata sync was started
**Descriptions** - warning worker: less than 0 repo metadata sync was started for 9h0m0s **Next steps** - Check worker logs for errors. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-syncer_sync_start). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_syncer_sync_start" ] ``` *Managed by the Sourcegraph Services team.*95th repositories sync duration
**Descriptions** - warning worker: 30s+ 95th repositories sync duration for 5m0s **Next steps** - Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-syncer_sync_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_syncer_sync_duration" ] ``` *Managed by the Sourcegraph Services team.*95th repositories source duration
**Descriptions** - warning worker: 30s+ 95th repositories source duration for 5m0s **Next steps** - Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-source_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_source_duration" ] ``` *Managed by the Sourcegraph Services team.*repositories synced
**Descriptions** - warning worker: less than 0 repositories synced for 9h0m0s **Next steps** - Check network connectivity to code hosts - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-syncer_synced_repos). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_syncer_synced_repos" ] ``` *Managed by the Sourcegraph Services team.*repositories sourced
**Descriptions** - warning worker: less than 0 repositories sourced for 9h0m0s **Next steps** - Check network connectivity to code hosts - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-sourced_repos). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_sourced_repos" ] ``` *Managed by the Sourcegraph Services team.*repositories scheduled due to hitting a deadline
**Descriptions** - warning worker: less than 0 repositories scheduled due to hitting a deadline for 9h0m0s **Next steps** - Check worker logs. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-sched_auto_fetch). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_sched_auto_fetch" ] ``` *Managed by the Sourcegraph Services team.*scheduler loops
**Descriptions** - warning worker: less than 0 scheduler loops for 9h0m0s **Next steps** - Check worker logs for errors. This is expected to fire if there are no user added code hosts - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-sched_loops). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_sched_loops" ] ``` *Managed by the Sourcegraph Services team.*repos that haven't been fetched in more than 8 hours
**Descriptions** - warning worker: 1+ repos that haven't been fetched in more than 8 hours for 25m0s **Next steps** - Check worker logs for errors. Check for rows in gitserver_repos where LastError is not an empty string. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-src_repoupdater_stale_repos). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_src_repoupdater_stale_repos" ] ``` *Managed by the Sourcegraph Services team.*repositories schedule error rate
**Descriptions** - critical worker: 1+ repositories schedule error rate for 25m0s **Next steps** - Check worker logs for errors - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-sched_error). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_worker_sched_error" ] ``` *Managed by the Sourcegraph Services team.*repos that have failed cleanup more than 5 times consecutively
**Descriptions** - critical worker: 1+ repos that have failed cleanup more than 5 times consecutively **Next steps** - Check worker logs for cleanup errors. Check for rows in gitserver_repos where failed_cleanup_attempts > 5. Failure to optimize repositories consistently will eventually lead to bad performance problems, and needs to be addressed. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-src_repoupdater_cleanup_failed_repos). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_worker_src_repoupdater_cleanup_failed_repos" ] ``` *Managed by the Sourcegraph Services team.*the total number of external services
**Descriptions** - critical worker: 20000+ the total number of external services for 1h0m0s **Next steps** - Check for spikes in external services, could be abuse - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-src_repoupdater_external_services_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_worker_src_repoupdater_external_services_total" ] ``` *Managed by the Sourcegraph Services team.*the total number of queued sync jobs
**Descriptions** - warning worker: 100+ the total number of queued sync jobs for 1h0m0s **Next steps** - **Check if jobs are failing to sync:** "SELECT * FROM external_service_sync_jobs WHERE state = `errored`"; - **Increase the number of workers** using the `repoConcurrentExternalServiceSyncers` site config. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-repoupdater_queued_sync_jobs_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_repoupdater_queued_sync_jobs_total" ] ``` *Managed by the Sourcegraph Services team.*the total number of completed sync jobs
**Descriptions** - warning worker: 100000+ the total number of completed sync jobs for 1h0m0s **Next steps** - Check worker logs. Jobs older than 1 day should have been removed. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-repoupdater_completed_sync_jobs_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_repoupdater_completed_sync_jobs_total" ] ``` *Managed by the Sourcegraph Services team.*the percentage of external services that have failed their most recent sync
**Descriptions** - warning worker: 10%+ the percentage of external services that have failed their most recent sync for 1h0m0s **Next steps** - Check worker logs. Check code host connectivity - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-repoupdater_errored_sync_jobs_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_repoupdater_errored_sync_jobs_percentage" ] ``` *Managed by the Sourcegraph Services team.*remaining calls to GitHub graphql API before hitting the rate limit
**Descriptions** - warning worker: less than 250 remaining calls to GitHub graphql API before hitting the rate limit **Next steps** - Consider creating a new token for the indicated resource (the `name` label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-github_graphql_rate_limit_remaining). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_github_graphql_rate_limit_remaining" ] ``` *Managed by the Sourcegraph Services team.*remaining calls to GitHub rest API before hitting the rate limit
**Descriptions** - warning worker: less than 250 remaining calls to GitHub rest API before hitting the rate limit **Next steps** - Consider creating a new token for the indicated resource (the `name` label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-github_rest_rate_limit_remaining). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_github_rest_rate_limit_remaining" ] ``` *Managed by the Sourcegraph Services team.*remaining calls to GitHub search API before hitting the rate limit
**Descriptions** - warning worker: less than 5 remaining calls to GitHub search API before hitting the rate limit **Next steps** - Consider creating a new token for the indicated resource (the `name` label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-github_search_rate_limit_remaining). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_github_search_rate_limit_remaining" ] ``` *Managed by the Sourcegraph Services team.*remaining calls to GitLab rest API before hitting the rate limit
**Descriptions** - critical worker: less than 30 remaining calls to GitLab rest API before hitting the rate limit **Next steps** - Try restarting the pod to get a different public IP. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-gitlab_rest_rate_limit_remaining). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_worker_gitlab_rest_rate_limit_remaining" ] ``` *Managed by the Sourcegraph Services team.*number of entities with outdated permissions
**Descriptions** - warning worker: 100+ number of entities with outdated permissions for 5m0s **Next steps** - **Enabled permissions for the first time:** Wait for few minutes and see if the number goes down. - **Otherwise:** Increase the API rate limit to [GitHub](https://sourcegraph.com/docs/admin/code_hosts/github#github-com-rate-limits), [GitLab](https://sourcegraph.com/docs/admin/code_hosts/gitlab#internal-rate-limits) or [Bitbucket Server](https://sourcegraph.com/docs/admin/code_hosts/bitbucket_server#internal-rate-limits). - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-perms_syncer_outdated_perms). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_perms_syncer_outdated_perms" ] ``` *Managed by the Sourcegraph Services team.*95th permissions sync duration
**Descriptions** - warning worker: 30s+ 95th permissions sync duration for 5m0s **Next steps** - Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-perms_syncer_sync_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_perms_syncer_sync_duration" ] ``` *Managed by the Sourcegraph Services team.*error rate for periodic goroutine executions
**Descriptions** - warning worker: 0.01reqps+ error rate for periodic goroutine executions for 15m0s **Next steps** - Check service logs for error details related to the failing periodic routine - Check if the routine depends on external services that may be unavailable - Look for recent changes to the routine`s code or configuration - More help interpreting this metric is available in the [dashboards reference](dashboards#worker-goroutine_error_rate). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_goroutine_error_rate" ] ``` *Managed by the Sourcegraph Services team.*percentage of periodic goroutine executions that result in errors
**Descriptions** - warning worker: 5%+ percentage of periodic goroutine executions that result in errors **Next steps** - Check service logs for error details related to the failing periodic routine - Check if the routine depends on external services that may be unavailable - Consider temporarily disabling the routine if it`s non-critical and causing cascading issues - More help interpreting this metric is available in the [dashboards reference](dashboards#worker-goroutine_error_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_goroutine_error_percentage" ] ``` *Managed by the Sourcegraph Services team.*mean blocked seconds per conn request
**Descriptions** - warning worker: 0.1s+ mean blocked seconds per conn request for 10m0s - critical worker: 0.5s+ mean blocked seconds per conn request for 10m0s **Next steps** - Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed - Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf) - If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-mean_blocked_seconds_per_conn_request). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_mean_blocked_seconds_per_conn_request", "critical_worker_mean_blocked_seconds_per_conn_request" ] ``` *Managed by the Sourcegraph Platform team.*CPU usage
**Descriptions** - warning worker: 95%+ CPU usage for 10m0s **Next steps** - Consider increasing CPU limits or scaling out. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-cpu_usage_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_cpu_usage_percentage" ] ``` *Managed by the Sourcegraph Code Understanding team.*memory (RSS)
**Descriptions** - warning worker: 90%+ memory (RSS) for 10m0s **Next steps** - Consider increasing memory limits or scaling out. - More help interpreting this metric is available in the [dashboards reference](dashboards#worker-memory_rss). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_memory_rss" ] ``` *Managed by the Sourcegraph Code Understanding team.*container cpu usage total (1m average) across all cores by instance
**Descriptions** - warning worker: 99%+ container cpu usage total (1m average) across all cores by instance **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the worker container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-container_cpu_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_container_cpu_usage" ] ``` *Managed by the Sourcegraph Code Understanding team.*container memory usage by instance
**Descriptions** - warning worker: 99%+ container memory usage by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of worker container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-container_memory_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_container_memory_usage" ] ``` *Managed by the Sourcegraph Code Understanding team.*container cpu usage total (90th percentile over 1d) across all cores by instance
**Descriptions** - warning worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the worker service. - **Docker Compose:** Consider increasing `cpus:` of the worker container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-provisioning_container_cpu_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_provisioning_container_cpu_usage_long_term" ] ``` *Managed by the Sourcegraph Code Plane team.*container memory usage (1d maximum) by instance
**Descriptions** - warning worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the worker service. - **Docker Compose:** Consider increasing `memory:` of the worker container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-provisioning_container_memory_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_provisioning_container_memory_usage_long_term" ] ``` *Managed by the Sourcegraph Code Plane team.*container cpu usage total (5m maximum) across all cores by instance
**Descriptions** - warning worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the worker container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-provisioning_container_cpu_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_provisioning_container_cpu_usage_short_term" ] ``` *Managed by the Sourcegraph Code Plane team.*container memory usage (5m maximum) by instance
**Descriptions** - warning worker: 90%+ container memory usage (5m maximum) by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of worker container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-provisioning_container_memory_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_provisioning_container_memory_usage_short_term" ] ``` *Managed by the Sourcegraph Code Plane team.*container OOMKILL events total by instance
**Descriptions** - warning worker: 1+ container OOMKILL events total by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of worker container in `docker-compose.yml`. - More help interpreting this metric is available in the [dashboards reference](dashboards#worker-container_oomkill_events_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_container_oomkill_events_total" ] ``` *Managed by the Sourcegraph Code Plane team.*maximum active goroutines
**Descriptions** - warning worker: 10000+ maximum active goroutines for 10m0s **Next steps** - More help interpreting this metric is available in the [dashboards reference](dashboards#worker-go_goroutines). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_go_goroutines" ] ``` *Managed by the Sourcegraph Code Plane team.*maximum go garbage collection duration
**Descriptions** - warning worker: 2s+ maximum go garbage collection duration **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-go_gc_duration_seconds). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_worker_go_gc_duration_seconds" ] ``` *Managed by the Sourcegraph Code Plane team.*percentage pods available
**Descriptions** - critical worker: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod worker` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p worker`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_worker_pods_available_percentage" ] ``` *Managed by the Sourcegraph Code Plane team.*maximum duration since last successful site configuration update (all "worker" instances)
**Descriptions** - critical worker: 300s+ maximum duration since last successful site configuration update (all "worker" instances) **Next steps** - This indicates that one or more "worker" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself. - Check for relevant errors in the "worker" logs, as well as frontend`s logs. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#worker-worker_site_configuration_duration_since_last_successful_update_by_instance). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_worker_worker_site_configuration_duration_since_last_successful_update_by_instance" ] ``` *Managed by the Sourcegraph Platform team.*requests per second per replica over 10m
**Descriptions** - warning searcher: 5+ requests per second per replica over 10m **Next steps** - More help interpreting this metric is available in the [dashboards reference](dashboards#searcher-replica_traffic). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_replica_traffic" ] ``` *Managed by the Sourcegraph Code Understanding team.*unindexed search request errors every 5m by code
**Descriptions** - warning searcher: 5%+ unindexed search request errors every 5m by code for 5m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-unindexed_search_request_errors). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_unindexed_search_request_errors" ] ``` *Managed by the Sourcegraph Code Understanding team.*maximum duration since last successful site configuration update (all "searcher" instances)
**Descriptions** - critical searcher: 300s+ maximum duration since last successful site configuration update (all "searcher" instances) **Next steps** - This indicates that one or more "searcher" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself. - Check for relevant errors in the "searcher" logs, as well as frontend`s logs. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-searcher_site_configuration_duration_since_last_successful_update_by_instance). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_searcher_searcher_site_configuration_duration_since_last_successful_update_by_instance" ] ``` *Managed by the Sourcegraph Platform team.*error rate for periodic goroutine executions
**Descriptions** - warning searcher: 0.01reqps+ error rate for periodic goroutine executions for 15m0s **Next steps** - Check service logs for error details related to the failing periodic routine - Check if the routine depends on external services that may be unavailable - Look for recent changes to the routine`s code or configuration - More help interpreting this metric is available in the [dashboards reference](dashboards#searcher-goroutine_error_rate). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_goroutine_error_rate" ] ``` *Managed by the Sourcegraph Services team.*percentage of periodic goroutine executions that result in errors
**Descriptions** - warning searcher: 5%+ percentage of periodic goroutine executions that result in errors **Next steps** - Check service logs for error details related to the failing periodic routine - Check if the routine depends on external services that may be unavailable - Consider temporarily disabling the routine if it`s non-critical and causing cascading issues - More help interpreting this metric is available in the [dashboards reference](dashboards#searcher-goroutine_error_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_goroutine_error_percentage" ] ``` *Managed by the Sourcegraph Services team.*mean blocked seconds per conn request
**Descriptions** - warning searcher: 0.1s+ mean blocked seconds per conn request for 10m0s - critical searcher: 0.5s+ mean blocked seconds per conn request for 10m0s **Next steps** - Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed - Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf) - If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-mean_blocked_seconds_per_conn_request). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_mean_blocked_seconds_per_conn_request", "critical_searcher_mean_blocked_seconds_per_conn_request" ] ``` *Managed by the Sourcegraph Code Understanding team.*CPU usage
**Descriptions** - warning searcher: 95%+ CPU usage for 10m0s **Next steps** - Consider increasing CPU limits or scaling out. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-cpu_usage_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_cpu_usage_percentage" ] ``` *Managed by the Sourcegraph Code Understanding team.*memory (RSS)
**Descriptions** - warning searcher: 90%+ memory (RSS) for 10m0s **Next steps** - Consider increasing memory limits or scaling out. - More help interpreting this metric is available in the [dashboards reference](dashboards#searcher-memory_rss). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_memory_rss" ] ``` *Managed by the Sourcegraph Code Understanding team.*container cpu usage total (1m average) across all cores by instance
**Descriptions** - warning searcher: 99%+ container cpu usage total (1m average) across all cores by instance **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the searcher container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-container_cpu_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_container_cpu_usage" ] ``` *Managed by the Sourcegraph Code Understanding team.*container memory usage by instance
**Descriptions** - warning searcher: 99%+ container memory usage by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of searcher container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-container_memory_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_container_memory_usage" ] ``` *Managed by the Sourcegraph Code Understanding team.*container cpu usage total (90th percentile over 1d) across all cores by instance
**Descriptions** - warning searcher: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the searcher service. - **Docker Compose:** Consider increasing `cpus:` of the searcher container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-provisioning_container_cpu_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_provisioning_container_cpu_usage_long_term" ] ``` *Managed by the Sourcegraph Code Understanding team.*container memory usage (1d maximum) by instance
**Descriptions** - warning searcher: 80%+ container memory usage (1d maximum) by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the searcher service. - **Docker Compose:** Consider increasing `memory:` of the searcher container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-provisioning_container_memory_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_provisioning_container_memory_usage_long_term" ] ``` *Managed by the Sourcegraph Code Understanding team.*container cpu usage total (5m maximum) across all cores by instance
**Descriptions** - warning searcher: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the searcher container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-provisioning_container_cpu_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_provisioning_container_cpu_usage_short_term" ] ``` *Managed by the Sourcegraph Code Understanding team.*container memory usage (5m maximum) by instance
**Descriptions** - warning searcher: 90%+ container memory usage (5m maximum) by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of searcher container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-provisioning_container_memory_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_provisioning_container_memory_usage_short_term" ] ``` *Managed by the Sourcegraph Code Understanding team.*container OOMKILL events total by instance
**Descriptions** - warning searcher: 1+ container OOMKILL events total by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of searcher container in `docker-compose.yml`. - More help interpreting this metric is available in the [dashboards reference](dashboards#searcher-container_oomkill_events_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_container_oomkill_events_total" ] ``` *Managed by the Sourcegraph Code Understanding team.*maximum active goroutines
**Descriptions** - warning searcher: 10000+ maximum active goroutines for 10m0s **Next steps** - More help interpreting this metric is available in the [dashboards reference](dashboards#searcher-go_goroutines). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_go_goroutines" ] ``` *Managed by the Sourcegraph Code Understanding team.*maximum go garbage collection duration
**Descriptions** - warning searcher: 2s+ maximum go garbage collection duration **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-go_gc_duration_seconds). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_searcher_go_gc_duration_seconds" ] ``` *Managed by the Sourcegraph Code Understanding team.*percentage pods available
**Descriptions** - critical searcher: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod searcher` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p searcher`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#searcher-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_searcher_pods_available_percentage" ] ``` *Managed by the Sourcegraph Code Understanding team.*CPU usage
**Descriptions** - warning syntect-server: 95%+ CPU usage for 10m0s **Next steps** - Consider increasing CPU limits or scaling out. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntect-server-cpu_usage_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_syntect-server_cpu_usage_percentage" ] ``` *Managed by the Sourcegraph Code Understanding team.*memory (RSS)
**Descriptions** - warning syntect-server: 90%+ memory (RSS) for 10m0s **Next steps** - Consider increasing memory limits or scaling out. - More help interpreting this metric is available in the [dashboards reference](dashboards#syntect-server-memory_rss). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_syntect-server_memory_rss" ] ``` *Managed by the Sourcegraph Code Understanding team.*container cpu usage total (1m average) across all cores by instance
**Descriptions** - warning syntect-server: 99%+ container cpu usage total (1m average) across all cores by instance **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the syntect-server container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntect-server-container_cpu_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_syntect-server_container_cpu_usage" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage by instance
**Descriptions** - warning syntect-server: 99%+ container memory usage by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of syntect-server container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntect-server-container_memory_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_syntect-server_container_memory_usage" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (90th percentile over 1d) across all cores by instance
**Descriptions** - warning syntect-server: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the syntect-server service. - **Docker Compose:** Consider increasing `cpus:` of the syntect-server container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntect-server-provisioning_container_cpu_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_syntect-server_provisioning_container_cpu_usage_long_term" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage (1d maximum) by instance
**Descriptions** - warning syntect-server: 80%+ container memory usage (1d maximum) by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the syntect-server service. - **Docker Compose:** Consider increasing `memory:` of the syntect-server container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntect-server-provisioning_container_memory_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_syntect-server_provisioning_container_memory_usage_long_term" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (5m maximum) across all cores by instance
**Descriptions** - warning syntect-server: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the syntect-server container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntect-server-provisioning_container_cpu_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_syntect-server_provisioning_container_cpu_usage_short_term" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage (5m maximum) by instance
**Descriptions** - warning syntect-server: 90%+ container memory usage (5m maximum) by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of syntect-server container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntect-server-provisioning_container_memory_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_syntect-server_provisioning_container_memory_usage_short_term" ] ``` *Managed by the Sourcegraph Platform team.*container OOMKILL events total by instance
**Descriptions** - warning syntect-server: 1+ container OOMKILL events total by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of syntect-server container in `docker-compose.yml`. - More help interpreting this metric is available in the [dashboards reference](dashboards#syntect-server-container_oomkill_events_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_syntect-server_container_oomkill_events_total" ] ``` *Managed by the Sourcegraph Platform team.*percentage pods available
**Descriptions** - critical syntect-server: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod syntect-server` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p syntect-server`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#syntect-server-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_syntect-server_pods_available_percentage" ] ``` *Managed by the Sourcegraph Platform team.*average resolve revision duration over 5m
**Descriptions** - warning zoekt: 15s+ average resolve revision duration over 5m **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#zoekt-average_resolve_revision_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_average_resolve_revision_duration" ] ``` *Managed by the Sourcegraph Code Understanding team.*the number of repositories we failed to get indexing options over 5m
**Descriptions** - warning zoekt: 100+ the number of repositories we failed to get indexing options over 5m for 5m0s - critical zoekt: 100+ the number of repositories we failed to get indexing options over 5m for 35m0s **Next steps** - View error rates on gitserver and frontend to identify root cause. - Rollback frontend/gitserver deployment if due to a bad code change. - View error logs for `getIndexOptions` via net/trace debug interface. For example click on a `indexed-search-indexer-` on https://sourcegraph.com/-/debug/. Then click on Traces. Replace sourcegraph.com with your instance address. - More help interpreting this metric is available in the [dashboards reference](dashboards#zoekt-get_index_options_error_increase). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_get_index_options_error_increase", "critical_zoekt_get_index_options_error_increase" ] ``` *Managed by the Sourcegraph Code Understanding team.*CPU usage
**Descriptions** - warning zoekt: 95%+ CPU usage for 10m0s **Next steps** - Consider increasing CPU limits or scaling out. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#zoekt-cpu_usage_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_cpu_usage_percentage" ] ``` *Managed by the Sourcegraph Code Understanding team.*memory (RSS)
**Descriptions** - warning zoekt: 90%+ memory (RSS) for 10m0s **Next steps** - Consider increasing memory limits or scaling out. - More help interpreting this metric is available in the [dashboards reference](dashboards#zoekt-memory_rss). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_memory_rss" ] ``` *Managed by the Sourcegraph Code Understanding team.*CPU usage
**Descriptions** - warning zoekt: 95%+ CPU usage for 10m0s **Next steps** - Consider increasing CPU limits or scaling out. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#zoekt-cpu_usage_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_cpu_usage_percentage" ] ``` *Managed by the Sourcegraph Code Understanding team.*memory (RSS)
**Descriptions** - warning zoekt: 90%+ memory (RSS) for 10m0s **Next steps** - Consider increasing memory limits or scaling out. - More help interpreting this metric is available in the [dashboards reference](dashboards#zoekt-memory_rss). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_memory_rss" ] ``` *Managed by the Sourcegraph Code Understanding team.*process memory map areas percentage used (per instance)
**Descriptions** - warning zoekt: 60%+ process memory map areas percentage used (per instance) - critical zoekt: 80%+ process memory map areas percentage used (per instance) **Next steps** - If you are running out of memory map areas, you could resolve this by: - Enabling shard merging for Zoekt: Set SRC_ENABLE_SHARD_MERGING="1" for zoekt-indexserver. Use this option if your corpus of repositories has a high percentage of small, rarely updated repositories. See [documentation](https://sourcegraph.com/docs/code-search/features#shard-merging). - Creating additional Zoekt replicas: This spreads all the shards out amongst more replicas, which means that each _individual_ replica will have fewer shards. This, in turn, decreases the amount of memory map areas that a _single_ replica can create (in order to load the shards into memory). - Increasing the virtual memory subsystem`s "max_map_count" parameter which defines the upper limit of memory areas a process can use. The default value of max_map_count is usually 65536. We recommend to set this value to 2x the number of repos to be indexed per Zoekt instance. This means, if you want to index 240k repositories with 3 Zoekt instances, set max_map_count to (240000 / 3) * 2 = 160000. The exact instructions for tuning this parameter can differ depending on your environment. See https://kernel.org/doc/Documentation/sysctl/vm.txt for more information. - More help interpreting this metric is available in the [dashboards reference](dashboards#zoekt-memory_map_areas_percentage_used). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_memory_map_areas_percentage_used", "critical_zoekt_memory_map_areas_percentage_used" ] ``` *Managed by the Sourcegraph Code Understanding team.*indexed search request errors every 5m by code
**Descriptions** - warning zoekt: 5%+ indexed search request errors every 5m by code for 5m0s **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#zoekt-indexed_search_request_errors). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_indexed_search_request_errors" ] ``` *Managed by the Sourcegraph Code Understanding team.*maximum active goroutines
**Descriptions** - warning zoekt: 10000+ maximum active goroutines for 10m0s **Next steps** - More help interpreting this metric is available in the [dashboards reference](dashboards#zoekt-go_goroutines). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_go_goroutines" ] ``` *Managed by the Sourcegraph Code Understanding team.*maximum go garbage collection duration
**Descriptions** - warning zoekt: 2s+ maximum go garbage collection duration **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#zoekt-go_gc_duration_seconds). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_go_gc_duration_seconds" ] ``` *Managed by the Sourcegraph Code Understanding team.*maximum active goroutines
**Descriptions** - warning zoekt: 10000+ maximum active goroutines for 10m0s **Next steps** - More help interpreting this metric is available in the [dashboards reference](dashboards#zoekt-go_goroutines). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_go_goroutines" ] ``` *Managed by the Sourcegraph Code Understanding team.*maximum go garbage collection duration
**Descriptions** - warning zoekt: 2s+ maximum go garbage collection duration **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#zoekt-go_gc_duration_seconds). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_zoekt_go_gc_duration_seconds" ] ``` *Managed by the Sourcegraph Code Understanding team.*percentage pods available
**Descriptions** - critical zoekt: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod indexed-search` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p indexed-search`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#zoekt-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_zoekt_pods_available_percentage" ] ``` *Managed by the Sourcegraph Code Understanding team.*average prometheus rule group evaluation duration over 10m by rule group
**Descriptions** - warning prometheus: 30s+ average prometheus rule group evaluation duration over 10m by rule group **Next steps** - Check the Container monitoring (not available on server) panels and try increasing resources for Prometheus if necessary. - If the rule group taking a long time to evaluate belongs to `/sg_prometheus_addons`, try reducing the complexity of any custom Prometheus rules provided. - If the rule group taking a long time to evaluate belongs to `/sg_config_prometheus`, please [open an issue](https://github.com/sourcegraph/sourcegraph/issues/new?assignees=&labels=&template=bug_report.md&title=). - More help interpreting this metric is available in the [dashboards reference](dashboards#prometheus-prometheus_rule_eval_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_prometheus_rule_eval_duration" ] ``` *Managed by the Sourcegraph Services team.*failed prometheus rule evaluations over 5m by rule group
**Descriptions** - warning prometheus: 0+ failed prometheus rule evaluations over 5m by rule group **Next steps** - Check Prometheus logs for messages related to rule group evaluation (generally with log field `component="rule manager"`). - If the rule group failing to evaluate belongs to `/sg_prometheus_addons`, ensure any custom Prometheus configuration provided is valid. - If the rule group taking a long time to evaluate belongs to `/sg_config_prometheus`, please [open an issue](https://github.com/sourcegraph/sourcegraph/issues/new?assignees=&labels=&template=bug_report.md&title=). - More help interpreting this metric is available in the [dashboards reference](dashboards#prometheus-prometheus_rule_eval_failures). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_prometheus_rule_eval_failures" ] ``` *Managed by the Sourcegraph Services team.*alertmanager notification latency over 1m by integration
**Descriptions** - warning prometheus: 1s+ alertmanager notification latency over 1m by integration **Next steps** - Check the Container monitoring (not available on server) panels and try increasing resources for Prometheus if necessary. - Ensure that your [`observability.alerts` configuration](https://sourcegraph.com/docs/admin/observability/alerting#setting-up-alerting) (in site configuration) is valid. - Check if the relevant alert integration service is experiencing downtime or issues. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-alertmanager_notification_latency). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_alertmanager_notification_latency" ] ``` *Managed by the Sourcegraph Services team.*failed alertmanager notifications over 1m by integration
**Descriptions** - warning prometheus: 0+ failed alertmanager notifications over 1m by integration **Next steps** - Ensure that your [`observability.alerts` configuration](https://sourcegraph.com/docs/admin/observability/alerting#setting-up-alerting) (in site configuration) is valid. - Check if the relevant alert integration service is experiencing downtime or issues. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-alertmanager_notification_failures). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_alertmanager_notification_failures" ] ``` *Managed by the Sourcegraph Services team.*prometheus configuration reload status
**Descriptions** - warning prometheus: less than 1 prometheus configuration reload status **Next steps** - Check Prometheus logs for messages related to configuration loading. - Ensure any [custom configuration you have provided Prometheus](https://sourcegraph.com/docs/admin/observability/metrics#prometheus-configuration) is valid. - More help interpreting this metric is available in the [dashboards reference](dashboards#prometheus-prometheus_config_status). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_prometheus_config_status" ] ``` *Managed by the Sourcegraph Services team.*alertmanager configuration reload status
**Descriptions** - warning prometheus: less than 1 alertmanager configuration reload status **Next steps** - Ensure that your [`observability.alerts` configuration](https://sourcegraph.com/docs/admin/observability/alerting#setting-up-alerting) (in site configuration) is valid. - More help interpreting this metric is available in the [dashboards reference](dashboards#prometheus-alertmanager_config_status). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_alertmanager_config_status" ] ``` *Managed by the Sourcegraph Services team.*prometheus tsdb failures by operation over 1m by operation
**Descriptions** - warning prometheus: 0+ prometheus tsdb failures by operation over 1m by operation **Next steps** - Check Prometheus logs for messages related to the failing operation. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-prometheus_tsdb_op_failure). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_prometheus_tsdb_op_failure" ] ``` *Managed by the Sourcegraph Services team.*prometheus scrapes that exceed the sample limit over 10m
**Descriptions** - warning prometheus: 0+ prometheus scrapes that exceed the sample limit over 10m **Next steps** - Check Prometheus logs for messages related to target scrape failures. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-prometheus_target_sample_exceeded). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_prometheus_target_sample_exceeded" ] ``` *Managed by the Sourcegraph Services team.*prometheus scrapes rejected due to duplicate timestamps over 10m
**Descriptions** - warning prometheus: 0+ prometheus scrapes rejected due to duplicate timestamps over 10m **Next steps** - Check Prometheus logs for messages related to target scrape failures. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-prometheus_target_sample_duplicate). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_prometheus_target_sample_duplicate" ] ``` *Managed by the Sourcegraph Services team.*container cpu usage total (1m average) across all cores by instance
**Descriptions** - warning prometheus: 99%+ container cpu usage total (1m average) across all cores by instance **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the prometheus container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-container_cpu_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_container_cpu_usage" ] ``` *Managed by the Sourcegraph Services team.*container memory usage by instance
**Descriptions** - warning prometheus: 99%+ container memory usage by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of prometheus container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-container_memory_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_container_memory_usage" ] ``` *Managed by the Sourcegraph Services team.*container cpu usage total (90th percentile over 1d) across all cores by instance
**Descriptions** - warning prometheus: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the prometheus service. - **Docker Compose:** Consider increasing `cpus:` of the prometheus container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-provisioning_container_cpu_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_provisioning_container_cpu_usage_long_term" ] ``` *Managed by the Sourcegraph Services team.*container memory usage (1d maximum) by instance
**Descriptions** - warning prometheus: 80%+ container memory usage (1d maximum) by instance for 336h0m0s **Next steps** - **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the prometheus service. - **Docker Compose:** Consider increasing `memory:` of the prometheus container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-provisioning_container_memory_usage_long_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_provisioning_container_memory_usage_long_term" ] ``` *Managed by the Sourcegraph Services team.*container cpu usage total (5m maximum) across all cores by instance
**Descriptions** - warning prometheus: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the prometheus container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-provisioning_container_cpu_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_provisioning_container_cpu_usage_short_term" ] ``` *Managed by the Sourcegraph Services team.*container memory usage (5m maximum) by instance
**Descriptions** - warning prometheus: 90%+ container memory usage (5m maximum) by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of prometheus container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-provisioning_container_memory_usage_short_term). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_provisioning_container_memory_usage_short_term" ] ``` *Managed by the Sourcegraph Services team.*container OOMKILL events total by instance
**Descriptions** - warning prometheus: 1+ container OOMKILL events total by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of prometheus container in `docker-compose.yml`. - More help interpreting this metric is available in the [dashboards reference](dashboards#prometheus-container_oomkill_events_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_prometheus_container_oomkill_events_total" ] ``` *Managed by the Sourcegraph Services team.*percentage pods available
**Descriptions** - critical prometheus: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod prometheus` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p prometheus`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#prometheus-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_prometheus_pods_available_percentage" ] ``` *Managed by the Sourcegraph Services team.*executor active handlers
**Descriptions** - critical executor: 0 active executor handlers and > 0 queue size for 5m0s **Next steps** - Check to see the state of any compute VMs, they may be taking longer than expected to boot. - Make sure the executors appear under Site Admin > Executors. - Check the Grafana dashboard section for APIClient, it should do frequent requests to Dequeue and Heartbeat and those must not fail. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#executor-executor_handlers). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_executor_executor_handlers" ] ``` *Managed by the Sourcegraph Code Understanding team.*executor operation error rate over 5m
**Descriptions** - warning executor: 100%+ executor operation error rate over 5m for 1h0m0s **Next steps** - Determine the cause of failure from the auto-indexing job logs in the site-admin page. - This alert fires if all executor jobs have been failing for the past hour. The alert will continue for up to 5 hours until the error rate is no longer 100%, even if there are no running jobs in that time, as the problem is not know to be resolved until jobs start succeeding again. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#executor-executor_processor_error_rate). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_executor_executor_processor_error_rate" ] ``` *Managed by the Sourcegraph Code Understanding team.*maximum active goroutines
**Descriptions** - warning executor: 10000+ maximum active goroutines for 10m0s **Next steps** - More help interpreting this metric is available in the [dashboards reference](dashboards#executor-go_goroutines). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_executor_go_goroutines" ] ``` *Managed by the Sourcegraph Code Plane team.*maximum go garbage collection duration
**Descriptions** - warning executor: 2s+ maximum go garbage collection duration **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#executor-go_gc_duration_seconds). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_executor_go_gc_duration_seconds" ] ``` *Managed by the Sourcegraph Code Plane team.*rate of growth of events export queue over 30m
**Descriptions** - warning telemetry: 1+ rate of growth of events export queue over 30m for 1h0m0s - critical telemetry: 1+ rate of growth of events export queue over 30m for 36h0m0s **Next steps** - Check the "number of events exported per batch over 30m" dashboard panel to see if export throughput is at saturation. - Increase `TELEMETRY_GATEWAY_EXPORTER_EXPORT_BATCH_SIZE` to export more events per batch. - Reduce `TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL` to schedule more export jobs. - See worker logs in the `worker.telemetrygateway-exporter` log scope for more details to see if any export errors are occuring - if logs only indicate that exports failed, reach out to Sourcegraph with relevant log entries, as this may be an issue in Sourcegraph`s Telemetry Gateway service. - More help interpreting this metric is available in the [dashboards reference](dashboards#telemetry-telemetry_gateway_exporter_queue_growth). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_telemetry_telemetry_gateway_exporter_queue_growth", "critical_telemetry_telemetry_gateway_exporter_queue_growth" ] ``` *Managed by the Sourcegraph Services team.*events exporter operation errors every 30m
**Descriptions** - warning telemetry: 0+ events exporter operation errors every 30m **Next steps** - Failures indicate that exporting of telemetry events from Sourcegraph are failing. This may affect the performance of the database as the backlog grows. - See worker logs in the `worker.telemetrygateway-exporter` log scope for more details. If logs only indicate that exports failed, reach out to Sourcegraph with relevant log entries, as this may be an issue in Sourcegraph`s Telemetry Gateway service. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#telemetry-telemetrygatewayexporter_exporter_errors_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_telemetry_telemetrygatewayexporter_exporter_errors_total" ] ``` *Managed by the Sourcegraph Services team.*events export queue cleanup operation errors every 30m
**Descriptions** - warning telemetry: 0+ events export queue cleanup operation errors every 30m **Next steps** - Failures indicate that pruning of already-exported telemetry events from the database is failing. This may affect the performance of the database as the export queue table grows. - See worker logs in the `worker.telemetrygateway-exporter` log scope for more details. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#telemetry-telemetrygatewayexporter_queue_cleanup_errors_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_telemetry_telemetrygatewayexporter_queue_cleanup_errors_total" ] ``` *Managed by the Sourcegraph Services team.*events export backlog metrics reporting operation errors every 30m
**Descriptions** - warning telemetry: 0+ events export backlog metrics reporting operation errors every 30m **Next steps** - Failures indicate that reporting of telemetry events metrics is failing. This may affect the reliability of telemetry events export metrics. - See worker logs in the `worker.telemetrygateway-exporter` log scope for more details. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#telemetry-telemetrygatewayexporter_queue_metrics_reporter_errors_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_telemetry_telemetrygatewayexporter_queue_metrics_reporter_errors_total" ] ``` *Managed by the Sourcegraph Services team.*failed writes to events export queue over 5m
**Descriptions** - warning telemetry: 1%+ failed writes to events export queue over 5m - critical telemetry: 2.5%+ failed writes to events export queue over 5m for 5m0s **Next steps** - Look for error logs related to `inserting telemetry events`. - Look for error attributes on `telemetryevents.QueueForExport` trace spans. - More help interpreting this metric is available in the [dashboards reference](dashboards#telemetry-telemetry_v2_export_queue_write_failures). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_telemetry_telemetry_v2_export_queue_write_failures", "critical_telemetry_telemetry_v2_export_queue_write_failures" ] ``` *Managed by the Sourcegraph Services team.*failed write V2 events to V1 'event_logs' over 5m
**Descriptions** - warning telemetry: 5%+ failed write V2 events to V1 'event_logs' over 5m **Next steps** - Error details are only persisted in trace metadata as it is considered non-critical. - To diagnose, enable trace sampling across all requests and look for error attributes on `telemetrystore.v1teewrite` spans. - More help interpreting this metric is available in the [dashboards reference](dashboards#telemetry-telemetry_v2_event_logs_write_failures). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_telemetry_telemetry_v2_event_logs_write_failures" ] ``` *Managed by the Sourcegraph Services team.*(off by default) user metadata exporter operation errors every 30m
**Descriptions** - warning telemetry: 0+ (off by default) user metadata exporter operation errors every 30m **Next steps** - Failures indicate that exporting of telemetry events from Sourcegraph are failing. This may affect the performance of the database as the backlog grows. - See worker logs in the `worker.telemetrygateway-exporter` log scope for more details. If logs only indicate that exports failed, reach out to Sourcegraph with relevant log entries, as this may be an issue in Sourcegraph`s Telemetry Gateway service. - This exporter is DISABLED by default. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#telemetry-telemetrygatewayexporter_usermetadata_exporter_errors_total). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_telemetry_telemetrygatewayexporter_usermetadata_exporter_errors_total" ] ``` *Managed by the Sourcegraph Services team.*spans refused per receiver
**Descriptions** - warning otel-collector: 1+ spans refused per receiver for 5m0s **Next steps** - Check logs of the collector and configuration of the receiver - More help interpreting this metric is available in the [dashboards reference](dashboards#otel-collector-otel_span_refused). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_otel-collector_otel_span_refused" ] ``` *Managed by the Sourcegraph Platform team.*span export failures by exporter
**Descriptions** - warning otel-collector: 1+ span export failures by exporter for 5m0s **Next steps** - Check the configuration of the exporter and if the service being exported is up - More help interpreting this metric is available in the [dashboards reference](dashboards#otel-collector-otel_span_export_failures). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_otel-collector_otel_span_export_failures" ] ``` *Managed by the Sourcegraph Platform team.*exporter enqueue failed spans
**Descriptions** - warning otel-collector: 0+ exporter enqueue failed spans for 5m0s **Next steps** - Check the configuration of the exporter and if the service being exported is up. This may be caused by a queue full of unsettled elements, so you may need to decrease your sending rate or horizontally scale collectors. - More help interpreting this metric is available in the [dashboards reference](dashboards#otel-collector-otelcol_exporter_enqueue_failed_spans). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_otel-collector_otelcol_exporter_enqueue_failed_spans" ] ``` *Managed by the Sourcegraph Platform team.*spans dropped per processor per minute
**Descriptions** - warning otel-collector: 0+ spans dropped per processor per minute for 5m0s **Next steps** - Check the configuration of the processor - More help interpreting this metric is available in the [dashboards reference](dashboards#otel-collector-otelcol_processor_dropped_spans). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_otel-collector_otelcol_processor_dropped_spans" ] ``` *Managed by the Sourcegraph Platform team.*container cpu usage total (1m average) across all cores by instance
**Descriptions** - warning otel-collector: 99%+ container cpu usage total (1m average) across all cores by instance **Next steps** - **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `cpus:` of the otel-collector container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#otel-collector-container_cpu_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_otel-collector_container_cpu_usage" ] ``` *Managed by the Sourcegraph Platform team.*container memory usage by instance
**Descriptions** - warning otel-collector: 99%+ container memory usage by instance **Next steps** - **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. - **Docker Compose:** Consider increasing `memory:` of otel-collector container in `docker-compose.yml`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#otel-collector-container_memory_usage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_otel-collector_container_memory_usage" ] ``` *Managed by the Sourcegraph Platform team.*percentage pods available
**Descriptions** - critical otel-collector: less than 50% percentage pods available for 15m0s **Next steps** - Determine if the pod was OOM killed using `kubectl describe pod otel-collector` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`. - Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p otel-collector`. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#otel-collector-pods_available_percentage). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "critical_otel-collector_pods_available_percentage" ] ``` *Managed by the Sourcegraph Platform team.*rate of growth of in-flight questions over 1h
**Descriptions** - warning deepsearch: 1+ rate of growth of in-flight questions over 1h for 30m0s - critical deepsearch: 1+ rate of growth of in-flight questions over 1h for 1h0m0s **Next steps** - Check for questions that are never stopping. - Check for slow LLM responses or tool execution times. - Review `deepsearch_question_processing_duration` for processing time trends. - More help interpreting this metric is available in the [dashboards reference](dashboards#deepsearch-deepsearch_questions_in_flight_growth). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_deepsearch_deepsearch_questions_in_flight_growth", "critical_deepsearch_deepsearch_questions_in_flight_growth" ] ``` *Managed by the Sourcegraph Code Understanding team.*question processing error rate over 5m
**Descriptions** - warning deepsearch: 10%+ question processing error rate over 5m for 10m0s - critical deepsearch: 20%+ question processing error rate over 5m for 10m0s **Next steps** - Check frontend logs for `Worker failed to process question` errors. - Review LLM stream errors in the `LLM streaming` panel below. - Check for upstream LLM provider issues. - More help interpreting this metric is available in the [dashboards reference](dashboards#deepsearch-deepsearch_question_processing_error_rate). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_deepsearch_deepsearch_question_processing_error_rate", "critical_deepsearch_deepsearch_question_processing_error_rate" ] ``` *Managed by the Sourcegraph Code Understanding team.*fatal LLM stream errors over 5m
**Descriptions** - warning deepsearch: 20+ fatal LLM stream errors over 5m **Next steps** - Check frontend logs for `fatal error in LLM stream`. - More help interpreting this metric is available in the [dashboards reference](dashboards#deepsearch-deepsearch_llm_stream_fatal_errors). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_deepsearch_deepsearch_llm_stream_fatal_errors" ] ``` *Managed by the Sourcegraph Code Understanding team.*error rate over 5m
**Descriptions** - warning externalapi: 10%+ error rate over 5m for 10m0s - critical externalapi: 25%+ error rate over 5m for 10m0s **Next steps** - Check frontend logs for external API errors. - Review individual RPC method error rates below. - More help interpreting this metric is available in the [dashboards reference](dashboards#externalapi-externalapi_error_rate). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_externalapi_externalapi_error_rate", "critical_externalapi_externalapi_error_rate" ] ``` *Managed by the Sourcegraph Platform team.*percentage of operations resulting in error by method
**Descriptions** - warning background-jobs: 5%+ percentage of operations resulting in error by method - critical background-jobs: 50%+ percentage of operations resulting in error by method **Next steps** - Review logs for the specific operation to identify patterns in errors. Check database connectivity and schema. If a particular method is consistently failing, investigate potential issues with that operation`s SQL query or transaction handling. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#background-jobs-error_percentage_by_method). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_background-jobs_error_percentage_by_method", "critical_background-jobs_error_percentage_by_method" ] ``` *Managed by the Sourcegraph Services team.*percentage of operations resulting in error by domain
**Descriptions** - warning background-jobs: 5%+ percentage of operations resulting in error by domain - critical background-jobs: 50%+ percentage of operations resulting in error by domain **Next steps** - Review logs for the specific domain to identify patterns in errors. Check database connectivity and schema. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#background-jobs-error_percentage_by_domain). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_background-jobs_error_percentage_by_domain", "critical_background-jobs_error_percentage_by_domain" ] ``` *Managed by the Sourcegraph Services team.*time spent running the resetter
**Descriptions** - warning background-jobs: 10s+ time spent running the resetter **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#background-jobs-resetter_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_background-jobs_resetter_duration" ] ``` *Managed by the Sourcegraph Services team.*number of times the resetter failed to run
**Descriptions** - warning background-jobs: 1reqps+ number of times the resetter failed to run **Next steps** - Check application logs for the failing domain to check for errors. High failure rates indicate a bug in the code handling the job, or a pod frequently dying. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#background-jobs-resetter_failures). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_background-jobs_resetter_failures" ] ``` *Managed by the Sourcegraph Services team.*number of stalled records marked as 'failed'
**Descriptions** - warning background-jobs: 50+ number of stalled records marked as 'failed' **Next steps** - Check application logs for the failing domain to check for errors. High failure rates indicate a bug in the code handling the job, or a pod frequently dying. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#background-jobs-failed_records). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_background-jobs_failed_records" ] ``` *Managed by the Sourcegraph Services team.*90th percentile of stall duration
**Descriptions** - warning background-jobs: 300s+ 90th percentile of stall duration **Next steps** - Learn more about the related dashboard panel in the [dashboards reference](dashboards#background-jobs-stall_duration_p90). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_background-jobs_stall_duration_p90" ] ``` *Managed by the Sourcegraph Services team.*total number of jobs queued across all domains
**Descriptions** - warning background-jobs: 1e+06+ total number of jobs queued across all domains **Next steps** - Check for stuck workers or investigate the specific domains with high queue depth. Check worker logs for errors and database for high load. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#background-jobs-aggregate_queue_size). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_background-jobs_aggregate_queue_size" ] ``` *Managed by the Sourcegraph Services team.*maximum time a job has been in queue across all domains
**Descriptions** - warning background-jobs: 86400s+ maximum time a job has been in queue across all domains **Next steps** - Investigate which domain has jobs stuck in queue. If the queue is growing, consider scaling up worker instances. - Learn more about the related dashboard panel in the [dashboards reference](dashboards#background-jobs-max_queue_duration). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json "observability.silenceAlerts": [ "warning_background-jobs_max_queue_duration" ] ``` *Managed by the Sourcegraph Services team.*