Observability Engineer

Position Overview The Observability Engineer builds and improves the capabilities used to understand the health, performance, and reliability of Goosehead applications and services. This hands-on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry.

The engineer works with Software Engineering, DevOps, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool.

Core outcome: Make production health visible, alerts actionable, and failures easier to diagnose.

Key Responsibilitie s · Design, implement, and improve observability for production applications, APIs, integrations, and critical services using OpenTelemetry for instrumenting metrics, logs, traces, and custom telemetry. · Partner with Engineering to instrument applications using OpenTelemetry for structured logging, distributed tracing, and custom metrics. · Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that make availability, latency, errors, dependencies, and customer impact clear. · Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs). · Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts; ensure alerts support effective NOC triage and escalation. · Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis. · Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements. · Partner with DevOps and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry. · Build scripts and automation using PowerShell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows. · Help integrate observability and alert/event data with ITSM and incident workflows, including future ServiceNow capabilities. · Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams. · Maintain service-based error budgets which our development teams use to dictate feature work.

Qualifications · Minimum 3 years of hands-on experience in Observability Engineering, SRE, DevOps, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4-5 years preferred. · Hands-on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting. · Experience instrumenting applications or services using OpenTelemetry for custom metrics, structured logs, and distributed tracing. · Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues. · Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues. · Hands-on scripting or automation experience using PowerShell, Python, Bash, APIs, or comparable technologies. · Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service-health concepts. · Working knowledge of cloud environments and the relationship between application and infrastructure telemetry. · Strong troubleshooting, analytical, communication, and cross-team collaboration skills. · Associate or bachelor's degree in Information Technology, Computer Science, Engineering, or related field preferred; equivalent technical experience considered.

Preferred Qualifications · Hands-on experience with New Relic, including OpenTelemetry services, logs, dashboards, alerting, distributed tracing, or service maps. · Experience with comparable observability platforms such as Datadog, Dynatrace, Splunk Observability, AppDynamics, Azure Monitor/Application Insights, Grafana, or Prometheus. · Practical OpenTelemetry implementation experience and experience defining reusable observability or instrumentation standards. · Experience establishing SLIs/SLOs or service-health measures for production applications. · Experience integrating observability platforms with incident, event, notification, or ITSM workflows; ServiceNow experience is a plus. · Experience supporting Microsoft Azure or other public cloud environments and integrating observability with CI/CD or deployment processes. · Familiarity with ITIL Incident Management and Problem Management practices.