Implementing proactive alerting

Proactive alerting enables teams to stay ahead of data quality issues. You can set up an open-source approach using tools that integrate with Airflow or use Astro, which offers built-in support for data quality monitoring.

Other ways to learn

See also:

• Blog post: Data Products: It's not what you call them that matters. It’s what you do with them.

Why to use proactive alerting

The use of proactive alerting adds value by helping ensure:

• Clear expectations for data quality: You can use proactive alerts to establish and evaluate data products against expectations for quality metrics. For example, you can track the accuracy, completeness, consistency, and timeliness of the data in your pipelines.

• Accountability: By formalizing standards for data quality, proactive alerting establishes accountability. If the data do not meet the agreed-upon standards, the responsible party (whether it’s an internal team or a third-party provider) can be held accountable for remediation.

• Performance measurement: Proactive alerts provide a framework for setting and evaluating assets against benchmarks and metrics that allow organizations to track performance over time, identify issues, and measure improvements.

• Proactive management: With proactive alerts in place, organizations can proactively manage data quality. Regular monitoring helps catch and address issues before they impact business processes or decision-making.

• Risk mitigation: High-quality data are essential for making informed decisions. Proactive alerting helps mitigate risks associated with poor data quality by ensuring that data meet required standards, reducing the likelihood of errors or misleading information.

• Continuous improvement: Proactive alerting often includes provisions for regular reviews and updates. This encourages continuous improvement in data quality practices and helps teams adapt to changing needs or new challenges.

• Customer satisfaction: For organizations that provide data as a service, proactive alerting helps build trust with clients by guaranteeing a certain level of data quality. This can lead to higher customer satisfaction and retention over time.

• Legal and compliance requirements: In regulated industries, maintaining high data quality is often a legal requirement. Proactive alerting helps ensure compliance with these regulations by formalizing data quality standards and monitoring processes.

• Resource allocation: Proactive alerting helps in resource planning and allocation. By defining expectations for quality, organizations can allocate resources more effectively to meet these standards and more easily handle any issues that arise.

In essence, proactive alerting helps create a structured approach to managing and ensuring data quality, which is critical for effective decision-making and operational efficiency.

When to use proactive alerting

Proactive alerting is most useful when you need to intervene before a table is not delivered on time or an update fails. When performance and governance are not concerns, but you do need to know when a particular task or DAG fails, alerts may suffice. For this use case, Astronomer recommends Astro alerts, which do not require modification of DAG code. Airflow notifications provide some of the same functionality.

By contrast, when you need to implement robust performance and data quality monitoring, a standardized approach at the organizational level, incorporating analytics, is recommended. In this case, proactive alerts such as SLAs are needed.

info

Astro Alerts can also alert you in cases of issues with task duration, but they cannot be applied to data products. Use Astro Observe (public preview) when you want to put an alert in place for an entire data product.

Implementing service-level agreements (SLAs)

SLAs are an increasingly crucial tool for ensuring that the data used in decision-making processes meet predefined standards for timeliness and freshness, among other key data quality and performance metrics.

Airflow SLAs

Airflow has a built-in feature for evaluating SLAs for task run duration. To set an SLA for a task, you pass a datetime.timedelta object to the operator's sla parameter. For more guidance, see Airflow service-level agreements.

caution

The functionality of Airflow SLAs has known limitations. Use with caution.

SLA evaluations on Astro

Using Astro Observe (public preview), you can create SLAs to evaluate the on-time delivery of a data product or its freshness. You can also view all SLAs in effect for the data product along with the current status and overall hit rate (success rate) of each at a glance.

Implementing data quality checks

Checking the quality of your data is essential to getting actionable insights from your data pipelines. A number of third-party tools support orchestrating data quality checks directly from Airflow DAGs. For more information, see Data quality and Airflow.

To learn more about observability on Astro, including platform support for proactive alerting, see Enhance data observability with Astro.