Skip to main content

Create and use data products with Astro Observe

public preview
This feature is in Public Preview.

You can use Astro Observe to create Data Products, which are abstractions that give you observability into the health and performance of data pipelines. Data products include assets, which can include tasks, datasets, warehouse tables, and local files, where you can configure notifications when asset performance or access meets certain criteria. Data products allow you to configure alerts and receive insights into the performance of the pipelines where you integrate them.

The following procedures describe how to create data products, view your assets, and leverage them to gain insight across pipelines and deployments.

tip

To learn more about data pipeline observability using Astro tools and Data Products features specifically, see the following resources:

Prerequisites​

  • Organization Billing Admin or higher user permissions
  • At least one deployment with a DAG. You can use either an OSS Airflow deployment or an Astro Deployment.

Asset Catalog​

In the Astro UI, when you click Observe and then Asset Catalog in the Organization menu, you can view and filter the different data assets in your Workspace. Assets include information about the attribute details and historical data regarding task runs.

In this view you can see the following information about your Assets, including the following:

  • Name
  • Asset type
  • Namespace
  • DAG ID
  • Owner

You can click on any asset to view more detailed information about the asset, such as historical metrics and metadata.

To learn more about the data quality of your assets, and how this impacts your Airflow performance, see Data quality and Airflow.

Data Products​

In the Data Products page of the Observe menu, you can view any of your existing data products or choose to create a new one. You can also create a Service Level Agreement (SLA), which you can configure to assess the on-time delivery of a data product, data timeliness, or how recently the data product has refreshed data, data freshness.

At a glance, the Data Products landing page shows you a summary of the data products that you have already created, including a graph of the SLA Hit Rate, visualizing how frequently the data product meets the terms of an SLA. It also shows other details, like the owner and how recently the data product was updated.

At-a-glance view of the data products for a particular Organization.

In addition to the summary view of all data products, by clicking on a specific data product, you can view additional information and choose to create a new SLA alert for your data product.

An example of a detailed view of a data product.

Create a Data Product​

You can create a data product in the Data Products summary page.

  1. Click + Data Product.
  2. Enter a Name for your data product and an optional description.
  3. Choose whether the data product belongs to a User or a Team and select a specific user or team that serve as the point of contact for it.
  4. Select the final downstream Assets in your pipeline to create your data product.
  5. Click Create Data Product.

After you make a data product, the page refreshes so you can view your Data Product Details where you can configure additional SLAs and alerts for the data product.

Create an SLA​

If you have existing data products, you can create an SLA for them. Astro allows you to create Freshness and Timeliness SLAs. Based on the criteria you define in your SLA, Astro uses the rate of success your data product meets an SLA to generate insights into your data pipelines and make recommendations about creating proactive alerts.

  1. In the Data Products page, click the specific data product for which you want to create an SLA.
  2. Click the SLA Evaluations tab and click + Add SLA.
  3. Add a Name for your SLA.
  4. Choose the SLA Type:
    • Timeliness: Create an SLA that is a window of time. During this time, if your asset completes its defined actions, Astro considers it a success. Define the window of time by choosing Days of the week, Verification time, and Lookback period.
    info

    Timeliness SLAs only support Standard Time. If you want Local Time support, you must adjust the SLA's UTC time when the time changes from Standard Time to Daylight Savings Time or from Daylight Savings Time to Standard Time.

    • Freshness: Create an SLA that defines how recently you want the data product to update. Configure this SLA by defining a Freshness Policy by the number of minutes, hours, or days.
  5. Click Create SLA.

After you create an SLA, you can configure alerts and proactive alerts.

Create an alert​

After you create an SLA, Astro keeps a record of the rate at which your data product hits or misses the SLA. You must configure an Alert or a Proactive Alert to receive notifications when your pipeline experiences an SLA miss or when an upstream process might cause an SLA miss or a failure.

  1. In the Data Products page, click the specific data product you want to create an SLA for.
  2. Click the Alerts tab and click + Add Alert.
  3. Choose the Type of alert and Severity. The following alert types are available:
    • Data Product SLA Violation: Send an alert when a data product asset has violated its SLA definition.
    • Data Product Proactive SLA: Astro monitors the upstream dependencies of the data product assets, and proactively sends an elert if delays in the upstream dependencies might eventually cause SLA misses.
    • Data Product Proactive Failure: Send an alert when a dependent asset upstream of your data product has failed.
  4. Define the conditions that the alert applies to. These conditions vary depending on the type of alert you want to set up.
  5. Select or add a Notification Channel where you want to send your alert. For more information about configuring notification channels, see Alert Notification Channels.
  6. (Optional) Customize the alert name.
  7. Click Create alerts.

Insights​

Astro Observe also provides Insights into the performance of the assets you include in your data products. The platform bases these recommendations on how frequently your assets meet the definitions of your SLAs. Some recommendations include:

  • If an asset has a high variability in meeting or missing the SLA due to constant upstream delays, Astro might display a recommendation in your Data Product Details page to create a Proactive SLA Alert on the asset.
  • If an asset frequently fails to meet an SLA due to consistent upstream failures, Astro might recommend in your Data Product Details page to create a Proactive Failure Alert on the upstream asset.

Was this page helpful?