If you need granularity and dependencies between your dbt models, like the team at Updater does, you may need to deconstruct the entire dbt DAG in Airflow. It’s important to note that whichever approach you choose, this is just a first step your actual production needs may have more requirements.
If you’re just looking to get started or just don’t want to deal with containers, using the BashOperator to call the dbt CLI can be a great way to begin scheduling your dbt workloads with Airflow. Tools like the astro-cli can make this easier, but at the end of the day, there’s no getting around the need for Kubernetes resources for the Gitlab approach. However, setting up development environments, CI/CD, and managing the arrays of containers can mean a lot of overhead for some teams. Furthermore, you’ll create a library of images containing your dbt models that can be run on any containerized environment. If you have DevOps resources available to you, and your team is comfortable with concepts like Kubernetes pods and containers, you can use the KubernetesPodOperator to run each job in a Docker image so that you never have to think about Python dependencies. Use the KubernetesPodOperator for each dbt job, as data teams have at places like Gitlab and Snowflake.īoth approaches are equally valid the right one will depend on the team and use case at hand.Use the dbt CLI+ BashOperator with Airflow (If you take this route, you can use an external secrets manager to manage credentials externally), or.Most of the time, users choose to either:
Once the data has been ingested, dbt Core can be used to model it for consumption. You can build your own custom connector (like the Astronomer Data Team did for ingesting Zendesk data), or you can choose to go with a solution like Fivetran or Airbyte, and simply orchestrate that with Airflow. Almost every data team has a workflow where data has to be ingested into the warehouse before being modeled into views for consumption, and Airflow and dbt Core can be used to see a single view of the end-to-end flow.Īirflow’s flexibility allows you to bring your own approach to the ingestion layer. Whether you are early on your dbt journey or a longtime user, there’s a good chance you’re already using dbt Core. Let’s dive into what different implementations are available. The best choice for you will depend on things like the resources available to your team, the complexity of your use case, and how long your implementation might need to be supported.īut the first big decision will depend on whether you’re using dbt Core or Cloud.
There are a lot of different ways Airflow and dbt can be used together - including options for dbt Core and, for those using dbt Cloud, a new dbt Cloud Provider, co-developed by Astronomer and the team at dbt Labs, that’s ready for use by all OSS Airflow users. Both are vital parts of any modern data stack. And because both orchestration and transformation have clearly defined boundaries, so too do Airflow and dbt: in a sense, each one picks up where the other leaves off. The best way to describe the relationship between Airflow and dbt might be spiritual alignment.īoth tools exist to facilitate collaboration across data teams, addressing problems - data orchestration, in Airflow’s case, and data transformation, in dbt’s - that go hand in hand. Streamline your data pipeline workflow and unleash your productivity, without the hassle of managing Airflow.
0 kommentar(er)
