The modern data stack has accumulated a set of overlapping tools, and the question of what belongs where has become genuinely confusing. Teams run Airflow next to dbt next to Fivetran next to Prefect, and the mental model for which tool owns which problem is rarely explicit. When it isn't, those tools end up stepping on each other — transformation logic in the orchestrator, scheduling logic in dbt, connection management scattered across three config systems.
This article is a framework for drawing the line clearly. The short version: orchestrators schedule and coordinate execution. Transformation tools compute new data. Neither should do the other's job.
What an orchestrator actually does
An orchestrator's job is to answer the question: in what order, at what times, and under what conditions should these tasks run?
That means DAG execution, task dependencies, retry policy, scheduling, and cross-pipeline coordination. Airflow excels at this. Prefect handles it with a more developer-friendly API. Dagster adds asset-awareness to the model — rather than scheduling tasks, you define assets (materialized data objects) and let the scheduler figure out what needs to run to produce them.
None of these tools transform data themselves. Airflow's operators call external systems — they run a Python function, trigger a Spark job, invoke a dbt command, or run a SQL query against a database. The orchestrator is the coordinator. The computation happens elsewhere.
An orchestrator also does not (and should not) understand the content of the data flowing through it. It knows whether a task succeeded or failed. It knows how long it took. It does not know whether the revenue column in the output table contains sensible values. And it doesn't know whether the schema of the raw data matches what downstream models expect.
This last point is worth sitting with. An orchestrator that runs a Fivetran sync, waits for it to report "success," and then immediately starts a dbt model run — that sequence is logically correct. But the orchestrator has no mechanism to know whether Fivetran's "success" means "all expected columns landed with the expected types" or "the pipeline ran without throwing an exception." Those are different guarantees.
What dbt actually does
dbt transforms data that is already in your warehouse. It does not move data into the warehouse. It does not monitor upstream sources. It does not care whether the raw data it reads was ingested by Fivetran, Airbyte, a custom connector, or a Python script someone ran manually three weeks ago.
dbt's model: you write SQL SELECT statements. dbt compiles them into DDL/DML and runs them against your warehouse. The output is a materialized table or view. You can chain models (one model reads from another), define tests on the outputs, and document lineage.
dbt is, fundamentally, a SQL-based transformation framework. Its execution layer understands incremental models (append new rows only), full refresh (rebuild the table entirely), and the dependency graph between models. What it does not understand is anything that happens before raw tables exist in the warehouse.
The sources.yml file in dbt declares the raw tables that your models depend on. It does not control how those tables get populated. The freshness checks in dbt (source freshness) will tell you whether a raw table received new data recently. They will not tell you whether that new data has the schema your model expects.
This matters practically: a dbt model that references raw.orders.order_amount will execute successfully even if order_amount is now all NULLs because the source renamed it to total_value. The model compiles. It runs. It materializes. The output is wrong but dbt has no way to know that — dbt's job is to execute SQL, not to validate that the SQL produces meaningful results.
Where the confusion comes from
The confusion arises from three places:
1. Both tools deal with "pipelines." dbt calls its model chains pipelines. Airflow calls its DAGs pipelines. Prefect calls its workflows pipelines. The word is overloaded, so teams talk past each other about what they mean.
The cleaner vocabulary: dbt handles transformation pipelines (raw → modeled → aggregated). Orchestrators handle execution pipelines (when and in what order do the steps run). EL tools (Fivetran, Airbyte) handle ingestion pipelines (data moves from source to raw table).
2. Airflow operators blur the line. The Airflow dbt operator, for example, triggers a dbt run from inside an Airflow DAG. This is fine — Airflow is coordinating when dbt runs. But some teams start putting transformation SQL directly into Airflow tasks (via PythonOperator or the SQLExecuteQueryOperator). Now transformation logic lives in the orchestrator. This is harder to test, harder to version, and loses dbt's lineage and documentation features.
3. Prefect's Python-first model makes it easy to blend concerns. A Prefect flow can run a SQL transformation, call an API, and write results to S3 — all in one flow. This is powerful but creates the risk of putting transformation logic where it's invisible to data catalogs, data quality frameworks, and downstream teams who want to understand where their data comes from.
The resulting anti-pattern is a data stack where transformation logic is split across dbt models, Airflow PythonOperator tasks, and Prefect flows — and the only way to understand the full transformation chain is to read all three systems simultaneously. This is not a theoretical risk; it's the state of many teams that have grown their stack incrementally without a deliberate layer model.
Where Queryvine fits in this picture
Queryvine is not a general-purpose orchestrator and is not a transformation tool. It operates specifically in the ingestion layer — the space between your upstream sources (SaaS APIs, databases, event streams) and the point where raw data enters your warehouse.
Its job is schema-drift-aware orchestration of that ingestion layer. When a source's schema changes, Queryvine intercepts before data moves, evaluates drift rules, and either adapts automatically or pauses and alerts. dbt never sees the bad rows. The warehouse never receives the corrupted data.
A typical stack with Queryvine looks like this: Queryvine connects to sources, fingerprints schemas, applies drift rules, and lands data in raw tables with guaranteed schema consistency. dbt picks up from those raw tables. Airflow or Prefect coordinates the timing of Queryvine runs and dbt runs as a DAG. Each tool does one thing.
When you run qv pipeline init --source salesforce.opportunities --destination raw.sf_opportunities, Queryvine registers the current schema fingerprint and the drift rules for that pipeline. Subsequent runs check the fingerprint before any rows move. The orchestrator's DAG just needs to know whether the Queryvine run succeeded — it doesn't need to understand what "schema fingerprint mismatch" means.
The framework: which tool owns which question
| Question | Owner |
|---|---|
| When does this task run? | Orchestrator (Airflow / Prefect / Dagster) |
| What runs before this task can start? | Orchestrator |
| Did this source's schema change since last run? | Schema-drift layer (Queryvine) |
| How should I respond to a renamed column? | Schema-drift layer |
| How do I compute monthly revenue from raw orders? | Transformation tool (dbt) |
| Do the revenue values make semantic sense? | Data quality tests (dbt / Great Expectations) |
| How do I move data from Salesforce to Snowflake? | EL tool (Fivetran / Airbyte) or Queryvine connector |
| Did the ingestion produce a consistent schema? | Schema-drift layer |
The cost of blurred boundaries
When tools step outside their lane, the costs compound slowly:
Transformation in the orchestrator: Airflow tasks that run SQL transformations directly bypass dbt's lineage graph. When a downstream analyst asks "what does this metric count?", there's no dbt docs generate output to point them to. The answer is in an Airflow task definition that only the data engineer who wrote it can navigate.
Scheduling in dbt: dbt Cloud can schedule its own runs. For small teams with simple pipelines, this is fine. For larger teams where dbt runs depend on upstream ingestion completing, scheduling inside dbt creates coupling. When the ingestion takes longer than usual, the dbt run starts on stale data and produces wrong results with a "succeeded" status. The orchestrator should control when dbt runs, based on whether the upstream data actually landed.
No schema awareness at ingestion: When neither the EL tool nor the orchestrator watches for schema changes, the only detection mechanism is dbt tests running on data that's already corrupted. You catch the problem in the transformation layer, but the corrupted data is already in the raw table. Backfilling it requires coordination with the source system, which may not be possible.
A note on Dagster's asset model
Dagster's software-defined assets (SDAs) offer a different framing that partially addresses the boundary confusion. Rather than scheduling tasks, you declare assets — the data objects your pipeline produces — and let Dagster infer the execution graph from asset dependencies. An asset for modeled.monthly_revenue knows it depends on raw.orders and raw.customers, and Dagster will run the upstream materializations first.
This is a cleaner model for expressing data lineage in the orchestrator. It doesn't, however, solve the schema awareness problem. A Dagster asset that materializes raw.orders from an API source still won't detect whether the API's schema changed. The asset's definition specifies what to do when it runs; it doesn't specify what the schema of the upstream source must look like for the run to be valid.
The schema contract is still a separate concern, regardless of whether your orchestrator uses task-based or asset-based modeling. Dagster + schema-drift detection is still the right layering. Dagster alone doesn't make schema drift a solved problem.
Putting it together
The cleanest architecture keeps the layers explicit: ingestion with schema-drift detection → raw tables → dbt transformations → modeled tables → BI. The orchestrator coordinates timing across all of these, but owns none of the actual computation or schema management.
When something goes wrong — and something always goes wrong — a clean layer separation makes root cause analysis tractable. Schema issue? Check the drift events in the ingestion layer. Logic issue? Check the dbt model. Timing issue? Check the DAG. Missing data? Check freshness at the source and at each stage of the pipeline.
Blurred layers don't just create technical debt. They create debugging debt — the cost of tracing a failure through systems where responsibilities overlap and no single tool has a complete view of what happened. A team that has spent four hours trying to determine whether a wrong dashboard number came from a Fivetran schema miss, an Airflow transformation task, or a dbt model logic error has experienced this directly. Clean layer boundaries make that four-hour investigation a 15-minute one.