Schema Drift

Queryvine monitors every pipeline run for structural changes to source tables. When drift is detected, it classifies the change, applies your configured rules, and takes action before broken data reaches your warehouse.

Change detection

At the start of each pipeline run, Queryvine computes a schema fingerprint of the source (column names, data types, nullability, ordering). This fingerprint is compared to the version captured on the previous run. If they differ, a drift event is generated.

Queryvine detects the following change types:

• column_rename — a column's name changed (detected via Levenshtein similarity + type matching)
• type_change — a column's data type changed (e.g., INT to BIGINT, or VARCHAR to TEXT)
• type_widen — a numeric type widened in a backward-compatible way (e.g., INT32 to INT64)
• column_drop — a column was removed from the source table
• column_add — a new column appeared in the source table
• nullability_change — a column changed from NOT NULL to nullable or vice versa
• ordering_change — column ordinal positions changed (typically benign, but configurable)

Severity levels

Each drift event has a severity level that determines default alerting behavior:

YAML drift rules

Override the defaults by editing your pipeline's drift_rules.yaml. Rules are evaluated top-to-bottom; the first match wins.

pipeline: orders-pipeline
drift_rules:
  # Safe numeric widening — always auto-migrate
  - type: type_widen
    from_types: [int32, float32]
    severity: info
    action: auto_migrate

  # amount_* columns: rename is expected, remap and continue
  - type: column_rename
    column_pattern: "^amount_"
    severity: warning
    action: alert_continue
    remap: true

  # All other column drops are critical
  - type: column_drop
    severity: critical
    action: pause_and_alert
    alert_channels: [slack, pagerduty]

Actions

• pause_and_alert — halt pipeline execution and send alerts. Pipeline resumes only after manual acknowledgment or via API.
• alert_continue — send alerts but let the run finish. Downstream data may reflect the change.
• auto_migrate — silently update the destination schema to match. A migration event is logged but no alert is sent unless configured.
• ignore — suppress the event entirely. Useful for known noisy changes like ordering_change.