Status: experimental, flag-gated (nexus.incremental.enabled, default false). With the flag off, every model behaves exactly as before.

By default every dbt-nexus run rebuilds the whole pipeline: every identifier, every edge, and a full connected-components traversal over the entire identity graph. Cost scales with total history. Incremental mode makes the identity-resolution subgraph — and the final source models that feed it — process only what arrived since the last run.

The full design rationale and spec live in the package repo: docs/incremental-identity-resolution.md. This page is the operator's view.

The idea: prior state is a contracted graph

Identifier edges only ever arrive (nothing retracts them), so resolved components can never split — they can only accrete new identifiers or merge with each other. That monotonicity means the previous run's identifier → entity_id mapping is a lossless summary of all prior connectivity: each existing entity enters the next run's graph as a single contracted "super-node", and connected components run over just (touched super-nodes + new identifiers + new edges). Cost scales with batch size, not history size. Old edges are never re-read.

Each resulting component classifies three ways:

prior entities in component	outcome
0	born — a new entity id is minted
1	accreted — the entity keeps its id; new identifiers join it
2+	merged — the entity with the most mapping rows survives; every row of each loser is re-pointed to the survivor

Watermarks are always on ingestion time (_ingested_at), never occurred_at — a backfilled 2019 event arriving today still enters the graph, and produces exactly the partition it would have produced arriving in 2019. Re-processing a batch is a no-op (safe under at-least-once delivery).

What changes semantically

Entity ids become stable instead of reproducible. The default (full refresh) scheme derives an entity id from the component's lexicographically-first identifier — reproducible from scratch, but the id churns whenever a new identifier happens to sort first. Incremental mode flips the contract: an entity keeps its id for life, and the id changes only across merges, which are enumerated in the resolution log. The practical rule stays what it always was: look entities up by identifier; treat entity ids as internal join keys. A --full-refresh renumbers entities and starts a new epoch — treat it as a rare, announced event.

Merges become data. nexus_resolution_log (built only when the flag is on) is an append-only record of every resolution decision: born, accreted, repointed (a merge — carries previous_entity_id), and full_resolution epochs. It is the downstream invalidation stream ("which entity ids changed this run") and doubles as an outbound merge/alias protocol for external tools. It is also the only table in the package that cannot be rebuilt from source data — it records what the pipeline concluded and when. It's protected with full_refresh: false; treat it as system-of-record.

Splits stay out of the standard flow. Removing edges (retroactive noise filtering, deletions) can split components, and splits are not incrementally computable. Incremental mode therefore refuses to compile with edge_quality autofiltering enabled, and split repair is a full-refresh (or future per-component surgery) concern.

Enabling it in a client project

# dbt_project.yml
vars:
  nexus:
    incremental:
      enabled: true

What turns incremental under the flag:

• the ER chain: nexus_entity_identifiers, nexus_entity_identifiers_edges, nexus_resolved_person_identifiers, nexus_resolved_group_identifiers, plus the new nexus_resolution_log
• the package's final source models for gmail and google_calendar (<source>_events, _entity_identifiers, _entity_traits, _relationship_declarations)

The hard requirement: _ingested_at on every ER source

The core identifiers union keeps one watermark shared across every entities: true source, so ingestion-timestamp discipline is all-or-nothing: a source emitting rows behind the shared watermark silently loses them, and a source stamping ahead of real time (now(), clock skew) drags the watermark past every other source's upcoming rows. Enabling the flag therefore hard-requires every enabled ER source's <source>_entity_identifiers model to expose a stable, non-null _ingested_at (real ingestion/load time — never occurred_at, never now()):

• the build fails at compile time naming offenders if the column is missing from a built source relation;
• the packaged test_incremental_sources_ingested_at_not_null test catches null values at the sources (a null would fail the watermark predicate and vanish silently — the core table can never witness its own missing rows).

Note this is about timestamps, not materialization — a source model may stay a full-rebuild table forever; only its rows' stamps must be truthful. For static/seed sources with no loader timestamp, stamp a data-vintage literal held in a var (SRT's upwork_data_vintage is the reference) and bump it when the data is re-exported: every row re-offers, which is an idempotent no-op for already-absorbed rows and exactly how backfills enter correctly. The package's segment source is known non-compliant today (it stamps current_timestamp()); enabling segment with incremental mode fails loudly until its real loaded_at is threaded through.

Client-local source finals can adopt the same incremental recipe — see the slide-rule-tech project for the reference pattern: nexus.nexus_incremental_materialization() + nexus.nexus_incremental_source_filter() + a unique-key merge + an incremental-only batch dedup. Verify a candidate's _ingested_at empirically against a built table, not by grepping the SQL — columns get used upstream and dropped from final selects, and a moving now() stamp looks fine in any single snapshot.

First build over an existing deployment

Tables built before the flag lack the columns the incremental predicates read. Every flagged model carries an upgrade guard that fails loudly with the exact remediation:

dbt run --full-refresh --select +<model>+

The clean path when enabling on an existing warehouse is a one-time full refresh of the ER subgraph (and the flagged source finals), which becomes the epoch the incremental runs continue from. First builds on an empty schema need nothing special — the first run is the epoch.

How it's tested

The package repo carries integration_tests/ — a consumer-shaped dbt project run against scratch DuckDB databases with a simulated ingestion clock: whole scenarios live in one seed, a runner advances it_now across dbt invocations (one batch per run), and after every step the incremental partition is compared against a from-scratch shadow resolution (co-membership equality; ids deliberately ignored). Sequence properties — id stability, log append-only, idempotent re-runs, empty-batch no-ops, watermark advancement — are asserted by the runner between runs.

cd dbt-nexus/integration_tests
python3 run.py

Eight scenarios cover birth, accretion, merges, within-batch merge chains, cross-batch merge chains, re-observation watermarks, out-of-order ingestion, and group-type merges with cross-entity-type edges.

Operational notes

• The delta contract for downstream consumers: the set of entity ids affected by a run = new entities + repointed log entries. Nothing else changed. (Downstream models — participants, traits, states — are still full rebuilds today; this contract is what their future incrementalization keys off.)
• reobserved rows in the resolved tables are watermark bookkeeping (a known identifier seen again), not resolution changes; they never enter the log.
• Watermark boundary ties: the delta predicate is strictly > on _ingested_at. Run after loads complete, or accept that boundary-sharing stragglers wait for the next run's rows to pull the watermark past them.
• max_recursion changes meaning: it bounds within-batch chain length over the contracted graph (small), not historical component diameter.