Prerequisite — Ingesting source data with os-nexus

Connect source accounts in Nexus and use the existing BigQuery warehouse as their destination, with MCP tools for pipeline setup and sync verification.

Module 2 used seed data. Module 3 uses source data landed by Nexus before dbt transforms it into entities, events, and relationships.

Learning Objectives

  • Reuse the warehouse created in lesson 1.2
  • Connect source accounts through Nexus integrations
  • Configure and verify ingestion with the Nexus MCP
  • Map actual destination tables into the course's dbt examples

Where ingestion fits

Source APIs → Nexus ingestion → BigQuery raw tables → dbt + dbt-nexus

Nexus manages source authentication through Nango and runs ingestion. dbt reads the landed tables and builds your modeled outputs. Connecting Google for warehouse setup does not also authorize Gmail or Calendar; source accounts need their own connections.

Step 1: Reopen your connected warehouse

In your Nexus workspace, open Integrations → BigQuery and reopen the completed setup from 1.2 Setting up BigQuery. Record its warehouse ID, actual project ID, dataset, and location.

For this course, the destination dataset is os_nexus. The setup grants Nexus write access to that selected dataset. It does not grant permission to create arbitrary additional datasets.

A new warehouse does not replace your workspace's default warehouse or move existing pipelines. Select it explicitly when wiring each source.

Step 2: Connect a source account

Open Integrations, choose the source integration available in your workspace, and complete its connection flow. Start with one account so you can verify a complete sync before adding more.

Authorize the intended Google account for Gmail or Calendar, or select the pages you want Nexus to access when connecting Notion. The available connector determines the source-specific settings and models you can sync.

Step 3: Wire the source to this warehouse with MCP

Use the Nexus MCP to set the destination explicitly. Some integration UI flows use a source-named namespace such as notion; for the warehouse created in this course, the connection namespace must match the approved dataset, os_nexus.

Ask your assistant:

Wire my connected source account to the warehouse from my completed BigQuery setup. Use its warehouse ID and os_nexus as the connection namespace. Check whether this account already has a pipeline before creating one, and show me the destination and models before syncing.

The assistant uses list_ingestion_pipelines to inspect existing wiring and create_ingestion_pipeline for a new pipeline:

Argument Value to use
organizationId Your workspace's ID
providerConfigKey The connected source integration's provider key
nangoProviderConnectionId The source's Nango provider connection ID
dataWarehouseId The warehouse ID returned by your completed BigQuery setup
connectionNamespace os_nexus, or the actual dataset approved in your setup
sourceConfig Connector-specific settings, when needed, encoded as a JSON string

Use the source connection, not the Google Cloud setup connection. The create tool returns an existing pipeline if that account is already wired; it does not move it to a new destination. If the existing pipeline points elsewhere, review that configuration with your Nexus administrator before syncing.

Step 4: Run and verify the initial sync

After reviewing the source and destination, ask the assistant to run an initial sync. It calls trigger_sync with your workspace ID and the pipeline's connectionId, then follows the returned run using get_sync_status. The pipeline connection ID is different from the Nango provider connection ID.

Check that the run completed and wrote records. If it fails, inspect the reported error before retrying the same pipeline. A dataset permission error can mean the namespace points outside the approved destination; check that mapping first.

Record the actual destination table names and schemas. Repeat for the other source accounts once the first source works. A successful warehouse setup verifies access; this source sync verifies that real source data has arrived.

Step 5: Map raw data into dbt

Nexus source ingestion commonly stores JSON in _raw_record, with metadata such as _ingested_at, _connection_id, _stream_id, and _sync_id. Other write paths use _raw_data; inspect the actual table schema before writing extraction SQL.

The course's later examples use simplified tables such as messages.sender_email and tasks.assignee_id. Build staging models to extract those typed fields from the raw payload first. Connector output does not automatically include all of these teaching tables.

The names gmail, google_calendar, and notion in dbt source() calls are logical source names. Map database, schema, and each table's identifier to the actual relations you prepared. They do not require three additional BigQuery datasets. For staging models inside the same dbt project, use ref() instead of source().

Your dbt identity needs read access to the ingestion dataset and write access to its development dataset. Use the dbt connection from Module 1; there is no Nexus service-account JSON key to export into your project.

Hands-On Exercise

  1. Connect one source account in Nexus.
  2. Wire it to your completed warehouse with the approved dataset namespace.
  3. Trigger a sync and confirm that records landed in the intended tables.
  4. Record the table mappings and stage the fields the course models need.
  5. Repeat for the other sources when ready.

If you are following the course with sample data, keep using dbt seeds and reference them with ref(). You do not need to manually provision another BigQuery project or service account to practice the models.

Next Lesson

Continue to 3.2 Introduction to dbt-nexus. The source model walkthrough is in 3.4 Setting up sources.