09
Advanced  ·  Module 09 of 14

Advanced
Data Vault 2.0

Push beyond standard DV2 patterns into advanced territory: temporal modeling, graph database mapping, MDM via Same-As Links, and the nuanced art of separating hard from soft business rules.

5 ConceptsCore topics
~60 minReading time
AdvancedLevel
Modules 01–08Prerequisites
Module Progress 0%

Concept 01

Anchor Modeling Comparison

🧠 ELI5 — Explain Like I’m Five

Anchor Modeling is another enterprise data modeling approach that, like Data Vault, separates entity identity from descriptive attributes and tracks full history. The key difference: in Anchor Modeling, each attribute gets its OWN table — instead of grouping related attributes into satellites. This gives maximum flexibility but creates an enormous number of tables. Imagine a customer with 20 attributes: Data Vault might use 3–4 satellites; Anchor Modeling would use 20 separate tables.

Structure at a Glance

Both methodologies share the same core philosophy: separate identity from description, track every historical change, and never delete data. The divergence is in the granularity of storage.

  • Data Vault 2.0 groups related attributes into satellites split by rate of change and source system. One satellite = one rate of change + one source.
  • Anchor Modeling takes this to the extreme: each individual attribute (first name, last name, email address) lives in its own table — called an “Attribute” table. Relationships between Anchors are called “Ties” (analogous to Links).
Aspect Data Vault 2.0 Anchor Modeling
Identity Store Hub (business key only) Anchor (identity only)
Attributes Satellites (groups of related columns) One table per attribute
Relationships Links Ties
History Satellite row per change Attribute table row per change
Number of Tables ✓ Moderate Very high
Query Complexity Medium High (many joins required)
Schema Flexibility Add new satellites Add new attribute tables
Tool Support ✓ Strong (dbt, WhereScape) Limited
DATA VAULT 2.0 ANCHOR MODELING HUB CUSTOMER SAT_DEMOGRAPHICS name · email · dob SAT_FINANCIALS balance · tier · limit SAT_ADDRESS street · city · country SAT_STATUS status · segment 4 satellites — grouped attributes ANCHOR CUSTOMER FIRST_NAME LAST_NAME EMAIL PHONE DOB COUNTRY STATUS BALANCE 8 attribute tables — one per column VS

When Anchor Modeling Might Be Preferred

Anchor Modeling has genuine advantages in specific edge cases:

  • When the attribute schema is extremely volatile and individual attributes change at very different, unpredictable rates
  • When you need maximum attribute-level granularity with separate versioning per column, not per group
  • When the schema evolves so rapidly that even defining satellite groupings is premature

In practice, Data Vault 2.0’s satellite splitting by rate of change achieves most of Anchor Modeling’s benefits with significantly lower table count, simpler queries, and a vastly better tool ecosystem.

💡

Architect insight: Most real-world evaluations that considered Anchor Modeling ultimately chose Data Vault because of tooling maturity (dbt, WhereScape, Datavault4dbt) and query simplicity. DV2 satellite splitting — one satellite per rate-of-change group per source — captures 95% of Anchor Modeling’s flexibility with a fraction of the table count and join complexity.

Concept 02

Temporal Data Vault — Bi-Temporal Modeling

🧠 ELI5 — Explain Like I’m Five

Most data has two timestamps: when something happened in the real world (“valid time” or “business date”) and when we recorded it in our system (“transaction time” or “load date”). Normal Data Vault only tracks transaction time automatically. Bi-temporal modeling tracks BOTH.

This lets you answer two distinct questions: “What did we BELIEVE to be true on January 1?” and “What do we NOW KNOW was actually true on January 1?” — two completely different questions with potentially very different answers.

Two Independent Time Axes

Transaction Time (TT) — when data was stored in the system. In Data Vault this is the LOAD_DATE column. It is system-controlled and immutable. This is what standard Data Vault already provides automatically.

Valid Time (VT) — when the fact was true in the real world. Added to the satellite payload as VALID_FROM and VALID_TO columns. Provided by the source system or business process; it can predate the Load Date when data arrives late.

TRANSACTION TIME (LOAD_DATE) → VALID TIME (BUSINESS DATE) → Jan 1 Feb 1 Mar 1 Apr 1 May 1 Jan 1 Feb 1 Mar 1 Apr 1 Real-time snapshot TT = VT = Jan 1 Late-arriving data Loaded Feb, true in Jan Loaded Mar 1 True since Jan 15 Real-time data Backdated correction Individual event
SQL — Bi-Temporal Satellite: Account Balance
-- Bi-temporal satellite: two independent time dimensions
CREATE TABLE SAT_ACCOUNT_BALANCE_BITEMPORAL (
    HUB_ACCOUNT_HK   CHAR(32)      NOT NULL,
    -- Transaction Time: when loaded into Data Vault (immutable, system-set)
    LOAD_DATE        TIMESTAMP     NOT NULL,
    LOAD_END_DATE    TIMESTAMP     NULL,
    -- Valid Time: when the fact was true in the real world (business-provided)
    VALID_FROM       DATE          NOT NULL,  -- business effective date
    VALID_TO         DATE          NULL,      -- NULL = currently valid
    HASH_DIFF        CHAR(32)      NOT NULL,
    RECORD_SOURCE    VARCHAR(100)  NOT NULL,
    -- Payload
    BALANCE          DECIMAL(18,4),
    BALANCE_TYPE     VARCHAR(20),
    CONSTRAINT pk_sat_ab_bt PRIMARY KEY (HUB_ACCOUNT_HK, LOAD_DATE)
);
SQL — Bi-Temporal Query: “What did we believe on Jan 1?”
-- Q: Balance as of Jan 1, 2023 AS WE UNDERSTOOD IT on that date
--    Both Valid Time AND Transaction Time anchored to '2023-01-01'
SELECT
    sat.BALANCE,
    sat.BALANCE_TYPE,
    sat.VALID_FROM,
    sat.LOAD_DATE   AS WHEN_RECORDED
FROM SAT_ACCOUNT_BALANCE_BITEMPORAL sat
WHERE sat.HUB_ACCOUNT_HK = 'abc123...'
  -- Valid Time filter: the fact was true on Jan 1
  AND sat.VALID_FROM     <= '2023-01-01'
  AND (sat.VALID_TO      IS NULL OR sat.VALID_TO      > '2023-01-01')
  -- Transaction Time filter: we had loaded it by Jan 1
  AND sat.LOAD_DATE      <= '2023-01-01'
  AND (sat.LOAD_END_DATE IS NULL OR sat.LOAD_END_DATE > '2023-01-01');

-- Swap LOAD_DATE filter to CURRENT_TIMESTAMP() to ask:
-- "What do we NOW KNOW was true on Jan 1?" (late corrections included)

Real-World Use Cases

  • Late-arriving banking data — A payment made Dec 31 arrives Jan 3. VALID_FROM = Dec 31, LOAD_DATE = Jan 3. Both dates are preserved independently.
  • Regulatory reporting — Prove what you reported to a regulator on a specific date versus what actually happened. These frequently differ due to subsequent corrections.
  • Insurance backdating — A policy approved Feb 1 may be backdated to Jan 1. Bi-temporal handles this without compromising data integrity.
  • Compliance audits — Reconstruct the exact system view any process had at any historical point, independent of subsequent corrections or late arrivals.

When to add bi-temporal: Not every project needs it. Add VALID_FROM/VALID_TO to satellite payloads only when business or regulatory requirements explicitly demand “as-of” analysis across both time dimensions. The complexity cost is real — only introduce it when the business case is unambiguous.

Concept 03

Soft Business Rules — The Right Way

🧠 ELI5 — Explain Like I’m Five

“Soft” business rules are interpretive transformations — code lookups, derivations, classifications — that add business meaning but could change over time. The cardinal sin in Data Vault is putting soft rules in the Raw Vault. If the rule changes, you’ve permanently altered your audit trail. Soft rules belong in the Business Vault or Information Mart, where they can evolve without touching raw history.

Think of the Raw Vault as a court reporter who writes down exactly what was said. The Business Vault is the analyst who interprets what it means. Never let the court reporter make interpretive choices.

Rule Taxonomy

Every transformation you write can be classified. That classification determines which layer it belongs in:

Rule Type Layer Examples Practical Test
Hard Rules Staging only TRIM() / UPPER() on BK; filter NULL BKs; hash key computation; split composite keys; data type casting Business analyst recognizes output as the same data the source sent
Soft Rules Business Vault or Info Mart Code table lookups (“US” → “United States”); risk tier classification; derived aggregates; segment assignment; currency conversion Output requires interpretation or business judgment to derive
Gray Area Depends Filtering NULL BK = hard (structural). Filtering “invalid” business records = soft (store with error satellite) Would a business rule change alter the outcome? If yes → soft
SQL — BAD: Soft Rule in Staging (DO NOT DO THIS)
-- ✗ BAD: soft business rule baked into staging / load layer
--   If STATUS_CODE meanings change, raw history is permanently corrupted
SELECT
    src.CUSTOMER_ID,
    CASE
        WHEN src.STATUS_CODE = 'A' THEN 'Active'
        WHEN src.STATUS_CODE = 'I' THEN 'Inactive'
        ELSE 'Unknown'
    END AS STATUS_DESCRIPTION   -- soft rule in the WRONG layer!
FROM raw_source.customers src;
SQL — GOOD: Soft Rule in Business Vault Computed Satellite
-- ✓ GOOD: soft rule lives in a Business Vault computed satellite (BSAT)
--   Raw Vault keeps STATUS_CODE='A' unchanged forever
--   BSAT re-derives the label whenever the rule or reference table changes
INSERT INTO BSAT_CUSTOMER_STATUS_RESOLVED
SELECT
    sat.HUB_CUSTOMER_HK,
    CURRENT_TIMESTAMP()                                        AS LOAD_DATE,
    MD5(sat.STATUS_CODE || '||' || ref.STATUS_DESCRIPTION)    AS HASH_DIFF,
    'BV.STATUS_RESOLUTION_RULE_V3'                             AS RECORD_SOURCE,
    sat.STATUS_CODE,
    ref.STATUS_DESCRIPTION,    -- looked up from BV reference table
    ref.IS_ACTIVE_FLAG
FROM SAT_CUSTOMER_STATUS sat
JOIN BREF_STATUS_CODES ref
    ON ref.STATUS_CODE = sat.STATUS_CODE
WHERE sat.LOAD_END_DATE IS NULL;   -- current satellite records only

-- When mapping changes (e.g., 'A' now means 'Archived'):
--   1. Update BREF_STATUS_CODES
--   2. Re-run the BSAT load — new rows reflect new rule
--   3. Raw Vault untouched — complete audit trail intact

✓ Hard Rules (Staging Layer)

  • TRIM() and UPPER() on business key columns
  • Filter rows where business key IS NULL
  • Compute hash keys (MD5/SHA-256 of BK)
  • Compute HashDiff (MD5 of all payload cols)
  • Split composite keys into components
  • Cast data types to vault standard types
  • Add LOAD_DATE and RECORD_SOURCE metadata

✗ Soft Rules (NEVER in Raw Vault)

  • Code / lookup table joins (“A” → “Active”)
  • Risk tier or customer segment derivation
  • Aggregate calculations (SUM, COUNT, AVG)
  • Currency or unit of measure conversion
  • Fiscal calendar / quarter derivation
  • Record filtering for “data quality” reasons
  • Any CASE WHEN that applies business logic
⚠

The dangerous gray area — record filtering: Filtering rows with a NULL business key is a hard rule (structural requirement for vault integrity). Filtering rows because they are “invalid” by some business definition is a soft rule. Store those records in the Raw Vault (with a companion error satellite flagging the issue) and let downstream business rules decide. Suppressing records at the Raw Vault layer destroys auditability and prevents future recovery if the “invalid” definition later changes.

Concept 04

Data Vault & Master Data Management

🧠 ELI5 — Explain Like I’m Five

Master Data Management (MDM) is about finding the “golden record” when multiple systems disagree about the same customer, product, or supplier. Your CRM calls them “CRM-1001”, your ERP calls them “ERP-XYZ-99” — same person, different identities. Data Vault doesn’t replace MDM tools, but provides the perfect foundation: the Raw Vault preserves all source identities, and Same-As Links store the matching decisions made by MDM processes — without destroying any original records.

DV + MDM Integration Architecture

  1. Raw Vault — Each source system’s entities load into Hubs with their own business keys. No merging at this layer. CRM’s “CRM-1001” and ERP’s “ERP-XYZ-99” coexist as separate Hub rows.
  2. MDM Process (external or embedded) — Matching algorithms (probabilistic, deterministic, or manual review) identify which records represent the same real-world entity and produce match decisions.
  3. Same-As Links — Business Vault — Each MDM decision is stored as a SAL row: DRIVER (canonical/golden identity) + DEPENDENT (matched alternative). A satellite on the SAL tracks confidence score, match algorithm, and effective dates.
  4. Business Vault Views — Apply the SAL mapping to resolve all source identities to the canonical key, producing a unified “golden customer” view.
  5. Information Marts — Consume the golden view for reporting and analytics, unaware of the underlying source fragmentation.
SOURCE RAW VAULT BIZ VAULT MART CRM CRM-1001 ERP ERP-XYZ-99 WEB USR-44821 HUB_CUSTOMER BK: CRM-1001 HUB_CUSTOMER BK: ERP-XYZ-99 HUB_CUSTOMER BK: USR-44821 MDM MATCH ENGINE probabilistic / deterministic SAL_CUSTOMER DRIVER ↔ DEPENDENT VW_GOLDEN_CUST resolved identity view INFORMATION MART single golden customer view
SQL — Same-As Link: Storing MDM Match Decisions
-- SAL_CUSTOMER: stores MDM match decisions in the Business Vault
-- Decision: "CRM-1001" IS THE SAME ENTITY AS "ERP-XYZ-99"
INSERT INTO SAL_CUSTOMER (
    SAL_CUSTOMER_HK,
    DRIVER_HUB_CUSTOMER_HK,
    DEPENDENT_HUB_CUSTOMER_HK,
    LOAD_DATE,
    RECORD_SOURCE
)
VALUES (
    MD5('CRM-1001' || '||' || 'ERP-XYZ-99'),
    MD5('CRM-1001'),     -- DRIVER: canonical / golden identity
    MD5('ERP-XYZ-99'),   -- DEPENDENT: matched alternative key
    CURRENT_TIMESTAMP(),
    'MDM.MASTER_DATA_HUB.MATCH_PROCESS_V2'
);

-- A satellite on the SAL stores match metadata:
-- confidence score, algorithm, manual_review_flag, effective dates
SQL — Business Vault View: Golden Customer Resolution
-- Business Vault view: resolve SAL to produce a unified "golden" key
-- COALESCE: if this hub key is a DEPENDENT in the SAL, resolve to its DRIVER
CREATE VIEW VW_BV_CUSTOMER_GOLDEN AS
SELECT
    COALESCE(
        sal.DRIVER_HUB_CUSTOMER_HK,
        h.HUB_CUSTOMER_HK
    )                      AS GOLDEN_HK,    -- canonical identity
    h.HUB_CUSTOMER_HK      AS ORIGINAL_HK,   -- source identity preserved
    h.CUSTOMER_BK,
    h.RECORD_SOURCE
FROM HUB_CUSTOMER h
LEFT JOIN SAL_CUSTOMER sal
    ON sal.DEPENDENT_HUB_CUSTOMER_HK = h.HUB_CUSTOMER_HK;

-- CRM-1001, ERP-XYZ-99, and USR-44821 now share one GOLDEN_HK
-- Original source keys remain intact and individually queryable
👥

Key advantage: Match decisions are auditable and reversible. If an MDM algorithm incorrectly linked two different people, you insert a new satellite row on the SAL marking it inactive. The incorrect decision is preserved in history, the correction is recorded with its own timestamp, and the golden view automatically reflects the updated state — with no data ever destroyed.

Concept 05

Interview Q&A — Advanced DV

What is bi-temporal modeling and why would you add it to a Data Vault? ▶

Bi-temporal modeling tracks two independent time dimensions: Transaction Time (when data was stored in the system) and Valid Time (when the fact was true in the real world). Data Vault naturally provides Transaction Time via LOAD_DATE. Adding Valid Time means adding VALID_FROM and VALID_TO columns to satellite payloads.

This enables: late-arriving data corrections (a payment that arrived late gets backdated to its actual payment date while preserving when we recorded it); regulatory reporting (prove exactly what we reported vs what actually happened on a given date); and policy backdating (an insurance policy approved on Feb 1 that becomes effective Jan 1).

Not every project needs bi-temporal — add it only when business or regulatory requirements explicitly demand “as-of” analysis on both time dimensions. Default to standard Data Vault (transaction time only) unless the business case is clear and documented.

How does Data Vault support Master Data Management without a dedicated MDM tool? ▶

The Raw Vault preserves all source identities exactly as received — no merging, no “winner” selection at load time. CRM’s “CRM-1001” and ERP’s “ERP-XYZ-99” coexist as separate Hub rows regardless of whether they represent the same real-world entity.

When MDM matching decisions are made (by algorithms or manual review), they are stored as Same-As Links (SALs) in the Business Vault. The SAL records: which key is the DRIVER (canonical/golden identity) and which key is the DEPENDENT (matched alternative). Business Vault views then apply SAL mapping to produce the unified “golden record”.

Key benefits: (1) original source identities are never destroyed; (2) match decisions are versioned and auditable via the SAL and its own satellite; (3) if a match decision is wrong, insert a new SAL satellite row marking it inactive — the history of the incorrect match is preserved while the correction takes effect immediately in downstream views.

Where exactly is the line between a hard business rule and a soft business rule? ▶

Hard rules are structural, source-faithful operations that do not change the meaning of data: key normalization (TRIM/UPPER), NULL filtering on business keys, hash key computation, data type casting, splitting composite keys. They run in staging. Practical test: could a business analyst look at the output and recognize it as the same data the source sent? If yes, it is a hard rule.

Soft rules add interpretation: code lookups, derivations, classifications, aggregations. They run in the Business Vault or Information Mart. The key question: “Would a change in business policy alter the output of this transformation?” If yes, it is a soft rule and it must never touch the Raw Vault.

The most dangerous gray area is filtering records for “data quality” reasons. Filtering on NULL business keys is a hard rule (structural requirement). Filtering on “this record looks invalid” is a soft rule — you are applying a business judgment that could change. Load the record into the Raw Vault with a companion error satellite, and let downstream rules determine handling.

How do you handle a GDPR “right to erasure” request in a Data Vault that is insert-only? ▶

This is one of the trickiest architectural challenges in regulated Data Vault implementations. Four defensible approaches exist:

1. Pseudonymization — Replace the customer’s business key and all PII columns with a pseudonym stored in a separate lookup table you control and can delete. The vault retains the pseudonym, but the link to real identity is severed. This is the most common and legally cleanest approach under GDPR.

2. Vault encryption with key deletion — Encrypt all PII columns with a customer-specific key. To “forget,” delete that key. Data physically remains in the vault but becomes permanently unreadable without the key. Technically elegant, operationally complex.

3. Right-to-erasure satellite — Insert a new satellite row with an ERASURE_DATE flag. All downstream marts and views filter out customers with this flag. The raw vault still has data, but no system surfaces it. Simplest to implement; data physically remains.

4. Physical deletion of satellite payload rows — Delete payload rows from satellites while keeping Hub and Link shells for referential integrity. This is the most legally defensible physical deletion, but violates Data Vault’s insert-only principle. Treat it as a formally approved, documented, regulated exception. Never delete Hubs or Links.

✅ Module 09 Key Takeaways

  • Anchor Modeling takes attribute granularity to the extreme — one table per attribute vs Data Vault’s grouped satellites. DV2 satellite splitting achieves most of Anchor Modeling’s flexibility with far fewer tables, simpler queries, and mature tooling support.
  • Bi-temporal modeling adds a Valid Time dimension (VALID_FROM/VALID_TO) to satellite payloads alongside Data Vault’s native Transaction Time (LOAD_DATE). Use it only when the business explicitly requires “what did we believe vs what actually happened” analysis across both dimensions.
  • Hard business rules (structural, source-faithful: TRIM, UPPER, hash computation, NULL BK filtering) belong in staging. Soft rules (interpretive: code lookups, derivations, classifications, aggregations) belong in the Business Vault or Information Mart — never in the Raw Vault where they permanently corrupt the audit trail.
  • Data Vault integrates with MDM through Same-As Links in the Business Vault: Raw Vault preserves all source identities unmodified, MDM match decisions are stored as SAL rows (DRIVER + DEPENDENT), and Business Vault views apply the SAL mapping to produce a golden record view that is fully auditable and reversible.
  • GDPR right-to-erasure in an insert-only vault requires a deliberate architectural choice made at design time: pseudonymization (sever the identity link), encryption with key deletion (data unreadable), right-to-erasure satellite (suppress without deletion), or formal physical deletion of satellite payloads with governance sign-off as a documented regulatory exception.