Workflows

This section documents the common operational procedures for managing the SUSE Paid Advertising Data Warehouse V2. Each workflow covers a specific process with step-by-step instructions, expected outcomes, and troubleshooting guidance.

Available Workflows

Daily Pipeline

The automated daily data flow from advertising platforms through Dataslayer into BigQuery staging tables, then refreshed into final tables. Covers the full schedule, failure handling, and verification steps.

Adding a New Platform Creative Table

Step-by-step process for onboarding a new advertising platform's creative metadata into the warehouse. Includes SQL templates for table creation, MERGE logic, and performance view setup.

Debugging Refresh Failures

How to diagnose and resolve issues when the adlevel or creative refresh scripts encounter errors. Covers log tables, status codes, common failure modes, and diagnostic queries.

Schema Changes

Safe procedures for modifying table schemas in the warehouse. Covers type changes, column additions, and the critical rule of keeping staging and final tables in sync.

General Principles

All workflows in this warehouse follow these principles:

1. No data loss -- Refresh operations preserve historical data not present in current staging
2. Platform isolation -- Errors in one platform do not block other platforms from processing
3. Logging -- All operations log their status to refresh_log_v2 or creative_refresh_log_v2
4. Idempotent operations -- Running a refresh multiple times produces the same result
5. Test before deploy -- Changes are validated on staging tables before touching final tables