Build a Scraper to Monitor Google’s New Total Campaign Budgets

Hook: Stop guessing when Google shifts your campaigns — detect "total campaign budgets" switches and track spend patterns automatically

Marketers and ad ops teams in 2026 are juggling short-lived promos, flash sales, and event-driven campaigns while Google increasingly automates budget pacing with the new total campaign budgets feature. The pain is real: you need early detection when a campaign flips to a total budget model and reliable telemetry to see how pacing changes over days and weeks. This guide shows a practical, production-ready hybrid approach that combines the Google Ads API for authoritative telemetry with Playwright scraping of the Ads UI as a fallback and verification layer.

Executive summary (most important first)

Use the Google Ads API as your primary data source to monitor campaign configuration and spend. If the API does not yet expose the new "total campaign budget" flag for every account, run a lightweight Playwright check against the authenticated Ads UI to detect UI markers or settings. Persist both sources into a timeseries store, perform diff-based detection for mode switches, and run automated analytics to surface pacing changes over days/weeks. Add alerting and dashboards. This hybrid model is resilient to API lag, UI rollout differences, and UI/API schema drift.

Why hybrid? The 2026 context

In late 2025 and early 2026 Google expanded total campaign budgets to Search and Shopping campaigns, building on what Performance Max received in prior years. The ecosystem reaction: faster budget pacing but less transparency in daily allocation. At the same time, Ads API endpoints are evolving faster than some client libraries and UI rollouts vary by region. That makes a pure-API approach brittle if you need immediate detection or if the API doesn't expose the exact field you want.

Hence a hybrid approach: use the API first (authoritative, rate-limited, structured), and use Playwright scraping of the Ads web UI as a fallback/verification for features not yet reflected in the API or to catch rollout-notifications and banners.

Top-level architecture

1. Ingest: Google Ads API (primary) + Playwright UI checks (fallback/verifier)
2. Storage: Timeseries DB (InfluxDB/ClickHouse/Postgres + partitioned ad_telemetry tables)
3. Processing: Daily and hourly jobs that diff campaign metadata and spend metrics
4. Analytics: Pacing, burn curves, spend variance, attribution to total-budget mode
5. Alerts: Slack/PagerDuty when campaign flips or pacing deviates from expected

Pre-reqs and security considerations

• Google Ads API credentials (OAuth2 with refresh token) for each advertiser/client account. Use manager accounts (MCC) where appropriate.
• Playwright (Python or Node) installed on ephemeral instances or a container. Use the stealth techniques sparingly and ethically.
• Secure storage for tokens: secrets manager (AWS Secrets Manager, GCP Secret Manager) and least-privilege service accounts.
• Proxy layer for UI checks only when necessary. Respect Google Terms of Service and client consent — only monitor accounts you own or manage.

Ethics & compliance: Only monitor accounts you control. Automated access to accounts must follow OAuth2 consent flows and client policies. Use Playwright checks for your accounts or with explicit client authorization.

Step 1 — Use Google Ads API to fetch campaign config and spend

Start by pulling authoritative data every hour. The Google Ads API provides campaign-level metrics and budget resources. Your initial query should fetch:

• Campaign resource: id, name, status, start_date, end_date
• CampaignBudget resource: amount_micros, delivery_method, explicitly set fields
• Metrics: cost_micros, clicks, impressions over 1d, 7d, 30d rolling windows
• Any available new flag fields (e.g., budget_type, budget_period, total_budget)

Python example: basic Google Ads query

This snippet uses the official google-ads library (Python). Replace placeholders with your credentials and adapt to the Ads API version your account uses.

# pip install google-ads
from google.ads.googleads.client import GoogleAdsClient
from google.ads.googleads.errors import GoogleAdsException

# Initialize (load from google-ads.yaml or env)
client = GoogleAdsClient.load_from_storage('google-ads.yaml')
customer_id = 'INSERT_CUSTOMER_ID'

query = '''
SELECT
  campaign.id,
  campaign.name,
  campaign.status,
  campaign.start_date,
  campaign.end_date,
  campaign_budget.id,
  campaign_budget.name,
  campaign_budget.amount_micros,
  campaign_budget.delivery_method,
  segments.date,
  metrics.cost_micros,
  metrics.impressions,
  metrics.clicks
FROM campaign
WHERE campaign.status != 'REMOVED'
''' 

ga_service = client.get_service('GoogleAdsService')
response = ga_service.search(customer_id=customer_id, query=query, page_size=1000)

for row in response:
    # normalize and persist rows into timeseries store
    print(row.campaign.id, row.campaign.name, row.metrics.cost_micros)

Notes: The exact field names and availability depend on the Ads API version (2025/2026 updates added budget schema improvements). Always check the client library docs and run a discovery query.

Step 2 — Detect the "total campaign budget" flag: API-first strategy

Where supported, the API will expose a budget schema field (for example, a hypothetical campaign_budget.total_amount_micros or budget_period). Implement a detection layer that:

1. Parses campaign and campaign_budget resources for candidate fields.
2. Normalizes values to a canonical structure: {campaign_id, budget_mode: 'daily'|'total'|'unknown', total_amount, start_date, end_date}.
3. Persists latest canonical state and diffs against prior snapshots.

Example normalization pseudocode:

def normalize_budget(row):
    budget = {}
    if hasattr(row.campaign_budget, 'total_amount_micros'):
      budget['mode'] = 'total'
      budget['total_amount'] = row.campaign_budget.total_amount_micros / 1e6
    elif row.campaign_budget.delivery_method == 'STANDARD' and row.campaign_budget.amount_micros:
      budget['mode'] = 'daily'
      budget['daily_amount'] = row.campaign_budget.amount_micros / 1e6
    else:
      budget['mode'] = 'unknown'
    return budget

Step 3 — Playwright fallback: verify UI and catch rollout artifacts

If the API doesn’t return an explicit flag (common during staged rollouts), use Playwright to authenticate into the Ads UI and check the campaign edit screen or the campaign list for UI markers: banners saying "Total campaign budget enabled", new select dropdowns, or campaign budget cards that show a date range and total amount.

Why Playwright and how to run it safely

• Playwright can run headful or headless; headful is often more stable for complex, JS-driven Ads UI.
• Use one authenticated session per advertiser and schedule low-frequency checks (e.g., every 4–12 hours) to reduce load and detection risk.
• Store session cookies safely and perform re-login using OAuth2 browser flow when needed.

Playwright (Python) snippet to verify campaign budget UI

# pip install playwright
from playwright.sync_api import sync_playwright

def check_campaign_ui(email, password, campaign_id):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)  # headfull recommended for login reliability
        context = browser.new_context()
        page = context.new_page()

        # Navigate to Google Ads campaign page (example URL - adapt to your region/account)
        page.goto('https://ads.google.com/aw/campaigns?ocid={campaign_id}'.format(campaign_id=campaign_id))

        # Wait for login flow to finish (implement robust login handling)
        # Look for UI element that indicates a total campaign budget
        try:
            page.wait_for_selector("text=Total campaign budget", timeout=10000)
            found = True
        except:
            found = False

        # Alternatively, inspect campaign budget card for date-range and total amount
        card = page.query_selector('div.campaign-budget-card')
        if card:
            text = card.inner_text()
            if 'Total' in text and 'through' in text:
                found = True

        context.close()
        browser.close()
        return found

Important: Replace the URL patterns and selectors by inspecting the current Ads UI. Playwright selectors and Ads DOM change often; keep selectors in a small, easy-to-update module.

Step 4 — Persist telemetry and detect switches

Store both API-derived canonical budget state and Playwright verification results. Your schema (conceptual) can look like:

• campaign_state(campaign_id, timestamp, mode, total_amount, start_date, end_date, source = 'api'|'ui')
• campaign_daily_spend(campaign_id, date, cost, clicks, impressions)

Run a detection job that looks for transitions where the last known mode != current mode. When a change to 'total' is detected, mark the change with a reason: API flag, UI banner, or manual confirmation.

def detect_switches(latest_state, previous_state):
    if latest_state['mode'] != previous_state['mode']:
      return {
        'campaign_id': latest_state['campaign_id'],
        'from': previous_state['mode'],
        'to': latest_state['mode'],
        'detected_at': now_iso()
      }
    return None

Step 5 — Analytics: how to measure spend patterns over days/weeks

Once you detect a switch, the interesting question is: how does pacing change? Implement these analytics:

• Burn curve: cumulative spend vs. campaign progression (day 0..N)
• Expected pacing (linear allocation) vs. actual pacing (observed) — compute percentage deviation
• Spend acceleration: week-over-week or day-over-day change in daily spend rate
• Performance signals: CTR, CPC, conversions per spend bucket before/after

Example: compute burn curve and deviation

def burn_curve(spend_rows, total_budget, start_date, end_date):
    days = (end_date - start_date).days + 1
    expected_daily = total_budget / days
    cumulative_expected = 0
    cumulative_actual = 0
    results = []
    for d in range(days):
        day = start_date + timedelta(days=d)
        actual = spend_rows.get(day, 0)
        cumulative_actual += actual
        cumulative_expected += expected_daily
        results.append({
            'day': day.isoformat(),
            'cumulative_expected': cumulative_expected,
            'cumulative_actual': cumulative_actual,
            'deviation_pct': (cumulative_actual - cumulative_expected) / cumulative_expected * 100 if cumulative_expected>0 else 0
        })
    return results

Visualize results in Grafana or a lightweight Plotly dashboard to identify frontloading or backloading patterns during the campaign period.

Step 6 — Alerting and runbooks

Configure alerts based on detection rules:

• Immediate alert when campaign switch to 'total' is detected (include source and evidence: API response, screenshot link)
• Warning when cumulative spend deviates > X% from linear pacing at predefined checkpoints (day 1, 3, 7, 50% campaign elapsed)
• Performance fallback: if ROAS drops by Y% after switch, notify marketing and pause automation rules

Include a runbook for each alert that outlines steps: verify in Ads UI, check conversion tracking, contact account manager, temporarily adjust bidding, or update campaign creatives.

Operational hardening (anti-bot, scaling, and maintenance)

2026 trends to account for:

• Google aggressively hardens public UI against automated scraping. Keep Playwright checks minimal and authenticated with client consent.
• Rate limits and API quota changes: implement exponential backoff, batching, and MCC-based aggregation where possible.
• Schema drift: Ads API and UI selectors will change. Keep a small test suite that validates critical selectors and queries every deploy.
• Observability: log every detection event, API response hashes, and screenshots (sanitize PII) for auditability.

Scaling pattern

1. Tier 1: hourly API sync for all accounts (cheap, structured)
2. Tier 2: Playwright UI check only when API yields ambiguous state or during rollout windows (targeted)

Sample full workflow: timeline and code orchestration

Example orchestration using a Kubernetes cronjob (or cloud functions):

1. 00:00 UTC — Google Ads API hourly sync job runs, writes campaign_state + daily metrics.
2. 00:15 UTC — Detect switches job diffs latest vs previous state; if ambiguous, enqueue Playwright verification.
3. 01:00 UTC — Playwright worker picks tasks (one session per account), captures evidence, updates campaign_state with source='ui'.
4. 01:30 UTC — Analytics job computes burn curves for any campaign in 'total' mode and writes to dashboard datastore.
5. Alerts fired immediately if thresholds hit.

Case study (hypothetical, 2026-style)

Retailer X turned on total campaign budgets for a two-week launch. Using the hybrid monitor, the analytics discovered the campaign frontloaded 60% of the budget in the first 48 hours (Google's optimizer prioritizing early learning). The team received an alert at 36 hours and adjusted bidding strategies — they recovered ROAS in the second week by narrowing target audiences and switching to conversion-focused bidding. The hybrid system provided both the API's spend numbers and a Playwright-captured screenshot showing the campaign's budget card and date range.

Common pitfalls and troubleshooting

• False positives: UI banners can be regional or for opt-in beta. Cross-check API and UI before notifying stakeholders.
• Selector brittleness: Ads DOM is complex. Maintain a selector health test and fallback CSS/XPath rules.
• Auth friction: Google SSO flows may require manual login. Use refresh tokens and expect occasional reconsent flows for long-lived sessions.
• Quota throttling: request higher API quotas proactively if you scale to many accounts.

Advanced strategies and future-proofing (2026+)

• Server-side telemetry: push more granular spend events into BigQuery via the Google Ads API's streaming features (where available).
• Model-based anomaly detection: build an ML model that predicts expected spend path based on historical campaigns and alert on statistically significant deviations post-switch.
• Feature flags and A/B: instrument your monitoring system to split campaigns into control and test to understand how total-budget mode affects performance across segments.
• Integrate multi-channel: combine with Data Warehouse signals (CRM, conversion imports) to analyze ROAS after budget mode switches.

Developer checklist (quick)

• Get OAuth credentials and MCC access for client accounts
• Implement hourly Google Ads API sync
• Normalize campaign budget state and persist
• Implement Playwright UI verifier for ambiguous cases
• Compute burn curves and set alert thresholds
• Document runbooks and logging for every detection event

Final thoughts and 2026 predictions

As Google pushes more automation into budget pacing, ad ops will shift from minute-to-minute micromanagement to monitoring and policy orchestration. In 2026, expect:

• Greater adoption of campaign-level total budgets across Search and Shopping
• Ads API parity improving, but UI rollouts still patchy — hybrid telemetry remains valuable
• Increased demand for explainability tools that correlate optimizer behavior with business metrics

Adopting a hybrid Playwright + Google Ads API approach gives you early detection, verifiable evidence, and a robust foundation for automated analytics and alerts.

Actionable next steps

1. Inventory client/ad accounts and ensure OAuth access for monitoring.
2. Wire up the hourly Google Ads API sync and baseline your campaign_state table.
3. Add Playwright verification for 10–20 high-priority campaigns, tune selectors, and store screenshots.
4. Build burn-curve dashboards and set deviation thresholds for alerts.

Call to action

If you want a ready-to-deploy starter kit: a Dockerized Playwright verifier, a Google Ads API sync worker, and a sample analytics notebook for burn curves — reach out or clone the example repo we maintain. Instrument your campaigns today so you can detect switches to total campaign budgets the moment they happen, and turn automated telemetry into actionable optimization.

Related Reading

• Mac mini M4 vs M4 Pro: Which Model Gives You the Best Value?
• Budget Breakfast Tech: Where to Score Deals on Cereal Makers and Kitchen Appliances
• How Total Campaign Budgets Can Help You Send Urgent Recall Notices Without Overspending
• AI & Analytics for Fitness Creators: Using Data-Driven Discovery to Find Your Hit Class Format
• Pitching a Channel to Legacy Media: How to Sell a YouTube Concept to Broadcasters