Google Cloud API outage — Northwind impact

Post-incidentP1Started 17:51 UTC13 JunReplay · 12 Jun 2025
13 JunWhat actually happened
T+5
T+10
T+20
T+60
T+110
T+130
T+190
T+220
Report
STAGE 9 / 9

Analysis · Next day · 13 Jun

What GPT-6 Astra did for this stage, and how it was checked.

Reference data

Analysis pipeline

1

Evidence in

Replay4 new · 35 total

Only evidence up to 13 Jun

2

Extract

GPT-6 Astra—Run this stage to see

Checkable statements with source ids

3

Reconcile

GPT-6 Astra1 upgraded · 3 merged · 5 new

Merged into the ledger, every change recorded

4

Policy check

Code1 capped

Classes capped by provenance, not by wording

5

Communicate

GPT-6 Astra4 updates

Every sentence cites a claim or one of our actions

6

Verify

Code + GPT-6 Astra38/38 sentences

Rules in code, then support checked by GPT-6 Astra

No GPT-6 Astra run for this stage yet. Showing reference data.

The post-incident pack keeps its reference drafts.

Golden replay check

22/22 match

22/22 claims match the golden ledger (1 after the policy cap) · 0 unsupported sentences

ClaimExpectedGPT-6 Astra proposedAfter policyResult
C-003Root cause: a new Service Control quota-policy feature had no error handling and no feature flag; a policy change with blank fields crashed it globallyCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-005Google Cloud has an incident affecting multiple products since 17:51 UTCCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-006Our failing requests are Google Cloud API calls (Firestore, Firebase Auth) returning 503CONFIRMEDCONFIRMEDCONFIRMEDMatch
C-007No Northwind deploys since 14:00, so not caused by our changeCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-008The GCP console won't load for our engineersCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-009Northwind runs only in Google Cloud us-central1, with no second regionCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-010Northwind staff reach internal tools through Cloudflare AccessCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-012Cloudflare's outage is caused by Google Cloud's outageUNCONFIRMEDREPORTEDUNCONFIRMEDCapped, then match
C-013AWS and Azure are also downUNCONFIRMEDUNCONFIRMEDUNCONFIRMEDMatch
C-015Downdetector: Google Cloud problem reports peaked at 14,695 (18:31); Cloudflare at 3,071 (18:49)REPORTEDREPORTEDREPORTEDMatch
C-018All Google Cloud locations except us-central1 have fully recoveredCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-021Google's support account on X says there are no known service disruptions (contradicts the status page)CONFLICTINGCONFLICTINGCONFLICTINGMatch
C-023Cloudflare's Workers KV outage was caused by the failure of a third-party service it depends on (not named)CONFIRMEDCONFIRMEDCONFIRMEDMatch
C-024Google has applied a mitigation in us-central1CONFIRMEDCONFIRMEDCONFIRMEDMatch
C-026All Cloudflare services have been restoredCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-027Most Google Cloud products have fully recovered; a few still see residual impactCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-028Customer impact ran 17:51–21:10 UTC (3 h 19 min): errors from 17:51, alert 17:52, baseline 21:10CONFIRMEDCONFIRMEDCONFIRMEDMatch
C-029Google's mini report states the duration as 3 hoursCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-030All Google Cloud products recovered by 01:27 UTC on 13 June (Vertex AI last)CONFIRMEDCONFIRMEDCONFIRMEDMatch
C-031Google: some customers' monitoring on Google Cloud also failed, leaving them without a signalCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-032Cloudflare: its outage was not the result of an attackCONFIRMEDCONFIRMEDCONFIRMEDMatch
C-033Google's fixes: fail open, audit globally replicated data, feature-flag critical changes, enforce backoff, improve external communicationsCONFIRMEDCONFIRMEDCONFIRMEDMatch

Must not say

Decided by provenance, not by how convincing the text sounds

CONFIRMED only with an organisation's own channel about itself or our internal systems (claims about us). REPORTED needs 2+ independent non-official sources, 1 monitor / press source, or 1 official source about someone else; anything else is at most UNCONFIRMED. CONFLICTING when credible sources disagree (both kept). UNKNOWN for questions nothing answers. A provider's status page outranks its other channels.