Incident Response

import { Aside } from ‘@astrojs/starlight/components’;

Severity Levels

Level	Name	Description	Response
P0	Critical	Site down, booking broken, data breach	Immediate (< 15 min)
P1	High	Core page broken, form failing, DB unreachable	< 1 hour
P2	Medium	Non-critical feature broken, visual regression	< 4 hours
P3	Low	Minor bug, cosmetic issue	Next business day

Response Process

Detect — Better Uptime alert / Sentry spike / user report
Assess — Check /api/status, Vercel logs, service status pages
Contain — Rollback bad deploy or enable Cloudflare “Under Attack” mode
Resolve — Apply fix via hotfix branch; see Release Process
Post-mortem — Write brief post-mortem for P0/P1 incidents

Scenario Playbooks

Site Returning 5xx Errors

# 1. Check if a recent deployment caused it
# Vercel → Deployments → compare deploy time vs. incident start

# 2. If deployment caused it → instant rollback (see Deployment Runbook)

# 3. Otherwise check:
#    - Vercel function logs
#    - Environment variables still set
#    - External services: Supabase, Resend, HubSpot status pages

Booking Form Not Submitting

Common causes:

Missing environment variable (SUPABASE_SERVICE_ROLE_KEY, RESEND_API_KEY)
Supabase connection refused
Turnstile site/secret key mismatch
Rate limit exceeded (Upstash Redis)

# Check Vercel function logs for /api/booking
# Test Turnstile with the Cloudflare test key locally

Temporary mitigation: Update contact page to show fallback email address. Deployable in < 5 minutes.

Suspected Data Breach

Immediate actions (within 15 minutes):

Rotate ALL secrets immediately:
- Supabase service role key
- Resend API key
- HubSpot private app token
- Cloudflare R2 access keys
- Turnstile secret key
- Upstash Redis credentials
Update rotated secrets in Vercel Environment Variables
Invalidate all active Supabase sessions
Review access logs (Supabase, Vercel, Cloudflare Security Events)
If leads table was accessed improperly:
- Document what was accessed (row count, time range)
- GDPR: EU resident PII breach → 72-hour notification requirement
- Notify Brett Johnson + legal counsel immediately

DDoS on Booking Form

# Cloudflare absorbs most attacks automatically
# Check CF Security → Events for threat traffic

# If not caught automatically:
# CF dashboard → Security → "I'm Under Attack" mode
# (adds JS challenge to ALL visitors — use as last resort)

# WAF rule to add if targeted:
# Block IPs sending > 20 req/min to /api/booking

Key Contacts & URLs

Resource	URL
Vercel Dashboard	https://vercel.com/dashboard
Vercel Status	https://vercel-status.com
Supabase Dashboard	https://app.supabase.com
Supabase Status	https://status.supabase.com
Cloudflare Dashboard	https://dash.cloudflare.com
Cloudflare Status	https://www.cloudflarestatus.com
Sentry	https://sentry.io
Upstash Console	https://console.upstash.com
Lead Engineer	Jesse Lucus / VLN Security