# Terraform Requirements Q&A Status: accepted for initial Terraform planning Date: 2026-05-15 Project: `long-velocity-496203-v9` Scope: production-ready infrastructure design, with `dev` provisioned first This document records the infrastructure questions, decisions, and rationale before writing Terraform. It is the current source of truth where it conflicts with earlier infrastructure notes. ## Summary | Area | Decision | Why | |---|---|---| | Environments | Build `dev` first, keep `prod` variables/modules ready for later | We are still in development, but the Terraform structure should not paint us into a corner | | GCP projects | Use one project for now: `long-velocity-496203-v9` | Simpler to operate now; use strict names and state prefixes to avoid mixing dev/prod | | Region | Start in `us-central1` | Good default for US launch; future Israel expansion is a migration, not a toggle | | Scale | Design for 1M RPM capability, but do not keep 1M RPM warm capacity idle | High max limits are cheap; warm Cloud Run, Spanner, Redis, LB/CDN/Armor capacity is not free | | Compute | One backend image, two Cloud Run services: API and SSE | One build pipeline, separate scaling behavior | | Database | Cloud Spanner PostgreSQL dialect, regional `us-central1` | Durable source of truth with room to scale | | Redis | Memorystore for volatile auth/cache/rate-limit state only | Likes/votes and durable counters belong in Spanner | | Storage | Private GCS bucket, signed uploads, CDN for finalized asset reads | Cloud Run should not proxy binary uploads or hot media reads | | Edge | Global HTTPS Load Balancer, Cloud CDN, Cloud Armor | Required for domain, routing, TLS, caching, and edge protection | | Auth OTP | Custom phone OTP with third-party SMS provider | GCP has no clean generic SMS OTP sender without Firebase/Identity Platform changing the auth model | | CI/CD | GitHub Actions primary, Artifact Registry, manual Terraform apply for now | Simple, production-ready, and controlled | | Observability | Cloud Logging, Error Reporting, Trace, uptime checks, alerting | Needed before real traffic | | Out of scope | Cloud Tasks/Pub/Sub, moderation APIs, Firebase, Firestore, Cloud SQL, GKE | Not required by the current product/infrastructure requirements | ## 1. Environments ### Q: What environments do we need? Decision: `dev` and `prod`, but Terraform should provision only `dev` for now. Why: - Development is still active, so `prod` resources would add cost and operational noise now. - Terraform modules should already support `prod` so the later move is a variable/config change plus a planned apply, not a rewrite. Recommended layout: - `duuble-dev-*` resources now. - `duuble-prod-*` resources later. - Same Terraform modules for both. - Separate state prefixes, for example `env/dev` and `env/prod`. ### Q: Separate GCP projects or one project? Decision: one project for now: `long-velocity-496203-v9`. Why: - Simpler permissions and billing now. - Production isolation is better long term, but not required before the first dev environment. Risk: - A single project makes accidental cross-environment access easier. We reduce that risk with strict naming, separate state prefixes, resource-level IAM, and clear labels. ### Q: Who owns billing and budget alerts? Decision: the project owner owns billing. Terraform should add budget alerts once the billing account ID and monthly thresholds are known. Why: - Budget alerts are tied to a billing account, not only a project. - The alert recipient should include `google@duuble.com`. Open input: - Billing account ID. - Budget alert thresholds. ## 2. Region and Scale ### Q: Primary region? Decision: `us-central1`. Why: - The first user base is mostly US. - It is a strong default GCP region with broad service availability. ### Q: Can Israel be enabled later with a simple switch? Decision: keep `primary_region` configurable, but do not treat Israel expansion as a simple switch. Why: - Cloud Run region changes are manageable. - Stateful systems are different: Spanner, Redis, and GCS location changes require migration planning. - Future Israel expansion should be designed as a second rollout, likely involving `me-west1` if available for the required services. ### Q: Is 1M RPM a day-one requirement? Decision: the architecture should be able to scale toward 1M RPM, but we will not pay for full warm capacity while unused. Why: - Cloud Run `max_instances = 1000` is mostly a limit until traffic arrives. - Cloud Run `min_instances`, Spanner processing units/nodes, Redis size/tier, Load Balancer, Cloud Armor, and CDN all have real cost even when idle or lightly used. - The right approach is high maximum limits, modest warm capacity, load testing, then raising warm capacity before a known launch or campaign. ### Q: Multi-region now? Decision: no. Single-region compute and database first, with backups enabled. Why: - The launch geography is the US. - Multi-region stateful infrastructure adds complexity and cost before the product needs it. ## 3. Cloud Run ### Q: One container image or separate API/SSE images? Decision: one container image, two Cloud Run services. Services: - `duuble-dev-api` - `duuble-dev-sse` - Later: `duuble-prod-api`, `duuble-prod-sse` Artifact image: - `us-central1-docker.pkg.dev/long-velocity-496203-v9/duuble/backend:` Why: - One build pipeline. - One backend version per deploy. - API and SSE still scale independently. - Terraform and CI stay simpler. Tradeoffs: - API and SSE are coupled at release time. - The image contains code both services may not execute. - A bad image can affect both services, so rollout and rollback need to handle both services intentionally. ### Q: Runtime version? Decision: pin the app to the current .NET LTS target, not an unpinned `latest`. Current target: - .NET 10 LTS for implementation planning. Why: - `latest` makes builds less reproducible. - LTS gives a clearer production support window. ### Q: Container port? Decision: `8080`, with the app listening on the Cloud Run `PORT` environment variable. Why: - `8080` is the common container default. - Respecting `$PORT` keeps the app portable across Cloud Run revisions. ### Q: Min/max instances? Decision: use different defaults for dev and prod. Dev defaults: - API min instances: `0` - API max instances: `20` - SSE min instances: `0` - SSE max instances: `20` Prod starting point: - API min instances: `5` - API max instances: `1000` - SSE min instances: `1` - SSE max instances: `1000` Prod launch tuning: - Do not set API min instances to the full 1M RPM warm target by default. - If load tests show about 60 API instances are needed for warm 1M RPM, raise min instances only before a launch that actually needs that capacity. Why: - Dev should be cost-controlled. - Prod should have enough warm capacity for low-latency normal traffic. - Maximum limits allow rapid scaling without keeping all capacity hot. ### Q: Concurrency? Decision: - API concurrency: `80` - SSE concurrency: `1000` Why: - The existing docs already model the API around `80`. - SSE connections are mostly idle long-lived streams, so they need a much higher concurrency setting. ### Q: Request timeouts? Decision: - Normal API timeout: `30s` default. - SSE timeout: `3600s`. Why: - `30s` keeps broken or overloaded requests from tying up capacity. - Long-running work should not happen in request/response paths. - SSE needs the full long-lived timeout. Tradeoff: - `60s` can hide slow operations and increase instance pressure. - `30s` forces the API to stay responsive. Endpoints that truly need longer should be justified individually. ### Q: Direct Cloud Run URL or Load Balancer only? Decision: Load Balancer only. Cloud Run ingress: - `internal-and-cloud-load-balancing` Why: - Cloud Armor, CDN, TLS, routing, and domain policy all live at the Load Balancer. - Direct Cloud Run URLs bypass edge security and caching. ## 4. Spanner ### Q: Instance and database names? Decision: - Dev instance: `duuble-dev-spanner` - Prod instance later: `duuble-prod-spanner` - Database: `duuble-db` ### Q: Dialect? Decision: PostgreSQL dialect. ### Q: Regional or multi-region? Decision: regional `us-central1` first. Why: - Lower complexity and cost. - Backups and PITR cover the current disaster recovery requirement. - Multi-region is a later business decision, not needed for the first US launch. ### Q: Launch capacity? Decision: - Dev: start at `100` processing units. - Prod: start at `1000` processing units / 1 node or use managed autoscaling if selected, then adjust from load tests. Why: - Spanner charges for provisioned capacity. - Guessing production capacity for 1M RPM without load tests is not reliable. ### Q: PITR and backups? Decision: - PITR: `7d` - Backup: daily snapshot, 30-day retention Why: - PITR handles bad deploys, accidental writes, and migration mistakes. - Daily backups provide longer restore points. ### Q: Who runs schema migrations? Decision: - Terraform creates the initial schema. - Long-term migrations should move to CI before production. Why: - Terraform can bootstrap the first schema. - Ongoing schema evolution in Terraform is awkward and risky for application migrations. ## 5. Redis ### Q: Memorystore Basic or Standard HA? Decision: - Dev: Basic tier. - Prod: Standard HA tier. Why: - Basic is cheaper and acceptable for development. - Standard HA is better for production because Redis loss would temporarily log users out, break OTP challenges, and reset rate-limit state. ### Q: Redis version? Decision: pin Redis `7.2` unless GCP support forces a different stable version at implementation time. Why: - Pinning avoids surprise behavior changes. ### Q: Memory size? Decision: - Dev: `1GB`. - Prod: start at `5GB` to `10GB`, then tune from metrics. Why: - Redis is used for volatile auth/cache/rate-limit state, not durable social data. ### Q: Authorized VPC? Decision: - Create an environment VPC, for example `duuble-dev-vpc`. - Attach Memorystore to that VPC. - Connect Cloud Run using Direct VPC egress unless implementation constraints require a Serverless VPC Access connector. Why: - Memorystore is private networking. - Direct VPC egress is simpler when supported for the needed traffic path. ### Q: What belongs in Redis? Use Redis for: - Refresh token/session state. - OTP challenges. - Rate-limit buckets. - Active user HyperLogLog counters. - Share-link or hot lookup cache. Do not use Redis as source of truth for: - Likes. - Votes. - Durable counters. - User profiles. - Posts. Why: - Redis data loss is acceptable for temporary auth/cache state. - Likes, votes, and durable counters must survive Redis restarts/failures and belong in Spanner. ### Q: Persistence/backups? Decision: no Redis persistence/backups for the initial plan. Why: - The planned Redis state is intentionally volatile. - If later Redis starts holding data that cannot be lost, that is a design problem to fix before enabling persistence. ### Q: Refresh token TTL? Decision: `30d`. ## 6. Storage ### Q: Asset bucket name? Decision: - Dev: `duuble-assets-bucket-long-velocity-496203-v9-dev` - Prod later: `duuble-assets-bucket-long-velocity-496203-v9-prod` ### Q: Bucket region? Decision: `US` multi-region. Why: - The app starts in the US. - Asset serving benefits from multi-region storage more than the primary database does. ### Q: Public or private assets? Decision: - Bucket remains private with uniform bucket-level access. - Uploads use signed `PUT` URLs. - Finalized asset reads should go through CDN URLs. Why: - Direct public bucket access is harder to control. - CDN gives better performance and future cache policy control. ### Q: What should be cacheable? Decision: cache every response that is safe to cache. Important boundary: - Public/static GET responses and finalized assets should be cacheable. - Personalized or authenticated data must be marked `private` or `no-store` by the app unless it is explicitly safe to share across users. Why: - "Cache everything" cannot mean globally caching private user responses. - Terraform can enable CDN broadly, but the application must emit correct `Cache-Control` headers. Initial CDN policy: - API GETs: cache only when the app marks them public. - Finalized immutable assets: long TTL, preferably object paths that never change. - Temporary upload paths: no CDN caching. ### Q: Upload size and MIME policy? Decision: - Images: up to `20MB`. - PDFs: up to `20MB`. - Videos: up to `100MB`. Allowed initial MIME types: - `image/jpeg` - `image/png` - `image/webp` - `application/pdf` - `video/mp4` - `video/webm` Optional later mobile formats: - `image/heic` - `image/heif` Why: - The current API docs support images and PDFs. - HEIC/HEIF can be added if the mobile client needs native iOS uploads; otherwise the client can normalize to JPEG/WebP. ### Q: Temporary upload cleanup? Decision: delete temporary uploads after `24h`. Why: - Failed or abandoned signed URL uploads should not accumulate cost. ### Q: Asset bucket versioning? Decision: disabled for asset buckets if object paths are immutable. Why: - Immutable object paths make overwrite protection less useful. - Versioning can multiply storage cost. Separate decision: - Terraform state bucket should have versioning enabled. ### Q: Soft delete? Decision: enable 14-day soft delete if available and cost is acceptable. Why: - It gives a short recovery window for accidental deletes without full object versioning. ## 7. Load Balancer and Domain ### Q: Final API domain? Decision: - Production domain: `api.duuble.com`. - Dev should use `dev-api.duuble.com`. Why: - Avoid coupling the final production hostname to the dev environment. ### Q: SSL certificate? Decision: Google-managed certificate through Certificate Manager. Why: - No manual certificate renewal. - Works cleanly with the global HTTPS Load Balancer. ### Q: IPv6? Decision: - Prod: yes. - Dev: optional. Why: - IPv6 is useful for broad client compatibility. - Dev can stay simpler if DNS/IP setup slows work down. ### Q: HTTP redirect? Decision: redirect HTTP to HTTPS. Why: - The API should only be used over TLS. ### Q: Path routing? Decision: - `/api/v1/notifications/stream` routes to the SSE Cloud Run service. - Everything else routes to the primary API service. Why: - SSE needs different timeout and concurrency settings. ### Q: Health check path? Decision: add `/healthz`. Why: - Uptime checks need a cheap endpoint. - The endpoint should not depend on expensive downstream calls by default. ## 8. Cloud CDN ### Q: Is CDN needed for assets or only API? Decision: CDN should front both selected API GET responses and finalized asset reads. Why: - Hot feed media can become much more expensive if every read goes to GCS or Cloud Run. - Public/static API responses should not hit Cloud Run repeatedly. ### Q: TTLs? Initial policy: - API public GET default TTL: `60s`. - API public GET max TTL: `300s`. - Finalized immutable assets default TTL: `86400s` / 1 day. - Finalized immutable assets max TTL: `31536000s` / 1 year. Why: - API data changes more often. - Asset paths should be immutable, so long asset caching is safe. ### Q: Cache key? Decision: - Include host, path, and query string. - Do not include cookies by default. - Do not cache responses with `Authorization` unless the app explicitly marks the response safe for shared caching. Why: - Query strings can affect API and signed asset URLs. - Cookies and Authorization can accidentally split or leak cache state. ### Q: CI cache invalidation? Decision: not required for normal media because object paths should be immutable. Why: - New deploys or uploads should publish new object paths or updated API cache headers. - CI invalidation can be added later for emergency purges. ## 9. Cloud Armor ### Q: Required from day one? Decision: yes. Why: - It protects Cloud Run from basic edge abuse before traffic reaches paid compute. ### Q: Global rate limit? Decision: `2,000 requests/minute/IP`. Why: - Coarse protection against single-IP floods while allowing normal heavy clients. ### Q: Stricter Auth/OTP limits? Decision: yes. Use both: - Cloud Armor path/IP-level throttles. - Redis/app-level per phone/device/user limits. Why: - Cloud Armor sees source IPs and paths. - The app knows phone/user identity and can prevent abuse across rotating IPs. ### Q: Adaptive Protection? Decision: yes. Why: - It gives anomaly detection and suggested edge rules for L7 attacks. ### Q: reCAPTCHA Enterprise? Decision: yes, from day one for high-risk flows. Initial flows: - OTP request. - OTP verify. - Post creation if abuse appears. Why: - It helps distinguish automated abuse before Cloud Run work is spent. Tradeoffs: - Adds mobile/client integration work. - Adds cost and operational tuning. - Can create false positives if thresholds are too aggressive. ### Q: Block Tor/VPN/datacenter IPs? Decision: do not blanket block them at first. Rate-limit or challenge them. Required allowlist: - `57.129.0.112/32` Why: - The owner uses that IP path. - Some legitimate users may be behind VPNs, office networks, or cloud egress. ### Q: Country restrictions? Decision: allow all countries for now. ## 10. Secrets and IAM ### Q: What secrets exist? Terraform should create Secret Manager containers for: - `JWT_SIGNING_KEY` - `JWT_ISSUER` - `JWT_AUDIENCE` - `REFRESH_TOKEN_PEPPER` - `OTP_CODE_PEPPER` - `GOOGLE_OAUTH_CLIENT_ID` - `GOOGLE_OAUTH_CLIENT_SECRET` - `APPLE_CLIENT_ID` - `APPLE_TEAM_ID` - `APPLE_KEY_ID` - `APPLE_PRIVATE_KEY` - `SMS_PROVIDER_API_KEY` - `EMAIL_PROVIDER_API_KEY` - `RECAPTCHA_SECRET` Non-secret config can be regular environment variables: - Project ID. - Region. - Spanner instance/database names. - Redis host/port. - GCS bucket name. - CDN base URL. ### Q: Should Terraform manage secret values? Decision: Terraform creates secret containers, not secret values. Why: - Keeps secret values out of Terraform state. - Simple operational model: create the named secret once, then manually add versions in Secret Manager or through a secure CI secret flow. ### Q: One service account or separate accounts? Decision: one Cloud Run runtime service account for dev. Why: - Simple and adequate for the first environment. ### Q: Project-level or resource-level IAM? Decision: prefer resource-level IAM where practical. Why: - Project-level IAM is simpler but grants broad access. - Resource-level IAM limits blast radius, especially for buckets and secrets. Use project-level IAM only when: - GCP service requirements make resource-level binding impractical. - The role is intentionally broad for a deployer/admin identity. ### Q: Who can deploy? Decision: - Humans with approved IAM can deploy manually. - GitHub Actions is the primary CI/CD deployer. - Cloud Build may also be granted deploy permissions if we later use it, but it is not the primary pipeline now. ## 11. CI/CD ### Q: GitHub Actions or Cloud Build? Decision: GitHub Actions primary. Why: - It is simpler if the repository is already on GitHub. - Workload Identity Federation avoids long-lived GCP keys. ### Q: Artifact Registry naming? Decision: - Repository: `duuble` - Image: `backend` - Full image: `us-central1-docker.pkg.dev/long-velocity-496203-v9/duuble/backend:` ### Q: Deploy branch? Decision: merge to `main`. ### Q: Terraform apply automatic or manual? Decision: - Terraform plan can run automatically. - Terraform apply is manual for now. Why: - Safer while the infrastructure is still being established. ### Q: Separate Terraform state bucket per environment? Decision: one state bucket, separate prefixes. Example: - `env/dev` - `env/prod` Why: - Simpler bucket management. - Prefixes keep states separate. ### Q: What is state locking and do we need it? Decision: yes, use backend locking. Meaning: - State locking prevents two Terraform runs from writing the same state at the same time. Why: - Without locking, concurrent applies can corrupt state or overwrite each other. ## 12. Observability ### Q: Logs retention? Decision: `30d` to start. Why: - Enough for debugging without keeping logs too long. ### Q: Error Reporting? Decision: yes. ### Q: Uptime checks? Decision: yes. Initial check: - `https://api.duuble.com/healthz` for prod. - Dev equivalent if a dev domain is configured. ### Q: Alert channel? Decision: - Email: `google@duuble.com` ### Q: Metrics alerts? Decision: yes. Initial alerts: - Cloud Run 5xx rate. - Load Balancer 5xx rate. - Cloud Armor 429 spike. - API p95 and p99 latency. - SSE connection errors. - Spanner CPU and latency. - Redis memory usage. - Redis connection errors. - Cloud Run max instance saturation. - Budget alerts after billing details are available. ### Q: Request tracing? Decision: yes. Why: - Trace data is needed to understand p95/p99 latency across Cloud Run, Spanner, Redis, and external providers. ## 13. Hard Requirement Check ### Q: Provision Cloud Tasks or Pub/Sub? Decision: no. Why: - Current requirements do not need async jobs. - If moderation, fanout jobs, background processing, or reliable external delivery become required, revisit this. ### Q: Provision moderation APIs? Decision: no. Why: - Moderation APIs were removed from the current scope. ### Q: Provision Firebase, Firestore, Cloud SQL, or GKE? Decision: no. Why: - Current docs and decisions use custom auth, Spanner, Redis, GCS, and Cloud Run. - OTP will use a third-party SMS provider rather than Firebase phone auth. ### Q: Terraform quality level? Decision: production-ready infrastructure modules, with dev applied first. Why: - We want the structure, IAM, state, naming, and modules to be correct before production exists. ## Remaining Open Inputs These should be answered before production apply, but they do not block a dev Terraform skeleton: 1. Billing account ID and budget thresholds. 2. SMS provider for OTP delivery. 5. Secret values and reCAPTCHA/app attestation credentials. 6. Whether HEIC/HEIF uploads are accepted directly or normalized by the mobile client. 7. Whether Cloud Build should actually be enabled as a deployer or only GitHub Actions plus humans.