# Automated Content Moderation

Status: deferred. Do not provision moderation APIs, Cloud Tasks, or Pub/Sub for the current Terraform plan. The active infrastructure decision record is [10-terraform-requirements-qa.md](./10-terraform-requirements-qa.md), which explicitly removes moderation services from the initial scope. The rest of this document is retained only as a possible future reference.

To ensure Duuble remains a safe environment, we leverage Google Cloud's suite of pre-trained Machine Learning APIs. Instead of building manual review queues for everything or training custom models, we insert GCP AI guardrails directly into the ingestion pipelines.

## 1. Text Moderation (Perspective API / Cloud Natural Language)

Used for Post captions, Comments, Hub descriptions, and User Bios.

*   **API:** Perspective API (by Jigsaw/Google) or Cloud Natural Language API.
*   **Mechanism:** Synchronous. The mobile client submits a `POST /posts` request. Before writing the text to Cloud Spanner, the Cloud Run API calls the Perspective API.
*   **Attributes Analyzed:** `TOXICITY`, `SEVERE_TOXICITY`, `IDENTITY_ATTACK`, `INSULT`, `PROFANITY`, `THREAT`.
*   **Action Flow:**
    *   If `TOXICITY_SCORE > 0.85` (Highly toxic): API returns `422 Unprocessable Entity - SAFETY_VIOLATION`. The post completely fails to create.
    *   If `0.60 < TOXICITY_SCORE <= 0.85` (Borderline): The post is created but marked in the database as `flagged_for_review = true`. It is immediately hidden from public feeds until an Admin approves it.
    *   If `TOXICITY_SCORE <= 0.60` (Safe): Post is created normally.

## 2. Image Moderation (Cloud Vision API)

Used whenever a user uploads an avatar, banner, or attaches a photo to a Post/Comment.

*   **API:** Cloud Vision API (`SafeSearch Annotation`).
*   **Mechanism:** Asynchronous (Event-Driven). 
*   **Architecture Flow:**
    1. User uploads the image to the GCS Bucket via a Signed URL.
    2. The Cloud Run API calls `POST /uploads/complete`.
    3. The application enqueues an asynchronous Task to Cloud Tasks (or publishes to Pub/Sub).
    4. A background worker reads the GCS object, sends it to Cloud Vision, and requests SafeSearch.
*   **Attributes Analyzed:** Adult (Pornography), Spoof (Memes/manipulation), Medical, Violence, Racy.
*   **Action Flow:**
    *   Cloud Vision returns likelihoods (e.g., `VERY_UNLIKELY` to `VERY_LIKELY`).
    *   If Adult or Violence is `LIKELY` or `VERY_LIKELY`: The background worker immediately deletes the file from GCS, deletes the associated object from Spanner, and issues an automated strike to the User's account.
    *   Otherwise: The image remains visible.

## 3. Video Moderation (Cloud Video Intelligence API)

Used for user-attached video files. Video processing is computationally expensive, so it strictly happens asynchronously.

*   **API:** Cloud Video Intelligence API.
*   **Mechanism:** Asynchronous (Event-Driven Context).
*   **Architecture Flow:**
    1. User uploads the `.mp4` file directly to GCS via Signed URL.
    2. Cloud Run executes `POST /uploads/complete` but marks the asset record as `status: PROCESSING`. The UI tells the user "Video is being processed."
    3. An event triggers the Video Intelligence API, pointing it directly at the URI (e.g., `gs://duuble-uploads/video_uuid.mp4`).
*   **Features Used:** Explicit Content Detection.
*   **Action Flow:**
    *   The API scans the video frames over time.
    *   Upon completion, it hits a webhook or publishes a Pub/Sub message back to the Cloud Run API.
    *   If Explicit Content is detected in any segment: The video is wiped from GCS, the DB record is marked `REJECTED`, and the user is notified.
    *   If Safe: The DB record `status` changes to `READY`, and the video appears in feeds.