PrivacyPilot

Automated Privacy Policy Intelligence Engine with Verbatim Quote Verification & GDPR Assessment

Overview • Key Capabilities • Architecture • GDPR Rubric • Quickstart • Benchmarks & Research

---

Overview

PrivacyPilot is a privacy intelligence platform designed to decode complex privacy disclosures, terms of service, and cookie policies into verifiable, structured assessments.

The Problem

The Consent Paradox: Over 97% of digital service users accept terms of service without reading them. Standard privacy policies span an average of 30 pages of dense legal terminology designed primarily for liability protection rather than consumer clarity.

• Opaque Data Monetization: Critical disclosures regarding third-party data brokerage, biometric tracking, location history retention, and deletion obstacles remain concealed in fine print.
• LLM Hallucination Risks: Naive AI summarizers frequently generate false assurances or hallucinate nonexistent privacy protections, creating compliance and security vulnerabilities.

The PrivacyPilot Solution

PrivacyPilot automates domain discovery, extracts legal policy text, executes Keyword-Hybrid Retrieval (RAG) across 10 GDPR compliance dimensions, and runs post-analysis string verification to guarantee zero-hallucination outputs.

Deterministic Verification Rule: Every claim generated by the LLM is programmatically matched against original scraped text. If an excerpt cannot be verified byte-for-byte in the source document, it is marked as Not clearly disclosed.

---

Key Capabilities

Automated Document Discovery & Crawling

• Same-Origin Crawler: Accepts any domain (e.g., https://www.mozilla.org) and automatically discovers linked /privacy, /terms, /cookies, and /notice documents on the same origin.
• SSRF-Hardened Execution: Blocks private IPv4/IPv6 ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 127.0.0.1), loopback targets, and AWS metadata endpoints with a strict 2MB streaming cap.

Keyword-Hybrid Retrieval Engine (RAG)

• Structural Chunking: Documents are parsed along section boundaries into ~1,800 character chunks.
• Dimension Scoring: Calculates term overlap for 10 distinct privacy dimensions to select the top 2 highest-scoring chunks per dimension.
• Logged Retrieval Traces: Exports audit logs (dimension → keywords → chunkId → sourceUrl) for complete research reproducibility.

Dual Anti-Hallucination Pipeline

• 1. Verbatim String Matching: The verifyEvidenceQuote post-processor validates LLM quotes against collected source text.
• 2. Live-Site Text Fragment Deep Linking: Generates standard WICG Text Fragments (#:~:text=...). Clicking a finding opens the live website with the exact quote highlighted in the user's browser.

10-Dimension GDPR Risk Index

• Layered Transparency Model: Aligned with ICO guidance and USENIX Security research, delivering an overall Transparency Score (0–5), Risk Percentage (%), and Risk Level (Very High to Low).
• Legal Alignment: Direct mapping to EU GDPR Articles 5, 12–22, and 32.

Multi-Provider Bring Your Own Key (BYOK) Architecture

• Decentralized Credentials: Users maintain API credentials (Groq, OpenAI, Anthropic) directly within their authenticated user settings.
• Local Offline Execution: Native support for local Ollama instances (llama3.2), enabling private on-device analysis without external cloud calls.

---

System Architecture

graph TD
    UserClient[User / Browser App] -->|HTTPS / Auth| NextFrontend[Next.js 15 Frontend on Vercel]
    NextFrontend -->|Reactive Sync| ConvexCore[Convex Cloud Backend]

    subgraph Convex Engine
        ConvexCore --> CacheEngine[In-Flight & Content Hash Cache]
        ConvexCore --> ScraperNode[SSRF-Safe Crawler Node Action]
        ScraperNode --> HybridRAG[Section Chunking & Keyword Hybrid Engine]
        HybridRAG --> LLMAction[LLM Execution Node Action]
    end

    subgraph LLM Providers
        LLMAction -->|Cloud API| GroqLLM[Groq llama-3.3-70b / OpenAI]
        LLMAction -->|Local Host| OllamaLLM[Local Ollama llama3.2]
    end

    LLMAction --> QuoteVerifier[Post-Verification Engine]
    QuoteVerifier -->|String Match & WICG Fragment| EvidenceStore[Verified Evidence Engine]
    EvidenceStore --> ConvexCore

Loading

Data Pipeline Sequence

[ Input Target URL ]
       │
       ▼
[ Normalize Host & Compute Content Hash ] ──► ( Cache Hit? Return Instant Result )
       │
       ▼
[ SSRF-Safe Multi-Page Crawler ] ──► Discover /privacy, /terms, /cookies
       │
       ▼
[ Section Boundary Parsing ] ──► Split text into ~1800 char chunks
       │
       ▼
[ Keyword-Hybrid Retrieval ] ──► Rank top 2 chunks for each of 10 dimensions
       │
       ▼
[ LLM Inference ] ──► Structured JSON extraction (Groq / Ollama)
       │
       ▼
[ Quote Verification ] ──► Match quotes vs source ──► Strip hallucinations
       │
       ▼
[ WICG Deep Linking ] ──► Generate live text fragment URLs (#:~:text=...)
       │
       ▼
[ Reactive UI Push ] ──► Live update to dashboard via Convex Subscriptions

---

GDPR Compliance Rubric

PrivacyPilot evaluates written disclosures from 1 (Opaque / Harmful) to 5 (Explicit / User-Protective) across 10 compliance dimensions:

Dimension	GDPR Reference	Scope of Evaluation
Data Collection	Art. 13(1)(a), Art. 14	Evaluates categories of personal data collected and explicit processing purposes.
Data Sharing	Art. 13(1)(e), Art. 19	Identifies third-party data broker disclosure, commercial transfers, and ad networks.
User Rights & Opt-Out	Art. 12–22	Checks for access, rectification, erasure (Right to be Forgotten), and objection paths.
Cookie & Tracking	ePrivacy / Art. 5(3)	Assesses disclosures on cookies, fingerprinting, telemetry, and opt-out controls.
Data Retention	Art. 5(1)(e), Art. 13(2)(a)	Evaluates explicit storage limits, retention schedules, and automatic deletion policies.
Security Measures	Art. 32	Measures technical safeguards, encryption standards, and breach notification mechanisms.
Policy Changes	Art. 13, Art. 14	Evaluates advance notification procedures before terms and policy updates take effect.
Children's Privacy	COPPA / Art. 8	Verifies age restrictions, parental consent frameworks, and minor protection policies.
International Transfer	Art. 44–49	Checks for Standard Contractual Clauses (SCCs), adequacy decisions, and cross-border safeguards.
Contact & Dispute	Art. 13(1)(a), Art. 77	Evaluates DPO contact information and instructions for lodging supervisory authority complaints.

Scoring Formulation

• Dimension Score ($S_d$): Individual assessment score from $1.0$ to $5.0$.
• Transparency Score ($T$): Arithmetic mean of all 10 dimension scores. $$\text{Transparency Score} = \frac{1}{10} \sum_{i=1}^{10} S_i$$
• Risk Percentage ($R$): $$\text{Risk %} = \left(\frac{5 - T}{4}\right) \times 100$$

---

Quickstart

Environment Requirements

• Node.js: v18.0.0 or higher
• npm: v9.0.0 or higher
• Ollama (Optional): Required only for local offline execution (Download Ollama)

1. Repository Setup

git clone https://github.com/VishalPainjane/PrivacyPilot.git
cd PrivacyPilot
npm install

2. Backend Initialization (Convex)

# Terminal 1: Launch Convex development environment
npx convex dev

# Configure Auth JWT keys
npm run setup:auth

3. Frontend Execution

# Terminal 2: Launch Next.js web application
cd web
npm run dev

Navigate to http://localhost:3000 in your web browser.

---

Offline Local LLM Execution (Ollama)

Analyze privacy policies locally without cloud API keys or external server dependencies:

# 1. Pull the default Llama 3.2 model
ollama pull llama3.2

# 2. Run local analysis CLI script
npm run test:ollama -- https://www.mozilla.org/en-US/privacy/

Or configure environment variables for local Python/Streamlit development:

LLM_PROVIDER=ollama
OLLAMA_BASE_URL=http://127.0.0.1:11434/v1
OLLAMA_MODEL=llama3.2

---

Benchmarks & Research

PrivacyPilot includes an automated benchmarking suite to evaluate retrieval performance, crawling latency, and extraction fidelity:

# Run retrieval accuracy benchmark (Keyword Hybrid vs Baseline)
node scripts/benchmark-retrieval.mjs

# Run crawler latency benchmark across top domains
node scripts/benchmark-scrape.mjs

# Run section extraction coverage tests
node scripts/benchmark-extraction.mjs

Academic References

1. Xie et al., USENIX Security 2025: "Evaluating Privacy Policies under Modern Privacy Laws At Scale" — Inspection & Coverage task methodologies for LLMs.
2. CompAi, ASE 2024: "Automated GDPR Completeness Checking" — Formal verification of policy requirements.
3. Springer Computing, 2024: "LLMs for Privacy Policy Analysis" — Comparative advantage of zero/few-shot LLMs over rule-based classifiers.
4. ICO Guidelines: "Layered Privacy Notices and Explicit Controls" (UK Information Commissioner's Office).

---

Project Structure

PrivacyPilot/
├── convex/                   # Convex Reactive Backend
│   ├── schema.ts             # Database schema (users, analyses, cache, userLlmSettings)
│   ├── auth.ts               # Convex Auth integration (Password, Google, GitHub)
│   ├── analyze.ts            # Keyword hybrid retrieval & LLM execution
│   ├── scrape.ts             # SSRF-safe policy crawler node action
│   ├── pipeline.ts           # End-to-end reactive orchestration pipeline
│   ├── llmSettings.ts        # User BYOK API key management & validation
│   └── lib/                  # Citation verification, security rules & rubric
├── web/                      # Next.js 15 Frontend
│   ├── app/                  # App Router pages (auth, dashboard, analyze, results)
│   ├── components/           # UI components (ScoreCard, FindingsList, EvidencePanel)
│   └── lib/                  # Client-side utility functions & type definitions
├── scripts/                  # Benchmarking, crawl testing & setup tools
├── docs/                     # Technical architecture & evaluation documentation
│   ├── ARCHITECTURE.md       # Detailed system design & network flow specifications
│   ├── EVALUATION_METHODOLOGY.md # GDPR mapping & rubric specifications
│   ├── PRODUCT_EXPERIENCE.md # Product design rationale & UX research
│   └── DEPLOYMENT.md         # Production deployment guide
├── main2.py                  # Legacy Python pipeline reference
└── app.py                    # Legacy Streamlit prototype

---

Security Specifications

• SSRF Protection: Validates every requested URL against loopback interfaces, RFC 1918 private IP subnets (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), and cloud metadata endpoints.
• Prompt Injection Containment: Scraped text is sanitized and wrapped within explicit structural delimiters prior to LLM submission.
• Credential Isolation: User API keys saved via BYOK settings are encrypted, masked upon retrieval, and accessible exclusively inside server-side Convex actions.

---

Legal Disclaimer

PrivacyPilot generates automated analytical reports based strictly on publicly available policy text. It does not constitute legal advice or a certified legal GDPR compliance audit.

---

License

Distributed under the MIT License. See LICENSE for details.