Skip to content
BranchesTags

Repository files navigation

Duplicate POI Checker

License Python FastAPI Contributions Welcome

A fast, reliable batch-processing tool for verifying Point of Interest (POI) names against geolocation APIs. Perfect for data validation, deduplication, and location verification workflows.

🌐 Live Demo

Try the app here: https://duplicate-pois-checker.onrender.com

Quick Test

  1. Upload an Excel file with POI_Name column
  2. Enter your geolocation API token
  3. Click "Start Processing"
  4. Download results as Excel

Dashboard

Dashboard

Upload File

Upload

Results

Results

✨ Features

  • 🚀 Batch Processing - Check thousands of POIs in minutes
  • 📁 Multi-Format Support - Excel (.xlsx, .xls) and CSV files
  • ⏸️ Resume Capability - Pause and resume without reprocessing
  • 🔐 VPN Relay Support - Access WAF-protected APIs securely
  • 💾 Session Persistence - Auto-saves progress, never lose data
  • 📊 Real-Time Stats - Live dashboard with progress tracking
  • 📥 Export Results - Download as Excel with one click
  • 🔄 Auto-Cleanup - Configurable session retention
  • ⚡ Async Operations - Non-blocking API calls for speed

📋 Table of Contents

🚀 Quick Start

Prerequisites

  • Python 3.8 or higher
  • Turso Database account (free tier available)
  • API token for your geolocation service

Installation

# Clone the repository
git clone https://github.com/the-ripper77/Duplicate-POIs-Checker.git
cd Duplicate-POIs-Checker

# Install dependencies
pip install -r requirements.txt

Configuration

Create a .env file in the project root:

# Turso Database (Get from turso.tech)
TURSO_DATABASE_URL=libsql://your-db-name.turso.tech
TURSO_AUTH_TOKEN=your_auth_token_here

# Your Geolocation API
API_BASE=https://your-api.com/api/search?name=

# VPN Relay (Optional - for WAF-protected APIs)
RELAY_URL=https://your-ngrok-url.ngrok.io
RELAY_SECRET=your-secret-key-here

Run

python main.py

Visit http://localhost:8000 in your browser.

📖 Usage

1. Upload File

  • Click "Upload Excel/CSV File"
  • Select file with POI_Name column
  • Confirm upload

2. Enter API Token

  • Paste your geolocation API bearer token
  • Click "Start Processing"

3. Monitor Progress

  • Watch real-time stats on the dashboard
  • See Total / Found / Not Found / Pending counts

4. Download Results

  • Click "Download Results" when done
  • Excel file with columns: POI_Name, Found, Found_Name

🔐 VPN Relay Setup

For APIs behind WAF with VPN-only access:

Step 1: On Your Computer

Open 3 terminals:

Terminal 1 - Start Relay Server:

python relay_server.py
# Output: Listening on 0.0.0.0:8001

Terminal 2 - Expose with ngrok:

ngrok http 8001
# Copy the forwarding URL: https://xxx.ngrok.io

Terminal 3 - Connect OpenVPN:

Open OpenVPN software and connect to your configuration

Step 2: On Render

Add environment variables:

  • RELAY_URL = https://xxx.ngrok.io (from ngrok)
  • RELAY_SECRET = same secret from your .env

Done! Your Render app now accesses your API through your VPN. 🎉

🌐 Deployment

Render

# 1. Push to GitHub
git push origin main

# 2. Go to render.com
# 3. Create Web Service from your GitHub repo
# 4. Add environment variables
# 5. Deploy!

Startup Command: python main.py

Docker

docker build -t poi-checker .
docker run -p 8000:8000 --env-file .env poi-checker

Railway / Heroku

Similar process - connect GitHub repo, add env vars, deploy!

📚 API Documentation

POST /upload/

Upload Excel or CSV file with POI data.

Request:

multipart/form-data: file
Cookie: session_id (auto-generated)

Response:

{
  "success": true,
  "count": 150,
  "message": "Successfully loaded 150 POIs",
  "resume": false
}

POST /check-poi/

Check a single POI against the API.

Request:

Form: poi_id, api_token
Cookie: session_id

Response:

{
  "id": 1,
  "poi_name": "Kathmandu Valley",
  "found": "Yes",
  "found_name": "Kathmandu Valley, Bagmati, Nepal"
}

GET /stats/

Get progress statistics.

Response:

{
  "stats": {
    "total": 150,
    "found": 95,
    "not_found": 42,
    "pending": 13
  },
  "pois": [...]
}

GET /download/

Download results as Excel file.

POST /reset/

Clear all results for current session.

POST /test-api/

Test API connection with a single POI.

🛠️ Tech Stack

Layer Technology
Backend FastAPI, Uvicorn
Database Turso (SQLite HTTP)
Async aiohttp
Data Processing Pandas
Frontend HTML5, Vanilla JS, Tailwind CSS
Infrastructure Render, Turso, ngrok

🐛 Troubleshooting

Upload fails with "POI_Name column missing"

Ensure your file has a column exactly named POI_Name (case-sensitive).

"Relay error: 404"

  • ✅ Check relay_server.py is running
  • ✅ Check ngrok tunnel is active
  • ✅ Verify RELAY_URL in Render environment

"Relay error: 502"

  • ✅ OpenVPN disconnected? Reconnect.
  • ✅ Is relay_server.py still running? Restart it.
  • ✅ Check your computer internet connection.

Slow processing

  • Verify API is responding: Use /test-api/ endpoint
  • Check network latency to your API
  • Consider increasing batch size in api.py

"No session found"

Clear browser cookies or open in private/incognito mode.

🤝 Contributing

Contributions are welcome! Here's how to get started:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Development Setup

# Clone your fork
git clone https://github.com/YOUR_USERNAME/Duplicate-POIs-Checker.git

# Install dev dependencies
pip install -r requirements.txt

# Run locally
python main.py

Code Style

  • Follow PEP 8
  • Use type hints where possible
  • Write docstrings for functions
  • Add comments for complex logic

📝 Database Schema

CREATE TABLE pois (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT NOT NULL,
    poi_name TEXT,
    found TEXT DEFAULT '',
    found_name TEXT DEFAULT '',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_session_id ON pois(session_id);

⚙️ Configuration Reference

Environment Variables

Variable Required Description
TURSO_DATABASE_URL Yes Turso database URL
TURSO_AUTH_TOKEN Yes Turso authentication token
API_BASE Yes Geolocation API endpoint
RELAY_URL No ngrok relay URL (for VPN)
RELAY_SECRET No Relay authentication secret
RELAY_PORT No Relay server port (default: 8001)

Customization

Change session retention:

# In main.py
await cleanup_old_sessions(days=14)  # Default: 7 days

Modify batch size:

# In api.py
batch_size = 100  # Default: 50

Adjust API timeout:

# In api.py
timeout=aiohttp.ClientTimeout(total=60)  # Default: 50 seconds

📊 Performance

Metric Value
Batch size 50-100 POIs
API timeout 50-60 seconds
Session retention 7 days (configurable)
Max file size Limited by server memory
Concurrent users Unlimited (session-isolated)

Throughput Examples:

  • 1,000 POIs: ~10 minutes (100ms per API call)
  • 5,000 POIs: ~50 minutes
  • Parallelizable with worker processes

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Copyright 2026 Duplicate POI Checker Contributors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.

👥 Authors

  • the-ripper77 - Initial work - GitHub

🙋 Support & Community

⭐ Show Your Support

Give a ⭐️ if this project helped you!

About

Duplicate POI Checker website was built during my intern in Pathao Nepal. There was a particular problem in our team, we all have to open vs code, run py script, and sometime if server crash, we had to redo the process. it was time consuming and inefficiency. so i built a website that working in a session wise flow and cloud database as storage.

duplicate-pois-checker.onrender.com/

Topics

cssfastapihtmljsngorkpythonrailwayturso

Resources

Readme
Apache-2.0 license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages