Production-ready web scraping platform with PostgreSQL, WebUI, REST API, and price monitoring.
⭐ View on GitHub# 1. Clone the repository
git clone https://github.com/blixten85/scraper.git
cd scraper
# 2. Create .env file with your settings
cp .env.example .env
nano .env
# 3. Create directories and set permissions
mkdir -p ${DOCKER}/scraper/{logs,postgres,playwright-cache}
sudo chown -R 999:999 ${DOCKER}/scraper/postgres
# 4. Create Discord webhook (optional)
echo "your-discord-webhook-url" > ${CONFIG}/.secrets/discord_webhook
# 5. Start the platform
docker compose up -d
# 6. Open WebUI
# http://localhost:3000| Service | Port | Description |
|---|---|---|
scraper_db | 5432 (internal) | PostgreSQL database |
scraper_engine | 5001 (internal) | Main engine - scrapes sites |
scraper_api | 8000 | REST API + Swagger docs |
scraper_webui | 3000 | Web interface |
scraper_alerts | - | Discord notifications |
# Get all products
curl -H "X-API-Key: ${API_KEY}" http://localhost:8000/products
# Search products
curl -H "X-API-Key: ${API_KEY}" "http://localhost:8000/products?search=RTX"
# Get price drops
curl -H "X-API-Key: ${API_KEY}" "http://localhost:8000/deals?min_drop_percent=10"
# Export to CSV
curl -H "X-API-Key: ${API_KEY}" http://localhost:8000/export/csv > products.csv| Version |  |
| Status | ✅ Production Ready |
| Last Updated |  |
MIT - see LICENSE
Created by blixten85 🚀