Name		Name	Last commit message	Last commit date
test_celery		test_celery
README.md		README.md
docker-compose.yml		docker-compose.yml
dockerfile		dockerfile
requirements.txt		requirements.txt

Repository files navigation

scaleable-crawler-with-docker-cluster

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine Check out here for full document

About

a scaleable and efficient crawelr with docker cluster , crawl million pages in 2 hours with a single machine

celery cluster crawler distributed docker python rabbitmq scaleable

95 stars

5 watching

Report repository

Releases

Packages

Used by

Contributors

Languages