Distributed PTT (踢踢踢) Crawler
- python > 3.6.0
pip install -r requirements.txt
pip install -r scraptt-pipeline/requirements.txt
Retrieve all board names (obligatory) scrapy crawl meta
Retrieve data from certain boards: scrapy crawl ptt -a boards=movie scrapy crawl ptt -a boards=movie,Gossiping
Create docker image
docker build . -t scraptt
Create a stack
docker stack deploy -c docker-stack.yml scraptt
CockroachDB monitor page:
Scrapyd monitor page:
Remove stack:
docker stack rm scraptt
docker network rm scraptt-network
docker volume rm scraptt-db
docker volume rm scraptt
docker rm `docker ps -a | grep scraptt | awk '{print $1}'`