Skip to content

使用Flask、MongoDB、Redis、Celery與Docker,定時向Booking.com爬取住房資訊,同時撰寫相關API,進行住房查詢,並且導入Swagger進行API的測試,加入Celery Flower和Grafana監測隊列任務與API效能,以及使用Logstash蒐集Nginx日誌。

Notifications You must be signed in to change notification settings

LinYuMingBejing/booking.com

Repository files navigation

Booking.com

Enviroment:

  • Ubuntu: 16.04
  • Python: 3.6.2
  • Backend: Flask
  • Test: Swagger
  • Database: MongoDB, Redis
  • Asynchronous: Celery (Redis)
  • Monitoring Tool: Supervisor, Prometheus, Grafana
  • Collecting Logs: Logstash

Install Docker

$ sudo apt-get update

$ sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

$ sudo apt-key fingerprint 0EBFCD88

$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu
$(lsb_release -cs)
stable"

$ sudo apt-get update

$ sudo apt-get install docker-ce docker-ce-cli containerd.io

Install Docker Compose

$ sudo apt install docker-compose

Deploy Booking.com Application

  • Deploy
$ cd ./docker_stg
$ sudo docker-compose up --build -d
  • Check docker process
$ sudo docker ps
  • Get docker log
$ sudo docker-compose logs --tail=20 -f flask

Swagger

swagger

Monitor process

$ supervisorctl

supervisorctl

supervisor_web

Monitor uwsgi application

uwsgitop 127.0.0.1:5002

uwsgitop

Monitor celery worker

flower

Prometheus Monitor

Prometheus

Grafana Monitor

Grafana

Nginx Log Monitor

Note1: How to allow certain ips connect mongodb?

  • vim /etc/mongod.conf
    # bindIp: 127.0.0.1
    bindIp: 0.0.0.0  
  • restart mongodb
$ sudo service mongod restart
$ sudo service mongod status
  • firewall settings
$ sudo ufw enable
$ sudo ufw deny  from 192.168.18.0/24 to any port 27017
$ sudo ufw allow from 192.168.18.0/24 to any port 27017
$ sudo ufw status

Note2: How to create multiple user on mongodb?

  • create user
> use booking;
switched to db booking
> db.createUser(
  {
    user: "dbadmin",
    pwd: "StrongPassword",
    roles: [ { role: "readWrite", db: "booking" } ]
  }
)
> exit
bye
  • vim /etc/mongod.conf
security:
  authorization: enabled
  • restart mongodb
$ sudo service mongod restart
$ sudo service mongod status
Reference

About

使用Flask、MongoDB、Redis、Celery與Docker,定時向Booking.com爬取住房資訊,同時撰寫相關API,進行住房查詢,並且導入Swagger進行API的測試,加入Celery Flower和Grafana監測隊列任務與API效能,以及使用Logstash蒐集Nginx日誌。

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published