Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate scraper from backend #86

Open
CTrando opened this issue Mar 17, 2019 · 3 comments
Open

Separate scraper from backend #86

CTrando opened this issue Mar 17, 2019 · 3 comments
Assignees
@snowme34 @CTrando

Comments

@CTrando
Copy link
Collaborator

CTrando commented Mar 17, 2019

No description provided.

CTrando added a commit that referenced this issue Mar 17, 2019
@CTrando
#86: Creating backup database file, editing config for local development
3ee13cb
@snowme34
Copy link
Collaborator

Is there any hint about the dependencies of the scraper? Any reliable reference for which files the scraper uses?

snowme34 added a commit that referenced this issue Apr 1, 2019
@snowme34
#86: fix docker network communication
2bf8b28
snowme34 added a commit that referenced this issue Apr 1, 2019
@snowme34
#86: create separated directory and Dockerfile for scraper
dfefafe
@snowme34
Copy link
Collaborator

snowme34 commented Apr 1, 2019

Ideally the scraper will be run on a totally separated server. So services like backend should have no knowledge of its presence.

snowme34 added a commit that referenced this issue Apr 1, 2019
@snowme34
#86: decouple scraper from backend
04fce31
snowme34 added a commit that referenced this issue Apr 1, 2019
@snowme34
Revert "#86: create separated directory and Dockerfile for scraper"
6ebc0b7 
This reverts commit dfefafe.
snowme34 added a commit that referenced this issue Apr 1, 2019
@snowme34
Revert "#86: fix docker network communication"
5f536c1 
This reverts commit 2bf8b28.
@snowme34
Copy link
Collaborator

snowme34 commented Apr 1, 2019

Things to worry about:

  • the config file is duplicated
  • requirements file should be pruned
  • Dockerfile should be pruned? Perhaps they can share the same layer underneath if we don't touch it

Things to do in next stage:

  • use images from docker hub for productoin

CTrando pushed a commit that referenced this issue Apr 3, 2019
@snowme34 @CTrando
#86: decouple scraper from backend
59321d7
CTrando added a commit that referenced this issue Apr 3, 2019
@CTrando
#86: Allowing the scraper to work with multiple quarters - backend is…
0d19ed8 
… not yet hooked up yet
snowme34 added a commit that referenced this issue Apr 9, 2019
@snowme34
#86 trying to add default quarter support for backend
4cc3bb1
snowme34 added a commit that referenced this issue Apr 9, 2019
@snowme34
#86 fix import path
490ca06
snowme34 added a commit that referenced this issue Apr 19, 2019
@snowme34
#86 trying to prune Dockerfile
682a549
snowme34 added a commit that referenced this issue Apr 19, 2019
@snowme34
#86 working on pruning requirements.txt
4452996
snowme34 added a commit that referenced this issue Apr 19, 2019
@snowme34
#86 add note for possible caching bug
3c6a080
snowme34 added a commit that referenced this issue Apr 19, 2019
@snowme34
#86 disable WI19, no one cares
9ad923d
snowme34 added a commit that referenced this issue Apr 19, 2019
@snowme34
#86 add mysql non-root user, based on config.ini
498ce1e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@snowme34 snowme34

@CTrando CTrando

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@snowme34 @CTrando