Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTPS connection to scrapyd instances not possible #190

Open
franga2000 opened this issue Jul 27, 2021 · 1 comment
Open

HTTPS connection to scrapyd instances not possible #190

franga2000 opened this issue Jul 27, 2021 · 1 comment

Comments

@franga2000
Copy link

Describe the bug
Running scrapyd instances across multiple servers at the moment requires either a complicated tunneling setup (like Wireguard) or directly exposing the scrapyd HTTP interface to the public Web. In the latter case, information (including basic auth credentials) is transmitted in clear text, which is obviously a problem.

Since scrapyd uses completely standard HTTP, one easy way to secure it is to use a HTTPS reverse proxy to encrypt the connection. However, the use of plain HTTP is hard-coded in various locations in scrapydweb. This means that the best we can do is set up a HTTP→HTTPS redirect, but that still means the connection goes through in plaintext first, exposing credentials.

To Reproduce
Steps to reproduce the behavior:

  1. Set up HTTPS reverse proxy for scrapyd
  2. Enter it into scrapydweb
  3. The connection doesn't work, because it's still plain HTTP

Expected behavior
An HTTPS connection is established from the start.

@andre-response
Copy link

Were you able to solve this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants