Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Add proxy auth test coverage (#496) * Cover all proxy auth scenarios * Add test_proxy_auth_works_with_mixed_case_basic_string * Update tox from 3.21.3 to 3.21.4 (#497) * Update autopep8 from 1.5.4 to 1.5.5 (#499) * Update pylint from 2.6.0 to 2.6.2 (#501) * Fix mypy errors (#504) * Update tox from 3.21.4 to 3.22.0 (#502) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update mypy from 0.790 to 0.812 (#503) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update pylint from 2.6.2 to 2.7.1 (#506) * Update coverage from 5.4 to 5.5 (#508) * Update pylint from 2.7.1 to 2.7.2 (#509) * Update tox from 3.22.0 to 3.23.0 (#510) * Update twine from 3.3.0 to 3.4.1 (#517) * Update flake8 from 3.8.4 to 3.9.0 (#514) * Update autopep8 from 1.5.5 to 1.5.6 (#516) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update py-spy from 0.3.3 to 0.3.5 (#522) * DeepSource: Code quality issues (#523) * Update pylint from 2.7.2 to 2.7.3 (#524) * Bump y18n from 3.2.1 to 3.2.2 in /dashboard (#526) Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2. - [Release notes](https://github.com/yargs/y18n/releases) - [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md) - [Commits](https://github.com/yargs/y18n/commits) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update pylint from 2.7.3 to 2.7.4 (#527) * Move wheel package to testing (#531) * Update pytest from 6.2.2 to 6.2.3 (#532) * Update flake8 from 3.9.0 to 3.9.1 (#538) * Update rope from 0.18.0 to 0.19.0 (#539) * Update pylint from 2.7.4 to 2.8.0 (#540) * Update pylint from 2.8.0 to 2.8.1 (#541) * Update pylint from 2.8.1 to 2.8.2 (#542) * Update autopep8 from 1.5.6 to 1.5.7 (#543) * Update typing-extensions from 3.7.4.3 to 3.10.0.0 (#544) * Update pytest from 6.2.3 to 6.2.4 (#545) * Update tox from 3.23.0 to 3.23.1 (#546) * Update py-spy from 0.3.5 to 0.3.6 (#547) * Update flake8 from 3.9.1 to 3.9.2 (#549) * Bump hosted-git-info from 2.8.5 to 2.8.9 in /dashboard (#548) Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.5 to 2.8.9. - [Release notes](https://github.com/npm/hosted-git-info/releases) - [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md) - [Commits](npm/hosted-git-info@v2.8.5...v2.8.9) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Bump lodash from 4.17.19 to 4.17.21 in /dashboard (#550) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.19 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.19...4.17.21) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update pytest-cov from 2.11.1 to 2.12.0 (#552) * Update py-spy from 0.3.6 to 0.3.7 (#555) * Bump ws from 7.4.0 to 7.4.6 in /dashboard (#556) Bumps [ws](https://github.com/websockets/ws) from 7.4.0 to 7.4.6. - [Release notes](https://github.com/websockets/ws/releases) - [Commits](websockets/ws@7.4.0...7.4.6) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update pylint from 2.8.2 to 2.8.3 (#558) * Update pytest-cov from 2.12.0 to 2.12.1 (#561) * Bump glob-parent from 5.1.1 to 5.1.2 in /dashboard (#564) Bumps [glob-parent](https://github.com/gulpjs/glob-parent) from 5.1.1 to 5.1.2. - [Release notes](https://github.com/gulpjs/glob-parent/releases) - [Changelog](https://github.com/gulpjs/glob-parent/blob/main/CHANGELOG.md) - [Commits](gulpjs/glob-parent@v5.1.1...v5.1.2) --- updated-dependencies: - dependency-name: glob-parent dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update pylint from 2.8.3 to 2.9.3 (#573) * Update tox from 3.23.1 to 3.24.0 (#575) * Update twine from 3.4.1 to 3.4.2 (#576) * Update pylint from 2.9.3 to 2.9.5 (#577) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update wheel from 0.36.2 to 0.37.0 (#585) * Update codecov from 2.1.11 to 2.1.12 (#582) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update typing-extensions from 3.10.0.0 to 3.10.0.2 (#599) * Update pytest from 6.2.4 to 6.2.5 (#598) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update tox from 3.24.0 to 3.24.3 (#592) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update pylint from 2.9.5 to 2.10.2 (#591) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Bump path-parse from 1.0.6 to 1.0.7 in /dashboard (#586) Bumps [path-parse](https://github.com/jbgutierrez/path-parse) from 1.0.6 to 1.0.7. - [Release notes](https://github.com/jbgutierrez/path-parse/releases) - [Commits](https://github.com/jbgutierrez/path-parse/commits/v1.0.7) --- updated-dependencies: - dependency-name: path-parse dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update py-spy from 0.3.7 to 0.3.9 (#604) * Update rope from 0.19.0 to 0.20.1 (#611) * Update pylint from 2.10.2 to 2.11.1 (#609) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update tox from 3.24.3 to 3.24.4 (#607) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * add paramiko types for mypy compliance (#613) * Update py-spy from 0.3.9 to 0.3.10 (#616) * Update coverage from 5.5 to 6.0 (#618) * Update pytest-cov from 2.12.1 to 3.0.0 (#619) * Update types-paramiko from 2.7.0 to 2.7.1 (#620) * Update coverage from 6.0 to 6.0.1 (#623) * Update paramiko from 2.7.2 to 2.8.0 (#624) * Update coverage from 6.0.1 to 6.0.2 (#628) * Update types-paramiko from 2.7.1 to 2.7.2 (#629) * Update types-paramiko from 2.7.2 to 2.7.3 (#630) * Update rope from 0.20.1 to 0.21.0 (#631) * Update flake8 from 3.9.2 to 4.0.1 (#627) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Add support for 3.10 (#637) * Add support for 3.10 * Upgrade to v2 actions * v3.4.0 (#638) * Build docker from 3.10-alpine * Bump version to 3.4.0 * Add instructions for how to run dashboard * Order of menu * Override dashboard png path until submitted * Add some doc string for top-level Proxy class. Also some TODOs and warnings regarding PID file overwrite * Allow HttpProxyBasePlugin implementations to register custom descriptors for read/write events * Remove hardcoded adblock regex into json config. Update upstream filter to block facebook, not google * ProxyPoolPlugin and ReverseProxyPlugin must now be updated to use get/read/write descriptor APIs * Add get/read/write descriptor API for HttpWebServerBasePlugin too * Surface actual listening port via flags.port * Update autopep8 from 1.5.7 to 1.6.0 (#632) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Fix the `typing_extensions` runtime dependency version (#641) * Only use `typing_extensions` below Python 3.8 * Rely on unconstrained `typing-extensions` version * Update coverage from 6.0.2 to 6.1 (#640) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Fix path to dashboard.png (#643) * [ImgBot] Optimize images (#644) *Total -- 1,253.27kb -> 774.20kb (38.23%) /Dashboard.png -- 1,072.21kb -> 617.66kb (42.39%) /ProxyPy.png -- 56.83kb -> 38.15kb (32.86%) /shortlink.gif -- 124.23kb -> 118.39kb (4.7%) Signed-off-by: ImgBotApp <ImgBotHelp@gmail.com> Co-authored-by: ImgBotApp <ImgBotHelp@gmail.com> * Update coverage from 6.1 to 6.1.1 (#646) * Async proxy pool, Event manager, Custom access log, Expose loop to plugins (#645) * Async proxy pool * Async proxy pool * Late upstream initialization and exception guards * Close upstream proxy connection on client connection close * Refactor into EventManager * Fix tests accounting in the event manager * Ensure each process initializes logger * pragma no cover * Teardown connection when proxy pool upstream proxy closes * Add ability to customize access log format and add additional context to it * Maintain total size for response bytes in access logs * Fix tests broken due to new plugin methods missing mock * Update pubsub_eventing to use EventManager to avoid entire bootstrapping step * Add tox envs for building dists via PEP 517 (#647) * Invoke self-install via PEP517 in the CI * Add tox envs for building dists via PEP 517 * Add linting dists via tox to GHA * Upgrade `master` to `develop` in `setup.py` * Simplify `python_requires` in `setup.py` * Convert dynamic `setup.py` into static `setup.cfg` * Revert "Upgrade `master` to `develop` in `setup.py`" (#650) This reverts commit 06f6f56. * Add a no-op check job to GHA for branch protection (#652) This patch adds an "empty" job that depends on all the important ones making it possible to use just this `check` job in the branch protection settings. This reduces the maintenance burden by preventing the need to update these settings on any changes to the job declarations. * Add a config for YAMLLint (#653) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Add a config for flake8 (#654) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Correct spelling mistakes caught by `codespell` (#656) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Add a config for pylint (#655) * Add an initial auto-generated PyLint config * Align pylint line length settings in with flake8 * Colorize the pylint report output * Disable all currently violated PyLint rules Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Organize the linting setup around tox+pre-commit (#657) * Add a no-op check job to GHA for branch protection This patch adds an "empty" job that depends on all the important ones making it possible to use just this `check` job in the branch protection settings. This reduces the maintenance burden by preventing the need to update these settings on any changes to the job declarations. * Add a config for YAMLLint * Add a config for flake8 * Add an initial auto-generated PyLint config * Align pylint line length settings in with flake8 * Colorize the pylint report output * Correct spelling mistakes caught by `codespell` * Disable all currently violated PyLint rules * Start managing the linters setup with pre-commit Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Enable yamllint for all previously ignored files (#658) * Add autocancellation of the stale PR GHA jobs (#663) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update twine from 3.4.2 to 3.5.0 (#665) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Enable the `add-trailing-comma` pre-commit fixer (#661) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Migrate the pytest invocation to `tox` (#662) * Fix the `commands` setting in the tox config * Wire up `pytest` invocations via tox * Pre-install mypy deps in `pre-commit.ci` (#666) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Fix the YTT201 flake8 violation (#667) This approach makes sure that if there's ever Python 4, it wouldn't fall back to the Python 2 behavior silently. Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Make names in the GHA lib workflow short (#669) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Update outdated sections of the codebase (#670) * Remove autopep8, is redundant now after recent CI changes * Add pyenv .python-version to .gitignore * Update year * Add lib-pytest target so that pytest can run in isolation * Add git-push hook which will also run the lint. By default now git-pre-commit hook will only run pytest. * Update outdated sections of README * Update requirement to match setup.cfg install_requires * Deprecate proxy.start and TestCase.PROXY_PORT Proxy port during test is now available as self.PROXY.pool.flags.port. Also now TestCase utilize ephemeral port strategy instead of calling get_available_port utility method. * Rename to git-pre-push * Ideally public repo dont require CODECOV_TOKEN but codecov integration is broken since introduction of codecov-action@v2 (instead of codecov binary invocation) * Issue is possibly with codecov@v2 action, fallback to codecov. See https://github.com/abhinavsingh/proxy.py/runs/4110423084\?check_suite_focus\=true and codecov/uploader#223 * Revert back to v2 * Make pytest emit XML coverage (#673) This patch should make GHA report coverage to codecov properly. * CustomDnsResolver plugin, CloudflareDnsResolver plugin, Allow plugins to configure network interface (#671) * Add CustomDnsResolver plugin. Addresses #535 and #664 * Add cloudflare DNS resolver plugin * Lint fixes * Add tests for missing core modules (#674) * Rename is_py3 to is_py2 for more logical guard * Add stubs for missing tests, add few more tests for core modules * Lint fixes * Line too long fix * Remove unnecessary KeyboardInterrupt * Consistent workflow names * Update homebrew formulae. Doesnt seems to work now * test_enable_dashboard and test_enable_events * test_enable_dashboard and test_enable_events * Fix problem where empty plugin string was passed as plugin module * test_enable_devtools and remove redundant guards for None and "" which was there due to a bug * Use core loop for reverse proxy async IO operations (#675) * Make reverse proxy plugin use proxy.py core loop for async io operations * Address lint errors * Deprecate on_websocket_close and replace with on_client_connection_close * Lint fixes * Retry on SSLWantReadError and SSLWantWriteError * Collect coverage for `tests/` (#677) This is necessary to make sure all the tests actually get executed somewhere and allows finding dead code in this area. * Allow pylint to utilize all available CPU cores (#676) This is a nice addition to the linter's responsiveness. Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Add a config for Coverage.py (#679) * Add a config for pytest (#680) * Ignore a `DeprecationWarning` in `pytest` config (#683) This was added in Python 3.10 and needs to be dealt with properly. * Adopt BaseTcpServerHandler within HttpProtocolHandler (#681) * Rename .server to .upstream * Lint fixes * Mark internal methods with _ prefix * Fix broken test * lint changes * Wah, double client :D * Avoid selector initialization for threadless mode * remove unused imports * Now HttpProtocolHandler implements BaseTcpServerHandler * Consistent return and guard againt upstream.closed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * AcceptorPool as context manager * Group multiprocessing imports together * Use com.jaxl bundle identifier as proxy.py will eventually move under jaxl org * revisit devtools integration :) * Emit all necessary events for devtools integration * Lint fixes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Inline the dist description in pkg metadata (#684) This is necessary because setuptools is about to deprecate supporting LF in one-line metadata fields. Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Add a pytest-based test for catching import loops (#678) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Test built artifacts in the CI instead of Git src (#682) This approach helps make sure that what is tested in the project is the same as what the end-users get installed on their machines. It will catch failures to include important failes into the packages that get uploaded to PyPI. Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * Fix python 3.10 @ ubuntu pytest ignore (#685) * Use suggested fix in #683 to remove pytest ignore * Handle should not flush now which can block, instead let base tcp handler do its magic * test speed up, doc string, logging enhancements * Move macOS to the end of workflows * Fix mypy warnings * must_flush_before_shutdown until entire client buffer has been flushed (#686) * Readme Updates (#687) * FilterByUpstreamHostPlugin now uses facebook as example * Advertise that cloudflare plugin require httpx library * Advertise Dashboard as WIP * More clarity around how plugins can be loaded * Advertise how to get direct access to Chrome DevTools websocket endpoint * Move plugin developer and contributor guide below the fold * Update README.md Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * bash to console and consistent prompt sign * Remove pytest commented out ignore clause * Add dummy CustomNetworkInterface plugin documentation Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * Response bodies can be empty or missing + HttpParser refactoring (#688) * Responses can have None body, remove assertions, update modify chunk plugin to not modify chunks for responses with no content * Address mypy warning after removing assertion * Reusable get_body_or_chunks * Order methods by public/private, mark private ones with _ prefix * HttpParser.url deprecation notice (renamed to _url). Add zero-copy todo * [ModifyChunkResponsePlugin] Only bail out of body is not expected (#690) * DEFAULT_CA_FILE is now certifi/cacert.pem (#691) * Add FAQ: OSError when wrapping client for TLS Interception * Silence exception log for several valid "cert verification failed" by client during tls interception * Lint checks * Move exception handling within wrap_server/wrap_client methods * Lint fixes * Use certifi/cacert.pem as default --ca-file flag value * Address tests after DEFAULT_CA_FILE change * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Expose pre-commit tool hooks execution times (#692) * Work (#693) * Refactor work acceptor and executor * Lint fixes * Fix expression-not-assigned pylint error * Pool (#694) * Refactor pool * mypy fixes * Fix import (relative) * Add WebScraper example skeleton & ConnectionPool skeleton * Add ConnectionPool class * Integrate ConnectionPool with proxy server (experimental) * Lint fixes * Remove unused imports. TODO: Put pool behind a flag. Default to false for now * Make ConnectionPool multiprocess safe. Later we want to make it safe but without using locks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unused imports * Return created flag from acquire * Guard connection pool behind --enable-conn-pool flag * Flag belongs within connection pool class * spelling * self.upstream = None only for pool config Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Top-level notion of work not client (#695) * Top-level notion of work not client * Update ssl echo server example * Rename `Proxy.initialize` as `FlagParser.initialize` (#696) * Move Proxy.initialize within FlagParser.initialize. Also move other staticmethods from within proxy class into utils * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused imports * Fix `import-outside-toplevel` error * add `make lib-flake8` and `make lib-mypy` targets Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add a `--unix-socket-path` flag (#697) * Add a `--unix-socket-path` flag. When available `--hostname` and `--port` flags are ignored. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * `print` statement is allowed only in `flags.py` and `version-check.py`. All other places must use a `logger` instance * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add guard for `AF_UNIX` on Windows * Comment out assertion on Windows for now Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Trigger workflows only when necessary file changes are detected (#699) * Trigger workflows only when necessary file changes are detected * Address yamllint * Trigger workflows even when workflow file itself is changed * Also include `examples` directory, may be one day we will have tests for them * `brew` and `docker` workflows must also only execute for python file changes. This will skip them for pure README.md changes * Execute `lib` workflows for package artifact changes * Disable static web server test on GHA environment (flaky on Ubuntu) (#700) * Add `lib-dep` makefile target (#701) * Put core flags where they belong (#702) * Move flags to where they belong * Move `get_default_plugins` within FlagParser as it depends upon args TODO: We need plugin dependency system * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update types-paramiko from 2.7.3 to 2.7.4 (#704) * Fix `--enable-dashboard` flags (#707) * Expose within __all__ * Enable `--numprocesses=auto` for `pytest.ini` * make lib-lint * Also consider `--plugins` flag when bootstrapping plugins * Add `from .dashboard import ProxyDashboard` in top-level `__init__.py` to make `ProxyDashboard` flags auto discoverable * Move `--enable-dashboard` to top-level * Move logging utility within `Logger` class * Consider comma separated --plugin and --plugins during discover_plugins * Refactor plugin related utilities in Plugins module * mypy and lint * Fix unused import * Safe to use tempdir on Github actions to avoid race conditions??? * pki (generically disk based file) based tests are flaky on macOS under parallel execution * Move pid file write/remove within `AcceptorPool` (#708) * Move pid file write/remove within AcceptorPool * Remove unused * `EventManager` is also a context manager (#709) * `EventManager` is also a context manager * unused * Rename `EventManager.event_queue` to `EventManager.queue` * `--threadless` default for `Python 3.8+` on `mac` and `linux` (#710) * Explicit `multiprocessing.Manager.shutdown` Multiprocessing manager is used within eventing core. From doc, it appears to start a BaseManager which starts a server???? Seriously???? Anyways, using multiprocessing manager is a PITA and mistake, as it doesn't even give us performance we expect. Our proxy server can handle more requests than what multiprocess manager can exchange between processes. * `--threadless is now ON by default for `Python 3.8+` on `mac` and `linux` environments * Clarity around why multiprocessing.Manager must be deprecated * Add `--threaded` flag which can be used to fallback for environments where `--threadless` is now default * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * never used * Update README * Use `threaded=True` in tests which were written for threaded model * Fix issue where sharing manager between global event queue and subscriber can lead to TypeError Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * `OP_NO_TLSv1_1` by default for upstream connection negotiations (#712) * print mode via acceptor pool * `OP_NO_TLSv1_1` by default for upstream connection negotiations. Fixes #639 * Proper fix for flaky static web server test. Diff in payload was due to a different compression algorithm being used. * mypy fixes * Add more context in intro (#713) * Add more context in intro * Take another pass * Add threadless in log section * Add an explicit config for darglint (#717) * Add `--num-acceptors` flag + Allow `work_klass` via `Proxy` context manager kwargs (#714) * Allow overriding work_klass via Proxy context manager kwargs * Decouple acceptor and executor pools * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add `--num_acceptors` flag and better load balancing * Remove unused * Lint errors * Another arg not kwarg * Move start work staticmethods within ExecutorPool * mypy fixes * Update README with `--num-acceptors` flag * Rename `Proxy.pool` to `Proxy.acceptors` * Add SetupShutdownContextManager abstraction * Match --num-acceptors logic with PR description * Rename executor utility methods and add docstring * Remove work_klass from constructors and pass it via flags * Update docstring for pools as they no longer accept a work_klass argument * Turn work_klass into a flag. main() no longer accepts input_args (only kwargs opts). Similarly, Proxy doesnt accept any input_args now (only kwargs opts) * Expose default work klass in README * Expose `HttpProtocolHandler` and `HttpProtocolHandlerPlugin` within `proxy.http` module * Start to fix tests * Fix tests * mypy and flake8 * Trailing comma * Remove unused var * Unused arg * uff Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update badges to match new GHA workflows (#718) * Add `Listener`, Web server close on header, use `Pipe` instead of `Manager` in eventing core (#720) * Abstract out a Listener class * unused * Use connection instead of manager queue * For web close connection of client requested via headers * Remove eventing WIP module * Sub and Unsub ack * Fix tests * mypy and flake8 * comma * Move callback within EventSubscriber constructor * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Skip test_unix_path_listener on Windows * Spelling fix Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Test cov (#721) * nocover for abstract classes * Add event manager test * Assert call args * Sponsored by `Jaxl Innovations Private Limited` * Allow `--plugins` flag to be used multiple times (#725) * deprecate server_file_or_404 * Optionally compress static content. Currently only if content length higher than 300 * trailing comma * Allow `--plugins` flag to be used multiple times Following are valid invocation: 1) `--plugins A` 2) `--plugins A,B` 3) `--plugins A --plugins B` 4) `--plugins A,B --plugins C` * mypy * Flake8 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * correct type * Add `HttpParser.is_https_tunnel()` utility method * mypy * lint checks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add `--proxy-pool` flag (#727) * Add `--proxy-pool` flag * lint checks * Custom Url Parser (#730) * Custom Url parser for our needs * lint fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix proxy_pool plugin as scheme can be None if not present in the Url * Address the ambiguous ipv6:port scenario along with valid cases * lint checks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * docstring * Abstract into `http.parser` module * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix #398 HTTP/1.0 related issue * lint checks Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add more info in log context (#732) * Provide more info in log context, ideally we could just pass client/upstream/request/response objects but for now passing dict is ok * lint checks * Allow `access_log` format override by web plugins (#733) * Return DEFAULT_404_RESPONSE by default from static server when file doesnt exist * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix web server with proxy test Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add `--auth-plugin` flag to override default basic auth plugin (#734) * Allow override of `--auth-plugin`. This can be used in scenarios where a hardcoded `--basic-auth` credentials cannot be used across all your users. May-be you want to connect to a database before authorizing the request. For all such scenario, disable in-build default auth plugin by providing `--auth-plugin` flag. * Allow usage of `--auth-plugin` flag without having to provide `--basic-auth` flag. When `--basic-auth` flag is used, default auth plugin will load. When `--basic-auth` and `--auth-plugin` flag is used, custom auth plugin will load * Address test_main broken after auth_plugin flag * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update README * Fix long line Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Switch versioning to be SCM-based (#715) Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> * `--enable-proxy-protocol` : HAProxy Protocol v1 (#735) * Introduce `--haproxy-protocol` flag * Complete proxy protocol v1 implementation, enable using `--enable-proxy-protocol` flag * link checks * Advertise support for haproxy protocol in readme * Add make target `lib-scm-version` * `make lib-version` is now `make lib-check` * Dont enforce -dev part of version within README * Add provision to update readme flags using check * Wrap help text within console * Add closing ticks * Remove verbose logging and update homebrew formulae (may be fixed?) * Reuse a pre-existing `version_tuple` from SCM * No abstract method for proxy plugin (#738) * Remove abstractmethod for proxy plugin base class, remove unused methods from bundled plugins * Move httpStatusCodes, httpMethods and Url within top-level proxy.http package * Implement publishing via GHA (#716) * Enable release-testpypi (#741) * Fix dist version in CI/CD on pushes to `develop` (#743) * Fix homebrew formulae URL to install from git and not zip because of scm changes (#744) * Add an example of loading external plugins * Move GHA e2e integration job to pytest (#746) Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Make Git archives `setuptools-scm` compatible (#737) Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Add initial Sphinx docs * Handle invalid/malformed data from clients in HttpParser (#740) * add validation in _process_line in parser.py add validation in _process_line in proxy/http/parser/parser.py * quick fail when parsing request quick fail when parsing request add test case for parsing invalid http request * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unnecessary checks and empty line remove unnecessary checks and empty line * minor fix minor fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * solve exception expression conflict solve exception expression conflict * use NotImplementedError temporary measure use change HttpProtocolException to NotImplementedError for a temporary measure * change exception type in test change exception type in test * remove unnecessary import remove unnecessary import Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Add all IANA assigned HTTP method names (#751) * Ignore `docs/_build` * Remove `v` prefix from VERSION. Also added a `lib-doc` target * Raise a `ValueError` instead of `NotImplementedError` * Add all registered http method verbs * Generate `_scm_version.py` on every `make` invocation. Fix `v` prefix bug. * `+proxy` for emails * Make explicit that this script writes to file * `PROXY_AGENT_HEADER_VALUE` still needs the `v` :) * Fix Makefile to use tox for docs (#752) * Fix Makefile to use tox for docs * Resolve `open` based upon operating system Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Include GitHub's CoC and security in Sphinx docs (#755) Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Add community contribution guidelines (#757) * [IntegrationTest] Use `127.0.0.1` as target address and a random port (#756) * Use `127.0.0.1` as target address and a random port * Fix spellcheck-docs * Transparent `ProxyPy.png` (#759) * Add `TcpUpstreamConnectionHandler` class (#760) * Add `TcpUpstreamConnectionHandler` which can be used as standalone or as mixin * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * `TcpUpstreamConnectionHandler` is now an abstract class * Fix mypy * `nitpick_ignore` the `proxy.core.base.tcp_upstream.TcpUpstreamConnectionHandler` class * Add mypy exception for now * Fix flake * Fix docstring Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add `.vscode` settings (#761) * Added `DEFAULT_SELECTOR_SELECT_TIMEOUT` (#762) * Enhancements (#763) * Update `make lib-profile` * Optimize `utils.find_http_line` * Pass work to executors within their own multiprocessing lock * Fix tests * Add `(_py_class_role, Url)` to fix rtfd :D * Run push workflow only for push to master and develop branch (#764) * Run push workflow only for push to master and develop branch * remove comments from `settings.json` * Single workflow (#765) * To optimize pull all workflows within check guard * give space between version and to * Move all other workflow under `other-checks` guard * Dont run brew/dashboard/docker until lib checks have passed * Additional check bridging slows it down * Name diff via emoji * More emojis * 🐳 * Per GitHub it will try to maximize, lets fallback to GHA behavior * False position (not indented) * Remove tagline from readme, its in hero image now * Remove badge for deprecated workflows * [ImgBot] Optimize images (#766) /ProxyPy.png -- 985.28kb -> 884.17kb (10.26%) Signed-off-by: ImgBotApp <ImgBotHelp@gmail.com> Co-authored-by: ImgBotApp <ImgBotHelp@gmail.com> * Acceptors performance (#767) * Use threads for delegation. Now `run_once` lock before `accept` not `select` * Add support to use master proxy within proxy pool plugin. When used, proxy pool plugin will be a no-op for the master node * Fix acceptor tests now that mask is being used * Use `cached_property` for web server routes * Use `select(timeout=1)` otherwise acceptor wont join if total blocking * mypy, flake, doc spell fixes * R0205: Class `cached_property` inherits from object, can be safely removed from bases in python3 (useless-object-inheritance) * Include contributing guidelines in Sphinx docs (#771) * Fix tox invocation in the doc make target (#772) Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Declare project URLs in Python package metadata (#778) * Declare project URLs in Python package metadata * Add link to GHA discussions Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Async `get_events`, `handle_event`, `handle_readables`, `handle_writables` (#769) * Asynchronous `handle_event` and `LocalExecutor` thread * Bail out on first task completion * mypy * Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded) * Print open file diff from `benchmark.sh` * Add `--local-executor` flag, disabled by default for now until tests are updated * Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now) * Async `get_events` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address tests after async changes * mypy and flake8 * spelldoc * `check.py` and trailing comma * Rename to `_assertions.py` * Add missing `pytest-mock` and `pytest-asyncio` deps * Add `pytest-mock` to `pylint` deps * Correct use of `parameterize` and add `PT007` to flake8 ignores * Fix mypy hints broken for `< Python3.9` * Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix for pre-python-3.9 versions * `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions * Install setuptools via `lib-dep` until we recommend editable install * Deprecate support for `Python 3.6` * Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929 * Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906 * Make `Threadless` agnostic of `multiprocessing.Process` * Acceptors must dispatch to local executor in non-blocking fashion * No daemon for executor processes and fix shutdown logic * Only return fds from `_selected_events` not all events data * Refactor logic * Prefix private methods with `_` * `work_queue` and not `client_queue` * Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor` * Make `LocalExecutor` agnostic of `threading.Thread` * `LocalExecutor` now implements `Threadless` * `get_events` and `get_descriptors` now must return int and not sock. `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify` * Fix `main` tests * Apply suggestions from code review Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * Apply code review recommendations manually * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert back `Any` and use `addr or None` * Address `flake8` * Update tests to use `fileno` * Fix doc build * Fix doc spell, use tear down and not teardown * Doc updates * Add back support for `Python 3.6` * Acceptors dont need loop initialization * On Python 3.6 `asyncio.new_event_loop()` is necessary * Make doc happy * `--threaded` needs a new event loop for 3.7 too * Always use `asyncio.new_event_loop()` for threaded mode Added e2e integration tests (subprocess & curl) for all modes. * Lint fixes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * Prune unnecessary directories from `sdist` package (#779) * Prune unnecessary directories from `sdist` package * Remove `LICENSE` & `README.md` from `MANIFEST.in` Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * Explicitly ignore top-level images and include `.github` folder Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * Use proper email for the Git user in GHA (#781) * Use proper email for the Git user in GHA * Use an action to set the GHA user in Git * Publish a GitHub Release after tagging (#782) Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Create `dependabot.yml` (#783) * Create dependabot.yml * Lint fixes * Yamllint * Reviewers fix * Optimize (#780) * Optimize `find_http_line` which is in critical path * Update benchmark results * Keep the loop hot, TCP no delay, cleanup inactive check periodically * Check for shutdown signal with tick * Use non-reentrant `NonBlockingQueue` implementation instead of `queue.Queue` * Fix listener test * lint and doc * Convert `recv` errors as warnings and not exceptions (#787) * Update codecov config to include separate tests and lib sub-projects (#788) * Update codecov config to include separate tests and lib sub-projects * Yaml lint * Bump actions/cache from 2.1.5 to 2.1.7 (#784) Bumps [actions/cache](https://github.com/actions/cache) from 2.1.5 to 2.1.7. - [Release notes](https://github.com/actions/cache/releases) - [Commits](actions/cache@v2.1.5...v2.1.7) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * pip prod(deps): bump rope from 0.21.0 to 0.22.0 (#785) Bumps [rope](https://github.com/python-rope/rope) from 0.21.0 to 0.22.0. - [Release notes](https://github.com/python-rope/rope/releases) - [Changelog](https://github.com/python-rope/rope/blob/master/CHANGELOG.md) - [Commits](python-rope/rope@0.21.0...0.22.0) --- updated-dependencies: - dependency-name: rope dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * npm: bump chrome-devtools-frontend in /dashboard (#786) Bumps [chrome-devtools-frontend](https://github.com/ChromeDevTools/devtools-frontend) from 1.0.827632 to 1.0.944903. - [Release notes](https://github.com/ChromeDevTools/devtools-frontend/releases) - [Changelog](https://github.com/ChromeDevTools/devtools-frontend/blob/main/docs/release_management.md) - [Commits](https://github.com/ChromeDevTools/devtools-frontend/commits) --- updated-dependencies: - dependency-name: chrome-devtools-frontend dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * pip prod(deps): bump types-paramiko from 2.7.4 to 2.8.1 (#799) Bumps [types-paramiko](https://github.com/python/typeshed) from 2.7.4 to 2.8.1. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-paramiko dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump eslint-plugin-import from 2.22.1 to 2.25.3 in /dashboard (#798) Bumps [eslint-plugin-import](https://github.com/import-js/eslint-plugin-import) from 2.22.1 to 2.25.3. - [Release notes](https://github.com/import-js/eslint-plugin-import/releases) - [Changelog](https://github.com/import-js/eslint-plugin-import/blob/main/CHANGELOG.md) - [Commits](import-js/eslint-plugin-import@v2.22.1...v2.25.3) --- updated-dependencies: - dependency-name: eslint-plugin-import dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Build docker container for all linux target architectures`386`, `amd64`, `arm/v6`, `arm/v7`, `arm64/v8`, `ppc64le`, `s390x` (#797) * Fixes #768 * buildx build * We need `setuptools_scm` to run `Makefile` * Build `docker` container using `wheel` * Download package distribution * Adjust supported platforms * Alpine only supports v6 * . * May-be fix `lsb_release` * Enable `multiarch` env for docker buildx * Deprecate `DOCKER_IMAGE_TAG` which removes `write-scm-version.py` dependency. Also pass targetplatform as an argument to `container-buildx` make target * Remove `setuptools_scm` dep for docker workflow step * woof * Match all target platforms to match `python-alpine` docker * yamllint * Use `yamllint disable rule:line-length` for `Dockerfile` * Tag the container using `dist-version` * Replace + with . for dev version tag * Add `PROXYPY_CONTAINER_VERSION` step * Fix publishing a GitHub Release from GHA (#811) * pip prod(deps): bump coverage from 6.1.1 to 6.1.2 (#813) Bumps [coverage](https://github.com/nedbat/coveragepy) from 6.1.1 to 6.1.2. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@6.1.1...6.1.2) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix matching requested version with Git tag in GHA (#815) * Fix the tagging condition in GHA build job (#816) * Publish to TestPyPI from the release request jobs (#819) * Handle `KBI` (#821) Co-authored-by: pyup.io bot <github-bot@pyup.io> Co-authored-by: Aksh Gupta <akshgpt7@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua> Co-authored-by: imgbot[bot] <31301654+imgbot[bot]@users.noreply.github.com> Co-authored-by: ImgBotApp <ImgBotHelp@gmail.com> Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: JerryKwan <Jinzhong.Guan@gmail.com>
- Loading branch information
