-
-
Notifications
You must be signed in to change notification settings - Fork 573
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Async
get_events, handle_event, handle_readables, `handle_writa…
…bles` (#769) * Asynchronous `handle_event` and `LocalExecutor` thread * Bail out on first task completion * mypy * Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded) * Print open file diff from `benchmark.sh` * Add `--local-executor` flag, disabled by default for now until tests are updated * Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now) * Async `get_events` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address tests after async changes * mypy and flake8 * spelldoc * `check.py` and trailing comma * Rename to `_assertions.py` * Add missing `pytest-mock` and `pytest-asyncio` deps * Add `pytest-mock` to `pylint` deps * Correct use of `parameterize` and add `PT007` to flake8 ignores * Fix mypy hints broken for `< Python3.9` * Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix for pre-python-3.9 versions * `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions * Install setuptools via `lib-dep` until we recommend editable install * Deprecate support for `Python 3.6` * Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929 * Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906 * Make `Threadless` agnostic of `multiprocessing.Process` * Acceptors must dispatch to local executor in non-blocking fashion * No daemon for executor processes and fix shutdown logic * Only return fds from `_selected_events` not all events data * Refactor logic * Prefix private methods with `_` * `work_queue` and not `client_queue` * Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor` * Make `LocalExecutor` agnostic of `threading.Thread` * `LocalExecutor` now implements `Threadless` * `get_events` and `get_descriptors` now must return int and not sock. `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify` * Fix `main` tests * Apply suggestions from code review Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * Apply code review recommendations manually * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert back `Any` and use `addr or None` * Address `flake8` * Update tests to use `fileno` * Fix doc build * Fix doc spell, use tear down and not teardown * Doc updates * Add back support for `Python 3.6` * Acceptors dont need loop initialization * On Python 3.6 `asyncio.new_event_loop()` is necessary * Make doc happy * `--threaded` needs a new event loop for 3.7 too * Always use `asyncio.new_event_loop()` for threaded mode Added e2e integration tests (subprocess & curl) for all modes. * Lint fixes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
- Loading branch information
3 people
authored
Nov 23, 2021
1 parent
d554b94
commit 44d7243
Showing
43 changed files
with
1,675 additions
and
1,095 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| #!/bin/bash | ||
| # | ||
| # proxy.py | ||
| # ~~~~~~~~ | ||
| # ⚡⚡⚡ Fast, Lightweight, Programmable, TLS interception capable | ||
| # proxy server for Application debugging, testing and development. | ||
| # | ||
| # :copyright: (c) 2013-present by Abhinav Singh and contributors. | ||
| # :license: BSD, see LICENSE for more details. | ||
| # | ||
| usage() { | ||
| echo "Usage: ./helper/benchmark.sh" | ||
| echo "You must run this script from proxy.py repo root." | ||
| } | ||
|
|
||
| DIRNAME=$(dirname "$0") | ||
| if [ "$DIRNAME" != "./helper" ]; then | ||
| usage | ||
| exit 1 | ||
| fi | ||
|
|
||
| BASENAME=$(basename "$0") | ||
| if [ "$BASENAME" != "benchmark.sh" ]; then | ||
| usage | ||
| exit 1 | ||
| fi | ||
|
|
||
| PWD=$(pwd) | ||
| if [ $(basename $PWD) != "proxy.py" ]; then | ||
| usage | ||
| exit 1 | ||
| fi | ||
|
|
||
| TIMEOUT=1 | ||
| QPS=20000 | ||
| CONCURRENCY=100 | ||
| TOTAL_REQUESTS=100000 | ||
| OPEN_FILE_LIMIT=65536 | ||
| BACKLOG=OPEN_FILE_LIMIT | ||
| PID_FILE=/tmp/proxy.pid | ||
|
|
||
| ulimit -n $OPEN_FILE_LIMIT | ||
|
|
||
| # time python -m \ | ||
| # proxy \ | ||
| # --enable-web-server \ | ||
| # --plugin proxy.plugin.WebServerPlugin \ | ||
| # --backlog $BACKLOG \ | ||
| # --open-file-limit $OPEN_FILE_LIMIT \ | ||
| # --pid-file $PID_FILE \ | ||
| # --log-file /dev/null | ||
|
|
||
| PID=$(cat $PID_FILE) | ||
| if [[ -z "$PID" ]]; then | ||
| echo "Either pid file doesn't exist or no pid found in the pid file" | ||
| exit 1 | ||
| fi | ||
| ADDR=$(lsof -Pan -p $PID -i | grep -v COMMAND | awk '{ print $9 }') | ||
|
|
||
| PRE_RUN_OPEN_FILES=$(./helper/monitor_open_files.sh) | ||
|
|
||
| echo "CONCURRENCY: $CONCURRENCY workers, TOTAL REQUESTS: $TOTAL_REQUESTS req, QPS: $QPS req/sec, TIMEOUT: $TIMEOUT sec" | ||
| hey \ | ||
| -n $TOTAL_REQUESTS \ | ||
| -c $CONCURRENCY \ | ||
| -q $QPS \ | ||
| -t $TIMEOUT \ | ||
| http://$ADDR/http-route-example | ||
|
|
||
| POST_RUN_OPEN_FILES=$(./helper/monitor_open_files.sh) | ||
|
|
||
| echo $output | ||
|
|
||
| echo "Open files diff:" | ||
| diff <( echo "$PRE_RUN_OPEN_FILES" ) <( echo "$POST_RUN_OPEN_FILES" ) | ||
|
|
||
| # while true; do netstat -ant | grep .8899 | awk '{print $6}' | sort | uniq -c | sort -n; sleep 1; done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,5 @@ | ||
| #!/bin/bash | ||
|
|
||
| # | ||
| # proxy.py | ||
| # ~~~~~~~~ | ||
| # ⚡⚡⚡ Fast, Lightweight, Programmable, TLS interception capable | ||
|
|
||
Oops, something went wrong.