Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Work towards 1.0. Refactoring of runners/events/web ui. Getting rid of global state. #1266

Merged
merged 47 commits into from
Mar 10, 2020
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
78dba23
WIP: Refactor internal code to get rid of global LocustRunner instanc…
heyman Feb 23, 2020
2dd2087
Remove LocustRunner.hatch_rate (the start and spawn methods takes hat…
heyman Feb 23, 2020
021d928
Add Environment.host (move from options.host in order to not have to …
heyman Feb 23, 2020
81e12b0
Add Environment.reset_stats (move from options.reset_stats in order t…
heyman Feb 23, 2020
aae6bb1
stats_printer only needs access to RequestStats instance, and not the…
heyman Feb 23, 2020
32216d8
Merge branch 'master' into v1.0
heyman Feb 23, 2020
f7e4317
Log actual exception in retry exception handler
heyman Feb 25, 2020
4cdb808
Set strict_map_key=False when unserializing master/slave messages, si…
heyman Feb 25, 2020
48d94fc
Always round response times, stored in response_times dict, to integer.
heyman Feb 25, 2020
40a159f
Merge branch 'master' into v1.0
heyman Feb 25, 2020
ee30daa
Move LocustRunner.step_load into Environment.step_load
heyman Feb 26, 2020
3923cef
Fix broken CSV stats writer
heyman Feb 26, 2020
4ae8b2b
Speed up test
heyman Feb 26, 2020
46ad14c
Remove EventHook.__iadd__ and EventHook.__isub__ methods in favour of…
heyman Feb 27, 2020
ac79808
Make add_listener return handler, because it’ll make it possible to u…
heyman Feb 27, 2020
705fcae
Merge branch 'master' into v1.0
heyman Feb 27, 2020
1a7c3a4
Remove unused import
heyman Feb 27, 2020
21d4833
Fix bug where EventHook.fire(reverse=True) would permanently reverse …
heyman Feb 27, 2020
5a7704c
Add locust.event.init event hook that can be used by end-users to run…
heyman Feb 27, 2020
91019fa
Fix re-structured text syntax
heyman Feb 27, 2020
9c559f7
Fix example in docstring
heyman Feb 27, 2020
c12f615
Update extending-locust and API documentation page for the new API
heyman Feb 27, 2020
c4fcd0e
Remove deprecated info about installing libev on macOS
heyman Feb 27, 2020
fbc04ce
Fix example code that used old API for specifying wait time in millis…
heyman Feb 27, 2020
33a068b
Rewrote the “Common libraries” section in the documentation, and rena…
heyman Feb 27, 2020
489987f
Add documentation page about running Locust as a lib (so far it only …
heyman Feb 27, 2020
5f6937c
Update documentation on custom clients to new event API.
heyman Feb 27, 2020
23f2388
Update example to use new event API
heyman Feb 27, 2020
9e1a816
Remove LocustRunner.request_stats property
heyman Feb 28, 2020
8444d09
Remove HttpSession’s dependency on the Environment instance, and inst…
heyman Feb 28, 2020
507d546
More refactoring. Move stats and locust_clases from Environment onto …
heyman Feb 28, 2020
3f8d700
Add LocustRunner and WebUI instances as argument to the init event
heyman Mar 2, 2020
9fdeb12
Reinstate code for saving and restoring event listeners, since we sti…
heyman Mar 3, 2020
84153b0
Improve code for creating temporary locustfiles that can be used in t…
heyman Mar 3, 2020
d1f7c4e
Refactor code for parsing command line arguments.
heyman Mar 3, 2020
fd1ce43
Remove global locust.events.events Event instance, in favour of using…
heyman Mar 3, 2020
8a1f4b5
Update example (event arguments was changed)
heyman Mar 3, 2020
f6715f5
Rename locust.events module to locust.event
heyman Mar 3, 2020
cfba346
Add missing change from previous commit
heyman Mar 3, 2020
d5ec17e
Replace --heartbeat-liveness and --heartbeat-interval command line op…
heyman Mar 4, 2020
1cc9450
Add more config variables to Environment class (get rid of LocustRunn…
heyman Mar 4, 2020
b380197
Make sure init event is fired before the runner is started.
heyman Mar 5, 2020
b945a80
Do set process exit code to non-zero when CPU warning has been emitte…
heyman Mar 5, 2020
0758a8f
Remove master_host/master_port/master_bind_host/master_bind_port from…
heyman Mar 6, 2020
216f771
When stopping a test, log a warning message if the CPU went above 90%…
heyman Mar 6, 2020
3f9e0ad
Escape failure messages
heyman Mar 10, 2020
4115010
Merge branch 'master' into v1.0
heyman Mar 10, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions locust/clients.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,10 @@ class HttpSession(requests.Session):
response, even if the response code is ok (2xx). The opposite also works, one can use catch_response to catch a request
and then mark it as successful even if the response code was not (i.e 500 or 404).
"""
def __init__(self, base_url, *args, **kwargs):
def __init__(self, environment, base_url, *args, **kwargs):
super(HttpSession, self).__init__(*args, **kwargs)


self.locust_environment = environment
self.base_url = base_url

# Check for basic authentication
Expand Down Expand Up @@ -127,7 +128,7 @@ def request(self, method, url, name=None, catch_response=False, **kwargs):

if catch_response:
response.locust_request_meta = request_meta
return ResponseContextManager(response)
return ResponseContextManager(response, environment=self.locust_environment)
else:
if name:
# Since we use the Exception message when grouping failures, in order to not get
Expand All @@ -138,15 +139,15 @@ def request(self, method, url, name=None, catch_response=False, **kwargs):
try:
response.raise_for_status()
except RequestException as e:
events.request_failure.fire(
self.locust_environment.events.request_failure.fire(
request_type=request_meta["method"],
name=request_meta["name"],
response_time=request_meta["response_time"],
response_length=request_meta["content_size"],
exception=e,
)
else:
events.request_success.fire(
self.locust_environment.events.request_success.fire(
request_type=request_meta["method"],
name=request_meta["name"],
response_time=request_meta["response_time"],
Expand Down Expand Up @@ -186,9 +187,10 @@ class ResponseContextManager(LocustResponse):

_is_reported = False

def __init__(self, response):
def __init__(self, response, environment):
# copy data from response to this object
self.__dict__ = response.__dict__
self.locust_environment = environment
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it could be called just ”environment” instead of locust_environment?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefixed it with locust_ just to make sure that we won't clash with any (current or future) attribute in requests.Session.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea, this is gone now anyways :)


def __enter__(self):
return self
Expand Down Expand Up @@ -223,7 +225,7 @@ def success(self):
if response.status_code == 404:
response.success()
"""
events.request_success.fire(
self.locust_environment.events.request_success.fire(
request_type=self.locust_request_meta["method"],
name=self.locust_request_meta["name"],
response_time=self.locust_request_meta["response_time"],
Expand All @@ -247,7 +249,7 @@ def failure(self, exc):
if isinstance(exc, str):
exc = CatchResponseError(exc)

events.request_failure.fire(
self.locust_environment.events.request_failure.fire(
request_type=self.locust_request_meta["method"],
name=self.locust_request_meta["name"],
response_time=self.locust_request_meta["response_time"],
Expand Down
23 changes: 13 additions & 10 deletions locust/contrib/fasthttp.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,20 +64,21 @@ class FastHttpLocust(Locust):
The client support cookies, and therefore keeps the session between HTTP requests.
"""

def __init__(self):
super(FastHttpLocust, self).__init__()
def __init__(self, environment):
super().__init__(environment)
if self.host is None:
raise LocustError("You must specify the base host. Either in the host attribute in the Locust class, or on the command line using the --host option.")
if not re.match(r"^https?://[^/]+", self.host, re.I):
raise LocustError("Invalid host (`%s`), must be a valid base URL. E.g. http://example.com" % self.host)

self.client = FastHttpSession(base_url=self.host)
self.client = FastHttpSession(self.environment, base_url=self.host)


class FastHttpSession(object):
auth_header = None

def __init__(self, base_url, **kwargs):
def __init__(self, environment, base_url, **kwargs):
self.environment = environment
self.base_url = base_url
self.cookiejar = CookieJar()
self.client = LocustUserAgent(
Expand Down Expand Up @@ -183,20 +184,20 @@ def request(self, method, path, name=None, data=None, catch_response=False, stre

if catch_response:
response.locust_request_meta = request_meta
return ResponseContextManager(response)
return ResponseContextManager(response, environment=self.environment)
else:
try:
response.raise_for_status()
except FAILURE_EXCEPTIONS as e:
events.request_failure.fire(
self.environment.events.request_failure.fire(
request_type=request_meta["method"],
name=request_meta["name"],
response_time=request_meta["response_time"],
response_length=request_meta["content_size"],
exception=e,
)
else:
events.request_success.fire(
self.environment.events.request_success.fire(
request_type=request_meta["method"],
name=request_meta["name"],
response_time=request_meta["response_time"],
Expand Down Expand Up @@ -318,10 +319,12 @@ class ResponseContextManager(FastResponse):

_is_reported = False

def __init__(self, response):
def __init__(self, response, environment):
# copy data from response to this object
self.__dict__ = response.__dict__
self._cached_content = response.content
# store reference to locust Environment
self.environment = environment

def __enter__(self):
return self
Expand Down Expand Up @@ -356,7 +359,7 @@ def success(self):
if response.status_code == 404:
response.success()
"""
events.request_success.fire(
self.environment.events.request_success.fire(
request_type=self.locust_request_meta["method"],
name=self.locust_request_meta["name"],
response_time=self.locust_request_meta["response_time"],
Expand All @@ -380,7 +383,7 @@ def failure(self, exc):
if isinstance(exc, str):
exc = CatchResponseError(exc)

events.request_failure.fire(
self.environment.events.request_failure.fire(
request_type=self.locust_request_meta["method"],
name=self.locust_request_meta["name"],
response_time=self.locust_request_meta["response_time"],
Expand Down
16 changes: 9 additions & 7 deletions locust/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,18 +152,20 @@ class User(Locust):
_lock = gevent.lock.Semaphore() # Lock to make sure setup is only run once
_state = False

def __init__(self):
def __init__(self, environment):
super(Locust, self).__init__()
# check if deprecated wait API is used
deprecation.check_for_deprecated_wait_api(self)

self.environment = environment

with self._lock:
if hasattr(self, "setup") and self._setup_has_run is False:
self._set_setup_flag()
try:
self.setup()
except Exception as e:
events.locust_error.fire(locust_instance=self, exception=e, tb=sys.exc_info()[2])
self.environment.events.locust_error.fire(locust_instance=self, exception=e, tb=sys.exc_info()[2])
logger.error("%s\n%s", e, traceback.format_exc())
if hasattr(self, "teardown") and self._teardown_is_set is False:
self._set_teardown_flag()
Expand Down Expand Up @@ -218,12 +220,12 @@ class HttpLocust(Locust):
We don't need this feature most of the time, so disable it by default.
"""

def __init__(self):
super(HttpLocust, self).__init__()
def __init__(self, *args, **kwargs):
super(HttpLocust, self).__init__(*args, **kwargs)
if self.host is None:
raise LocustError("You must specify the base host. Either in the host attribute in the Locust class, or on the command line using the --host option.")

session = HttpSession(base_url=self.host)
session = HttpSession(self.environment, base_url=self.host)
session.trust_env = self.trust_env
self.client = session

Expand Down Expand Up @@ -362,7 +364,7 @@ def __init__(self, parent):
try:
self.setup()
except Exception as e:
events.locust_error.fire(locust_instance=self, exception=e, tb=sys.exc_info()[2])
self.locust.environment.events.locust_error.fire(locust_instance=self, exception=e, tb=sys.exc_info()[2])
logger.error("%s\n%s", e, traceback.format_exc())
if hasattr(self, "teardown") and self._teardown_is_set is False:
self._set_teardown_flag()
Expand Down Expand Up @@ -418,7 +420,7 @@ def run(self, *args, **kwargs):
except GreenletExit:
raise
except Exception as e:
events.locust_error.fire(locust_instance=self, exception=e, tb=sys.exc_info()[2])
self.locust.environment.events.locust_error.fire(locust_instance=self, exception=e, tb=sys.exc_info()[2])
if self.locust._catch_exceptions:
logger.error("%s\n%s", e, traceback.format_exc())
self.wait()
Expand Down
47 changes: 47 additions & 0 deletions locust/env.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
from .events import Events
from .stats import RequestStats


class Environment:
locust_classes = None
"""The locust user classes that is to be run"""

events = None
"""Event hooks used by Locust internally, as well as """

stats = None
"""Instance of RequestStats which holds the request statistics for this Locust test"""

options = None
"""Other environment options"""

runner = None
"""Reference to the runner instance"""

web_ui = None
"""Reference to the WebUI instance"""

host = None
"""Base URL of the target system"""

reset_stats = False
"""Determines if stats should be reset once all simulated users have been spawned"""

def __init__(self, locust_classes=None, options=None, host=None, reset_stats=False):
self.events = Events()
self.stats = RequestStats()
self.locust_classes = locust_classes
self.host = host
self.reset_stats = reset_stats
self.options = options

# set up event listeners for recording requests
def on_request_success(request_type, name, response_time, response_length, **kwargs):
self.stats.log_request(request_type, name, response_time, response_length)

def on_request_failure(request_type, name, response_time, response_length, exception, **kwargs):
self.stats.log_request(request_type, name, response_time, response_length)
self.stats.log_error(request_type, name, exception)

self.events.request_success.add_listener(on_request_success)
self.events.request_failure.add_listener(on_request_failure)
Loading