Work towards 1.0. Refactoring of runners/events/web ui. Getting rid of global state. #1266

heyman · 2020-02-23T16:08:50Z

Hey!

I've done some refactoring of Locust. The main purpose is to improve the code quality, and make some parts of Locust that is currently non public part of the public API.

It's now in a working state but since it's a few changes to the public API, as well as a lot of changes to the internal API, I think it would be a good idea to have these changes in a 1.0. Therefore - if others doesn't think it's a terrible idea - I suggest that we call a feature freeze for master (and implement all new features in the v1.0 branch) since it'll be a messy job to port new features from master.

Please note that it's a work in progress. All tests pass, but there are no documentation changes yet. I wanted to get it out in a Github PR as soon as possible in order to get more eyes on it.

Implemented changes

These are the changes that are currently implemented in this branch:

An Environment class has been introduced to work as a glue between the runner, locust classes, events and web UI. The main benefit of this is that we can get rid of the global locust.stats.global_stats and locust.runner.locust_runner instances. End users will also be able to run Locust as a lib/programatically (How to run test programatically #222), and it'll be easier to write (unit) tests for Locust.
locust.events has been moved to the events attribute of the Environment instance.
All Locust classes must now be passed an Environment instance reference (when instantiated) which is then stored on Locust.environment. It's then used by the HTTP client when it triggers request_success/request_failure events.
A WebUI class has been added in order to be able to spawn one (or multiple) web UI server(s) when running Locust programatically.
LocustRunner.start_hatching() has been renamed to LocustRunner.start()
LocustRunner.hatch_rate has been removed in favour of it just being an argument to start() and spawn_locusts().

More changes

Here are other changes that I think we should implement/consider for a 1.0. Most of them have been suggested/requested by others before at some point:

We need ~~to figure out~~ an API for some initialization hook in end users' test scripts. Since the the test script is imported before the Environment is constructed, we can't rely on letting end users put the code at the module level.
API:

# in end-user's test script
from locust.events import init

def locust_init(environment, **kwargs):
    pass
init.add_listener(locust_init)

# This works as well
from locust.events import init

@init.add_listener
def _(environment, **kwargs):
    pass

Clean up among the command line arguments.
Rename Locust to User (see Rename and restructure Locust/TaskSet #1264 for a longer discussion)
Remove the Locust.setup and Locust.teardown hooks in favour of adding test_start and test_stop events to Environment.events (Changes made during setup only effect the first user #1265).
Change how logging is handled. Currently we're wrapping sys.stdout and sys.stderr which seems wrong.
Remove @seq_task and instead add a SequentialTaskSet class that run all declared tasks sequentially.
Maybe rename host to base_url (Build URLs using urljoin #1217)?
Replace master/slave terminology with master/worker.
Re-work the step load feature. I'd like it to be something more general like a run plan with an API that one can hook into. It should also be able to both spawn and kill simulated users.

These are the changes that I could think of from the top of my head right now. It's just a start and we can edit this post to keep the above list up to date.

…e and RequestStats instance, by introducing Environment class that ties things together. Add WebUI class that creates the Flask app the web UI. Rename --port command line argument to --web-port (since we already use --web-host) Rename LocustRunner.start_hatching to start.

…ch_rate as an argument and there’s no need to store it on the runner instance)

…parse or mock options when using Locust as a lib)

…o not have to parse or mock options when using Locust as a lib)

… whole Environment

codecov · 2020-02-23T16:21:33Z

Codecov Report

Merging #1266 into master will increase coverage by 2.6%.
The diff coverage is 77.98%.

@@            Coverage Diff            @@
##           master    #1266     +/-   ##
=========================================
+ Coverage   77.52%   80.12%   +2.6%     
=========================================
  Files          20       23      +3     
  Lines        1984     2068     +84     
  Branches      313      319      +6     
=========================================
+ Hits         1538     1657    +119     
+ Misses        368      333     -35     
  Partials       78       78

Impacted Files	Coverage Δ
locust/clients.py	`91.66% <100%> (+0.26%)`	⬆️
locust/contrib/fasthttp.py	`88.43% <100%> (+1.23%)`	⬆️
locust/env.py	`100% <100%> (ø)`
locust/util/cache.py	`91.3% <100%> (+0.82%)`	⬆️
locust/main.py	`23.46% <12.72%> (-9.46%)`	⬇️
locust/argument_parser.py	`71.11% <71.11%> (ø)`
locust/stats.py	`88.19% <73.33%> (+2.48%)`	⬆️
locust/core.py	`91.25% <80%> (-0.84%)`	⬇️
locust/event.py	`90.62% <90.62%> (ø)`
locust/runners.py	`77.83% <91.89%> (+5.2%)`	⬆️
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed3b263...4115010. Read the comment docs.

cyberw · 2020-02-23T18:03:47Z

Cool stuff! I’ll have a deeper look soon.

cyberw · 2020-02-25T13:32:12Z

btw, I like all the suggested "more changes" (though I think most of them of deserve their own PRs instead of having a mega-PR :)

…nce it’s default has been set to True in msgpack 1.0. Fixes a crash bug when running Locust distributed with latest msgpack.

Python 3’s round() returns ints, so no need to call int(round()) any more.

heyman · 2020-02-25T15:02:13Z

btw, I like all the suggested "more changes" (though I think most of them of deserve their own PRs instead of having a mega-PR :)

You're right. We should probably branch of master in a 0.X branch, which we can use for bug fixes in the 0.14 version until we're ready to release a 1.0.

Alternatively, we could make separate pull requests to this PR's branch (at least I think that's possible on Github?), though it's probably better to keep the 1.0 changes in master, and make a separate branch for the 0.14 version.

cyberw · 2020-02-26T09:39:43Z

You're right. We should probably branch of master in a 0.X branch, which we can use for bug fixes in the 0.14 version until we're ready to release a 1.0.

Alternatively, we could make separate pull requests to this PR's branch (at least I think that's possible on Github?), though it's probably better to keep the 1.0 changes in master, and make a separate branch for the 0.14 version.

We can branch a emergency fix branch for 0.14 if and only if we need it. I think GH will have an easier time keeping track of PRs & stuff if everything is merged to master instead of having PRs to this branch. But either way is fine.

heyman · 2020-02-26T10:38:47Z

We can branch a emergency fix branch for 0.14 if and only if we need it

Sounds good!

heyman · 2020-02-26T10:49:53Z

Below is a working example where Locust is used as a lib instead of started through the default command (main.py).

Hopefully it's easier to grasp the changes to the internal API by looking at this example, compared to looking at all the changes in the PR.

import gevent
from locust import HttpLocust, TaskSet, task, between
from locust.runners import LocalLocustRunner
from locust.env import Environment
from locust.stats import stats_printer
from locust.log import setup_logging
from locust.web import WebUI

setup_logging("INFO", None)

class User(HttpLocust):
    wait_time = between(1, 3)
    host = "https://docs.locust.io"
    
    class task_set(TaskSet):
        @task
        def my_task(self):
            self.client.get("/")
        
        @task
        def task_404(self):
            self.client.get("/non-existing-path")

# setup Environment and Runner
env = Environment()
runner = LocalLocustRunner(environment=env, locust_classes=[User])
# start a WebUI instance
web_ui = WebUI(environment=env, runner=runner)
gevent.spawn(lambda: web_ui.start("127.0.0.1", 8089))
# start a greenlet that periodically outputs the current stats
gevent.spawn(stats_printer(env.stats))

# start the test
runner.start(1, hatch_rate=10)

# wait for the greenlets (indefinitely)
runner.greenlet.join()

cyberw · 2020-02-26T11:31:23Z

Below is a working example where Locust is used as a lib instead of started through the default command (main.py).

Cool stuff! Lets add a unit test that does something like that!

… only using add_listener() and remove_listener()

…se EventHook.add_listener as a decorator

…the list of listeners

… code that is dependent on the Environment instance (e.g. to set up request_success/request_failure listeners). Update examples/events.py to use the new API.

heyman · 2020-02-29T11:01:46Z

After some discussion with @cyberw on Slack I've now reworked the API slightly.

Environemnt.locust_classes has been moved to LocustRunner.locust_classes, and LocustRunner now takes locust_classes as a constructor argument in addition to an Environment instance.
Environment.stats has been moved to LocustRunner.stats.
Environment does no longer hold references to the LocustRunner instance or the WebUI instance (avoids circular class dependencies). Locust scripts and plugins that need access to these instances will instead have to retrieve them by listening to an init event which will be passed the instances as event arguments.

…ll have module level events (locust.events.init)

…ests

Add init_command_line_parser event that can be used by test scripts and plugins to add command line arguments.

… the init event to set up event listeners

Introduce global instance of Events() stored in locust.events which can be used to register event listeners at the module level of locustfiles. Remove some unused imports.

Rename locust.events module to locust.event Introduce global instance of Events() stored in locust.events which can be used to register event listeners at the module level of locustfiles.

…tions with constants

…ers' dependency on parsed command line options)

Remove seemingly unnecessary sleep call.

…d, as it seems much too flaky

… Environment. Now they are just keyword arguments to MasterLocustRunner/SlaveLocustRunner.

… during the test

cyberw · 2020-03-09T11:44:36Z

Maybe we should rename this PR (because the name is not descriptive of the changes made) and try to merge before it grows too big?

heyman · 2020-03-09T16:06:33Z

Yes, that sounds like a good idea. Though we should branch off the master into a 0.x branch (that we can use for critical bug fixes) before merging this.

heyman · 2020-03-10T15:19:33Z

This has now been merged to master. I've created a new branch (v0.x) from the previous master.

I've created a 1.0 milestone, and created stub issues for some of the items in my checkbox list under "More changed" in the OP.

cyberw · 2020-02-23T17:59:38Z

locust/clients.py

        # copy data from response to this object
        self.__dict__ = response.__dict__
+        self.locust_environment = environment


I think it could be called just ”environment” instead of locust_environment?

I prefixed it with locust_ just to make sure that we won't clash with any (current or future) attribute in requests.Session.

yea, this is gone now anyways :)

cyberw · 2020-02-23T18:47:27Z

locust/test/test_locust_class.py

@@ -256,8 +258,8 @@ class MyLocust2(Locust):
            host = "http://127.0.0.1"
            task_set = MyTaskSet2

-        l = MyLocust()
-        l2 = MyLocust2()
+        l = MyLocust(Environment(locust_classes=[MyLocust]))


Hmm. An Environment has a list of Locust classes, but a Locust requires an Environment to be instantiated? I am not sure what the solution is, but this is a typical sign that there is too tight coupling between Env/Locust.

Hm, maybe. Not sure though. One would never instantiate a Locust class like that except for in a test.

I think this is one of those things we spend a lot of time discussing later, so it is kind of old news :)

cyberw · 2020-02-29T11:56:26Z

examples/custom_xmlrpc_client/xmlrpc_locustfile.py

    """
    Simple, sample XML RPC client implementation that wraps xmlrpclib.ServerProxy and 
    fires locust events on request_success and request_failure, so that all requests 
    gets tracked in locust's statistics.
    """
+
+    _locust_environment = None


Maybe just call this _environment? Not that it is super important...

I just wanted to make sure that we wouldn't clash with anything in xmlrpclib.ServerProxy.

heyman added 6 commits February 23, 2020 13:04

Remove LocustRunner.hatch_rate (the start and spawn methods takes hat…

2dd2087

…ch_rate as an argument and there’s no need to store it on the runner instance)

Add Environment.host (move from options.host in order to not have to …

021d928

…parse or mock options when using Locust as a lib)

Add Environment.reset_stats (move from options.reset_stats in order t…

81e12b0

…o not have to parse or mock options when using Locust as a lib)

stats_printer only needs access to RequestStats instance, and not the…

aae6bb1

… whole Environment

Merge branch 'master' into v1.0

32216d8

heyman mentioned this pull request Feb 24, 2020

Re-introduce -n option to specify the number of requests #1085

Closed

Log actual exception in retry exception handler

f7e4317

heyman added 2 commits February 25, 2020 14:43

Set strict_map_key=False when unserializing master/slave messages, si…

4cdb808

…nce it’s default has been set to True in msgpack 1.0. Fixes a crash bug when running Locust distributed with latest msgpack.

Always round response times, stored in response_times dict, to integer.

48d94fc

Python 3’s round() returns ints, so no need to call int(round()) any more.

Merge branch 'master' into v1.0

40a159f

Move LocustRunner.step_load into Environment.step_load

ee30daa

heyman added 10 commits February 26, 2020 14:57

Fix broken CSV stats writer

3923cef

Speed up test

4ae8b2b

Remove EventHook.__iadd__ and EventHook.__isub__ methods in favour of…

46ad14c

… only using add_listener() and remove_listener()

Make add_listener return handler, because it’ll make it possible to u…

ac79808

…se EventHook.add_listener as a decorator

Merge branch 'master' into v1.0

705fcae

Remove unused import

1a7c3a4

Fix bug where EventHook.fire(reverse=True) would permanently reverse …

21d4833

…the list of listeners

Add locust.event.init event hook that can be used by end-users to run…

5a7704c

… code that is dependent on the Environment instance (e.g. to set up request_success/request_failure listeners). Update examples/events.py to use the new API.

Fix re-structured text syntax

91019fa

Fix example in docstring

9c559f7

heyman added 14 commits March 2, 2020 17:33

Add LocustRunner and WebUI instances as argument to the init event

3f8d700

Reinstate code for saving and restoring event listeners, since we sti…

9fdeb12

…ll have module level events (locust.events.init)

Improve code for creating temporary locustfiles that can be used in t…

84153b0

…ests

Refactor code for parsing command line arguments.

d1f7c4e

Add init_command_line_parser event that can be used by test scripts and plugins to add command line arguments.

Remove global locust.events.events Event instance, in favour of using…

fd1ce43

… the init event to set up event listeners

Update example (event arguments was changed)

8a1f4b5

Rename locust.events module to locust.event

f6715f5

Introduce global instance of Events() stored in locust.events which can be used to register event listeners at the module level of locustfiles. Remove some unused imports.

Add missing change from previous commit

cfba346

Rename locust.events module to locust.event Introduce global instance of Events() stored in locust.events which can be used to register event listeners at the module level of locustfiles.

Replace --heartbeat-liveness and --heartbeat-interval command line op…

d5ec17e

…tions with constants

Add more config variables to Environment class (get rid of LocustRunn…

1cc9450

…ers' dependency on parsed command line options)

Make sure init event is fired before the runner is started.

b380197

Remove seemingly unnecessary sleep call.

Do set process exit code to non-zero when CPU warning has been emitte…

b945a80

…d, as it seems much too flaky

Remove master_host/master_port/master_bind_host/master_bind_port from…

0758a8f

… Environment. Now they are just keyword arguments to MasterLocustRunner/SlaveLocustRunner.

When stopping a test, log a warning message if the CPU went above 90%…

216f771

… during the test

Escape failure messages

3f9e0ad

heyman changed the title ~~Locust 1.0~~ Work towards 1.0. Refactoring of runners/events/web ui. Getting rid of global state. Mar 10, 2020

Merge branch 'master' into v1.0

4115010

heyman merged commit 8046eab into master Mar 10, 2020

This was referenced Mar 10, 2020

Clean up among the command line arguments #1282

Closed

Rename Locust to User/LocustUser #1283

Closed

timran1 mentioned this pull request Mar 10, 2020

Enable statistically meaningful constant RPS load generation distributions #1281

Closed

heyman linked an issue Apr 13, 2020 that may be closed by this pull request

How to run test programatically #222

Closed

heyman deleted the v1.0 branch April 14, 2020 10:00

cyberw reviewed Apr 16, 2020

View reviewed changes

cyberw mentioned this pull request Apr 16, 2020

Add Runners, WebUI and Environment to the public API #1327

Merged

heldic mentioned this pull request Apr 28, 2020

Let hatch rate be provided by a callable #1353

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Work towards 1.0. Refactoring of runners/events/web ui. Getting rid of global state. #1266

Work towards 1.0. Refactoring of runners/events/web ui. Getting rid of global state. #1266

heyman commented Feb 23, 2020 •

edited

Loading

codecov bot commented Feb 23, 2020 •

edited

Loading

cyberw commented Feb 23, 2020

cyberw commented Feb 25, 2020 •

edited

Loading

heyman commented Feb 25, 2020

cyberw commented Feb 26, 2020

heyman commented Feb 26, 2020

heyman commented Feb 26, 2020 •

edited

Loading

cyberw commented Feb 26, 2020

heyman commented Feb 29, 2020

cyberw commented Mar 9, 2020 •

edited

Loading

heyman commented Mar 9, 2020

heyman commented Mar 10, 2020

cyberw Feb 23, 2020

heyman Apr 16, 2020

cyberw Apr 16, 2020

cyberw Feb 23, 2020

heyman Apr 16, 2020

cyberw Apr 16, 2020

cyberw Feb 29, 2020

heyman Apr 16, 2020

Work towards 1.0. Refactoring of runners/events/web ui. Getting rid of global state. #1266

Work towards 1.0. Refactoring of runners/events/web ui. Getting rid of global state. #1266

Conversation

heyman commented Feb 23, 2020 • edited Loading

Implemented changes

More changes

codecov bot commented Feb 23, 2020 • edited Loading

Codecov Report

cyberw commented Feb 23, 2020

cyberw commented Feb 25, 2020 • edited Loading

heyman commented Feb 25, 2020

cyberw commented Feb 26, 2020

heyman commented Feb 26, 2020

heyman commented Feb 26, 2020 • edited Loading

cyberw commented Feb 26, 2020

heyman commented Feb 29, 2020

cyberw commented Mar 9, 2020 • edited Loading

heyman commented Mar 9, 2020

heyman commented Mar 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heyman commented Feb 23, 2020 •

edited

Loading

codecov bot commented Feb 23, 2020 •

edited

Loading

cyberw commented Feb 25, 2020 •

edited

Loading

heyman commented Feb 26, 2020 •

edited

Loading

cyberw commented Mar 9, 2020 •

edited

Loading