Locust docker version 1.4.0 using 100% CPU on idle

[simakuutio]

Test environment:
Ubuntu 18.04.1
Docker version 19.03.6

Environment stays the same only the docker image version is changed.
Running the docker container with the same locustifle.py which is an example from https://hub.docker.com/r/locustio/locust

from locust import HttpUser, task

class QuickstartUser(HttpUser):
    def on_start(self):
        self.client.post
("/login", json={"username":"foo", "password":"bar"})

    @task
    def hello_world(self):
        self.client.get("/hello")
        self.client.get("/world")

    @task(3)
    def view_item(self):
        for item_id in range(10):
            self.client.get(f"/item?id={item_id}", name="/item")

When running locustio/locust version 1.3.2 the idle CPU load is around 0.0% and stable.

Running locustio/locust version 1.4.0 the idle CPU load goes up to 100.0% at once when the container is started and stays between 96.7% and 100.0%.

[max-rocket-internet]

What is consuming the CPU?

[heyman]

Maybe "caused" by #1626?

[heyman]

@simakuutio:

Since Locust 1.4, the default wait time for simulated locust users is zero. Therefore I recommend that you specify a wait time for QuickstartUser.

from locust import HttpUser, task, between

class QuickstartUser(HttpUser):
    # wait between 1 and 5 seconds between tasks
    wait_time = between(1, 5)
    ...

[simakuutio]

If this is the recommended way, example files should be also updated accordingly.

[heyman]

If this is the recommended way, example files should be also updated accordingly.

I agree.

[eskallberg]

Hi

The same result comes with or without wait_time parameter

Following example locustfile also uses 100% CPU

import time
from locust import HttpUser, task, between

class QuickstartUser(HttpUser):
    wait_time = between(1, 2)

    @task
    def index_page(self):
        self.client.get("/hello")
        self.client.get("/world")

    @task(3)
    def view_item(self):
        for item_id in range(10):
            self.client.get(f"/item?id={item_id}", name="/item")
            time.sleep(1)

        def on_start(self):
            self.client.post("/login", json={"username":"foo", "password":"bar"})

Command locust is using 100% CPU inside the Docker container

[cyberw]

I cant reproduce 100% cpu usage on any of the suggested test plans.

Do you mean 100% cpu on the load gen or on the server, or are you running them both on the same machine?

Is it possible that it is somehow related to the keyboard input, which was also introduced in 1.4?

[cyberw]

@eskallberg are you also on Ubuntu?

[eskallberg]

Yes

[cyberw]

@simakuutio Btw, when you say "idle CPU load" what do you mean exactly? I guess you mean "while running load", but that is kind of the opposite of "idle", so I just want to be sure?

[heyman]

Btw, when you say "idle CPU load" what do you mean exactly? I guess you mean "while running load"

That's what I thought as well. I can reproduce the issue with just a few number of simulated users for the original example when running against a HTTP server that can handle the load. I haven't been able to reproduce it when the test isn't running. About to test @eskallberg 's example in Docker container now.

[cyberw]

I can reproduce this in docker/Ubuntu and it also prints the following exception when shutting down. I'm 99% confident this is an issue with keyboard input (on Ubuntu, not MacOS, CentOS or Windows), not with wait times.

I will check with @DennisKrone if he can fix it.

docker run -p 8089:8089 -v $PWD:/mnt/locust locustio/locust -f /mnt/locust/quickstart.py
[2020-11-16 12:10:57,399] c07ede557d74/INFO/locust.main: Starting web interface at http://0.0.0.0:8089 (accepting connections from all network interfaces)
[2020-11-16 12:10:57,412] c07ede557d74/INFO/locust.main: Starting Locust 1.4.0
[2020-11-16 12:11:02,415] c07ede557d74/WARNING/root: CPU usage above 90%! This may constrain your throughput and may even give inconsistent response time measurements! See https://docs.locust.io/en/stable/running-locust-distributed.html for how to distribute the load over multiple CPU cores or machines
^CTraceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 88, in input_listener_func
    input = poller.poll()
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 36, in poll
    return sys.stdin.read(1)
  File "/usr/local/lib/python3.8/codecs.py", line 319, in decode
    def decode(self, input, final=False):
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 98, in input_listener_func
    poller.cleanup()
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 31, in cleanup
    termios.tcsetattr(self.stdin, termios.TCSANOW, self.tattr)
AttributeError: 'UnixKeyPoller' object has no attribute 'tattr'
2020-11-16T12:11:07Z <Greenlet at 0x7fcc45787e10: input_listener_func> failed with AttributeError

[2020-11-16 12:11:07,514] c07ede557d74/CRITICAL/locust.main: Unhandled exception in greenlet: <Greenlet at 0x7fcc45787e10: input_listener_func>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 88, in input_listener_func
    input = poller.poll()
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 36, in poll
    return sys.stdin.read(1)
  File "/usr/local/lib/python3.8/codecs.py", line 319, in decode
    def decode(self, input, final=False):
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 98, in input_listener_func
    poller.cleanup()
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 31, in cleanup
    termios.tcsetattr(self.stdin, termios.TCSANOW, self.tattr)
AttributeError: 'UnixKeyPoller' object has no attribute 'tattr'
KeyboardInterrupt
2020-11-16T12:11:07Z
[2020-11-16 12:11:07,527] c07ede557d74/INFO/locust.main: Running teardowns...
[2020-11-16 12:11:07,527] c07ede557d74/INFO/locust.main: Shutting down (exit code 2), bye.
[2020-11-16 12:11:07,527] c07ede557d74/INFO/locust.main: Cleaning up runner...
[2020-11-16 12:11:07,528] c07ede557d74/WARNING/locust.runners: CPU usage was too high at some point during the test! See https://docs.locust.io/en/stable/running-locust-distributed.html for how to distribute the load over multiple CPU cores or machines

[cyberw]

Aha, the problem is that when running docker, there is no interactive terminal. Add -it to your docker run command as a temp workaround.

[eskallberg]

I will try that and let you know how it goes with -it

About the idle, no heavy test running, just the container up.

[eskallberg]

I will try that and let you know how it goes with -it

About the idle, no heavy test running, just the container up.

with -it it works fine.

[heyman]

I can reproduce this in docker/Ubuntu and it also prints the following exception when shutting down. I'm 99% confident this is an issue with keyboard input (on Ubuntu, not MacOS, CentOS or Windows), not with wait times.

Nice find! The #1630 should fix it.

[cyberw]

Fixed in #1631

[Warns]

I have a similar issue when running locust on EC2 instances (t3.medium) in a distributed mode. For example 1 master 4 worker nodes. With a simple load testing with 20 users for example after some point all worker nodes hit 100% CPU. The tasks I'm running and the payload being passed to the tasks are pretty simple and no such load is expected with 20 users.

Sorry if this has been reported anywhere else I just wanted to follow up on this. We've also added the wait_time but to no avail.