Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add heartbeat to detect down slaves #927

Merged
merged 24 commits into from
Feb 6, 2019
Merged

Add heartbeat to detect down slaves #927

merged 24 commits into from
Feb 6, 2019

Commits on Dec 10, 2018

  1. Replace zmq sockets with one DEALER-ROUTER socket 

    The PUSH and PULL sockets being used caused hatch messages to get routed
to slaves that may have become unresponsive or crashed. This change
includes the client id in the messages sent out from the master which
ensures that hatch messages are going to slaves the are READY or
RUNNING.

This should also fix the issue locustio#911 where slaves are not receiving the
stop message. I think these issues are a result of PUSH-PULL sockets
using a round robin approach.
    Jonathan McCall committed Dec 10, 2018
    Configuration menu
    Copy the full SHA
    3d4b927 View commit details
    Browse the repository at this point in the history

  2. Remove client_id parameter from send_multipart method

    Jonathan McCall committed Dec 10, 2018
    Configuration menu
    Copy the full SHA
    095acbd View commit details
    Browse the repository at this point in the history

  3. Add heartbeat worker to server and client 

    The server checks to see if clients have expired and if they have
updates their status to "missing".

The client has a worker that will send a heartbeat on a regular
interval. The heart also relays the slave state back to the
master so that they stay in sync.
    Jonathan McCall committed Dec 10, 2018
    Configuration menu
    Copy the full SHA
    c02824e View commit details
    Browse the repository at this point in the history

  4. Use new clients.all property in heartbeat worker

    Jonathan McCall committed Dec 10, 2018
    Configuration menu
    Copy the full SHA
    3ff4080 View commit details
    Browse the repository at this point in the history

  5. Fix reporting of stopped state 

    Wait until all slaves are reporting in as ready before stating
that the master is stopped.
    Jonathan McCall committed Dec 10, 2018
    Configuration menu
    Copy the full SHA
    28b0bc9 View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2018

  1. Fix tests after changing ZMQ sockets to DEALER-ROUTER

    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    7b5e62d View commit details
    Browse the repository at this point in the history

  2. Change heartbeat log msg to info so that it does not appear in tests

    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    d019117 View commit details
    Browse the repository at this point in the history

  3. Add tests for zmqrpc.py

    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    a5c768c View commit details
    Browse the repository at this point in the history

  4. Remove commented imports, add note about sleep

    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    7e1c8e9 View commit details
    Browse the repository at this point in the history

  5. Support str/unicode diff in py2 vs py3

    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    6984741 View commit details
    Browse the repository at this point in the history

  6. Ensure failed zmqrpc tests clean up bound sockets

    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    197949a View commit details
    Browse the repository at this point in the history

  7. Create throw away variable for identity from from ZMQ message 

    I think this looks better than using msg[1].
    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    e617573 View commit details
    Browse the repository at this point in the history

  8. Replace usage of parse_options in tests with mock options 

    Using parse_options during test setup can conflict with test runners
like pytest. Essentially it will swallow up the options that are
meant to be passed to the test runner and instead treats them
as options being passed to the test.
    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    06062f7 View commit details
    Browse the repository at this point in the history

  9. Set coverage concurrency to gevent 

    Coverage breaks with gevent and does not fully report green threads
as having been tested. Setting concurrency in .coveragerc will
fix the issue. https://bitbucket.org/ned/coveragepy/issues/149/coverage-gevent-looks-broken
    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    5a416fc View commit details
    Browse the repository at this point in the history

  10. Add test that shows master heartbeat worker marks slaves missing

    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    9818dd3 View commit details
    Browse the repository at this point in the history

  11. Add assertions to test_zmqrpc.py

    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    7c0d155 View commit details
    Browse the repository at this point in the history

  12. Use unittest assertions

    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    bd56d22 View commit details
    Browse the repository at this point in the history

  13. Change assertion value to bytes object

    Jonathan McCall committed Dec 11, 2018
    Configuration menu
    Copy the full SHA
    e757591 View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2018

  1. Add cmdline options for heartbeat liveness and interval

    Jonathan McCall committed Dec 17, 2018
    Configuration menu
    Copy the full SHA
    5854b60 View commit details
    Browse the repository at this point in the history

  2. Add new option heartbeat_liveness to test_runners mock options

    Jonathan McCall committed Dec 17, 2018
    Configuration menu
    Copy the full SHA
    493db4c View commit details
    Browse the repository at this point in the history

  3. Ensure SlaveNode class uses heartbeat_liveness default or passed

    Jonathan McCall committed Dec 17, 2018
    Configuration menu
    Copy the full SHA
    72f16cb View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2018

  1. Ensure hatch data can be updated for slaves currently hatching

    Jonathan McCall committed Dec 18, 2018
    Configuration menu
    Copy the full SHA
    50b8e7d View commit details
    Browse the repository at this point in the history

  2. Add test for start hatching accepted slave states 

    Checks that start_hatching sends messages to ready, running, and
hatching slaves.
    Jonathan McCall committed Dec 18, 2018
    Configuration menu
    Copy the full SHA
    bb9b473 View commit details
    Browse the repository at this point in the history

  3. Remove unneeded imports of mock

    Jonathan McCall committed Dec 18, 2018
    Configuration menu
    Copy the full SHA
    25272c6 View commit details
    Browse the repository at this point in the history