UI stops updating stats/charts when connection is interupted

[max-rocket-internet]

Description of issue

Sometimes we have some connectivity issue to the locust master. When this happens I can see jquery just stops.

Here is what Chrome dev tools looks like:

Expected behavior

Could jquery retry on error?

Actual behavior

It fails and stops.

Environment settings

• Python version: 3
• Locust version: 0.11

Steps to reproduce (for bug reports)

Have an unreliable connection while running a load test.

[cgoldberg]

When this happens I can see jquery just stops.

If you have connectivity issues, your stats are likely borked already. You should use --no-web mode for high throughput workloads.

If there is something actionable to be done here to diagnose, I can re-open this issue... but as-is, this issue is much too vague.

[max-rocket-internet]

Cool. Don't ask for clarification. Don't explain which part is vague. Just close the issue. Very helpful.

[cgoldberg]

Cool. Don't ask for clarification. Don't explain which part is vague. Just close the issue.

@max-rocket-internet
When triaging issues, items that require work are left open while ones that do not are closed. There is nothing to be done here and I don't have time to follow up with individual reporters to tease out details and ask clarifying questions. If you want to submit a better report, go ahead and open a new issue or ask for this one to be re-opened and submit more information.

The issue stated "Sometimes we have some connectivity issue to the locust master". To even attempt at diagnosing your issue, I would need much more detail so I could reproduce it and see if it is even valid, then apply some sort of fix. I would need to know things like: Under what circumstances does this happen? What is your topology and your environment details? What is a "connectivity issue" in this context? What does your locustfile script contain? What is the workload you are running? But as a maintainer, I don't feel the responsibility to ask all of these clarifying questions. The onus is on you to report an issue with sufficient detail so something can be done about it. Perhaps this could be better communicated in the issue template.

Very helpful

It is not my intention to be helpful to you individually... I was attempting to be helpful to the overall project by keeping the issue tracker uncluttered.

[max-rocket-internet]

This is a tracker for the development of Locust, not a user support forum. It exists solely to keep track of development issues that need to be fixed.

OK, sorry, I didn't realise this.

To even attempt at diagnosing your issue, I would need much more detail so I could reproduce it

You can reproduce it in 1 minute:

1. Start a load test with the UI, any load test, doesn't matter
2. Wait for chart to start to be updated
3. In Chrome dev tools, in Network tab, change online to offline
4. Wait for 2 jquery updates to fail
5. Switch offline to online
6. No more updates of chart or stats

At this point, sure you can reload, but then the chart data is lost.

Under what circumstances does this happen? What is your topology and your environment details?

Just a long load test, like 3 hours. I am on a laptop, with wifi, I have meetings and small breaks, make coffee etc. In this time, it's not uncommon for this jquery request to fail at some point for whatever environmental reason. I'm just asking if it could automatically retry instead of giving up.

[cgoldberg]

In this time, it's not uncommon for this jquery request
to fail at some point for whatever environmental reason.
At this point, sure you can reload, but then the chart
data is lost.

I agree we should handle that error condition more gracefully. But if that failure is common, something is wrong... any stats produced near the time of the errors should be discarded.

[max-rocket-internet]

I agree we should handle that error condition more gracefully

🎉

any stats produced near the time of the errors should be discarded

Sure but I have 10 million requests listed here. If a few of the get requests to retrieve the current stats don't succeed, it's not a huge problem. I want the UI to keep updating because I want to know if the thing is still running, is it still ramping up etc.

[cgoldberg]

If a few of the get requests to retrieve the current
stats don't succeed, it's not a huge problem

I guess we disagree on that... I think even a single failure of the UI to update is a sign that something is very wrong, and the stats displayed are suspect and no longer useful.

I want the UI to keep updating because I want to know if the thing is still running, is it still ramping

If I were running Locust tests and saw UI errors, I would immediately stop the test and fix the issue before continuing. But perhaps you are just using Locust as a dumb load generator, and the fact that it is spewing errors and displaying useless data doesn't concern you (but then why do you care about stats anyway?)

I am not going to implement any changes around this myself since I don't find recovering a saturated master node to be of much value for me. However, if you feel strongly and want to implement something, I can review your Pull Request.

[max-rocket-internet]

a sign that something is very wrong

It's not. It just means my laptop was asleep, or I was walking to a meeting and lost WiFi, etc.

the fact that it is spewing errors

No errors. The load test is running great. Just jquery couldn't connect once from my browser to the master.

I don't find recovering a saturated master node to be of much value for me

It's nothing to do with my master node. My master is running on AWS. My slaves are on AWS. The target environment is on AWS. They are all very happy, there is no "saturation". I am in an office. jquery runs in my browser. Make sense?

[cgoldberg]

My master is running on AWS. My slaves are on AWS.
The target environment is on AWS. They are all very happy,
there is no "saturation". I am in an office. jquery runs in my
browser. Make sense?

Now that you actually describe your topology it makes sense. I didn't consider your master node was not localhost.