Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libbeat/publisher/pipeline: fix data races #19821

Merged
merged 1 commit into from
Jul 14, 2020
Merged

Conversation

axw
Copy link
Member

@axw axw commented Jul 13, 2020

What does this PR do?

Fix how we pass the initial queue consumer into eventConsumer.loop;
we were referencing c.consumer in a background goroutine, which can
race with updates to the consumer.

Update tests to properly load atomic variables. Changed serially updated
numEvents vars to basic, non-atomic types.

Why is it important?

I'm not sure if this particular data race would cause any issues in production,
but we (apm-server) rely on the race detector to pick up real issues. This is
causing our tests to fail.

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

(Does this change need a changelog entry? This is a fix to a recently merged change.)

How to test this PR locally

go test -race ./libbeat/publisher/pipeline

Fix how we pass the initial queue consumer into
eventConsumer.loop; we were referencing c.consumer
in a background goroutine, which can race with
updates to the consumer.

Update tests to properly load atomic variables.
Changed serially updated numEvents vars to basic,
non-atomic types.
@axw axw added bug Team:Services (Deprecated) Label for the former Integrations-Services team labels Jul 13, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 13, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 13, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [urso commented: jenkins run the tests please]

  • Start Time: 2020-07-13T13:07:40.324+0000

  • Duration: 127 min 17 sec

Test stats 🧪

Test Results
Failed 1
Passed 7447
Skipped 1427
Total 8875

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat OSS Unit tests / test_process – test_system.Test

    • Age: 1
    • Duration: 0.763
    • Error Details: False is not true : fd not found in any process events

Steps errors

Expand to view the steps failures

  • Name: Make -C filebeat testsuite

    • Description: make -C filebeat testsuite

    • Duration: 7 min 58 sec

    • Start Time: 2020-07-13T13:37:01.238+0000

    • log

  • Name: Mage update build test

    • Description: mage update build test

    • Duration: 5 min 10 sec

    • Start Time: 2020-07-13T13:37:03.933+0000

    • log

  • Name: Make -C auditbeat testsuite

    • Description: make -C auditbeat testsuite

    • Duration: 8 min 49 sec

    • Start Time: 2020-07-13T13:37:05.944+0000

    • log

  • Name: Mage update build test

    • Description: mage update build test

    • Duration: 4 min 49 sec

    • Start Time: 2020-07-13T13:37:19.985+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 1 min 27 sec

    • Start Time: 2020-07-13T14:01:20.004+0000

    • log

  • Name: Make -C libbeat testsuite

    • Description: make -C libbeat testsuite

    • Duration: 9 min 36 sec

    • Start Time: 2020-07-13T13:37:33.339+0000

    • log

  • Name: Make -C x-pack/libbeat testsuite

    • Description: make -C x-pack/libbeat testsuite

    • Duration: 6 min 49 sec

    • Start Time: 2020-07-13T13:37:31.524+0000

    • log

  • Name: Mage build unitTest

    • Description: mage build unitTest

    • Duration: 6 min 0 sec

    • Start Time: 2020-07-13T13:37:35.410+0000

    • log

  • Name: Mage goIntegTest

    • Description: mage goIntegTest

    • Duration: 5 min 49 sec

    • Start Time: 2020-07-13T13:37:37.246+0000

    • log

  • Name: Mage pythonIntegTest

    • Description: mage pythonIntegTest

    • Duration: 3 min 19 sec

    • Start Time: 2020-07-13T13:37:39.746+0000

    • log

  • Name: Mage build test

    • Description: mage build test

    • Duration: 8 min 45 sec

    • Start Time: 2020-07-13T13:41:44.481+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-07-13T13:50:13.782+0000

    • log

  • Name: Make -C packetbeat testsuite

    • Description: make -C packetbeat testsuite

    • Duration: 7 min 30 sec

    • Start Time: 2020-07-13T13:37:53.552+0000

    • log

  • Name: Make -C generator/_templates/metricbeat test-package

    • Description: make -C generator/_templates/metricbeat test-package

    • Duration: 7 min 18 sec

    • Start Time: 2020-07-13T13:41:25.084+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-13T15:14:29.085Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-13T15:14:29.085Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-13T15:14:29.085Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-07-13T15:14:29.085Z] Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
[2020-07-13T15:14:29.085Z] 	at hudson.FilePath.act(FilePath.java:1071)
[2020-07-13T15:14:29.085Z] 	at hudson.FilePath.act(FilePath.java:1058)
[2020-07-13T15:14:29.085Z] 	at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-07-13T15:14:29.085Z] 	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-07-13T15:14:29.085Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-07-13T15:14:29.085Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-07-13T15:14:29.085Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-07-13T15:14:29.085Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-07-13T15:14:29.085Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-07-13T15:14:29.085Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-07-13T15:14:29.085Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-07-13T15:14:29.085Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-07-13T15:14:29.085Z] No artifacts found that match the file pattern "**\build\TEST*.out". Configuration error?
[2020-07-13T15:14:30.386Z] Failed in branch Functionbeat
[2020-07-13T15:14:30.541Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats
[2020-07-13T15:14:30.860Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-13T15:14:30.873Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Lint
[2020-07-13T15:14:30.962Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-13T15:14:31.038Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Winlogbeat-oss
[2020-07-13T15:14:31.135Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-07-13T15:14:31.227Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-13T15:14:31.312Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-07-13T15:14:31.389Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Journalbeat-oss
[2020-07-13T15:14:31.514Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-07-13T15:14:31.613Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-13T15:14:31.728Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-13T15:14:31.831Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-13T15:14:31.946Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-13T15:14:32.057Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-07-13T15:14:32.160Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-13T15:14:32.271Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-07-13T15:14:32.385Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Filebeat-oss
[2020-07-13T15:14:32.470Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-07-13T15:14:32.556Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-07-13T15:14:32.650Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-13T15:14:32.750Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-13T15:14:32.835Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Packetbeat-oss
[2020-07-13T15:14:32.912Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Libbeat-oss
[2020-07-13T15:14:32.992Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-13T15:14:33.078Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-13T15:14:33.165Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Heartbeat-oss
[2020-07-13T15:14:33.254Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-07-13T15:14:33.343Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-13T15:14:33.424Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-13T15:14:33.526Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-07-13T15:14:33.619Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Metricbeat-Windows
[2020-07-13T15:14:33.706Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-07-13T15:14:33.787Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-13T15:14:33.881Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-07-13T15:14:33.969Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-13T15:14:34.071Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-07-13T15:14:34.173Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-13T15:14:34.282Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-13T15:14:34.387Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-07-13T15:14:34.478Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-07-13T15:14:34.569Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Heartbeat-Windows
[2020-07-13T15:14:34.670Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats/Functionbeat-Windows
[2020-07-13T15:14:35.051Z] + cat
[2020-07-13T15:14:35.051Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-13T15:14:35.051Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-13T15:14:41.647Z] runbld>>> runbld started
[2020-07-13T15:14:41.647Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-13T15:14:44.194Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-19821' in order of occurrence in the config (last value wins).
[2020-07-13T15:14:45.136Z] runbld>>> Debug logging enabled.
[2020-07-13T15:14:45.136Z] runbld>>> Storing result
[2020-07-13T15:14:45.399Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-13T15:14:45.399Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200713151444-5D31D8B0
[2020-07-13T15:14:45.399Z] runbld>>> Adding system facts.
[2020-07-13T15:14:46.341Z] runbld>>> Adding vcs info for the latest commit:  b1e4a51106cdeb64e11bce4e7a3bdc966f1ed829
[2020-07-13T15:14:46.341Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-13T15:14:46.341Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-13T15:14:46.341Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-13T15:14:46.341Z] Processing JUnit reports with runbld...
[2020-07-13T15:14:46.603Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-13T15:14:46.603Z] runbld>>> DURATION: 10ms
[2020-07-13T15:14:46.603Z] runbld>>> STDOUT: 40 bytes
[2020-07-13T15:14:46.603Z] runbld>>> STDERR: 49 bytes
[2020-07-13T15:14:46.603Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-13T15:14:46.603Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats
[2020-07-13T15:14:47.990Z] runbld>>> Storing build metadata: 
[2020-07-13T15:14:47.990Z] runbld>>> Adding test report.
[2020-07-13T15:14:47.990Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-19821/src/github.com/elastic/beats
[2020-07-13T15:14:48.564Z] runbld>>> Found 43 test output files
[2020-07-13T15:14:50.483Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 8729 Skipped: 1316
[2020-07-13T15:14:50.483Z] runbld>>> Storing result
[2020-07-13T15:14:50.483Z] runbld>>> FAILURES: 1
[2020-07-13T15:14:51.054Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-13T15:14:51.055Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200713151444-5D31D8B0
[2020-07-13T15:14:51.055Z] runbld>>> Email notification disabled by environment variable.
[2020-07-13T15:14:51.055Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-13T15:14:56.613Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-19821
[2020-07-13T15:14:56.711Z] [INFO] getVaultSecret: Getting secrets
[2020-07-13T15:14:56.784Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-13T15:14:57.536Z] + chmod 755 generate-build-data.sh
[2020-07-13T15:14:57.536Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19821/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19821/runs/2 FAILURE 7636951
[2020-07-13T15:14:57.536Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19821/runs/2/steps/?limit=10000 -o steps-info.json

@urso
Copy link

urso commented Jul 13, 2020

jenkins run the tests please

@urso
Copy link

urso commented Jul 13, 2020

Hm weird, Beats tests are run with -race enabled as well. Thanks for the fix.

(Does this change need a changelog entry? This is a fix to a recently merged change.)

No changelog required.

@axw
Copy link
Member Author

axw commented Jul 14, 2020

Test failures appear to be unrelated, merging.

@axw axw merged commit ebacd3b into elastic:master Jul 14, 2020
@axw axw deleted the fix-data-races branch July 14, 2020 02:38
axw added a commit to axw/beats that referenced this pull request Jul 14, 2020
Fix how we pass the initial queue consumer into
eventConsumer.loop; we were referencing c.consumer
in a background goroutine, which can race with
updates to the consumer.

Update tests to properly load atomic variables.
Changed serially updated numEvents vars to basic,
non-atomic types.

(cherry picked from commit ebacd3b)
@axw axw added the v7.9.0 label Jul 14, 2020
v1v added a commit to v1v/beats that referenced this pull request Jul 14, 2020
* upstream/master: (25 commits)
  [Elastic Agent] Send checkin payload to Fleet (elastic#19857)
  [Ingest Manager] Fixed tests across agent elastic#19877
  [Ingest Manager] Fix serialization test  elastic#19876
  Fix service start type mapping in windows/service metricset (elastic#19551)
  ci: Change comment trigger detection method (elastic#19827)
  Add 21 autogenerated filesets from rsa2elk devices (elastic#19713)
  [Ingest Manager] Agent config cleanup (elastic#19848)
  libbeat/publisher/pipeline: fix data races (elastic#19821)
  Update monitoring-internal-collection.asciidoc (elastic#19422) (elastic#19697)
  [Elastic Agent] Trust exchange endpoint must bind to 127.0.0.1 (elastic#19861)
  Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat (elastic#19159)
  Add azure billing metricset (elastic#19207)
  Add support for appinsights in the metricbeat azure module (elastic#18940)
  Add MySQL query metricset with lightweight module and SQL helper (elastic#18955)
  [Ingest Manager] Refuse invalid stream values in configuration (elastic#19587)
  Do not use vendor during integration tests (elastic#19839)
  LIBBEAT: Enhancement Convert dissected values from String to other basic data types and IP (elastic#18683)
  [Elastic Agent] Remove support for "logs" and only support logfile (elastic#19761)
  [CI] support windows-2012 (elastic#19773)
  Do not update go.mod during packaging and testing (elastic#19823)
  ...
axw added a commit that referenced this pull request Jul 15, 2020
Fix how we pass the initial queue consumer into
eventConsumer.loop; we were referencing c.consumer
in a background goroutine, which can race with
updates to the consumer.

Update tests to properly load atomic variables.
Changed serially updated numEvents vars to basic,
non-atomic types.

(cherry picked from commit ebacd3b)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
Fix how we pass the initial queue consumer into
eventConsumer.loop; we were referencing c.consumer
in a background goroutine, which can race with
updates to the consumer.

Update tests to properly load atomic variables.
Changed serially updated numEvents vars to basic,
non-atomic types.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Team:Services (Deprecated) Label for the former Integrations-Services team v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants