Fix issue when running many productions in on the same machine in parallel #1211

orelgueta · 2024-10-17T07:14:33Z

No one complained about tests failing with this branch, so I open this PR which should fix #1209.
I mark everyone to review, partly because it would actually be good for everyone to run tests locally and see if any of the integration tests fail and partly because it is short and the first one can do it.

Best way to run the tests is with the following command:

pytest -n auto --no-cov tests/integration_tests/ --count 5 -k simtools-simulate-prod

Closes #1209.

…_running

ctao-dpps-sonarqube · 2024-10-17T07:24:24Z

Analysis Details

0 Issues

0 Bugs
0 Vulnerabilities
0 Code Smells

Coverage and Duplications

No coverage information (93.10% Estimated after merge)
No duplication information (0.00% Estimated after merge)

Project ID: gammasim_simtools_AY_ssha9WiFxsX-2oy_w

View in SonarQube

tobiaskleiner · 2024-10-17T07:56:14Z

Did run the tests twice with above command - all tests passed.

orelgueta · 2024-10-17T08:03:51Z

Thanks! I will give the others a chance to run the tests as well before I merge.

EshitaJoshi · 2024-10-17T09:09:33Z

Failed on my laptop. I'm up to date with this branch and checked that all the unit tests passed. I ran the command above and got 65 failed, 15 passed in 28.45 s. All of them are AssertionError: Application failed.

GernotMaier · 2024-10-17T09:28:29Z

Also fails for me:

pytest --no-cov -n auto --count 5 --random-order tests/integration_tests/test_applications_from_config.py -k simtools-simulate-prod
======================================================================== test session starts ========================================================================
platform linux -- Python 3.11.7, pytest-8.3.3, pluggy-1.5.0 -- /workdir/env/bin/python3.11
cachedir: .pytest_cache
Using --random-order-bucket=module
Using --random-order-seed=104236

rootdir: /workdir/external/simtools
configfile: pyproject.toml
plugins: cov-5.0.0, mock-3.14.0, profiling-1.7.0, random-order-1.1.1, repeat-0.9.3, xdist-3.6.1
10 workers [80 items]
scheduling tests via LoadScheduling

tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_auto-help-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_north_check_output-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_pack_for_grid-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north_check_output-1-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_north_check_output-4-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_proton_20_deg_north_check_output-3-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north_for_trigger_rates-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_auto-version-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_pack_for_grid-3-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_az0deg_south_check_output-3-5]
[gw2] [  1%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_auto-help-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_auto-version-2-5]
[gw3] [  2%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_auto-version-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north_for_trigger_rates-3-5]
[gw2] [  3%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_auto-version-2-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_diffuse_20_deg_south_check_output-3-5]
[gw0] [  5%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north_for_trigger_rates-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_auto-help-4-5]
[gw0] [  6%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_auto-help-4-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_north_check_output-3-5]
[gw3] [  7%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north_for_trigger_rates-3-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north_for_trigger_rates-2-5]
[gw4] [  8%] FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_az0deg_south_check_output-3-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north-5-5]
[gw2] [ 10%] FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_diffuse_20_deg_south_check_output-3-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_north_check_output-1-5]
[gw1] [ 11%] FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_pack_for_grid-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_proton_20_deg_south_check_output-2-5]
[gw6] [ 12%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_pack_for_grid-3-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_pack_for_grid-4-5]
[gw8] [ 13%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north_check_output-1-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north_for_trigger_rates-4-5]
[gw5] [ 15%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_proton_20_deg_north_check_output-3-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_diffuse_20_deg_south_check_output-4-5]
[gw7] [ 16%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_north_check_output-4-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_auto-no_config-3-5]
[gw7] [ 17%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_auto-no_config-3-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north_for_trigger_rates-1-5]
[gw9] [ 18%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_north_check_output-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_az0deg_south_check_output-5-5]

....

DEBUG::corsika_runner(l109)::prepare_run_script::Extra commands to be added to the run script: None
DEBUG::corsika_runner(l110)::prepare_run_script::CORSIKA data will be set to /tmp/pytest-of-root/pytest-0/popen-gw4/test-data6/simtools-simulate-prod-gamma_diffuse_20_deg_south_check_output/simtools-output/corsika/data
INFO::job_manager(l107)::submit::Submitting script /tmp/pytest-of-root/pytest-0/popen-gw4/test-data6/simtools-simulate-prod-gamma_diffuse_20_deg_south_check_output/simtools-output/sub_scripts/sub_run000001_gamma_diffuse_za20deg_azm180deg_South_alpha_check_output.sh
INFO::job_manager(l108)::submit::Job output stream /tmp/pytest-of-root/pytest-0/popen-gw4/test-data6/simtools-simulate-prod-gamma_diffuse_20_deg_south_check_output/simtools-output/sub_logs/sub_run000001_gamma_diffuse_za20deg_azm180deg_South_alpha_check_output.out
INFO::job_manager(l109)::submit::Job error stream /tmp/pytest-of-root/pytest-0/popen-gw4/test-data6/simtools-simulate-prod-gamma_diffuse_20_deg_south_check_output/simtools-output/sub_logs/sub_run000001_gamma_diffuse_za20deg_azm180deg_South_alpha_check_output.err
INFO::job_manager(l110)::submit::Job log stream /tmp/pytest-of-root/pytest-0/popen-gw4/test-data6/simtools-simulate-prod-gamma_diffuse_20_deg_south_check_output/simtools-output/sub_logs/sub_run000001_gamma_diffuse_za20deg_azm180deg_South_alpha_check_output.job
INFO::job_manager(l129)::_submit_local::Running script locally
INFO::simulate_prod(l195)::main::Production run is complete for primary gamma showers coming from 180.0 deg azimuth and zenith angle of 20.0 deg at the South site, using the 6.0.0 simulation model.
INFO::simulate_prod(l157)::pack_for_register::Packing the output files for registering on the grid
INFO::simulator(l450)::get_file_list::Getting list of output files
INFO::simulator(l450)::get_file_list::Getting list of log files
INFO::simulator(l450)::get_file_list::Getting list of hist files
Traceback (most recent call last):
  File "/workdir/env/bin/simtools-simulate-prod", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/workdir/external/simtools/simtools/applications/simulate_prod.py", line 203, in main
    pack_for_register(logger, simulator, args_dict)
  File "/workdir/external/simtools/simtools/applications/simulate_prod.py", line 171, in pack_for_register
    tar.add(file_to_tar, arcname=Path(file_to_tar).name)
  File "/usr/lib64/python3.11/tarfile.py", line 2171, in add
    tarinfo = self.gettarinfo(name, arcname)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/tarfile.py", line 2044, in gettarinfo
    statres = os.lstat(name)
              ^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-0/popen-gw4/test-data6/simtools-simulate-prod-gamma_diffuse_20_deg_south_check_output/simtools-output/simtel/logs/run000001_gamma_diffuse_za20deg_azm180deg_South_alpha_check_output.hdata.zst'
------------------------------------------------------------------------- Captured log call -------------------------------------------------------------------------
INFO     root:test_applications_from_config.py:246 Test configuration from config file: {'APPLICATION': 'simtools-simulate-prod', 'TEST_NAME': 'gamma_diffuse_20_deg_south_check_output', 'CONFIGURATION': {'SIMULATION_SOFTWARE': 'corsika_simtel', 'LABEL': 'check_output', 'MODEL_VERSION': '6.0.0', 'SITE': 'South', 'ARRAY_LAYOUT_NAME': 'alpha', 'PRIMARY': 'gamma', 'RUN_NUMBER_START': 1, 'NUMBER_OF_RUNS': 1, 'AZIMUTH_ANGLE': 'South', 'ZENITH_ANGLE': 20, 'NSHOW': 10, 'VIEW_CONE': '0 deg 10 deg', 'ENERGY_RANGE': '100 GeV 500 GeV', 'CORE_SCATTER': '20 700 m', 'DATA_DIRECTORY': 'simtools-output', 'OUTPUT_PATH': 'simtools-output', 'SIM_TELARRAY_SEEDS': '1745,290', 'CORSIKA_TEST_SEEDS': True, 'PACK_FOR_GRID_REGISTER': True, 'LOG_LEVEL': 'DEBUG'}, 'INTEGRATION_TESTS': [{'OUTPUT_FILE': 'directory_for_grid_upload/run000001_gamma_diffuse_za20deg_azm180deg_South_alpha_check_output.log_hist.tar.gz'}, {'OUTPUT_FILE': 'directory_for_grid_upload/run000001_gamma_diffuse_za20deg_azm180deg_South_alpha_check_output.zst'}, {'EXPECTED_OUTPUT': {'trigger_time': [0, 50], 'pe_sum': [20, 1000], 'photons': [90, 1000]}}]}
INFO     root:test_applications_from_config.py:284 Temporary output path: /tmp/pytest-of-root/pytest-0/popen-gw4/test-data6/simtools-simulate-prod-gamma_diffuse_20_deg_south_check_output
INFO     root:test_applications_from_config.py:250 Model version: None
INFO     root:test_applications_from_config.py:251 Application configuration: {'APPLICATION': 'simtools-simulate-prod', 'TEST_NAME': 'gamma_diffuse_20_deg_south_check_output', 'CONFIGURATION': {'SIMULATION_SOFTWARE': 'corsika_simtel', 'LABEL': 'check_output', 'MODEL_VERSION': '6.0.0', 'SITE': 'South', 'ARRAY_LAYOUT_NAME': 'alpha', 'PRIMARY': 'gamma', 'RUN_NUMBER_START': 1, 'NUMBER_OF_RUNS': 1, 'AZIMUTH_ANGLE': 'South', 'ZENITH_ANGLE': 20, 'NSHOW': 10, 'VIEW_CONE': '0 deg 10 deg', 'ENERGY_RANGE': '100 GeV 500 GeV', 'CORE_SCATTER': '20 700 m', 'DATA_DIRECTORY': 'simtools-output', 'OUTPUT_PATH': 'simtools-output', 'SIM_TELARRAY_SEEDS': '1745,290', 'CORSIKA_TEST_SEEDS': True, 'PACK_FOR_GRID_REGISTER': True, 'LOG_LEVEL': 'DEBUG'}, 'INTEGRATION_TESTS': [{'OUTPUT_FILE': 'directory_for_grid_upload/run000001_gamma_diffuse_za20deg_azm180deg_South_alpha_check_output.log_hist.tar.gz'}, {'OUTPUT_FILE': 'directory_for_grid_upload/run000001_gamma_diffuse_za20deg_azm180deg_South_alpha_check_output.zst'}, {'EXPECTED_OUTPUT': {'trigger_time': [0, 50], 'pe_sum': [20, 1000], 'photons': [90, 1000]}}]}
INFO     root:test_applications_from_config.py:213 Writing config file: /tmp/pytest-of-root/pytest-0/popen-gw4/test-data6/simtools-simulate-prod-gamma_diffuse_20_deg_south_check_output/tmp_config.yml
INFO     root:test_applications_from_config.py:354 Running application: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw4/test-data6/simtools-simulate-prod-gamma_diffuse_20_deg_south_check_output/tmp_config.yml
====================================================================== short test summary info ======================================================================
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_az0deg_south_check_output-3-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw4/test-data0/simtools-simulate-prod-gamma_20_deg_az0deg...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_diffuse_20_deg_south_check_output-3-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw2/test-data2/simtools-simulate-prod-gamma_diffuse_20_de...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_pack_for_grid-5-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw1/test-data0/simtools-simulate-prod-gamma_20_deg_south_...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_pack_for_grid-4-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw6/test-data1/simtools-simulate-prod-gamma_20_deg_south_...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_proton_20_deg_south_check_output-2-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw1/test-data1/simtools-simulate-prod-proton_20_deg_south...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_diffuse_20_deg_south_check_output-4-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw5/test-data1/simtools-simulate-prod-gamma_diffuse_20_de...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_check_output-5-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw0/test-data4/simtools-simulate-prod-gamma_20_deg_south_...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_south_check_output-5-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw8/test-data3/simtools-simulate-prod-gamma_40_deg_south_...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_south_check_output-1-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw5/test-data3/simtools-simulate-prod-gamma_40_deg_south_...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_check_output-1-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw0/test-data5/simtools-simulate-prod-gamma_20_deg_south_...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_south_check_output-2-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw1/test-data3/simtools-simulate-prod-gamma_40_deg_south_...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_diffuse_20_deg_south_check_output-5-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw8/test-data4/simtools-simulate-prod-gamma_diffuse_20_de...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_check_output-2-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw3/test-data5/simtools-simulate-prod-gamma_20_deg_south_...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_diffuse_20_deg_south_check_output-2-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw7/test-data6/simtools-simulate-prod-gamma_diffuse_20_de...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_check_output-4-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw2/test-data8/simtools-simulate-prod-gamma_20_deg_south_...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_az0deg_south_check_output-2-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw5/test-data5/simtools-simulate-prod-gamma_20_deg_az0deg...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_proton_20_deg_south_check_output-5-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw6/test-data6/simtools-simulate-prod-proton_20_deg_south...
FAILED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_diffuse_20_deg_south_check_output-1-5] - AssertionError: Application failed: simtools-simulate-prod --config /tmp/pytest-of-root/pytest-0/popen-gw4/test-data6/simtools-simulate-prod-gamma_diffuse_20_de...
============================================================= 18 failed, 62 passed in 402.21s (0:06:42) ========================================================

This is exactly the same error as before.

orelgueta · 2024-10-17T09:47:48Z

This is not great because I cannot reproduce the errors...

(env) pytest --no-cov -n auto --count 5 --random-order tests/integration_tests/test_applications_from_config.py -k simtools-simulate-prod
==================================================================== test session starts =====================================================================
platform linux -- Python 3.11.7, pytest-8.3.3, pluggy-1.5.0 -- /workdir/env/bin/python3.11
cachedir: .pytest_cache
Using --random-order-bucket=module
Using --random-order-seed=32610

rootdir: /workdir/external/simtools
configfile: pyproject.toml
plugins: cov-5.0.0, mock-3.14.0, profiling-1.7.0, random-order-1.1.1, repeat-0.9.3, xdist-3.6.1
10 workers [80 items]
scheduling tests via LoadScheduling

tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_corsika_corsika7_particle_id-3-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_corsika_corsika7_particle_id-2-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_az0deg_south_check_output-4-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_north_check_output-5-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_north_check_output-1-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_north_check_output-4-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_proton_20_deg_north_check_output-4-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_south_check_output-2-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_south_pack_for_grid-1-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_40_deg_north_check_output-3-5]
[gw1] [  1%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_corsika_corsika7_particle_id-2-5]
tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_corsika_corsika7_particle_id-4-5]
[gw4] [  2%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_corsika_corsika7_particle_id-3-5]
...
[gw6] [100%] PASSED tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_north_check_output-1-5]

=============================================================== 80 passed in 232.56s (0:03:52) ===============================================================

I ran this 3 times and do not see an error.

How do we find the difference between our setups? My branch is as follows:

git show --summary
commit d5737230f4490447b7da04d694eecb55850d0a0d (HEAD -> fix_parallel_running, origin/fix_parallel_running)
Merge: 22dce4e2e 301d8b1b0
Author: ****** <*****>
Date:   Thu Oct 17 09:07:34 2024 +0200

    Merge branch 'main' of github.com:gammasim/simtools into fix_parallel_running

The hash of the container I am using is sha256:08170331c34a13041d91cd6e66294d886e449dd074bbf27a58b70e9f912f17dd and it was created 2024-10-15T13:57:25.76234547Z.

Does this information help?

GernotMaier · 2024-10-17T09:51:55Z

This seems to be the same as for me:

git show --summary

commit d5737230f4490447b7da04d694eecb55850d0a0d (HEAD -> fix_parallel_running, origin/fix_parallel_running)
Merge: 22dce4e2e 301d8b1b0
Author: **** <***>
Date:   Thu Oct 17 09:07:34 2024 +0200

    Merge branch 'main' of github.com:gammasim/simtools into fix_parallel_running

Running with remote DB in a podman container which I start with (exactly as in the manual):

docker run --rm -it -v "$(pwd)/:/workdir/external" ghcr.io/gammasim/simtools-dev:latest bash -c "source /workdir/env/bin/activate && cd /workdir/external/simtools && pip install -e . && bash"

I assume you are running this in a container and you are giving docker/podman many cores? I have assigned podman 10 cores.

EshitaJoshi · 2024-10-17T09:57:00Z

My branch is the same:


git show --summary
commit d5737230f4490447b7da04d694eecb55850d0a0d (HEAD -> fix_parallel_running, origin/fix_parallel_running)
Merge: 22dce4e2e 301d8b1b0
Author: ****** <*****>
Date:   Thu Oct 17 09:07:34 2024 +0200
Merge branch 'main' of github.com:gammasim/simtools into fix_parallel_running

I am not using docker, I am running it in a linux terminal with the simtools-dev conda environment using all 14 cores of my laptop.

tobiaskleiner · 2024-10-17T10:07:28Z

I run with podman container with 8 cores assigned.

orelgueta · 2024-10-17T10:08:39Z

Running with remote DB in a podman container which I start with (exactly as in the manual):
docker run --rm -it -v "$(pwd)/:/workdir/external" ghcr.io/gammasim/simtools-dev:latest bash -c "source /workdir/env/bin/activate && cd /workdir/external/simtools && pip install -e . && bash"

Yes, exactly the same.

I assume you are running this in a container and you are giving docker/podman many cores? I have assigned podman 10 cores.

Same, you can see 10 workers [80 items] in my printout.

I also tried downloading the latest container which was created 2024-10-16T10:24:21.183022679Z. I still do not get errors. However, since @EshitaJoshi does not use the container, maybe it is meaningless anyway.

tobiaskleiner · 2024-10-17T10:10:46Z

Which podman version are you using? Podman v5.2.4?

orelgueta · 2024-10-17T10:13:30Z

Which podman version are you using? Podman v5.2.4?

5.2.0 (which is weird since I just updated my client yesterday, but I guess it is different to the CLI).
But that should not be the issue I expect.

GernotMaier · 2024-10-17T10:22:20Z

Pulled the latest image (gammasimtools-0.7.1.dev603+gd5737230f) and I am using podman 5.2.0

orelgueta · 2024-10-17T10:28:59Z

I even used the same random order seed as Gernot and I still don't get the errors. I also printed the version and get

simtools-simulate-prod --version
simtools-simulate-prod 0.7.1.dev603+gd5737230f

EshitaJoshi · 2024-10-17T10:31:28Z

Oh, my version seems to be different:

simtools-simulate-prod --version
simtools-simulate-prod 0.7.1.dev388+g488a23271.d20240927

orelgueta · 2024-10-17T12:06:03Z

Oh, my version seems to be different:

simtools-simulate-prod --version
simtools-simulate-prod 0.7.1.dev388+g488a23271.d20240927

I think this is just because you do not run pip install -e . like we do whenever we start the docker. If you run git fetch upstream --tags and pip install -e ., your version should then be the same as ours but without any actual code changes.

orelgueta added 2 commits October 16, 2024 10:06

Write the tarball directly into the directory for grid

22dce4e

Merge branch 'main' of github.com:gammasim/simtools into fix_parallel…

d573723

…_running

orelgueta requested review from GernotMaier, tobiaskleiner and EshitaJoshi October 17, 2024 07:14

tobiaskleiner approved these changes Oct 17, 2024

View reviewed changes

tobiaskleiner self-requested a review October 25, 2024 12:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix issue when running many productions in on the same machine in parallel #1211

Fix issue when running many productions in on the same machine in parallel #1211

orelgueta commented Oct 17, 2024

ctao-dpps-sonarqube bot commented Oct 17, 2024

tobiaskleiner commented Oct 17, 2024

orelgueta commented Oct 17, 2024

EshitaJoshi commented Oct 17, 2024 •

edited

Loading

GernotMaier commented Oct 17, 2024

orelgueta commented Oct 17, 2024

GernotMaier commented Oct 17, 2024 •

edited by orelgueta

Loading

EshitaJoshi commented Oct 17, 2024

tobiaskleiner commented Oct 17, 2024

orelgueta commented Oct 17, 2024

tobiaskleiner commented Oct 17, 2024 •

edited

Loading

orelgueta commented Oct 17, 2024

GernotMaier commented Oct 17, 2024

orelgueta commented Oct 17, 2024

EshitaJoshi commented Oct 17, 2024

orelgueta commented Oct 17, 2024

Fix issue when running many productions in on the same machine in parallel #1211

Are you sure you want to change the base?

Fix issue when running many productions in on the same machine in parallel #1211

Conversation

orelgueta commented Oct 17, 2024

ctao-dpps-sonarqube bot commented Oct 17, 2024

Analysis Details

0 Issues

Coverage and Duplications

tobiaskleiner commented Oct 17, 2024

orelgueta commented Oct 17, 2024

EshitaJoshi commented Oct 17, 2024 • edited Loading

GernotMaier commented Oct 17, 2024

orelgueta commented Oct 17, 2024

GernotMaier commented Oct 17, 2024 • edited by orelgueta Loading

EshitaJoshi commented Oct 17, 2024

tobiaskleiner commented Oct 17, 2024

orelgueta commented Oct 17, 2024

tobiaskleiner commented Oct 17, 2024 • edited Loading

orelgueta commented Oct 17, 2024

GernotMaier commented Oct 17, 2024

orelgueta commented Oct 17, 2024

EshitaJoshi commented Oct 17, 2024

orelgueta commented Oct 17, 2024

EshitaJoshi commented Oct 17, 2024 •

edited

Loading

GernotMaier commented Oct 17, 2024 •

edited by orelgueta

Loading

tobiaskleiner commented Oct 17, 2024 •

edited

Loading