Simulate prod test

[VictorBarbosaMartins]

Done:
Here I am including some missing tests to check whether the files for either corsika or simtel_array output were generated by the production tool. We are constantly breaking this tool because the tests are not checking whether the output files exist. The integration tests for simulate_prod.py will fail at the moment.

I removed the test flag for sim_showers_for_trigger_rates because it was preventing us to simulate and de facto test the tool. To minimize computational time of the simulation, I reduced the number of events to 2.

To do:
The best would be to make the corrections to the tool before merging this and have the integration tests passing.

[GernotMaier]

I've looked into

pytest --no-cov "tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg]"

and files (although very small ones are generated for me at /tmp/pytest-of-root/pytest-1/test-data0/simtools-simulate-prod-gamma_20_deg/simtools-output/test-production//simtel-data/North/gamma/data/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.simtel.zst

The log files look reasonable and it runs through without errors.

There are issues, many statements like

Warning: problematic transmission between telescope and observation level.
Evaluation was between observation layer at 21.560000 m a.s.l.
and 31.000000 m above, plus 16.000000 m in the telescope.
Lookup in transmission table in wavelength range from 200 to 1000 nm,
   altitude range from 2.156000 to 100.000000 km a.s.l.,
   for wavelength 400 nm from 0.052560 km a.s.l., for airmass factor 1.000000
Starting altitude of 0.052560 km is out of range (2.156000 to 100.000000 km).
This is either an underground installation or the wrong transmission table.
At 400 nm we get a transm. of 0.000000 between tel. and obs.lev., and 0.000000 inside tel.
Your transmission table might not be appropriate for the given site.
We are ignoring this problematic part now but you should fix your set-up.

in the log file.

[VictorBarbosaMartins]

I've looked into

pytest --no-cov "tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg]"

and files (although very small ones are generated for me at /tmp/pytest-of-root/pytest-1/test-data0/simtools-simulate-prod-gamma_20_deg/simtools-output/test-production//simtel-data/North/gamma/data/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.simtel.zst

The log files look reasonable and it runs through without errors.

There are issues, many statements like

Warning: problematic transmission between telescope and observation level.
Evaluation was between observation layer at 21.560000 m a.s.l.
and 31.000000 m above, plus 16.000000 m in the telescope.
Lookup in transmission table in wavelength range from 200 to 1000 nm,
   altitude range from 2.156000 to 100.000000 km a.s.l.,
   for wavelength 400 nm from 0.052560 km a.s.l., for airmass factor 1.000000
Starting altitude of 0.052560 km is out of range (2.156000 to 100.000000 km).
This is either an underground installation or the wrong transmission table.
At 400 nm we get a transm. of 0.000000 between tel. and obs.lev., and 0.000000 inside tel.
Your transmission table might not be appropriate for the given site.
We are ignoring this problematic part now but you should fix your set-up.

in the log file.

Hi @GernotMaier, thanks for checking. This also works for me when running pytest in the container as you suggested. However, it does not work when running the application in the terminal (even in the master branch): simtools-simulate-prod --config tests/integration_tests/config/simulate_prod_gamma_20_deg.yml. The simtools-output/test-production/simtel-data/North/gamma/data/ directory is empty because of an error in the corsika part of the script as pointed out in the corsika log file in simtools-output/test-production/simtel-data/North/gamma/log/:

Setting up pipe 1 as: /workdir/sim_telarray/sim_telarray/bin/sim_telarray -c /workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg -I/workdir/external/simtools/simtools-output/test-production/model -C telescope_theta=20.0 -C telescope_phi=0.0 -C power_law=2.0 -C histogram_file=/workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/log/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.hdata.zst -C output_file=/workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/data/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.simtel.zst -C random_state=none -C show=all - | gzip > /workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/log/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.log.gz 2>&1 || exit
Configuration file is '/workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg'.
Preprocessor command is '/workdir/sim_telarray/sim_telarray/bin/pfp -v -I. -DWITH_LOW_GAIN_CHANNEL -DMAX_GAINS=2 -DWITH_GSL_RNG -DSIMTEL_VERSION=1635199717 -DSIMTEL_RELEASE=20211027 -I/workdir/external/simtools/simtools-output/test-production/model -I. -I/workdir/sim_telarray/sim_telarray/cfg -I/workdir/sim_telarray/sim_telarray/cfg/common -I/workdir/sim_telarray/sim_telarray/cfg/hess -I/workdir/sim_telarray/sim_telarray/cfg/hess2 -I/workdir/sim_telarray/sim_telarray/cfg/hess3 -I/workdir/sim_telarray/sim_telarray/cfg/hess5000 -I/workdir/sim_telarray/sim_telarray/cfg/CTA /workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg /tmp/cfg269.tmp'.
Unknown configuration item 'site_config_name'.
Unknown configuration item 'site_config_variant'.
Unknown configuration item 'site_config_version'.
Syntax error for configuration item 'EFFECTIVE_FOCAL_Length': Index out of range
Unknown configuration item 'fadc_compensate_pedestal'.
Unknown configuration item 'fadc_err_compensate_pedestal'.
Unknown configuration item 'fadc_lg_compensate_pedestal'.
Unknown configuration item 'fadc_lg_err_compensate_pedestal'.
Error in configuration data or incomplete configuration
Error in initialisation of the configuration
Error in configuration

Setup not complete.
Received signal 13
Broken pipe (unit 1) detected. Trying to handle this gracefully.
Closing pipe 1
No process left listening to me. Giving up.
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer

Could you please try to reproduce this?

[VictorBarbosaMartins]

Also the file you pointed out that exists is already being checked for in the integration test in this branch, which lead to a failure in the test. I am not entirely sure what is going on.

https://github.com/gammasim/simtools/actions/runs/8723725721/job/23932598219?pr=875#step:5:554

[GernotMaier]

@VictorBarbosaMartins , I can run python simtools/applications/simulate_prod.py --config tests/integration_tests/config/simulate_prod_gamma_20_deg.yml without any error messages.

Are you sure you are using a recent installation of simtools?

(there are some warnings regarding observation levels, which are related to something else)

[GernotMaier]

@VictorBarbosaMartins - everything runs fine with me locally without any errors. Can you check again?

Also please check the integration test which is failing on github (tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_pack_for_grid]). It is probably mostly related to the paths where output is located.

[VictorBarbosaMartins]

@VictorBarbosaMartins , I can run python simtools/applications/simulate_prod.py --config tests/integration_tests/config/simulate_prod_gamma_20_deg.yml without any error messages.

Are you sure you are using a recent installation of simtools?

(there are some warnings regarding observation levels, which are related to something else)

Hi @GernotMaier. The problem is not visible at first, one has to dig in. Naturally there is no error printed out, and that is the problem(!) The application runs smoothly (simtools-simulate-prod --config tests/integration_tests/config/simulate_prod_gamma_20_deg.yml), but there is no simtel file produced. If one checks the log file it doesnt look too bad, then when one looks at the corsika log file, a number of unknown configuration items break the process. I already posted the error here, but here it is again (in log_corsika_run000002_gamma_North_TestLayout_test-production.log):

Setting up pipe 1 as: /workdir/sim_telarray/sim_telarray/bin/sim_telarray -c /workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg -I/workdir/external/simtools/simtools-output/test-production/model -C telescope_theta=20.0 -C telescope_phi=0.0 -C power_law=2.0 -C histogram_file=/workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/log/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.hdata.zst -C output_file=/workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/data/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.simtel.zst -C random_state=none -C show=all - | gzip > /workdir/external/simtools/simtools-output/test-production/simtel-data/North/gamma/log/run000002_gamma_za020deg_azm000deg_North_TestLayout_test-production.log.gz 2>&1 || exit
Configuration file is '/workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg'.
Preprocessor command is '/workdir/sim_telarray/sim_telarray/bin/pfp -v -I. -DWITH_LOW_GAIN_CHANNEL -DMAX_GAINS=2 -DWITH_GSL_RNG -DSIMTEL_VERSION=1635199717 -DSIMTEL_RELEASE=20211027 -I/workdir/external/simtools/simtools-output/test-production/model -I. -I/workdir/sim_telarray/sim_telarray/cfg -I/workdir/sim_telarray/sim_telarray/cfg/common -I/workdir/sim_telarray/sim_telarray/cfg/hess -I/workdir/sim_telarray/sim_telarray/cfg/hess2 -I/workdir/sim_telarray/sim_telarray/cfg/hess3 -I/workdir/sim_telarray/sim_telarray/cfg/hess5000 -I/workdir/sim_telarray/sim_telarray/cfg/CTA /workdir/external/simtools/simtools-output/test-production/model/CTA-TestLayout-North-prod6_test-production.cfg /tmp/cfg217.tmp'.
Unknown configuration item 'site_config_name'.
Unknown configuration item 'site_config_variant'.
Unknown configuration item 'site_config_version'.
Syntax error for configuration item 'EFFECTIVE_FOCAL_Length': Index out of range
Unknown configuration item 'fadc_compensate_pedestal'.
Unknown configuration item 'fadc_err_compensate_pedestal'.
Unknown configuration item 'fadc_lg_compensate_pedestal'.
Unknown configuration item 'fadc_lg_err_compensate_pedestal'.
Error in configuration data or incomplete configuration
Error in initialisation of the configuration
Error in configuration

Setup not complete.
Received signal 13
Broken pipe (unit 1) detected. Trying to handle this gracefully.
Closing pipe 1
No process left listening to me. Giving up.
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer
Output error for I/O buffer

To worsen even more, I found out that the integration test of the output file (pytest --no-cov "tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg]") also does not raise an error but it is indeed not working. Tests are passing regardless of the name of the file one puts there.

I will look into these issues now but I will need help with the issue with the parameters. Please follow my suggestion above and check the corsika log file.

[VictorBarbosaMartins]

@VictorBarbosaMartins - everything runs fine with me locally without any errors. Can you check again?

Also please check the integration test which is failing on github (tests/integration_tests/test_applications_from_config.py::test_applications_from_config[simtools-simulate-prod_gamma_20_deg_pack_for_grid]). It is probably mostly related to the paths where output is located.

And this error is related to the other one. Here the simtel file is not produced, and an attempt to pack the output for registering on the grid fails. Given that this was working before, I assume that the error is not on the application/module but rather on the fact that simtel/corsika is not running properly. It is better to fix the error with running corsika/simtel_array before checking this failing integration test. I am looking into this rn.

[VictorBarbosaMartins]

The message above is quite clear on where the errors are coming. Wrong names of parameters. I also could not find those parameters in the DB. I am convinced these names were added/changed in the last updates (note also at the effective focal length error, which we also just changed the code there):

Unknown configuration item 'site_config_name'.
Unknown configuration item 'site_config_variant'.
Unknown configuration item 'site_config_version'.
Syntax error for configuration item 'EFFECTIVE_FOCAL_Length': Index out of range
Unknown configuration item 'fadc_compensate_pedestal'.
Unknown configuration item 'fadc_err_compensate_pedestal'.
Unknown configuration item 'fadc_lg_compensate_pedestal'.
Unknown configuration item 'fadc_lg_err_compensate_pedestal'.

I found some of these variables in the

https://gitlab.cta-observatory.org/cta-science/simulations/simulation-model/model_parameters/-/issues/16

They seem not to be part of the model and are written to the config file anyways. Do you see any straightforward solution @GernotMaier ?

[GernotMaier]

The command simtools-simulate-prod --config tests/integration_tests/config/simulate_prod_gamma_20_deg.yml is working fine for me (there are some warnings in the log files which needs to be addressed, but generally both CORSIKA and sim_telarray are working fine).

How is your setup?

I am running it in a container (podman or docker; doesn't matter):

podman run --rm -it -v "$(pwd)/:/workdir/external" ghcr.io/gammasim/simtools-dev:latest bash -c "source /workdir/env/bin/activate && cd /workdir/external/simtools && pip install -e . && bash

With a container from 2024-04-17T18:05:27.34498581Z (not the latest; but reasonable new).

Not sure what else we can check.

[VictorBarbosaMartins]

Ok, I think we are slowly clarifying things here. I fixed the test_applications_from_config[simtools-simulate-prod_gamma_20_deg_pack_for_grid] test, which is now working in my container. However, #887 will have to be merged first, otherwise the test will not run for checking the output files

[tobiaskleiner]

shower entries

[GernotMaier]

Fixed this typo.

[tobiaskleiner]

All good. One minor comment.

[VictorBarbosaMartins]

@tobiaskleiner, same here :D I guess we are ready to go?

[tobiaskleiner]

All good now. Thanks!

[VictorBarbosaMartins]

Thanks again!

[VictorBarbosaMartins]

I did some additional changes and will, therefore, wait until someone has another look. Thanks!

[VictorBarbosaMartins]

Merging here as well as this is approved and it is complementing the #906