Mirage has an issue with the linearized dark names when "segmentation" is done.

[KevinVolkSTScI]

When running a Mirage model for NIRISS imaging with 2 integrations in the exposure the Mirage code crashes. The issue is that the code creates two linearized dark files

jw01094001002_01101_00061_nis_uncal_seg001_linear_dark_prep_object.fits
jw01094001002_01101_00061_nis_uncal_seg002_linear_dark_prep_object.fits

from the original dark file

/ifs/jwst/wit/mirage_data/niriss/darks/raw/NISNIRISSDARK-172500017_15_496_SE_2017-09-07T05h28m22_dms_uncal.fits

but in the obs_generator.py routine when the code line 1212

seed_files = seed_dict[linDark]

is executed it has an empty list and then in the statements

if isinstance(seed_files[0], str):
print('\nSeed files:')
print(seed_files)

the code crashes because seed_files[0] is out of range for the indexing.

I inserted a few print statements into the code at this point

        print('Diagnostic: ', basename)
        #basename = os.path.join(temp_outdir, basename)

        if i > 0:
            print('Diagnostic: ', self.linDark[i])
            self.linear_dark = self.read_dark_file(self.linDark[i])

        print('Diagnostic: ', linDark)
        print('Diagnostic seed_dict: ', seed_dict)
        seed_files = seed_dict[linDark]
        print('Diagnostic: ', seed_files)
        if isinstance(seed_files[0], str):
            print('\nSeed files:')
            print(seed_files)

and the log plus the error messages look like this:

Creating output file name with segment number.
Diagnostic: jw01094001002_01101_00061-seg001_nis_uncal.fits
Diagnostic: /ifs/jwst/wit/niriss/kevin/new_car1094_simulations/jw01094001002_01101_00061_nis_uncal_seg001_linear_dark_prep_object.fits
Diagnostic seed_dict: {'/ifs/jwst/wit/niriss/kevin/new_car1094_simulations/jw01094001002_01101_00061_nis_uncal_seg001_linear_dark_prep_object.fits': [], '/ifs/jwst/wit/niriss/kevin/new_car1094_simulations/jw01094001002_01101_00061_nis_uncal_seg002_linear_dark_prep_object.fits': []}
Diagnostic: []
Traceback (most recent call last):
File "./runmodels.py", line 64, in
t1.create()
File "/home/kvolk/anaconda3/envs/mirage_30march2021/lib/python3.7/site-packages/mirage/imaging_simulator.py", line 82, in create
obs.create()
File "/home/kvolk/anaconda3/envs/mirage_30march2021/lib/python3.7/site-packages/mirage/ramp_generator/obs_generator.py", line 1183, in create
if isinstance(seed_files[0], str):
IndexError: list index out of range

The seed_dict variable does not have the file names in it as expected.

One can simply force the seed_dict values to equal the key names, but I hesitate to do this without knowing what the seed_dict variable is actually supposed to be doing.

[bhilbert4]

Can you send me the yaml file you were using for this? I just ran a NIRISS imaging case with 2 integrations and 10 groups with the raw dark you mentioned, and Mirage ran successfully. I thought that the results you are seeing were because you're starting with a raw dark and when that is processed, it is segmented because the version of the code you are using still has the extra low segmentation threshold. But I just tried running that scenario as well and Mirage finished successfully.

[KevinVolkSTScI]

You can see the .yaml files where this issue happened in directory /ifs/jwst/wit/niriss/kevin/new_car1094_simulations. The files where the problem happened are

jw01094001002_01101_00061_nis.yaml jw01094003002_01101_00061_nis.yaml
jw01094001002_01101_00062_nis.yaml jw01094003002_01101_00062_nis.yaml
jw01094001002_01101_00065_nis.yaml jw01094003002_01101_00065_nis.yaml
jw01094001002_01101_00066_nis.yaml jw01094003002_01101_00066_nis.yaml

[bhilbert4]

I just ran jw01094001002_01101_00061_nis.yaml with the file splitting threshold low enough to segment the file, and Mirage ran successfully. In this case, self.linDark was set to the list of dark files here: https://github.com/spacetelescope/mirage/blob/master/mirage/imaging_simulator.py#L85

And then seed_dict had the expected mapping of seed image files to dark files.

[KevinVolkSTScI]

OK so I am not clear about what was wrong in my run that caused the issue. I do not see a file splitting threshold in the .yaml file. Where is this set?

[bhilbert4]

The file splitting limit is set in constants.py. It's large enough (160 full frame groups) that basically only TSO observations should be split. https://github.com/spacetelescope/mirage/blob/master/mirage/utils/constants.py#L100

And in fact I just learned that in flight, only TSO mode observations can possibly be split.

So it's not clear to me why your run failed. But if you update the version of the code you are using, you should get the large file splitting limit, which will then mean that your observations won't be split.

[KevinVolkSTScI]

Which version of Mirage did you use to get it to work? When I looked at the constants.py file for my installation of 30 March 2021 the splitting limit was set to 38.*2048.*2048, but I have run things with 2 times 20 frames successfully including some of the runs done in the group in the directory where I ran into the issue. Or did you need to chance constants.py to get it to work?

[bhilbert4]

I merged the increased file splitting limit on Feb 12. For the tests today I brought the limit down to 2020482048 in order to make sure the files would be split. But you should see a limit of 16020482048 for code from 30 March.

If you're installation is using a local copy of the code, then you can be sure that you have an updated version by going into the directory where the code is and

1 .Be sure your mirage environment is activated
2. Using git remote -v, make sure that upstream is pointing to spacetelescope/mirage.git
3. And then pull down the new code

git fetch upstream master
git pull upstream master

Assuming you originally installed the code using pip install -e . then you should be all set. If you installed using pip install . then you'll have to run pip install . again to install the code changes.

[KevinVolkSTScI]

I just changed the value manually. But I am puzzled as to why the file had a value of 38*2048.*2048. I certainly did not edit the file. Something must have gotten messed up in the installation somehow. This is rather curious. But I just ran one of the models successfully so that is good.