Working base #389

Aske-Rosted · 2023-01-25T06:51:05Z

Addition of several features used with neutrino-generator-21220 data, which seems to differ slightly compared to previously used data sets.

added functionality includes

Skipping frame in I3 file in case of missing pulsemap (In the files I have only every other physics frame seems to be relevant)
A warning message for when a frame is skipped.
Added option for when the MCTree is named differently in my case "I3MCTree_preMuonProp" standard option still "I3MCTree"
Implementation of a function which renames detector features given a list of strings. (This one is throwing a pre-commit error that I could not handle on my own)
allows for passing name of features to perform coarsening on. Automatically finds the "time" feature assuming time is in the name of the feature.
adds a function which returns a random fraction of data without any specific selection criteria
adds functionality for loading data with with missing input values by forcing float (Warning message is printed. This might be a bit "hacky")
add an extractor for pulling zenith and azimuth comparisons from other algorithms.

Sorry about this being a bit a bunched up collection of added functionality.

…lass

update to current main branch

asogaard

Thanks @Aske-Rosted!

src/graphnet/models/detector/detector.py

src/graphnet/models/detector/icecube.py

src/graphnet/models/model.py

src/graphnet/training/utils.py

src/graphnet/data/dataset.py

src/graphnet/data/extractors/i3splinempeextractor.py

src/graphnet/data/extractors/__init__.py

src/graphnet/data/sqlite/sqlite_selection.py

src/graphnet/models/coarsening.py

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

Aske-Rosted · 2023-02-21T04:26:04Z

Hmm, the "unintended behavior" that you are describing is precisely the behavior I am looking for. Is there any scenario where you need information from a P-frame where one (or more) of your pulse maps is missing?

Aske-Rosted · 2023-02-21T07:30:22Z

I should now have addressed all the suggestions and corrections. I have implemented the handling of NullSplit frames in the way suggsted by @RasmusOrsoe as well as added some extra extraction features for the I3ParticleExtractor.

merge from newest upstream main

asogaard · 2023-02-21T07:54:38Z

Hmm, the "unintended behavior" that you are describing is precisely the behavior I am looking for. Is there any scenario where you need information from a P-frame where one (or more) of your pulse maps is missing?

Yes, see "Expected behaviour" here. :)

asogaard

Hi @Aske-Rosted,

Thanks for this nice PR. I am approving it now, but before merging I would like you to have a look at:

One suggestion to update a docstring.
One comment regarding the way NullSplit frames are skipped.

I'll leave it to you whether and how to resolve these. :)

src/graphnet/data/extractors/i3particleextractor.py

asogaard · 2023-02-21T07:58:17Z

src/graphnet/data/dataconverter.py

@@ -427,11 +427,13 @@ def _extract_data(self, fileset: FileSet) -> List[OrderedDict]:
        while i3_file_io.more():
            try:
                frame = i3_file_io.pop_physics()
+                assert self._is_not_null_split_frame(frame)


As I mention here, I am not particularly fond of this solution, and would propose (1) amore general frame-skipping method and (2) separating this logic from whether the frame is a P-frame. But the proposed code works and is not unreasonable, so I'll leave it at your discretion, @Aske-Rosted.

@Aske-Rosted I think Andreas suggestion is better than what I mentioned earlier.

Is the latest commit satisfactory? I am not 100% certain that all exceptions coming from the I3 modules will contain I3 in the exception name, but I would hope that they do...

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

asogaard · 2023-02-21T12:32:06Z

All good, @Aske-Rosted! :)

Working base

Aske-Rosted added 12 commits January 24, 2023 13:55

adding feature option to graphnet/models/detector/icecube/IceCube86 c…

ba81aae

…lass

Skipping frame if pulsemap missing

29ec105

message passing in case of missing pulsemap

08d13f5

added options for using differently named MCTree

27a5cb9

new implementation of feature renaming

dee0a9b

differently named feature in DOMCoarsening

72566b5

train_validation_test_dataloader

29e8ebb

Add function for no selection criteria sql db

b7b416e

Allow object type by forcing float with warning

8a87fa8

Add extractor for comparisons to other algorithms

079d419

Fixing according to pre-commit hooks

3e57582

remove @abstraclass

c52e051

MortenHolmRep requested a review from asogaard January 25, 2023 12:00

MortenHolmRep assigned asogaard Jan 25, 2023

Aske-Rosted added 3 commits January 31, 2023 12:56

allow for choosing parrellelising strategy

42d019e

Merge branch 'main' into working_base

2a02ed5

Merge branch 'main' into working_base

662c5e1

update to current main branch

asogaard assigned Aske-Rosted and asogaard and unassigned asogaard and Aske-Rosted Feb 14, 2023

asogaard requested changes Feb 14, 2023

View reviewed changes

asogaard assigned Aske-Rosted and unassigned asogaard Feb 14, 2023

Aske-Rosted and others added 6 commits February 15, 2023 11:11

Update src/graphnet/models/detector/detector.py

4d9d4af

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

Update src/graphnet/models/detector/icecube.py

adfd152

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

Update src/graphnet/models/model.py

b856bac

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

Update src/graphnet/training/utils.py

2d31e2e

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

Update src/graphnet/data/dataset.py

038785c

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

Update src/graphnet/data/extractors/i3splinempeextractor.py

b8e2077

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

Aske-Rosted and others added 4 commits February 21, 2023 13:12

Update src/graphnet/models/coarsening.py inputs

8f0c4e5

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

use input instead

53d1b23

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

renaming src/graphnet/data/extractors/i3comparisonextractor.py

baf8c27

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

renaming comparison extractor in __init__.py

8d4e8a9

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

Aske-Rosted added 6 commits February 21, 2023 13:45

added null split frame check

06f21a0

comparisonextractor -> particleextractor

c545490

comparisonextractor -> particleextractor

7a727e4

rm exceptional handling of missing pulsemap

7e172fe

fix missing capitalization

926f1e8

added additional values

0c8d54c

Aske-Rosted requested a review from asogaard February 21, 2023 07:30

Aske-Rosted assigned asogaard and unassigned asogaard and Aske-Rosted Feb 21, 2023

Aske-Rosted added 2 commits February 21, 2023 16:38

streamlining error handling.

e26a73c

Merge branch 'main' into working_base

a050176

merge from newest upstream main

asogaard approved these changes Feb 21, 2023

View reviewed changes

Aske-Rosted and others added 2 commits February 21, 2023 19:11

Update src/graphnet/data/extractors/i3particleextractor.py

b659190

Co-authored-by: Andreas Søgaard <andreas.sogaard@gmail.com>

only accept I3 exceptions

695bd90

asogaard assigned Aske-Rosted and unassigned asogaard Feb 21, 2023

Aske-Rosted merged commit 983818a into graphnet-team:main Feb 21, 2023

Aske-Rosted deleted the working_base branch February 22, 2023 02:06

RasmusOrsoe pushed a commit to RasmusOrsoe/graphnet that referenced this pull request Oct 25, 2023

Merge pull request graphnet-team#389 from Aske-Rosted/working_base

2c3019e

Working base

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Working base #389

Working base #389

Aske-Rosted commented Jan 25, 2023

asogaard left a comment

Aske-Rosted commented Feb 21, 2023

Aske-Rosted commented Feb 21, 2023

asogaard commented Feb 21, 2023

asogaard left a comment

asogaard Feb 21, 2023

RasmusOrsoe Feb 21, 2023

Aske-Rosted Feb 21, 2023

asogaard commented Feb 21, 2023

Working base #389

Working base #389

Conversation

Aske-Rosted commented Jan 25, 2023

asogaard left a comment

Choose a reason for hiding this comment

Aske-Rosted commented Feb 21, 2023

Aske-Rosted commented Feb 21, 2023

asogaard commented Feb 21, 2023

asogaard left a comment

Choose a reason for hiding this comment

asogaard Feb 21, 2023

Choose a reason for hiding this comment

RasmusOrsoe Feb 21, 2023

Choose a reason for hiding this comment

Aske-Rosted Feb 21, 2023

Choose a reason for hiding this comment

asogaard commented Feb 21, 2023