-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallel RFI filtering #35
Conversation
59d9ded
to
7625cf1
Compare
Although the products aren't quite right yet, this reduces the |
01aa599
to
bba1a5a
Compare
Depends on radiocosmology/caput#211 |
6b216ed
to
cb8b593
Compare
Interestingly, the mask being produced is different from that produced by the previous method. |
@jrs65 this produces the same mask now as previous revisions, so it should be good to go unless there are additional changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved, but please quickly change the comment, and condense the commits (if it makes sense to do so), before merging.
ch_util/rfi.py
Outdated
limit_range : slice, optional | ||
Data is limited to this range in the freqeuncy axis. Defaults to Ellipsis. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe change the comment as the default is not actually an Ellipsis (even though it's effectively the same).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, good catch, that's a remnant of a previous way I had tried implementing it
21a1974
to
e8d8605
Compare
Running number_deviations across inputs causes the entire operation to take place on a single process. This is modified to be distributed across many nodes.
e8d8605
to
2fe5c6d
Compare
In its current implementation, the RFI Filter step of the daily pipeline takes ~25 minutes. Most of this time is spent taking a rolling median, and it is done in a single process. This PR should address the issue by pre-calculating a median across frequencies to flatten the auto vis. data, then distributing across the frequency axis to distribute the computation across processes.