Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.2.4 #147

Merged
merged 166 commits into from
Aug 23, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
166 commits
Select commit Hold shift + click to select a range
6a72fce
update metadata w/ frame_id and secondary dt
cmarshak Feb 13, 2023
9b68415
update changelog; fix version num
cmarshak Feb 13, 2023
70ccb38
add frame-id to test data
cmarshak Feb 13, 2023
07ddf91
Update delivery_prep.py
cmarshak Feb 14, 2023
5a4617b
temporal baseline and frame id
cmarshak Feb 27, 2023
add751c
Merge branch 'dev' into frame-metadata
cmarshak Feb 28, 2023
dff9012
Merge branch 'fix_variable_name_error' into frame-metadata
cmarshak Feb 28, 2023
f28b198
fix datetime types
cmarshak Feb 28, 2023
8bf78a7
remove duplicate SET line
cmarshak Feb 28, 2023
772abf2
merge dev
cmarshak Mar 6, 2023
257c7b8
ensure start time is earliest
cmarshak Mar 6, 2023
6e1cc3b
split url and download cells
cmarshak Mar 6, 2023
47a52d7
ensure midnight crossing is correctly handled per DB
cmarshak Mar 6, 2023
a989999
Merge branch 'dev' into frame-metadata
cmarshak Mar 23, 2023
7c72c95
increase dataset version 3.0.0, include version in cmr metadata, and …
cmarshak Mar 23, 2023
cf387a7
dynamically update dataset version for test
cmarshak Mar 23, 2023
fd73e06
lint
cmarshak Mar 23, 2023
9b0f090
Iono workflow with masking
mgovorcin Mar 27, 2023
d4a04dc
add mergeBursts step
mgovorcin Mar 28, 2023
1357535
clean up
mgovorcin Mar 28, 2023
1c86380
Fix A001
mgovorcin Mar 28, 2023
9c6fc84
little fixes
mgovorcin Mar 28, 2023
541ced6
fixes lint
mgovorcin Mar 28, 2023
c7b19af
clean up whitespaces
mgovorcin Mar 28, 2023
3f24a6c
Pass coordinates from geotrans, not metadata arrays
sssangha Mar 29, 2023
1f6869e
Remove vestigial variables, attempt 2
sssangha Mar 29, 2023
8b40127
Update changelog
sssangha Mar 29, 2023
4b8019a
andrews suggestion
cmarshak Mar 29, 2023
7ccb526
revert localized data
cmarshak Mar 29, 2023
9c9c5a8
clean up
mgovorcin Mar 29, 2023
8367f05
Change arrays to float32 precision
sssangha Mar 30, 2023
db9dbb9
flake warnings
mgovorcin Mar 30, 2023
dbdec7b
fix flake8 warnings
mgovorcin Mar 30, 2023
ad3e379
wrong import order
mgovorcin Mar 30, 2023
0bee389
wrong order 2
mgovorcin Mar 30, 2023
0ca7cea
wrong order 3
mgovorcin Mar 30, 2023
f00617a
correct descrption
mgovorcin Mar 30, 2023
cbbfd2a
Merge pull request #124 from ACCESS-Cloud-Based-InSAR/sss_SETgeotrans
cmarshak Mar 30, 2023
c91a355
little clean up
mgovorcin Mar 31, 2023
56c70b0
little clean up
mgovorcin Mar 31, 2023
0c1491f
Merge pull request #123 from ACCESS-Cloud-Based-InSAR/iono_mask
cmarshak Mar 31, 2023
96de171
adding iono template
mgovorcin Apr 3, 2023
679a960
Merge pull request #126 from ACCESS-Cloud-Based-InSAR/dev
mgovorcin Apr 3, 2023
9531532
removed args paramter to use water mask true/false flag
mgovorcin Apr 3, 2023
e96e113
update change log
mgovorcin Apr 3, 2023
646d4df
added water mask flag to main
mgovorcin Apr 3, 2023
24eb992
Merge pull request #127 from ACCESS-Cloud-Based-InSAR/ionosphere
cmarshak Apr 3, 2023
37f00ee
increase DEM buffer
cmarshak Apr 5, 2023
005eef7
add approx
cmarshak Apr 5, 2023
1ecc665
Update docstring
cmarshak Apr 5, 2023
bfd8df1
Another docstring typo
cmarshak Apr 5, 2023
e6dbf9f
Merge pull request #129 from ACCESS-Cloud-Based-InSAR/dev
mgovorcin Apr 18, 2023
776f312
updating iono workflow
mgovorcin Apr 18, 2023
40fc06c
update the import order
mgovorcin Apr 18, 2023
6646ef2
import order 2
mgovorcin Apr 18, 2023
9f1d8d0
import order 3
mgovorcin Apr 18, 2023
66fec39
Update CHANGELOG.md
mgovorcin Apr 18, 2023
8fde8f4
updated changelog
mgovorcin Apr 18, 2023
2003812
Update CHANGELOG.md
mgovorcin Apr 18, 2023
cd67782
fix import order 3
mgovorcin Apr 18, 2023
f95c3da
update import order 4
mgovorcin Apr 18, 2023
37625f5
import order 5
mgovorcin Apr 18, 2023
f0c151a
import order 6
mgovorcin Apr 18, 2023
2097b65
import order 7
mgovorcin Apr 18, 2023
ab6d355
import order 9
mgovorcin Apr 18, 2023
82d7233
Merge pull request #128 from ACCESS-Cloud-Based-InSAR/increase-dem-bu…
cmarshak Apr 18, 2023
a09041a
added reference and applied comments
mgovorcin Apr 19, 2023
b48da88
Merge branch 'dev' into iono_processing_update
mgovorcin Apr 19, 2023
b8be858
import order 10
mgovorcin Apr 19, 2023
4812032
updated CHANGELOG
mgovorcin Apr 19, 2023
2fb49d1
add_crossS1A_B_support
mgovorcin Apr 24, 2023
a4dab42
add_cross_S1_B_support
mgovorcin Apr 24, 2023
cd9b5c4
Delete Untitled-1.ipynb
mgovorcin Apr 24, 2023
b490cd7
import order
mgovorcin Apr 24, 2023
1745ab2
update import order 2
mgovorcin Apr 24, 2023
4f833c4
update order 3
mgovorcin Apr 24, 2023
b2a2ff9
update_order3
mgovorcin Apr 24, 2023
9b7684e
update_import_order4
mgovorcin Apr 24, 2023
6566a95
turn off ESD when using iono computation
mgovorcin Apr 25, 2023
0d1cff6
set with azimuth time
cmarshak Apr 26, 2023
d2c650c
AREA v POINT convention
cmarshak Apr 26, 2023
e698a2a
better comments in main
cmarshak Apr 26, 2023
9cd9ad4
changelog
cmarshak Apr 27, 2023
131af7e
fix translation of coords
cmarshak Apr 27, 2023
cb7cc68
import test_dir to nb and azimuth time test
cmarshak Apr 27, 2023
949b8c9
sort imports
cmarshak Apr 27, 2023
de1aa2d
Merge branch 'dev' into SET-azimuth-time
cmarshak Apr 27, 2023
736c6fe
remove set nb
cmarshak Apr 27, 2023
cbeeeb6
add test data
cmarshak Apr 27, 2023
b4de015
use orbit files for geo2rdr from raider
cmarshak Apr 28, 2023
5901a92
rerun nb
cmarshak May 1, 2023
178c77e
include orbit data
cmarshak May 1, 2023
e282669
added derampinh before bridging
mgovorcin May 8, 2023
b52736f
fix falke8 comments
mgovorcin May 8, 2023
8c78434
fix #128
mgovorcin May 8, 2023
c1bb126
Add optimum number of threads parameter
jhkennedy May 15, 2023
928f2ea
fix slc tests
cmarshak May 22, 2023
30bcc96
fix warning catch
cmarshak May 22, 2023
6b8d5a0
changelog
cmarshak May 22, 2023
f16dbf5
put records outside of warning context
cmarshak May 22, 2023
b3be9b5
Merge pull request #133 from ACCESS-Cloud-Based-InSAR/fix-slc-test
cmarshak May 23, 2023
7e13559
Update isce2_topsapp/__main__.py
jhkennedy May 23, 2023
86894e1
Update isce2_topsapp/__main__.py
jhkennedy May 23, 2023
0cda6b3
Merge branch 'dev' into iono_processing_update
mgovorcin May 23, 2023
bbf2496
update changelog
mgovorcin May 23, 2023
0f74f88
Merge pull request #130 from ACCESS-Cloud-Based-InSAR/iono_processing…
cmarshak May 23, 2023
73a77fa
fix whitespace and comment
cmarshak May 24, 2023
47a64ae
Merge branch 'dev' into resource-competition
cmarshak May 24, 2023
e8e10fb
fix conflicts
cmarshak May 24, 2023
65f6c9d
Merge branch 'dev' into frame-metadata
cmarshak May 25, 2023
98dadaf
Update __main__.py with water mask
cmarshak May 25, 2023
1a140d4
Merge pull request #102 from ACCESS-Cloud-Based-InSAR/frame-metadata
cmarshak May 27, 2023
46e1c40
remove option of using num of threads
mgovorcin Jun 5, 2023
1d810d8
skip iono computation if all data is nan
mgovorcin Jun 6, 2023
5014d0f
skip using water mask if request is out of SWBD coverage
mgovorcin Jun 6, 2023
48d29ee
fix typo in water mask naming
mgovorcin Jun 6, 2023
8307b1d
add notes
mgovorcin Jun 6, 2023
454e814
remove whitespace
mgovorcin Jun 6, 2023
ff1f280
update CHANGELOG
mgovorcin Jun 6, 2023
9ee8b40
Merge pull request #138 from ACCESS-Cloud-Based-InSAR/iono
cmarshak Jun 8, 2023
9b013c1
fix azimuth time geo2rdr
cmarshak Jun 21, 2023
3131d4f
include additional test for azimuth timing
cmarshak Jun 21, 2023
b444fae
adding iono processing attributes to GUNW
mgovorcin Jun 26, 2023
db41266
fix for E128 in iono_proc
mgovorcin Jun 26, 2023
78a7593
iono_proc fix E128
mgovorcin Jun 26, 2023
ef1d2ef
update changelog
mgovorcin Jun 26, 2023
33a95aa
small fix for iono attr dict
mgovorcin Jun 26, 2023
0cc6c21
set burst jump estimation True by default
mgovorcin Jun 26, 2023
6eb679e
added introduced additional attributes to changelog
mgovorcin Jun 26, 2023
d60b151
Merge pull request #141 from ACCESS-Cloud-Based-InSAR/iono_processing…
cmarshak Jun 26, 2023
5eae2fd
fix typo
mgovorcin Jun 26, 2023
f90d1ac
Fix lat array bug
sssangha Jun 29, 2023
e2c966c
revert lat shift
cmarshak Jun 29, 2023
8759678
fix test ensuring secondary orbit used
cmarshak Jun 29, 2023
14f80fd
Add better comments
cmarshak Jun 29, 2023
ea7e814
fix interpolation at second edge case and better comments
cmarshak Jul 6, 2023
2f959e6
fix linear interpolation weights
cmarshak Jul 6, 2023
6d198b3
update resolutions to be positive
cmarshak Jul 6, 2023
459ac1d
Merge branch 'SET-azimuth-time' of github.com:ACCESS-Cloud-Based-InSA…
cmarshak Jul 6, 2023
701e530
bug fix for lon array
sssangha Jul 6, 2023
e50a6fe
add test that compares pysolid to plugin
cmarshak Jul 6, 2023
7e4e6c9
utilize all datasets for tests
cmarshak Jul 6, 2023
d4b62a6
fix test
cmarshak Jul 6, 2023
0c36cde
remove slc_id
cmarshak Jul 6, 2023
9620b89
update docstring
cmarshak Jul 6, 2023
2002698
lint
cmarshak Jul 6, 2023
12e292f
fix typing
cmarshak Jul 6, 2023
e5c7e55
fix acq_type in azi test
cmarshak Jul 6, 2023
f58fedf
more comments
cmarshak Jul 6, 2023
d52cb87
update docstring and rearrange for clarity
cmarshak Jul 6, 2023
26f511a
Merge pull request #131 from ACCESS-Cloud-Based-InSAR/SET-azimuth-time
cmarshak Jul 11, 2023
dbea910
Merge pull request #143 from ACCESS-Cloud-Based-InSAR/dev
mgovorcin Jul 12, 2023
b88c1aa
add packing of burstRamps
mgovorcin Jul 12, 2023
28dae6f
fix lint issues
mgovorcin Jul 12, 2023
4c2f977
small typos
mgovorcin Jul 12, 2023
ffe14b2
update layer_name typo
mgovorcin Jul 12, 2023
015be30
changing naming burst jumps to burst ramps
mgovorcin Jul 12, 2023
3f6b088
Update environment.yml
cmarshak Jul 12, 2023
23f32ff
Update CHANGELOG.md
cmarshak Jul 12, 2023
266400f
Merge pull request #144 from ACCESS-Cloud-Based-InSAR/Add-fsspect
cmarshak Jul 12, 2023
7075fc6
Merge branch 'dev' into iono_adding_burst_jumps
mgovorcin Jul 13, 2023
146384c
update changelog
mgovorcin Jul 13, 2023
354991e
Merge pull request #142 from ACCESS-Cloud-Based-InSAR/iono_adding_bur…
cmarshak Jul 13, 2023
baed31a
fix set bug
cmarshak Jul 26, 2023
5e9e433
Merge pull request #146 from ACCESS-Cloud-Based-InSAR/set-fix
cmarshak Jul 26, 2023
121e6c5
Merge pull request #132 from ACCESS-Cloud-Based-InSAR/resource-compet…
cmarshak Aug 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ conf/settings.conf

# Include template directory
!isce2_topsapp/templates/topsapp_template.xml
!isce2_topsapp/templates/topsapp_iono_template.xml


# Tests
Expand All @@ -106,6 +107,18 @@ tests/out/*.ipynb
tests/**/*.json
!tests/sample_loc_metadata.json
!tests/midnight_crossing_metadata.json
# Test for SET GUNW 1
!tests/set_test_data/S1-GUNW-A-R-064-tops-20210723_20210711-015000-00119W_00033N-PP-6267-v2_0_6.nc
!tests/set_test_data/S1B_OPER_AUX_POEORB_OPOD_20210812T111941_V20210722T225942_20210724T005942.EOF
!tests/set_test_data/S1B_OPER_AUX_POEORB_OPOD_20210731T111940_V20210710T225942_20210712T005942.EOF
# Test for SET GUNW 2
!tests/set_test_data/S1-GUNW-A-R-064-tops-20220113_20220101-015048-00119W_00034N-PP-3b1f-v2_0_5.nc
!tests/set_test_data/S1A_OPER_AUX_POEORB_OPOD_20220121T121549_V20211231T225942_20220102T005942.EOF
!tests/set_test_data/S1A_OPER_AUX_POEORB_OPOD_20220202T081558_V20220112T225942_20220114T005942.EOF
!tests/test_data/ref_manifest.xml
!tests/test_data/ref_metadata.xml
!tests/test_data/sec_manifest.xml
!tests/test_data/sec_metadata.xml

# Use geojson zip file for ARIA s1 frame data
!isce2_topsapp/data/s1_gunw_frame_footprints.geojson.zip
Expand Down
1 change: 1 addition & 0 deletions .trufflehog.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ Dockerfile
tests/test_data/sec_manifest.xml
tests/test_data/ref_manifest.xml
tests/test_data/
isce2_topsapp/iono_proc.py
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,34 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.2.4]

### Added
* `++omp-num-threads` parameter to the `main()` entrypoint to limit the number of threads used by ISCE2 during
multiprocessing.

### Fixed
* For Solid Earth Tide computation, use azimuth timing to calculate solid earth tide in `science/grids/imagingGeometry` reference frame using ISCE2 rdr2geo.
* Include topsapp_iono template.
* Increases DEM buffer to .4 from .1 to ensure the extent of at least two bursts (~40 km) are added when retrieving DEM (because estimated footprint can differ from what ISCE2 generates for a GUNW extent)
* Catch warnings in tests and match messages to ensure package warnings do not fail test suite
* Read low resolution Natural Earth land masses from public url due to removal from geopandas package.
* For ionosphere computation over water, includes masking conncomp zero, phase bridging, and modified adaptive gaussian filtering
* Fix for #135, skip iono computation if there are not land (all zero values) and skip using water mask if the area is outside of SWBD coverage
* Fix for #145 and SET - duplicate orbit xmls for computing azimuth time grid with ISCE2 geo2rdr (duplicate state vectors likely culprit). Ensures orbit object is intialized with unique set of orbit xmls passed. Also, localized metadata appropriately.

### Added
* localize_data within __main__.py added option to use/not use water mask for ionosphere processing
* Added option to estimate burst phase jumps in ionosphere computation
* Added additional attributes for ionosphere computation into GUNW ionosphere layer metadata: processing_steps, water_mask, mask_connected_component_zero (flag) , do_phase_bridging (flag), swath_mode (flag), swath_ramp_removal (flag), swath_mode_description, multilook_az_rg1, multilook_az_rg2, iono_height
* Added packing of additional attributes for ionosphere computation into GUNW
* fsspec is now required in environment due to burst processing.

## [0.2.3]

### Updated
* Explode footprints polygons
* Added support for using water mask in ionospheric correction computation

### Removed
* Python 3.8 Support
Expand All @@ -25,6 +47,7 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
* Exposes `frame-id` parameter for fixed frame cropping. Discussion, references, and examples in README.
* Latitude aligned frames and their expected extents are added as geojson in repository as zip file.
* Pins ISCE2 version to 2.6.1 and numpy / scipy to previous versions (see environment.yml) - to be amended when newest ISCE2 build is sorted out
* Includes `frame_id` and `temporal_baseline_days` in json metadata for CMR handshake. The former is the fixed frame id and the latter is the number of days between images (will be multiple of 6).
* Added support to compute and embed solid earth tide correction layers into GUNW products (see PR #91) - reference and secondary have own groups
* Raises warning if there is at least 80% of water in the IFG area using Natural Earth Land mask.

Expand Down
1 change: 1 addition & 0 deletions environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ dependencies:
- flake8-blind-except
- flake8-builtins
- flake8-import-order
- fsspec
- gdal
- geopandas
- hyp3lib>=1.7
Expand Down
3 changes: 2 additions & 1 deletion isce2_topsapp/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@
from isce2_topsapp.localize_aux_cal import download_aux_cal
from isce2_topsapp.localize_burst import BurstParams, download_bursts, get_region_of_interest
from isce2_topsapp.localize_dem import download_dem_for_isce2
from isce2_topsapp.localize_mask import download_water_mask
from isce2_topsapp.localize_orbits import download_orbits
from isce2_topsapp.localize_slc import download_slcs, get_asf_slc_objects
from isce2_topsapp.packaging import package_gunw_product
from isce2_topsapp.topsapp_proc import topsapp_processing


try:
__version__ = version(__name__)
except PackageNotFoundError:
Expand All @@ -25,6 +25,7 @@
'get_asf_slc_objects',
'get_region_of_interest',
'download_dem_for_isce2',
'download_water_mask',
'download_aux_cal',
'download_bursts',
'BurstParams',
Expand Down
Loading
Loading