Custom event #122

cmarshak · 2023-03-27T21:10:07Z

Provides CLI options for:

output resolution: 30 meters or 90 meters (the latter is default and the standardized GUNW resolution)
dense offset layers: azimuth and range offset estimates as separate layers
unfiltered coherence layer

If the output resolution is changed to 30 meters or any of the latter two layers are requested, then the initial token on product name is change from S1-GUNW to S1-GUNW-CUSTOM.

cmarshak · 2023-03-30T17:42:31Z

@gracebato

This is the first pass of a custom disasters product at 30 meters resolution for a coseismic IFG in Iraq. Might be useful to apply it to another earthquake or volcano. Here is the command used to generate the product:

isce2_topsapp --reference-scenes S1B_IW_SLC__1SDV_20171117T145926_20171117T145953_008323_00EBAB_AFB8 \
              --secondary-scenes S1A_IW_SLC__1SDV_20171111T150004_20171111T150032_019219_0208AF_EE89 \
              --output-resolution 30 \
              --unfiltered-coherence True \
              --dense-offsets True \
              --estimate-ionosphere-delay True \
              > topsapp_img.out 2> topsapp_img.err

The GUNW can be downloaded here: S1-GUNW-CUSTOM-A-R-072-tops-20171117_20171111-145939-00043E_00034N-PP-428e-v2_0_6.nc. Note the 30 meter resolution and additional layers increases the file size to 1.2 GB (!!!).

For the pixel offset layers, take a look in case you want some basic post-processing (nodata/smoothing/etc). Right now the plugin is taking the ISCE2 outputs directly and sticking them into the GUNW. I do not think we should multiply by the pixel resolution because that can be done very easily for any analysis later and it will be hard for users to track down the exact resolution we are using in case they want to do some other kind of feature tracking using this product.

Thanks for your feedback.

And to get a sense of the layout of layers, here is a screenshot from panoply.

@jhkennedy - this PR (once merged) will provide the last CLI options for the hyp3 api to be filled in (as discussed in the meta issue). Also, providing the output resolution will make this plugin adaptable for on-demand processing (may want to restrict the resolution to 50 or 60 meters because the product size is large).

cmarshak · 2023-09-11T18:59:13Z

I think we will also want to be able to manually set a bbox as it will greatly reduce computational overhead (so we don't have to process all SLCs covering the AOI) and operational accounting (so we can just collect all the relevant SLCs that intersect our AOI) to ensure we can specify precisely the AOI we will want. This will require some modification of the API and coordination with the hyp3 job specification.

If this is pursued, we will want a different naming convention.

review-notebook-app · 2023-11-08T01:02:31Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

cmarshak · 2023-11-20T17:47:57Z

Added

Provides CLI options for:
- output resolution: 30 meters or 90 meters (the latter is default and the standardized GUNW resolution)
- unfiltered coherence layer (True/False)
- Goldstein filtering power - power that phase is raised to in patch FFT - default .5
- Dense offsets layers (still in pixel units)
Codifies and documents (in readme) what is meant by standard GUNW with respect to exposed parameters including:
- 90 m resolution
- .5 value in the Goldstein filter for InSAR phase
- No ESD or dense offsets
- Additional layers: ionosophere, SET, and unfiltered coherence.
- uses pydantic to record relevant topsapp parameters for "standard" GUNW
Records parameters in the product including the CLI command to regenerate said product
If parameters are not standard uses prefix S1-GUNW_CUSTOM-...
Pydantic dependency for parameter accounting

Changed

The CLI now requires frame_id (use frame_id = -1 for old API and what is now considered a "non"-standard product)

cmarshak · 2023-11-20T17:50:23Z

Here is a sample of new global attributes (i.e. global metadata) for provenance:

  // global attributes:
  :product_type = "UNW GEO IFG";
  :Conventions = "CF-1.6";
  :title = "ARIA standard product UNW GEO IFG";
  :author = "David Bekaert, Grace Bato, Marin Govorcin, Andrew Johnston, Joe Kennedy, Charlie Marshak, Simran Sangha and ARIA-JPL";
  :institution = "Jet Propulsion Laboratory";
  :references = "https://aria.jpl.nasa.gov/";
  :ogr_geometry_field = "productBoundingBox";
  :ogr_layer_name = "productBoundingBox";
  :ogr_layer_type = "POLYGON";
  :source = "Contains modified Copernicus Sentinel data processed by ESA and ARIA NASA/JPL using the DockerizedTopsApp HyP3 plugin version 0.2.6.dev77+g484709f";
  :reference_scenes = "S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E", "S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2";
  :secondary_scenes = "S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F", "S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7";
  :frame_id = 25502L; // long
  :estimate_ionosphere_delay = "TRUE";
  :compute_solid_earth_tide = "TRUE";
  :output_resolution = 90L; // long
  :unfiltered_coherence = "TRUE";
  :goldstein_filter_power = 0.5; // double
  :dense_offsets = "FALSE";
  :wrapped_phase_layer = "FALSE";
  :esd_coherence_threshold = -1.0; // double
  :command_line_string = "isce2_topsapp ++gunw_slc --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 --frame-id 25502";
  :mean_perpendicularBaseline = -130.12419f; // float
  :mean_parallelBaseline = 79.829f; // float
  :mean_coherence = 0.4382714f; // float
  :version = "1c";

Note that we have the wrapped phase layer - I just removed that per Grace's suggestion.

cmarshak · 2023-11-20T18:09:08Z

Here are some sample products.

A standard product link: https://gunw-development-testing.s3.us-west-2.amazonaws.com/custom-events/S1-GUNW-A-R-164-tops-20220212_20220131-222829-00071W_00047N-PP-3d6c-v3_0_0.nc

Generated by this command:

isce2_topsapp ++gunw_slc --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 --frame-id 25502

A "custom" GUNW link:
https://gunw-development-testing.s3.us-west-2.amazonaws.com/custom-events/S1-GUNW_CUSTOM-A-R-164-tops-20220212_20220131-222829-00071W_00047N-PP-3d6c-v3_0_0.nc

Generated with this command:

isce2_topsapp ++gunw_slc --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 --frame-id 25502 --goldstein-filter-power 0.0 --output-resolution 30 --dense-offsets True

cmarshak · 2023-11-20T18:11:48Z

Here is some "standardization" documentation too: https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/custom-event/README.md#what-makes-an-aria-s1-gunw-product-standard

cmarshak · 2023-11-20T18:42:00Z

@gracebato - please take a look at the sample product - note it has dense offsets at 30 meters and goldstein filter turned off.

cmarshak · 2023-11-21T02:18:53Z

Per @dbekaert request, the new metadata at the top level is:

// global attributes:
  :product_type = "UNW GEO IFG";
  :Conventions = "CF-1.6";
  :title = "ARIA standard product UNW GEO IFG";
  :version = "1";
  :author = "David Bekaert, Grace Bato, Marin Govorcin, Andrew Johnston, Joe Kennedy, Charlie Marshak, Simran Sangha and ARIA-JPL";
  :institution = "Jet Propulsion Laboratory";
  :references = "https://aria.jpl.nasa.gov/";
  :ogr_geometry_field = "productBoundingBox";
  :ogr_layer_name = "productBoundingBox";
  :ogr_layer_type = "POLYGON";
  :source = "Contains modified Copernicus Sentinel data processed by ESA and ARIA NASA/JPL using the DockerizedTopsApp HyP3 plugin version 0.2.6.dev77+g484709f";
  :aria_frame_id = 25502L; // long
  :topsapp_command_line_string = "isce2_topsapp ++gunw_slc --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 --frame-id 25502 --goldstein-filter-power 0.0 --output-resolution 30 --dense-offsets True";
  :isce2_topsapp_version = "0.2.6.dev77+g484709f";
  :mean_perpendicularBaseline = -130.12419f; // float
  :mean_parallelBaseline = 79.829f; // float
  :mean_coherence = 0.29765356f; // float
  :product_geometry_wkt = "POLYGON ((-67.56389051312044 47.852140949747735, -67.47945065650076 47.48175879127325, -67.99492534034955 47.429567703811465, -68.52987513855085 47.37275123696082, -68.55727989287345 47.48198524930101, -69.09896846019329 47.421616017501826, -69.70919700836623 47.35015226169008, -69.78111489146 47.613618796674885, -70.29087990731554 47.55199732126649, -70.87876693003453 47.47767780134041, -71.1335260373605 48.33245909620376, -71.17732771160051 48.497691935550456, -71.32807577094489 48.9934664193567, -70.72357418792893 49.073063410471164, -70.1620736681961 49.14396169772953, -70.08244510661609 48.861110698101, -69.49265888652626 48.93246094155771, -68.89930636351852 49.001024725991755, -68.87209673148254 48.89524681271839, -68.35882950083834 48.952510206796376, -67.82862731394724 49.00911898266449, -67.78063949294818 48.80414813817185, -67.7838829742832 48.803513684598364, -67.56389051312044 47.852140949747735))";

The really long wkt string can be read as:

import xarray as xr
import shapely

with xr.open_dataset(netcdf_path) as ds:
    wkt_str = ds.attrs['product_geometry_wkt']
    geo = shapely.wkt.loads(wkt_str)

Which takes ~20 ms -- not sure what the equivalent would be with ARIA-Tools. Please confirm this is what is wanted @mgovorcin.

I still saved all the parameters as attributes in the science/grids group.

cmarshak · 2023-11-21T19:38:21Z

I had to fix a bug in which filter_strength in the xml was repeated (see below). This is now fixed. Will have to re-generate products to demonstrate successful end-to-end run.

<?xml version="1.0" encoding="UTF-8"?>
<topsApp>
  <component name="topsinsar">
    <component name="reference">
      <property name="orbit directory">orbits</property>
      <property name="auxiliary data directory">aux_cal</property>
      <property name="output directory">reference</property>
      <property name="safe">['S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2.zip', 'S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E.zip']</property>
    </component>
    <component name="secondary">
      <property name="orbit directory">orbits</property>
      <property name="auxiliary data directory">aux_cal</property>
      <property name="output directory">secondary</property>
      <property name="safe">['S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7.zip', 'S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F.zip']</property>
    </component>
    <property name="swaths">[1, 2, 3]</property>
    <property name="range looks">7</property>
    <property name="azimuth looks">3</property>
    <property name="filter strength">0.5</property>
    <property name="region of interest">[47.67792128636605, 48.83690336500213, -71.28001910050304, -67.52426023284436]</property>
    <property name="demFilename">full_res.dem.wgs84</property>
    <property name="geocodeDemFilename">low_res.dem.wgs84</property>
    <property name="do unwrap">True</property>
    <property name="unwrapper name">snaphu_mcf</property>
    <property name="do ionosphere correction">False</property>
    <property name="apply ionosphere correction">False</property>
    <property name="do ESD">False</property>
    <property name="ESD coherence threshold">-1.0</property>
    <property name="do dense offsets">True</property>
    <property name="Ampcor window width">64</property>
    <property name="Ampcor window height">64</property>
    <property name="use virtual files">True</property>
    <property name="filter strength">0.0</property>
    <property name="geocode list">['merged/phsig.cor', 'merged/filt_topophase.unw', 'merged/los.rdr', 'merged/topophase.flat', 'merged/filt_topophase.flat', 'merged/filt_topophase_2stage.unw', 'merged/topophase.cor', 'merged/filt_topophase.unw.conncomp', 'merged/dense_offsets.bil']</property>
  </component>
</topsApp>

cmarshak · 2023-11-21T19:43:52Z

@mgovorcin

Questions:

It appears that https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/dev/isce2_topsapp/templates/topsapp_iono_template.xml is not being used in ionosphere processing. May I delete it?
It appears the ionosphere processing is using the topsapp.xml generated from the original processing. Will the Goldstein filter strength impact the ionosphere processing?
(from above) - is the WKT global attribute still desired?

mgovorcin · 2023-11-21T20:19:27Z

@mgovorcin

Questions:

1. It appears that https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/dev/isce2_topsapp/templates/topsapp_iono_template.xml is not being used in ionosphere processing. May I delete it?

2. It appears the ionosphere processing is using the [topsapp.xml](https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/dev/isce2_topsapp/iono_proc.py#L38) generated from the original processing. Will the Goldstein filter strength impact the ionosphere processing?

3. (from above) - is the WKT global attribute still desired?

Hi @cmarshak

Yes you can delete it as it is not in use anymore. Ionosphere is done by calling topsApp modules with not need for additional template.
Filtering is coming after corregistration step, so should not affect ionosphere estimation
I would keep it there, as see a potential later. At the moment ariaTools uses same shapely.wkt.loads , however requires opening and reading NETCDF:$FILE_PATH:productBoundingBox with osgeo.ogr which adds couple of ms. If it is in metadata, all the needed information can be obtain with e.g. gdal.Info.

How I see it, if these metadata info are added to accompanying GUNW json metadata file, most of the ARIAtools spatio-temporal analysis could be done just by using/downloading metadata files, once the GUNWs that fall within AOI and fulfill spatio-temporal connectivity requirement are selected, only those files then can be download from VERTEX which can save some time on the downloading and later on extraction.

cmarshak · 2023-11-21T21:28:18Z

Thanks, Marin!

In regards to 3, you can get the approximate spatial extent of the GUNW from asf_search without even opening any netcdf. I am unfamiliar with the json file except in the s3 bucket we generate, but that doesn't get publicly distributed (does it?).

If you want additional metadata in CMR e.g. mean coherence, etc. that will be a bigger ask and requires coordination with ASF. It was a decent amount of work to get the temporal baseline and frame id there.

cmarshak · 2023-11-22T18:29:22Z

Here are some sample products.

A standard product link: https://gunw-development-testing.s3.us-west-2.amazonaws.com/custom-events/S1-GUNW-A-R-164-tops-20220212_20220131-222829-00071W_00047N-PP-3d6c-v3_0_0.nc

Generated by this command:
isce2_topsapp ++gunw_slc --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 --frame-id 25502
A "custom" GUNW link: https://gunw-development-testing.s3.us-west-2.amazonaws.com/custom-events/S1-GUNW_CUSTOM-A-R-164-tops-20220212_20220131-222829-00071W_00047N-PP-3d6c-v3_0_0.nc

Generated with this command:
isce2_topsapp ++gunw_slc --secondary-scenes S1A_IW_SLC__1SDV_20220131T222803_20220131T222830_041711_04F690_8F5F S1A_IW_SLC__1SDV_20220131T222828_20220131T222855_041711_04F690_28D7 --reference-scenes S1A_IW_SLC__1SDV_20220212T222803_20220212T222830_041886_04FCA3_2B3E S1A_IW_SLC__1SDV_20220212T222828_20220212T222855_041886_04FCA3_A3E2 --frame-id 25502 --goldstein-filter-power 0.0 --output-resolution 30 --dense-offsets True

These links should be updated with the latest fixes.

One thing to note at 30m and with Goldstein filter set to 0, the connected components are much different. Unsure as to why this is expected. It yields browse PNGs that are very different even though the area is the same.

The standard 90 m product:

The custom 30 m product with Goldstein power set to 0:

cmarshak added 2 commits March 27, 2023 13:34

dem 30 meters and draft of custom layers

ccbfa7b

select band number from isce2 outputs

6346f85

cmarshak marked this pull request as draft March 27, 2023 21:10

cmarshak added 14 commits March 27, 2023 14:43

fix args for resolution

a3f7905

fix typo

50749ab

fix localize dem

3bd3dc9

fix geocode rasterio profile for 30 m

53d5023

changelog

f56a5a8

correct version

74207b4

fix argparse typo; argggg

1cf984d

add unfiltered coherence

4711e82

more comments

a237c11

include pixel offsets, units for layers, and better comments

04cd8c2

include entry point for dense offsets and unfiltered coherence

7df9cb7

ensure json determines possible layers that can be added

cd12691

remove print dataset and add spacing in join statement

1009e2d

include pixel offsets in science/grids/data

b133ee8

cmarshak marked this pull request as ready for review March 30, 2023 17:42

cmarshak requested a review from gracebato March 30, 2023 17:42

cmarshak added 3 commits March 30, 2023 11:02

swap azimuth and then range

f7ed9e3

fix standard name

a3628f8

fix ampcor wind size

3001162

cmarshak mentioned this pull request Apr 5, 2023

Increase DEM buffer for Frames #128

Merged

cmarshak mentioned this pull request Apr 20, 2023

Instructions for trying this plugin ASFHyP3/hyp3-isce2#28

Closed

cmarshak mentioned this pull request Jun 27, 2023

Iono adding burst jumps #142

Merged

cmarshak added 2 commits November 7, 2023 16:23

merge dev

247f717

merge again

d92a6f1

cmarshak added 2 commits November 16, 2023 18:03

update localize-data

0583a8a

fix errant cell

5ef33a5

remove wrapped phase from params

3233976

cmarshak mentioned this pull request Nov 20, 2023

Standardize parameters for GUNW ASFHyP3/hyp3#1937

Merged

cmarshak and others added 3 commits November 20, 2023 16:57

update packaging

4b8054e

lint

62179c0

serialize 2d layers and attrs

144793d

cmarshak and others added 5 commits November 20, 2023 18:20

one comment for serialization

c851c61

fix packaging call

6b0f754

reorder main for clarity of variable inti

8dce8ab

fix language

fffddb9

fix filter bug

1d460a6

cmarshak added 2 commits November 21, 2023 11:50

default arg for iono

771782b

validate args

6c33e42

remove iono template

c45888b

cmarshak requested a review from sssangha November 22, 2023 18:26

cmarshak merged commit c3ae958 into dev Nov 29, 2023
7 checks passed

cmarshak deleted the custom-event branch November 29, 2023 18:11

cmarshak mentioned this pull request Feb 5, 2024

Update to python 3.10+ #162

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom event #122

Custom event #122

cmarshak commented Mar 27, 2023 •

edited

Loading

cmarshak commented Mar 30, 2023

cmarshak commented Sep 11, 2023 •

edited

Loading

review-notebook-app bot commented Nov 8, 2023

cmarshak commented Nov 20, 2023 •

edited

Loading

cmarshak commented Nov 20, 2023 •

edited

Loading

cmarshak commented Nov 20, 2023

cmarshak commented Nov 20, 2023

cmarshak commented Nov 20, 2023 •

edited

Loading

cmarshak commented Nov 21, 2023 •

edited

Loading

cmarshak commented Nov 21, 2023 •

edited

Loading

cmarshak commented Nov 21, 2023

mgovorcin commented Nov 21, 2023 •

edited

Loading

cmarshak commented Nov 21, 2023 •

edited

Loading

cmarshak commented Nov 22, 2023

Custom event #122

Custom event #122

Conversation

cmarshak commented Mar 27, 2023 • edited Loading

cmarshak commented Mar 30, 2023

cmarshak commented Sep 11, 2023 • edited Loading

review-notebook-app bot commented Nov 8, 2023

cmarshak commented Nov 20, 2023 • edited Loading

Added

Changed

cmarshak commented Nov 20, 2023 • edited Loading

cmarshak commented Nov 20, 2023

cmarshak commented Nov 20, 2023

cmarshak commented Nov 20, 2023 • edited Loading

cmarshak commented Nov 21, 2023 • edited Loading

cmarshak commented Nov 21, 2023 • edited Loading

cmarshak commented Nov 21, 2023

mgovorcin commented Nov 21, 2023 • edited Loading

cmarshak commented Nov 21, 2023 • edited Loading

cmarshak commented Nov 22, 2023

cmarshak commented Mar 27, 2023 •

edited

Loading

cmarshak commented Sep 11, 2023 •

edited

Loading

cmarshak commented Nov 20, 2023 •

edited

Loading

cmarshak commented Nov 20, 2023 •

edited

Loading

cmarshak commented Nov 20, 2023 •

edited

Loading

cmarshak commented Nov 21, 2023 •

edited

Loading

cmarshak commented Nov 21, 2023 •

edited

Loading

mgovorcin commented Nov 21, 2023 •

edited

Loading

cmarshak commented Nov 21, 2023 •

edited

Loading