Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI geoips describe data #751

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
Draft

CLI geoips describe data #751

wants to merge 9 commits into from

Conversation

evrose54
Copy link
Contributor

This PR is still in draft stage and isn't expected to get out of that for a while. Updates to Readers are likely needed.

This PR also branched off of #739, so some of the changes are linked to that PR.

Reviewer Checklist

  • Required existing tests pass (ie full_test.sh, others as appropriate)
  • NO REQUIRED existing tests (explain why not required)
  • Required unit tests added and pass for new/modified functionality
  • NO REQUIRED unit tests (explain why not required)
  • Required integration tests added and pass for new/modified functionality
  • NO REQUIRED integration tests (explain why not required)
  • Required documentation added for new/modified functionality
  • NO REQUIRED documentation (explain why not required)
  • Required release notes added for new/modified functionality
  • NO REQUIRED release notes (explain why not required)
  • Required updates to other repos complete
  • NO REQUIRED updates to other repos (explain why not required)

Related Issues

fixes #737

Testing Instructions

Will provide once command is stable.

Summary

This will be held as a draft until we can get the output of readers consistent. For example, when testing with reader abi_netcdf, different xarray_dicts are returned depending if an area_def is specified or not. If no area_def is specified, the xarray_dict will look like: {"LOW": xobj, "MED": xobj, "HIGH": xobj, "METADATA": xobj}. If area_def is specified, the xarray_dict will look like: {"sector_name": xobj, "METADATA": xobj}. Due to these inconsistencies, we cannot provide the same structure of data for each reader.

This PR also adds the command geoips --version, as requested by @mindyls. I additionally play along with validating the output of readers, though this is still in development as I need to figure out how to link BaseReaderPlugin:validate_output to the end of a reader plugin's __call__ method.

Initial stage of CLI command geoips describe data. This command will describe the data that will come out of a reader. Its signature is geoips describe data <reader_name> <file_paths>. Expected output will look along the lines of

Datasets:
  Resolution1:
    - ds_name1
    - ds_name2
  Resolution2:
    - ds_name3
    - ds_name4
Coords:
  coord_info:
Dims:
  dim_info:
Metadata:
  md_key1: info
  md_key2: info

Output

Here is the current example of running geoips describe data abi_netcdf /home/evan/geoips/geoips_packages/test_data/test_data_noaa_aws/data/goes16/20200918/1950/*, where we provide a postage sector for quick processing. Note that no resolutions are specified under Datasets.

(geoips) [evan@spring geoips]$ geoips desc data abi_netcdf /home/evan/geoips/geoips_packages/test_data/test_data_noaa_aws/data/goes16/20200918/1950/*
Metadata: 
  file_metadata: 
    B01: 
      file_info: 
        id: c081667d-0edd-45e2-a82a-62fc683a6d93
        dataset_name: OR_ABI-L1b-RadF-M6C01_G16_s20202621950205_e20202621959513_c20202621959567.nc
        naming_authority: gov.nesdis.noaa
        institution: DOC/NOAA/NESDIS > U.S. Department of Commerce, National Oceanic and Atmospheric Administration, National Environmental Satellite, Data, and Information Services
        project: GOES
        iso_series_metadata_id: a70be540-c38b-11e0-962b-0800200c9a66
        Conventions: CF-1.7
        Metadata_Conventions: Unidata Dataset Discovery v1.0
        keywords_vocabulary: NASA Global Change Master Directory (GCMD) Earth Science Keywords, Version 7.0.0.0.0
        standard_name_vocabulary: CF Standard Name Table (v35, 20 July 2016)
        title: ABI L1b Radiances
        summary: Single reflective band ABI L1b Radiance Products are digital maps of outgoing radiance values at the top of the atmosphere for visible and near-IR bands.
        license: Unclassified data.  Access is restricted to approved users only.
        keywords: SPECTRAL/ENGINEERING > VISIBLE WAVELENGTHS > VISIBLE RADIANCE
        cdm_data_type: Image
        orbital_slot: GOES-East
        platform_ID: G16
        instrument_type: GOES R Series Advanced Baseline Imager
        processing_level: National Aeronautics and Space Administration (NASA) L1b
        date_created: 2020-09-18T19:59:56.7Z
        production_site: RBU
        production_environment: OE
        production_data_source: Realtime
        timeline_id: ABI Mode 6
        scene_id: Full Disk
        spatial_resolution: 1km at nadir
        time_coverage_start: 2020-09-18T19:50:20.5Z
        time_coverage_end: 2020-09-18T19:59:51.3Z
      var_info: 
        band_id: [1]
        band_wavelength: [0.47]
        earth_sun_distance_anomaly_in_AU: 1.004645
        esun: 2017.1648
        kappa0: 0.0015719
        max_radiance_value_of_valid_pixels: 646.07336
        min_radiance_value_of_valid_pixels: -0.5803809
        missing_pixel_count: 1403
        nominal_satellite_height: 35786.023
        nominal_satellite_subpoint_lat: 0.0
        nominal_satellite_subpoint_lon: -75.2
        percent_uncorrectable_L0_errors: 0.0
        planck_bc1: --
        planck_bc2: --
        planck_fk1: --
        planck_fk2: --
        processing_parm_version_container: --
        saturated_pixel_count: 15303
        std_dev_radiance_value_of_valid_pixels: 97.364265
        time_bounds: [6.53730621e+08 6.53731191e+08]
        undersaturated_pixel_count: 0
        valid_pixel_count: 92168222
        x: [-0.151858   -0.15183    -0.151802   ...  0.151802    0.15182999  0.151858  ]
        y: [ 0.151858    0.15183     0.151802   ... -0.151802   -0.15182999 -0.151858  ]
        yaw_flip_flag: 0
        num_lines: 10848
        num_samples: 10848
      ll_info: 
        geospatial_eastbound_longitude: 6.2995
        geospatial_lat_center: 0.0
        geospatial_lat_nadir: 0.0
        geospatial_lon_center: -75.0
        geospatial_lon_nadir: -75.0
        geospatial_northbound_latitude: 81.3282
        geospatial_southbound_latitude: -81.3282
        geospatial_westbound_longitude: -156.2995
      projection: 
        inverse_flattening: 298.2572221
        latitude_of_projection_origin: 0.0
        longitude_of_projection_origin: -75.0
        perspective_point_height: 35786023.0
        semi_major_axis: 6378137.0
        semi_minor_axis: 6356752.31414
        grid_mapping: geostationary
      path: /home/evan/geoips/geoips_packages/test_data/test_data_noaa_aws/data/goes16/20200918/1950/OR_ABI-L1b-RadF-M6C01_G16_s20202621950205_e20202621959513_c20202621959567.nc
      satellite: G16
      sensor: ABI
      num_lines: 10848
      num_samples: 10848
  start_datetime: 2020-09-18 19:50:20.526449
  end_datetime: 2020-09-18 19:59:51.323902
  source_name: abi
  data_provider: noaa
  platform_name: goes-16
  area_definition: None
  area_id: Full-Disk
  interpolation_radius_of_influence: 2000.0
Datasets: 
  - B01Rad
  - B01Ref
  - B02Rad
  - B02Ref
  - B03Rad
  - B03Ref
  - B04Rad
  - B04Ref
  - B05Rad
  - B05Ref
  - B06Rad
  - B06Ref
  - B07BT
  - B07Rad
  - B08BT
  - B08Rad
  - B09BT
  - B09Rad
  - B10BT
  - B10Rad
  - B11BT
  - B11Rad
  - B12BT
  - B12Rad
  - B13BT
  - B13Rad
  - B14BT
  - B14Rad
  - B15BT
  - B15Rad
  - B16BT
  - B16Rad
  - latitude
  - longitude
  - satellite_azimuth_angle
  - satellite_zenith_angle
  - solar_azimuth_angle
  - solar_zenith_angle
Coordinates: 
Dimensions: 
  dim_0: 10
  dim_1: 10
Source Names: 
  - abi

WARNING: The GeoIPS CLI is currently under development and is subject to change.
Until this warning is removed, do not rely on the CLI to be static.
Please feel free to test the CLI and report any bugs or comments as an issue here:
https://github.com/NRLMMD-GEOIPS/geoips/issues/new/choose

@evrose54 evrose54 added enhancement New feature or request cli Command line interface tasks labels Aug 16, 2024
@evrose54 evrose54 marked this pull request as draft August 16, 2024 19:52
Base automatically changed from CLI-geoips-list-source-names to main October 22, 2024 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cli Command line interface tasks enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant