Add ocean and invalid reflectance masking (#36)

* merge main into add_ocean_mask

* update proteus version from 0.5.1 to 0.5.2

* update proteus version from 0.5.1 to 0.5.2 (2)

* update identation of messages logged/printed to the user in the function `save_as_cog()`

* add ocean masking using a shoreline shape file

* remove debug messages and update comments

* add runconfig field ocean_masking_shoreline_distance_km

* add cumulative reflectance no-data masking

* add cumulative reflectance no-data masking (2)

* refactor `save_dswx_product()` and only add CLOUD-masked classes to colortable if layer name is WTR

* add test for coverage of ancillary inputs

* update snow-, cloud-, and ocean-masked values from 8, 9, and 7 to 252, 253, and 254, respectively

* make ocean masked classes transparent

* make ocean masked classes transparent (2)

* update CLOUD_OCEAN_MASKED to 254

* Update global variable names and CONF layer with _CLEAR, _CLOUD, and _SNOW designations.

* Update global variable names and CONF layer with _CLEAR, _CLOUD, and _SNOW designations. (2)

* Apply suggestions from code review

Whitespace and update to a docstring.

Co-authored-by: Gustavo H. X. Shiroma <52007211+gshiroma@users.noreply.github.com>

* update to documentation per code review

* add new line to the end of the file

* rename shape_file to shapefile

* reduce shoreline margin from 10 km to 5 km

* update comments and docstrings

* close gdal dataset

* simplify code

* add comments for check_ancillary_inputs_coverage

* update messages printed to the user regarding Sun parameters

* rename `cloud_mask` to `cloud_layer`

* update _load_hls_band_from_file()

* add `temp_shapefile_filename` to the list of temporary files

* read SOFTWARE_VERSION from version.py

* update _load_hls_band_from_file() (2)

* handle anti-meridian

* give precedence to fill value over ocean mask

* update description of CLOUD layer

* change ocean masked color entry to match the open water as per S. Chan's request

* give precedence to clouds over the ocean mask as per S. Chan's request

* change default value for  per S. Chan's request

* change default of `exclude_psw_aggressive_in_browse` from `False` to `True` per chat with S. Chan

* give precedence to clouds over the ocean mask as per S. Chan's request (2)

* give precedence to clouds over the ocean mask as per S. Chan's request (3)

* remove all temporary files

* improve comments and variable names

* improve comments and variable names (2)

* set ocean masked values to transparent in browse images

* use dark blue for ocean-masked pixels

* remove temporary shape files

* rename variable from `temp_file test` to `temp_file`

* give precedence to the ocean mask over clouds

* check for file`s existence before adding it to temporary file`s list

* update OPERA Cal/Val release version from R3.2 to R3.3

* update workflow test dataset URL

Co-authored-by: Samantha Niemoeller <11642807+nemo794@users.noreply.github.com>

Author: gshiroma

README.md

@@ -98,7 +98,7 @@ Then, from inside the cloned repository, build the Docker image:
 Load the Docker container image onto your computer:
 
 ```bash
-docker load -i docker/dockerimg_proteus_cal_val_3.1.tar
+docker load -i docker/dockerimg_proteus_cal_val_3.3.tar
 ```
 
 See DSWx-HLS Science Algorithm Software (SAS) User Guide for instructions on processing via Docker.

bin/dswx_hls.py

@@ -51,7 +51,7 @@ def main():
         args.output_file,
         hls_thresholds = runconfig_constants.hls_thresholds,
         dem_file=args.dem_file,
-        dem_description=args.dem_description,
+        dem_file_description=args.dem_file_description,
         output_interpreted_band=args.output_interpreted_band,
         output_rgb_file=args.output_rgb_file,
         output_infrared_rgb_file=args.output_infrared_rgb_file,
@@ -62,7 +62,7 @@ def main():
         output_shadow_masked_dswx=args.output_shadow_masked_dswx,
         output_landcover=args.output_landcover,
         output_shadow_layer=args.output_shadow_layer,
-        output_cloud_mask=args.output_cloud_mask,
+        output_cloud_layer=args.output_cloud_layer,
         output_dem_layer=args.output_dem_layer,
         output_browse_image=args.output_browse_image,
         browse_image_height=args.browse_image_height,
@@ -72,9 +72,11 @@ def main():
         cloud_in_browse=args.cloud_in_browse,
         snow_in_browse=args.snow_in_browse,
         landcover_file=args.landcover_file,
-        landcover_description=args.landcover_description,
+        landcover_file_description=args.landcover_file_description,
         worldcover_file=args.worldcover_file,
-        worldcover_description=args.worldcover_description,
+        worldcover_file_description=args.worldcover_file_description,
+        shoreline_shapefile=args.shoreline_shapefile,
+        shoreline_shapefile_description=args.shoreline_shapefile_description,
         flag_offset_and_scale_inputs=args.flag_offset_and_scale_inputs,
         scratch_dir=args.scratch_dir,
         product_id=args.product_id,
@@ -85,6 +87,8 @@ def main():
         max_sun_local_inc_angle=args.max_sun_local_inc_angle,
         mask_adjacent_to_cloud_mode=args.mask_adjacent_to_cloud_mode,
         copernicus_forest_classes=args.copernicus_forest_classes,
+        ocean_masking_shoreline_distance_km = \
+            args.ocean_masking_shoreline_distance_km,
         flag_debug=args.flag_debug)
 
 

build_docker_image.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 
 IMAGE=opera/proteus
-t=cal_val_3.2
+t=cal_val_3.3
 echo "IMAGE is $IMAGE:$t"
 
 # fail on any non-zero exit codes

docker/Dockerfile

@@ -32,12 +32,12 @@ RUN set -ex \
 
 # set default user and workdir
 WORKDIR /home/conda
-ADD dist/proteus-0.5.1.tar.gz .
+ADD dist/proteus-0.5.2.tar.gz .
 
 USER root
-RUN mkdir -p proteus-0.5.1/build
+RUN mkdir -p proteus-0.5.2/build
 RUN export PATH=/opt/conda/bin:$PATH
-WORKDIR /home/conda/proteus-0.5.1
+WORKDIR /home/conda/proteus-0.5.2
 RUN python3 setup.py install
 WORKDIR /home/conda
 
@@ -48,7 +48,7 @@ ENV GDAL_DIR /opt/conda
 ENV PATH $GDAL_DIR/bin:$PATH
 ENV GDAL_DATA $GDAL_DIR/share/gdal
 
-ENV PYTHONPATH /home/conda/proteus-0.5.1/src/:$PYTHONPATH
-ENV PATH /home/conda/proteus-0.5.1/bin/:$PATH
+ENV PYTHONPATH /home/conda/proteus-0.5.2/src/:$PYTHONPATH
+ENV PATH /home/conda/proteus-0.5.2/bin/:$PATH
 
-CMD [ "/bin/bash" ]
+CMD [ "/bin/bash" ]

load_docker_tar.sh

@@ -1,3 +1,3 @@
 #!/bin/bash
 
-docker load -i docker/dockerimg_proteus_cal_val_3.2.tar
+docker load -i docker/dockerimg_proteus_cal_val_3.3.tar

src/proteus/core.py

@@ -27,7 +27,8 @@ def save_as_cog(filename, scratch_dir = '.', logger = None,
     if logger is None:
         logger = logging.getLogger('proteus')
 
-    logger.info('COG step 1: add overviews')
+    logger.info(f'saving file as COG: {filename}')
+    logger.info('    step 1: add overviews')
     gdal_ds = gdal.Open(filename, gdal.GA_Update)
     gdal_dtype = gdal_ds.GetRasterBand(1).DataType
     dtype_name = gdal.GetDataTypeName(gdal_dtype).lower()
@@ -48,7 +49,7 @@ def save_as_cog(filename, scratch_dir = '.', logger = None,
     if os.path.isfile(external_overview_file):
         os.remove(external_overview_file)
 
-    logger.info('COG step 2: save as COG')
+    logger.info('    step 2: save as COG')
     temp_file = tempfile.NamedTemporaryFile(
                     dir=scratch_dir, suffix='.tif').name
 
@@ -72,7 +73,7 @@ def save_as_cog(filename, scratch_dir = '.', logger = None,
 
     shutil.move(temp_file, filename)
 
-    logger.info('COG step 3: validate')
+    logger.info('    step 3: validate')
     try:
         from proteus.extern.validate_cloud_optimized_geotiff import main as validate_cog
     except ModuleNotFoundError:
@@ -82,10 +83,10 @@ def save_as_cog(filename, scratch_dir = '.', logger = None,
     argv = ['--full-check=yes', filename]
     validate_cog_ret = validate_cog(argv)
     if validate_cog_ret == 0:
-        logger.info(f'file "{filename}" is a valid cloud optimized'
+        logger.info(f'    file "{filename}" is a valid cloud optimized'
                     ' GeoTIFF')
     else:
-        logger.warning(f'file "{filename}" is NOT a valid cloud'
+        logger.warning(f'    file "{filename}" is NOT a valid cloud'
                        f' optimized GeoTIFF!')
 
 

src/proteus/defaults/dswx_hls.yaml

@@ -11,22 +11,32 @@ runconfig:
 
         dynamic_ancillary_file_group:
             # Digital elevation model file
+            # (REQUIRED if check_ancillary_inputs_coverage is True)
             dem_file:
 
             # Digital elevation model source description
-            dem_description:
+            dem_file_description:
 
             # Copernicus Global Land Service (CGLS) Land Cover Layer file
+            # (REQUIRED if check_ancillary_inputs_coverage is True)
             landcover_file:
 
             # Copernicus Global Land Service (CGLS) Land Cover Layer description
-            landcover_description:
+            landcover_file_description:
 
             # ESA WorldCover map file
+            # (REQUIRED if check_ancillary_inputs_coverage is True)
             worldcover_file:
 
             # ESA WorldCover map description
-            worldcover_description:
+            worldcover_file_description:
+
+            # NOAA GSHHS shapefile
+            # (REQUIRED if check_ancillary_inputs_coverage is True)
+            shoreline_shapefile:
+
+            # NOAA GSHHS shapefile description
+            shoreline_shapefile_description:
 
         primary_executable:
             product_type: DSWX_HLS
@@ -71,6 +81,9 @@ runconfig:
             # Copernicus CGLS Land Cover 100m forest classes
             copernicus_forest_classes: [20, 111, 113, 115, 116, 121, 123, 125, 126]
 
+            # Ocean masking distance from shoreline in km
+            ocean_masking_shoreline_distance_km: 1
+
             save_wtr: True    # Layer 1 - WTR
             save_bwtr: True   # Layer 2 - BWTR
             save_conf: True   # Layer 3 - CONF
@@ -109,7 +122,7 @@ runconfig:
             # class in the browse image. If True, PSW-Agg pixels will 
             # appear as Not Water. If False, this class will be displayed
             # in the browse image, same as in WTR. Default is False.
-            exclude_psw_aggressive_in_browse: False
+            exclude_psw_aggressive_in_browse: True
 
             # Define how Not Water (e.g. land) appears in the browse image.
             # Defaults to 'white'.