EAMxx: Issue with select level output with min and max requests

[bogensch]

I submitted an ne30pg2_ne30pg2 run with the following YAML output file:

%YAML 1.1
---
filename_prefix: output.scream.daily
averaging_type: max
max_snapshots_per_file: 100
track_fill: true
fields:
  physics_pg2:
    field_names:
    #2D vars
    - LongwaveCloudForcing
    - ShortwaveCloudForcing
    - ps
    - SeaLevelPressure
    - T_2m
    - qv_2m
    - surf_radiative_T
    - VapWaterPath
    - IceWaterPath
    - LiqWaterPath
    - RainWaterPath
    - ZonalVapFlux
    - MeridionalVapFlux
    - surf_evap
    - surf_sens_flux
    - surface_upward_latent_heat_flux
    - precip_liq_surf_mass_flux
    - precip_ice_surf_mass_flux
    - landfrac
    - ocnfrac
    - PotentialTemperature_at_700hPa
    - PotentialTemperature_at_1000hPa
    - omega_at_500hPa
    - RelativeHumidity_at_700hPa
    - pbl_height
output_control:
  frequency: 1
  frequency_units: ndays
  mpi_ranks_in_filename: false


Which produced the following error:

12: aborting job:
12: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 12
 0: terminate called after throwing an instance of 'std::runtime_error'
 0:   what():  Error! Could not retrieve the file. File not open.
 0:  - filename: output.scream.daily.MAX.ndays_x1.0001-01-01-00000.nc
 0: Context:
 0:  scorpio::release_file
 0:
 0: FAILED CONDITION: 's.files.count(filename)==1'
 0:
 0: BACKTRACE:
 0: /pscratch/sd/b/bogensch/dp_scream/codes/E3SM_master/components/eamxx/src/share/scorpio_interface/eamxx_scorpio_i >
 0:
 0:
 0: Program received signal SIGABRT: Process abort signal.
 0:
 0: Backtrace for this error:
 8: PIO: FATAL ERROR: Aborting... An error occured, Writing variables (number of variables = 25) to file (output.scr >
 4: PIO: FATAL ERROR: Aborting... An error occured, Writing variables (number of variables = 25) to file (output.scr >
 4: Obtained 10 stack frames.
 4: /pscratch/sd/b/bogensch/e3sm_scratch/pm-gpu/F2010-SCREAMv1.ne30pg2_ne30pg2.16x1.pbl.002b/bld/e3sm.exe() [0x34d7f

After many trials I discovered that if I use the same YAML file but omit the following variables:

- PotentialTemperature_at_700hPa
  - PotentialTemperature_at_1000hPa
  - omega_at_500hPa
  - RelativeHumidity_at_700hPa

My run is successful.

After further trial and error I only seem to get this error with the above variables when I request min or max output. Instant and average do not seem to be affected (I got successful runs with these).

Note that I am using most current master. I was previously getting these errors with master dated March 4 and I updated to current, but the error still persists.