`regional_atmaq_debug_oneapi` failing w/spack-stack v2.1.1

[gspetro-NOAA]

Description

Nearly all non-debug RTs were passing during spack-stack v2.1.0 testing. spack-stack v2.1.1 testing shows that the issue with the debug tests has been resolved. All debug tests now pass on Ursa except for regional_atmaq_debug_oneapi. This test is failing in AQM (rather than in FMS). The stack trace for the initial error says:

248: forrtl: error (65): floating invalid
248: Image              PC                Routine            Line        Source
248: libc.so.6          00001548DA66CBF0  Unknown               Unknown  Unknown
248: fv3.exe            0000000001E8FA5D  getdep_v                  595  aero_depv.F
248: fv3.exe            0000000001E8374C  aero_depv                 272  aero_depv.F
248: fv3.exe            0000000001AFF567  get_depv                  630  DEPV_DEFN.F
248: fv3.exe            00000000015DD10A  vdiff                     413  vdiffproc.F
248: fv3.exe            00000000013C6C81  cmaq_advance              186  cmaq_mod.F90
248: fv3.exe            00000000013C5369  cmaq_model_advanc         139  cmaq_model_mod.F90
248: fv3.exe            000000000136061E  aqm_comp_advance          219  aqm_comp_mod.F90
248: fv3.exe            000000000135F095  modeladvance              593  aqm_cap.F90

The err file can be found at /scratch4/NAGAPE/epic/Gillian.Petro/stmp/RT_RUNDIRS/Gillian.Petro/FV3_RT/rt_3587556/regional_atmaq_debug_oneapi/err, and my test directory on Ursa is at: /scratch3/NAGAPE/epic/Gillian.Petro/ursa/RTs/ufs-wm/pretests/3174-debug/tests.

We need an AQM developer to look into this issue before we move to spack-stack v2.1.1.

To Reproduce:

1. git clone --recursive -b feature/ss2.x https://github.com/gspetro-NOAA/ufs-weather-model.git 3174
2. cd 3174/modulefiles
3. Update the ufs_ursa.oneapi.lua modulefile to point to the v2.1.1 test stack on Ursa:

   -prepend_path("MODULEPATH", "/contrib/spack-stack/spack-stack-2.1.0/envs/ue-oneapi-2025.3.1/modules/Core")
   +prepend_path("MODULEPATH", "/contrib/spack-stack/spack-stack-2.1.1/envs/ue-oneapi-2025.3.1-build/modules/Core")
   

4. cd ../tests
5. Run the test: ./rt.sh -a <account> -n "regional_atmaq_debug oneapi"

Additional context

• needed for PR Upgrade WM to spack-stack 2.1.x #3174 & Issue Update UFS-WM to spack-stack/2.1.x #3067

Output

Log Excerpt:

UFSWM hash used in testing:
4f85fe2935af0014ec6a1593270a9c6bde28a333

...

BASELINE DIRECTORY: /scratch4/NAGAPE/epic/role-epic/UFS-WM_RT/NEMSfv3gfs/develop-20260615
COMPARISON DIRECTORY: /scratch4/NAGAPE/epic/Gillian.Petro/stmp/RT_RUNDIRS/Gillian.Petro/FV3_RT/rt_3587556

RT.SH OPTIONS USED:
* (-a) - HPC PROJECT ACCOUNT: epic
* (-c) - CREATE NEW BASELINES
* (-n) - RUN SINGLE TEST: regional_atmaq_debug
* (-k) - KEEP RUN DIRECTORY
* (-e) - USE ECFLOW

PASS -- COMPILE 'atmaq_debug_oneapi' [03:11, 01:50]
FAILED: RUN DID NOT COMPLETE -- TEST 'regional_atmaq_debug_oneapi' [, ]( MB)

SYNOPSIS:
Starting Date/Time: 20260622 21:06:20
Ending Date/Time: 20260622 21:28:40
Total Time: 00h:22m:26s
Compiles Completed: 1/1
Tests Completed: 0/1
Failed Tests:
* TEST regional_atmaq_debug_oneapi: FAILED: RUN DID NOT COMPLETE
-- LOG: /scratch3/NAGAPE/epic/Gillian.Petro/ursa/RTs/ufs-wm/pretests/3174-debug/tests/logs/log_ursa/run_regional_atmaq_debug_oneapi.log

NOTES:
A file 'test_changes.list' was generated with list of all failed tests.
You can use './rt.sh -c -b test_changes.list' to create baselines for the failed tests.
If you are using this log as a pull request verification, please commit 'test_changes.list'.

Result: FAILURE

[gspetro-NOAA]

@KaiWang-NOAA Is this something you would be able to look into? If not, who would you recommend?

[KaiWang-NOAA]

@KaiWang-NOAA Is this something you would be able to look into? If not, who would you recommend?
I will look into it later this week.

[rickgrubin-noaa]

@KaiWang-NOAA Is this something you would be able to look into? If not, who would you recommend?
I will look into it later this week.

@KaiWang-NOAA any status updates? Thanks.

[gspetro-NOAA]

@KaiWang-NOAA Is this something you would be able to look into? If not, who would you recommend?
I will look into it later this week.

@KaiWang-NOAA any status updates? Thanks.

@rickgrubin-noaa This test is now passing! The changes are included in PR #3174 with an addition to the GOCART section of CMakeLists.txt. 🙂

[KaiWang-NOAA]

@gspetro-NOAA Thank you for confirming this. My latest test run also passed without showing the error.