Derecho-specific issues for the WM spack-stack v2.1.x upgrade

[gspetro-NOAA]

This issue will document issues and conversation around the spack-stack v2.1.x upgrade on Derecho.

It is related to Issue #3067 and PR #3174 .

[gspetro-NOAA]

@rickgrubin-noaa @climbfuji The latest issue is that most oneapi debug tests are now failing in compile on Derecho with the bugfix provided in gspetro-NOAA#6.

The error message for all resembles this one from compile_s2s_32bit_sfs_debug_oneapi/err:

/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o): mapped section .debug_info has non-NULL buffer
UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o):(.debug_info+0x1916): undefined reference to `MPIR_C_MPI_ERRCODES_IGNORE'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o):(.debug_info+0x1930): undefined reference to `MPIR_C_MPI_ARGVS_NULL'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o):(.debug_info+0x194a): undefined reference to `MPIR_C_MPI_ARGV_NULL'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o):(.debug_info+0x1998): undefined reference to `MPIR_C_MPI_STATUSES_IGNORE'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o):(.debug_info+0x19b2): undefined reference to `MPIR_C_MPI_STATUS_IGNORE'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o): mapped section .debug_info has non-NULL buffer
UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o):(.debug_info+0xc3): undefined reference to `MPIR_C_MPI_ERRCODES_IGNORE'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o):(.debug_info+0xdc): undefined reference to `MPIR_C_MPI_ARGVS_NULL'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o):(.debug_info+0xf5): undefined reference to `MPIR_C_MPI_ARGV_NULL'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o):(.debug_info+0x140): undefined reference to `MPIR_C_MPI_STATUSES_IGNORE'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o):(.debug_info+0x159): undefined reference to `MPIR_C_MPI_STATUS_IGNORE'
make[2]: *** [CMakeFiles/ufs_model.dir/build.make:136: ufs_model] Error 1
make[1]: *** [CMakeFiles/Makefile2:531: CMakeFiles/ufs_model.dir/all] Error 2

I attempted running with the Cray compiler wrappers (ftn / cc / CC) in the modulefiles and got similar results with several printouts of this message:

No supported cpu target is set, CRAY_CPU_TARGET=x86-64 will be used.
Load a valid targeting module or set CRAY_CPU_TARGET

The easiest place to view the compile_s2s_32bit_sfs_debug_oneapi/err file is /glade/derecho/scratch/gpetro/FV3_RT/rt_99125/compile_s2s_32bit_sfs_debug_oneapi/err. I have several other run directories I can point to, but this one has the modulefiles and code as it is in feature/ss2.x without other changes.

[gspetro-NOAA]

There are also several tests failing for compiles that succeeded, but I will look into those errors tomorrow. Same for the MPAS tests, which are failing in compile again.

[climbfuji]

@rickgrubin-noaa @climbfuji The latest issue is that most oneapi debug tests are now failing in compile on Derecho with the bugfix provided in gspetro-NOAA#6.

The error message for all resembles this one from compile_s2s_32bit_sfs_debug_oneapi/err:

/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o): mapped section .debug_info has non-NULL buffer
UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o):(.debug_info+0x1916): undefined reference to `MPIR_C_MPI_ERRCODES_IGNORE'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o):(.debug_info+0x1930): undefined reference to `MPIR_C_MPI_ARGVS_NULL'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o):(.debug_info+0x194a): undefined reference to `MPIR_C_MPI_ARGV_NULL'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o):(.debug_info+0x1998): undefined reference to `MPIR_C_MPI_STATUSES_IGNORE'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_regional_bc.F90.o):(.debug_info+0x19b2): undefined reference to `MPIR_C_MPI_STATUS_IGNORE'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o): mapped section .debug_info has non-NULL buffer
UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o):(.debug_info+0xc3): undefined reference to `MPIR_C_MPI_ERRCODES_IGNORE'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o):(.debug_info+0xdc): undefined reference to `MPIR_C_MPI_ARGVS_NULL'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o):(.debug_info+0xf5): undefined reference to `MPIR_C_MPI_ARGV_NULL'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o):(.debug_info+0x140): undefined reference to `MPIR_C_MPI_STATUSES_IGNORE'
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: UFSATM/fv3/atmos_cubed_sphere/libfv3.a(fv_mp_mod.F90.o):(.debug_info+0x159): undefined reference to `MPIR_C_MPI_STATUS_IGNORE'
make[2]: *** [CMakeFiles/ufs_model.dir/build.make:136: ufs_model] Error 1
make[1]: *** [CMakeFiles/Makefile2:531: CMakeFiles/ufs_model.dir/all] Error 2

I attempted running with the Cray compiler wrappers (ftn / cc / CC) in the modulefiles and got similar results with several printouts of this message:

No supported cpu target is set, CRAY_CPU_TARGET=x86-64 will be used.
Load a valid targeting module or set CRAY_CPU_TARGET

The easiest place to view the compile_s2s_32bit_sfs_debug_oneapi/err file is /glade/derecho/scratch/gpetro/FV3_RT/rt_99125/compile_s2s_32bit_sfs_debug_oneapi/err. I have several other run directories I can point to, but this one has the modulefiles and code as it is in feature/ss2.x without other changes.

The Cray compiler wrappers aren't to be used. The entire stack was compiled with the vendor compiler wrappers.

The MPIR error you posted above are (should be?) fixed by lines 61-64 in file cmake/IntelLLVM.cmake in 1220017#diff-2f67bd87577369636044899c8d15036f7ad45e7c41e289cfadbcc4d865b67e58R61. Can you check if there is an additional -g somewhere in the compile/linker line that needs to become -g1?

[gspetro-NOAA]

The Cray compiler wrappers aren't to be used. The entire stack was compiled with the vendor compiler wrappers.

Good to know--thanks!

The MPIR error you posted above are (should be?) fixed by lines 61-64 in file cmake/IntelLLVM.cmake in 1220017#diff-2f67bd87577369636044899c8d15036f7ad45e7c41e289cfadbcc4d865b67e58R61. Can you check if there is an additional -g somewhere in the compile/linker line that needs to become -g1?

cmake/IntelLLVM.cmake only has -g assigned explicitly once. However, it seems that certain WM (sub)components set a -g flag, too, after the main WM compiler flags are set. I don't know where to find them all, but the out file definitely has a -g in the list of flags after the -g1 in several places.

UPDATE: In /glade/derecho/scratch/gpetro/FV3_RT/rt_99125/compile_s2s_32bit_sfs_debug_oneapi/build_fv3_s2s_32bit_sfs_debug_oneapi/UFSATM/fv3/atmos_cubed_sphere/CMakeFiles/fv3.dir/flags.make it lists -g1 at the start of the list but -g in the middle:

Fortran_FLAGS =  -g1 -traceback -fpp -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -align array64byte -qno-opt-dynamic-align -diag-disable 5462 -diag-disable 7712 -diag-disable 8291 -diag-disable 5194 -g -traceback -fpp -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -nowarn -align array64byte -qno-opt-dynamic-align -real-size 32 -O0 -check -check noarg_temp_created -check nopointer -warn -warn noerrors -fstack-protector-all -fpe0 -debug -ftrapuv -init=snan,arrays -module include/fv3 -fPIC -fiopenmp

[NickSzapiro-NOAA]

CICE, CDEPS, CMEPS, MOM6 (at least) all have "-g" in their CMakeLists
https://github.com/search?q=repo%3Aufs-community%2Fufs-weather-model+-g+%2FCMakeLists&type=code

[climbfuji]

CICE, CDEPS, CMEPS, MOM6 (at least) all have "-g" in their CMakeLists https://github.com/search?q=repo%3Aufs-community%2Fufs-weather-model+-g+%2FCMakeLists&type=code

Hmm. We could try to go back to the EPIC 2025.3.1 compilers or update to 2026.x.y. Since this is for 2.1.1, I am in favor of the former. I can try a 2025.3.1 stack over the next few days.

[gspetro-NOAA]

As for the non-debug tests, the oneapi ones that fail have messages like:

+ mpiexec -n 480 -ppn 120 --hostfile /var/spool/pbs/aux/6937542.desched1 ./fv3.exe
MPICH ERROR [Rank 0] [job id d191762f-2cba-49eb-aee3-b3fb64132ebd] [Tue Jul 28 15:02:55 2026] [dec0822] - Abort(1616527) (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(170).......:
MPID_Init(501)..............:
MPIDI_OFI_mpi_init_hook(580):
open_fabric(1559)...........: OFI fi_getinfo() failed (ofi_init.c:1559:open_fabric:No data available)

aborting job:
Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(170).......:
MPID_Init(501)..............:
MPIDI_OFI_mpi_init_hook(580):
open_fabric(1559)...........: OFI fi_getinfo() failed (ofi_init.c:1559:open_fabric:No data available)
MPICH ERROR [Rank 0] [job id d191762f-2cba-49eb-aee3-b3fb64132ebd] [Tue Jul 28 15:02:55 2026] [dec0822] - Abort(1616527) (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(170).......:
MPID_Init(501)..............:
MPIDI_OFI_mpi_init_hook(580):
open_fabric(1559)...........: OFI fi_getinfo() failed (ofi_init.c:1559:open_fabric:No data available)

This is the same error I had been seeing with oneapi RTs. However, not all oneapi tests fail. (For example, control_c48 passes.)