RELEASE_NOTES.md

Release notes for EESSI test suite

v1.0.0 (29 July 2026)

This is a major release of the EESSI test-suite

Bug fixes:

• Use command line args for mpirun to do process binding, as environment variables are ignored for some newer OpenMPI versions (#305).
• Some tests that should have run on 2 GPUs on the same node, would start running on 2 GPUs on different nodes (#291). This was fixed by setting the use_nodes_option scheduler option in the (common) ReFrame config (#310).
• Factor out common configuration, to avoid having to duplicate this in every ReFrame config file for eveyr system (#310).
• Fix issue where marking all files in the source dir as read-only caused issues from ReFrame 4.9.0 onwards (#318).
• Fix issue where importing pkg_resources fails on Python >= 3.12 (#334).
• Fix CI failure due to hpctestlib no longer being shipped in ReFrame >= 4.10 (#341)

Enhancements:

• Replace bench_name_ci with the more intuitive is_ci_test property (#301).
• Only do hard checking of required_mem_per_node and readonly_files in CI, but demote to a warning by default (#302).
• Add a check on process binding, to verify processes are bound correctly (#312, #338).
• Added used_cpus_per_task test property, which may be different from number of CPUs the test will request from the scheduler (test.num_cpus_per_task) (#322, #336). Useful to make sure e.g. OSU tests will bind to individual cores instead of a larger set of cores (#321).
• Use ReFrame programming environments to more easily test modules accross all EESSI versions, with a single ReFrame run (#326, #345).
• Migrated MetalWalls to use the EESSI_Mixin class (#332).

Other changes:

• Added Reframe configuration file for the HCA experimental RISC-V cluster at BSC (#314).
• Replace deprecated RegressionMixin with RegressionTestPlugin (#315).
• Update ReFrame configuration for Hortense (#316).
• Reduce verbosity on ReframeFatalErrors by not showing the traceback by default, but only when setting TRACEBACK=1 in your enviromment (#319).
• Update link that is reported when CPU autodetection fails (#324).
• Update ReFrame configuration for AWS Magic Castle (#327).
• Copy ReFrame's hpctestlib files into the EESSI test suite, since hpctestlib is no longer supported in ReFrame (#328).

v0.9.0 (23 December 2025)

This is a minor release of the EESSI test-suite

New tests:

• OpenFoam: added a large test case, that can run on larger scales (#281)
• ESPResSo: added a lattice-Boltzman (LB) test case, updated P3M and LJ test cases for ESPResSo 5.0 release (#300). New test cases also test waLBerla integration in ESPResSo.
• LPC3D: test case simulates NMR spectra and diffusion of a liquid within a disorder porous carbon, based on a mesoscopic model (#306).
• lmbpy-pssrt: test case for a fork of lbmpy that adds population-based methods (#308).
• waLBerla: test case for a stand-alone waLBerla installation. Runs a slightly modified version of one of waLBerla's lbm tutorials (#309).
• LAMMPS: added tests to test integrated LAMMPS + OBMD (open-boundary molecular dynamics) and LAMMPS + ALL (A Load balancing Library), and LAMMPS + ALL + OBMD functionality (#303).

v0.8.1 (6 November 2025)

This is a bug fix release of the EESSI test-suite

Bug fixes:

• Adding init.py to fix the docs. (#293)
• Add possibility for tests to use predownloaded data, and otherwise mark themselves as requiring an online node. Also: implemented this for TensorFlow (#275)
• Add Mare Nostrum ReFrame config (#277)

Other changes:

• Changes in Snellius config specifying the exact number of nodes and not leaving it up for interpretation by SLURM. (#292)
• Added CI check for being able to autogenerate the docs (#295)

v0.8.0 (5 November 2025)

This is a minor release of the EESSI test-suite

New tests:

• BLAS (gemm, hemm, herk, trmm, trsm) (#268)
• OpenFOAM CI (small scale, 1-128 MPI tasks) (#278)
• numpy (Dot product, SVD, Cholesky decomposition, Inversion, Eigen decomposition) (#276)

Bug fixes:

• Skip tests that require internet on offline machines. (#279)
• Fix output of full module path. (#274)

Enhancements:

• Print list of supported scales for each test class with non-standard scales or a filtered set. (#285)
• Set compact thread-binding for numpy test. (#286)
• Added a launcher variable for test classes rather than using a separate function in each test to use a non-standard launcher. (#286)
• BLAS test uses the launcher variable. (#288)

Other changes:

• Changes in VSC config. (#282)
• Fixing a typo. (#283)

v0.7.1 (30 June 2025)

This is a bug fix release of the EESSI test-suite

Bug fixes:

• Make sure automatic API docs generation also generates docs for OpenFOAM (#271)

Other changes:

• Update CI workflow to use ReFrame 4.8.1 by default, and use new argument passing functionality to pass the SLURM account (#270)

v0.7.0 (17 June 2025)

This is a minor release of the EESSI test-suite

New tests:

• OpenFOAM (#243)

Bug fixes:

• Fixes syntax issue in docstring that blocks automatic API doc generation (#265)
• Update VSC Tier-1 Hortense configuration file and CI script to fix problems (#250, #266)

Other changes:

• Set SLURM_EXPORT_ENV=ALLinprepare_cmds` of example config (#262)

v0.6.0 (19 March 2025)

This is a minor release of the EESSI test-suite.

WARNING: this release contains a breaking change (#258) for the format of the required ReFrame configuration file. Essentially, this requires making the following substitutions in the ReFrame configuration file.

• FEATURES[CPU] => FEATURES.CPU
• FEATURES[GPU] => FEATURES.GPU
• 'mem_per_node' => EXTRAS.MEM_PER_NODE
• DEVICE_TYPES[CPU] => DEVICE_TYPES.CPU
• DEVICE_TYPES[GPU] => DEVICE_TYPES.GPU
• GPU_VENDOR: GPU_VENDORS[NVIDIA] => EXTRAS.GPU_VENDOR: GPU_VENDORS.NVIDIA
• FEATURES[ALWAYS_REQUEST_GPUS] => FEATURES.ALWAYS_REQUEST_GPUS

See the updated documentation for more details.

Updates:

• Use NamedTuples to define constants instead of constants indexing a dictionary of constants #258

v0.5.2 (13 March 2025)

This is a bug fix release of the EESSI test-suite

Bug fix:

• Add exact_memory attribute to tests that are not inheriting from EESSI_Mixin to avoid '' object has no attribute 'exact_memory' failure #245, fixes #252

Other updates:

• Prefix mixin method names to avoid accidental overrides #246
• Use CI constant (instead of 'CI' as string) #247
• Ported ESPResSo to EESSI_Mixin #248
• Simplify executable_opts with new variable user_executable_opts #249
• Update example_settings.py config file #254

v0.5.1 (30 January 2025)

This is a bug fix release of the EESSI test-suite

Other updates:

• Update to release workflow instructions (#232)
• Document how to use a different version of the config files when using CI/run_reframe.sh (#235)
• Bump the use of the default ReFrame version for period tests that use CI/run_reframe.sh to ReFrame 4.7.3 (#236)
• Make CI/run_reframe.sh use the latest release of the test suite as default version (#237)

v0.5.0 (23 January 2025)

This is a minor release of the EESSI test-suite

New tests:

• MetalWalls (#164) (N.B. requires ReFrame >=4.7.0)

New features:

• Created eessi_mixin class for tests to inherit from. This class makes it easier to create portable tests, and also keeps the structure of those tests more uniform. (#177)
• Automatic versioning with setuptools_scm (#185)
• Ported tests to use the eessi_mixin class: CP2K (#193), MPI4py (#203), QuantumESPRESSO (#212), PyTorch (#213), TensorFlow (#221), OSU (#222)
• Improved support for doing periodic test suite runs on a local software stack using CI/run_reframe.sh (#200)
• Support for setting the exact required memory (#214)
• For periodic tests using CI/run_reframe.sh, allow using a different version of the test-suite repository for the ReFrame config files (#226)

Bug fixes:

• Fixed failing mktemp command in CI/run_reframe.sh when EESSI_CI_TEMPROOT isn't set (#227)
• Make tests that don't use eessi_mixin always request GPUs if this is specified in the test.

Other updates:

• Update lammps hook name to be more descriptive (#177)
• Add empty init files to allow auto-generating API docs (#192)
• Updates to ReFrame configs (#195, #198, #199, #202, #215, #216, #223, #225)
• Log if the CI tag gets set (#204)
• Reduce the amount of data that gets copied during ReFrame staging (#205, #220)
• Log CVMFS_REPO_NAME, CVMFS_SOFTWARE_SUBDIR, MODULEPATH and EESSI test-suite version (#206)
• Clarify error messages from EESSI mixin that warn test developers if certain variables aren't set on time in the child class (#208)
• For periodic tests using CI/run_reframe.sh, avoid cloning the full history (#218)
• Update release instructions (#230)

v0.4.0 (26 September 2024)

This is a minor release of the EESSI test-suite

New tests:

• PyTorch (with torchvision) (#130)
• LAMMPS (#131, #169)
• CP2K (#133)

Bug fixes:

• Find and report duplicate modules (#167)
• Specify memory as an extra resource in all ReFrame configs (#180)
• Make req_memory_per_node hook print a warning if memory resource is not configured in ReFrame config (#182)
• No longer test import of eessi.testsuite.utils directly in CI, as it fails (by design) for the ReFrame 4.6.2 version deployed in EESSI (#183)

Other updates:

• mpi4py reduction test for tutorial (#149, #172), to accompany documentation on writing portable tests for the EESSI test suite
• Various updates to Snellius ReFrame config (#165, #168, #187, #188)
• Make the use of EESSI optional in common_eessi_init, to better support testing of local software stacks (#166)
• Remove hard-coded SLURM account from karolina and hortense configs in favour of using the $SBATCH_ACCOUNT environment variable (#171)
• Add ReFrame config and CI config for EESSI Azure build cluster (#173, #176, #186)
• Add hook for measuring memory usage in a job (#174)
• Bump default ReFrame version used in CI config to 4.6.2 (#175)
• Use ReFrame warning's to print warnings, amking them easier to spot (#178)
• Limit runtime of ReFrame command in in CI config (#179)
• Add release notes and bump versions (#189)

v0.3.2 (29 June 2024)

This is a bugfix release of the EESSI test-suite

It includes:

• Add config for Deucalion (#162)

v0.3.1 (28 June 2024)

This is a bugfix release of the EESSI test-suite

It includes:

• Correct required memory per node to MiB in ESPResSo test (#158)
• Change behavior for assign_tasks_per_compute_unit(test, COMPUTE_UNIT[CPU]) on hyperthreading-enabled systems (#160)
• Use compact process binding in ESPResSo test (#160)

v0.3.0 (27 June 2024)

This is a minor release of the EESSI test-suite

It includes:

• Update config AWS MC cluster to use software.eessi.io (#126)
• Add test for QuantumESPRESSO (pw.x) (#128)
• Fix compact process binding for OpenMPI mpirun (#137)
• Use compact process binding for GROMACS (#139)
• Rename scale tags 1_cpn_2_nodes and 1_cpn_4_nodes (#140)
• Set SRUN_CPUS_PER_TASK for srun launcher (#141)
• Fix for "Failed to modify UD QP to INIT on mlx5_0" on Karolina CI runs (#142)
• Reduce the iteration count to make the OSU tests run faster, especially on slower interconnects (#143)
• Add test for ESPResSo (P3M) (#144)
• Use software.eessi.io repo in CI (#146)
• Add notes on release management to README (#148)
• Fix memory_per_node for Hortense (#151)
• Use MiB units for memory per node (#152)
• Added / updated memory for various systems in MiB units (#153)
• Add additional test for ESPRESSO (LJ) (#155)
• Bump default version used in CI (#157)

v0.2.0 (7 March 2024)

This is a minor release of the EESSI test-suite

It includes:

• Implement the CI for regular runs on a system (#93)
• Add OSU tests and update the hooks and configs to make the tests portable (#54, #95, #96, #97, #110, #116, #117, #118, #121)
• Add extra scales to filter tests(#94)
• add new hook to filter out invalid scales based on features in the config (#111)
• unify test names (#108)
• updates to CI workflow ((#102, #103, #104, #105)
• Update common_config (#114)
• Add common config item to redirect the report file to the same directory as e.g. the perflog (#122)
• Fix code formatting + enforce it in CI workflow (#120)

Bug fixes:

• Fix hook _assign_num_tasks_per_node (#98)
• fix import common-config vsc_hortense (#99)
• fix typo in partition names in configuration file for vsc_hortense (#106)

v0.1.0 (5 October 2023)

This is the first release of the EESSI test suite.

It includes:

• A well-structured eessi.testsuite Python package that provides constants, utilities, hooks, and tests, which can be installed with "pip install".
• Tests for GROMACS and TensorFlow in eessi.testsuite.tests.apps that leverage the functionality provided by eessi.testsuite.*.
• Examples of ReFrame configuration files for various systems in the config subdirectory.
• A common_logging_config() function to facilitate the ReFrame logging configuration.
• A set of standard device types and features that can be used in the partitions section of the ReFrame configuration file.
• A set of tags (CI + scale) that can be used to filter checks.
• Scripts that show how to run the test suite.

For documentation, see https://eessi.io/docs/test-suite .