Use shell scripts generated from config_machines.xml - #303
Merged
Conversation
27 tasks
xylar
force-pushed
the
customize-generated-scripts
branch
from
October 21, 2025 11:02
969c899 to
db32c1a
Compare
...from get_modules_env_vars_and_mpi_compilers() and get_spack_script()
Almost all shell templates appear to be interchangable with their auto-generated counterparts from machine_configs.xml. Perlmutter CPU and GPU still need a little trick to set $NERSC_HOST if it isn't defined.
Before this fix, the autogenerated shell scripts were not getting rendered as jinja templates
We no longer need any config options to create the spack script.
xylar
force-pushed
the
customize-generated-scripts
branch
from
October 24, 2025 11:31
db32c1a to
d04af42
Compare
xylar
force-pushed
the
customize-generated-scripts
branch
from
October 24, 2025 11:42
7e5efa0 to
9b5a43f
Compare
Collaborator
Author
TestingThis branch has been used as part of #304 in deploying E3SM-Unified on various supported machines. Testing of spack dependency builds as well as packages such as MPAS-Analysis, zppy, e3sm_diags and zstash suggests that the auto-generated scripts are working as expected. |
Collaborator
Author
|
@andrewdnolan, as soon as Aurora deployment has succeeded, could you review this so we can work our way to a 2.0.0 release candidate? |
This was referenced Oct 24, 2025
andrewdnolan
approved these changes
Oct 27, 2025
andrewdnolan
left a comment
andrewdnolan
left a comment
Collaborator
There was a problem hiding this comment.
I've looked through and this all seems good to me.
Based on the testing as part of deploying e3sm-unified release candidates last week, I think this is ready to go.
Collaborator
Author
|
Thanks again, @andrewdnolan! |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rather than manually updating shell scripts for each machine, compiler and MPI library, we auto-generate them from config machines. The option remains to provide template scripts as before but these would be used to modify, rather than replace, the auto-generated version (with huge savings in maintenance burden). At the moment, this appears to only be needed for Perlmutter, where a small trick is needed to set
$NERSC_HOSTif it isn't defined.The unneeded template scripts have been removed.
The unused
yaml_templatearguments toget_spack_script()andget_modules_env_vars_and_mpi_compilers()have been removed. This is an API breaking change and will mean the next version should be 2.0.0 (not 1.33.0).The options
modules_beforeandmodules_afterin the[spack]section of the config files have been removed and are no longer used. We never get modules from the YAML templates anymore, instead always getting them as part of the auto-generated scripts fromconfig_machines.xml. The modules are always loaded after the spack environment. Three machines previously hasmodules_before = True: Compy, Dane and Ruby. I believe this was unused in Compy's case because the modules came from template scripts instead of the YAML files in any case. For Dane and Ruby, we will have to keep an eye on whether behavior changes unexpectedly in future E3SM-Unified testing.Checklist
Testingcomment, if appropriate, in the PR documents testing used to verify the changes