Support new versioning scheme#137
Conversation
…y file. Update GLOWS L3b to handle new major.minor version format.
…s minor version only for compatibility
# Conflicts: # tests/periodically_run_tests.json
pleasant-menlo
changed the title
|
@bryan-harter @tech3371 @vineetbansal , can one of you take a look at this? |
…me to use VersionMap with a version per output descriptor.
…me to use VersionMap for GLOWS processing L3b, L3c, and L3d
|
Note we probably shouldn't merge this until we have all the metadata updates for release 1 done, so we can still build new containers against those. |
…s per product descriptor for GLOWS L3bcde
# Conflicts: # pyproject.toml # tests/glows/l3e/test_glows_l3e_utils.py # tests/periodically_run_tests.json # uv.lock
| "astropy_healpix==1.1.2", | ||
| "bitstring==4.3.0", | ||
| "imap-data-access==0.42.0.dev0", | ||
| "imap-data-access>=0.42.0.dev0", |
There was a problem hiding this comment.
0.42.0.dev1 is breaking backwards compatibility for versions. We could leave this at ==0.42.0.dev0 for the time being until 0.42.0.dev2 is released.
| l3d_initializer_result = GlowsL3DInitializer.should_process_l3d(l3bc_initializer_data.external_dependencies, | ||
| l3bs, l3cs) | ||
| l3bs, l3cs, l3d_major_version) | ||
| if l3d_initializer_result is None: |
There was a problem hiding this comment.
Can this early return just go away? If L3e needs to be regenerated due to a new major version, that should not be gated behind updates or a new major version of L3d
| if not latest_l3d_cr: | ||
| first_updated_cr = find_first_updated_cr(l3d_output.l3d_cdf_file_path, previous_l3d) | ||
| if first_updated_cr is None: | ||
| return None |
There was a problem hiding this comment.
This early return prevents us from generating any L3e if L3d has not changed. We may still want to process if the L3e major version has changed
…are no changes between the output L3d and the existing L3d
2 participants
Change Summary
Overview
Implemented handling of major version in IMAP L3 processing including the jobs that manage their own version numbers (cron jobs for GLOWS and maps).
The dependency JSON can now contain additional information about the requested version:
We currently assume that version information will only be provided for a single descriptor, because our existing dependency configuration only specifies a single output for each job. This may need to be revisited if the dependency system later expects several outputs for a single L3 job.
The parallel change to imap_processing was made in IMAP-Science-Operations-Center/imap_processing#3313.
See IMAP-Science-Operations-Center/sds-data-manager#1397 for the top-level issue.
Closes #118.
File changes
Version objects are now created from the dependency JSON when they contain version numbers and otherwise fall back to creating version objects from the command line arguments.
Testing
Invoking any job from the command line using a dependency JSON that includes the version numbers should result in an output with the corresponding major and minor version numbers.