Cost estimation not working with new version of Cromwell (v87)

[malachig]

Running the following command no longer works for immuno.wdl workflow runs that were performed with v87 of Cromwell. Previous testing was primarily with v71.

To reproduce the error:

cd /storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pancreas_leidos/analysis/TWJF-5120-39/gcp_immuno/final_results/workflow_artifacts/costs

export WORKFLOW_ID=b033e4be-8d5a-4bc1-9f6b-efa9387bd255

bsub -Is -q general-interactive -G $GROUP -a "docker(mgibio/cloudize-workflow:latest)" /bin/bash

python3 $WORKING_BASE/git/cloud-workflows/scripts/estimate_billing.py $WORKFLOW_ID $GCS_BUCKET_PATH/workflow_artifacts/$WORKFLOW_ID/artifacts/metadata/ > costs.json

The error encountered:

Traceback (most recent call last):
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pancreas_leidos/analysis/TWJF-5120-39/gcp_immuno/git/cloud-workflows/scripts/estimate_billing.py", line 265, in <module>
    cost = cost_workflow(args.metadata_dir.rstrip('/'), args.workflow_id)
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pancreas_leidos/analysis/TWJF-5120-39/gcp_immuno/git/cloud-workflows/scripts/estimate_billing.py", line 231, in cost_workflow
    call_costs_by_name[ck] = cost_workflow(location, call["subWorkflowId"])
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pancreas_leidos/analysis/TWJF-5120-39/gcp_immuno/git/cloud-workflows/scripts/estimate_billing.py", line 231, in cost_workflow
    call_costs_by_name[ck] = cost_workflow(location, call["subWorkflowId"])
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pancreas_leidos/analysis/TWJF-5120-39/gcp_immuno/git/cloud-workflows/scripts/estimate_billing.py", line 227, in cost_workflow
    call_costs_by_name[ck] = cost_task(call)
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pancreas_leidos/analysis/TWJF-5120-39/gcp_immuno/git/cloud-workflows/scripts/estimate_billing.py", line 151, in cost_task
    duration = from_iso(end_time) - from_iso(start_time)
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pancreas_leidos/analysis/TWJF-5120-39/gcp_immuno/git/cloud-workflows/scripts/estimate_billing.py", line 79, in from_iso
    return datetime.fromisoformat(datetime_str.rstrip('Z'))
AttributeError: type object 'datetime.datetime' has no attribute 'fromisoformat'

The .json files being parsed do seem to generally have the expected data, but perhaps the structure of this output has changed with Cromwell versions and is no longer meeting the expectations of estimate_billing.py

[malachig]

The existing PR from @ldhtnp may actually already address this?

#41

[malachig]

The above error may have actually just resulted from attempting to run without being in the cloudize docker environment?

[malachig]

Note that Cromwell release v89 is planning internal support for cost estimation

Cromwell can now provide estimated costs incurred by a workflow run on GCP, read more in CostEstimation in ReadTheDocs.

https://github.com/broadinstitute/cromwell/blob/develop/CHANGELOG.md

[malachig]

The cost estimation script from @ldhtnp seems to be working on GCP batch / Cromwell v88 results:
https://github.com/wustl-oncology/cloud-workflows/blob/main/scripts/gb_estimate_billing.py

[malachig]

With Cromwell v88 I have had success using gb_estimate_billing.py for some runs. But for other runs I have seen errors like this:

[ERROR] sequenceToTrimmedFastq.trimFastq has not completed running, cost cannot be calculated
Traceback (most recent call last):
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pipeline_tests/hcc1395/immuno_releases/dev3/gcp_immuno/git/cloud-workflows/scripts/gb_estimate_billing.py", line 260, in <module>
    cost = get_workflow_cost(args.metadata_dir.rstrip('/'), args.workflow_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pipeline_tests/hcc1395/immuno_releases/dev3/gcp_immuno/git/cloud-workflows/scripts/gb_estimate_billing.py", line 225, in get_workflow_cost
    call_costs[ck] = get_workflow_cost(metadata_dir, call[SUBWORKFLOW_KEY])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pipeline_tests/hcc1395/immuno_releases/dev3/gcp_immuno/git/cloud-workflows/scripts/gb_estimate_billing.py", line 225, in get_workflow_cost
    call_costs[ck] = get_workflow_cost(metadata_dir, call[SUBWORKFLOW_KEY])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pipeline_tests/hcc1395/immuno_releases/dev3/gcp_immuno/git/cloud-workflows/scripts/gb_estimate_billing.py", line 222, in get_workflow_cost
    if is_cached_task(call):
       ^^^^^^^^^^^^^^^^^^^^
  File "/storage1/fs1/gillandersw/Active/Project_0001_Clinical_Trials/pipeline_tests/hcc1395/immuno_releases/dev3/gcp_immuno/git/cloud-workflows/scripts/gb_estimate_billing.py", line 73, in is_cached_task
    return call[CACHED_KEY]["hit"] == True
           ~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'hit'

This example was from a seemingly clean run, where the sequenceToTrimmedFastq.trimFastq did succeed and the whole workflow was successful on the first attempt (no rerun or call caching involved).