Simplifying config parameters

[Mwiche96]

If appropriate, go to the Preview tab and select the appropriate sub-template:

• Data Integration
• Prepare Release

Otherwise, stick with the template below. And remove this text from the PR.

---

Closes #351

<--Fill in the number of the issue this PR address. This will automatically close the referenced issue
when this PR is merged -->

Changes proposed in this Pull Request

There are several parameters that are duplicated from the project base config which is the validation_dispatch_zambia.yaml and can be safely removed to aid in userbility. there is also an orphaned file config.zm.default.yaml which is not utilised anywhere in the workflow that can be removed also

Checklist

<--This checklist must be filled in. If the item is not applicable, tick anyway.-->

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented, including updates to docstrings for meaningful functions.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/user-guide/configuration.md and doc/tutorials/electricity-model.md.
• If config sections were added, renamed, or removed, update doc/assets/scripts/extract_config_snippets.py accordingly.
• Archives of the uploaded data do not have an enclosing folder and archive names correspond to the conventions of configs/bundle_config.yaml.
• A note for the release notes doc/release-notes.md is amended in the format of previous release notes, including reference to the requested PR.

[Mwiche96]

Within the scope of this PR is remove duplicated parameters that don't change from our project base config file

[ekatef]

Thank you @Mwiche96. Completely agree that our configs need tidying-up, and great to have an initial implementation sketch.

I think it's essential that we'll also sketch some architecture concept. To my understanding, there are the following questions to address.

1. Which use cases do we aim to cover with the config files? Could be good to list all versions of the workflow we currently have and think what are differences between them in terms of the relevant config parameters
2. Which naming conventions do we want to assume? E.g. config.{ }.yaml is used in the upstream as a pattern which makes it easier to distinguish between workflow-relevant configs and other yaml files
3. What is out ideas on organising the config files: where to keep what? We have configs, configs/scenarios, configs/scenarios_zambiandtest` folders and it would be good to agree on some strategy on how we are using them
4. What is a handy way to document our config management system?

[Mwiche96]

Proposals

1. The project will have its own default config similar to what config.default.yaml is in the repo. It shall be called config.zm.default.yaml
   this will serve as the first layer on top of the default repo config, all the follow up configs will build on top of this config
2. in terms of naming, the files shall follow the pattern config.zm.{}.yaml. this will follow the pattern upstream but also to ensure that there is a clear indication of Zambia spacific config files
3. Ideally, configs layer on top of one another. There are parameters that are the same as the default parameters in the layer above but only differ by values i.e base_voltage: 330 in project config vs base_voltage: 380. in the default config. Then there are instances where parameters are specific only to the project config and not the default repo config such as disaggregate_powerplants: false. For the first case, we need to ensure that we copy and paste the block of parameters that applies to what we need into the new config file we are creating (deleting the irrelevant parts of the block) and then proceed to change the values contained in the parameters in question. This will ensure ordering and visual consistency in the configs and hint towards similar set of parameters regardless of the config they exist in.
4. In terms of organising the configs, it is important to bare in mind the end user who in this case is ZESCO. While they will get support for the duration of the partnership, we can admit that this support may vary as resources shift over time. As such we can assume an approach that is already intuitively applied within PyPSA itself. We can have a top level zambia folder that will house all the configs related to Zambia. This neatly puts everything in an easy to reach manner.

	- Zambia configs
		- config.zm.default.yaml
		- config.zm.validation_dispatch.yaml
		- config.zm.capacity_expansion_base.yaml
		- scenarios
			- config.cap_exp_2025
			- config.cap_exp_2030

Alternatively we can maintain the currently organisation of the configs which is already familiar with the team and which still works given the naming conventions we would adopt for the configs.
5. Once adopted, we can place this methodology into the documentation to allow Zesco and us to reference it as we develop.

@ekatef and @tacwebservices what do you think about this logic?

[ekatef]

Thank you @Mwiche96, the concept sounds well-elaborated and perfectly relevant to me.

Adding also an insight we have obtained in our discussion for point four: having regional-specific configs placed grouped helps to avoid merge conflicts and facilitates maintenance. On that, I can also add the test configs may deserve some additional attention since they can create quite some issues when merging upstream.

It would be good to have an opinion by @tacwebservices since it's a point which has high relevance for modelling runs.

[ekatef]

Hey @Mwiche96, as discussed the implementation looks consistent and well-structured. I have gone trough the changes with a more in-depth review and can confirm that they look perfectly relevant for our task.

The only point I have noticed is gihtub being not great in showcasing this kind of changes. So, I'm adding here a list of improvements this PR brings in and some comments on them following-up our discussion.

1. All the files related to Zambia modelling have been moved into zambia_configs folder, including scenarios_zambia folder.
2. Agree that build_cutout_... configs must be absolutely renamed as well to match the overall pattern we have.
3. I see that you have switched-off the retrieval flags in the default config, and I think it's a better approach as compared with what we have in the upstream default right now.
4. It makes perfect sense having different weather years in the default and the dispatch validation configs.

Points 3 and 4 would have some implications for training since we'll need to remember to apply a dedicated config over our config structure to adjust parameters of a specific run with it's goal. But I think that the consistent structure of the configs will make things easier in any case.

[Mwiche96]

@ekatef thank you very much for the review. Yes on the retreival sections, I figured it was easier to have them sitting in one place as opposed to having them on a per config basis.