🌞 DWD Data Parser Suite

A lightweight and modular Python toolkit to download and parse open meteorological datasets from the Deutscher Wetterdienst (DWD) Climate Data Center.

This repository was developed as a side project to support Finn Nerlich’s Master's thesis:

"Efficiency Analysis of a Photovoltaic System – Development and Application of a Measurement Method for Determining the Efficiency of PV-Systems under Real Conditions" University of Bremen, 2025.

---

✨ Project Goals

This project was built to simplify access to DWD datasets for research purposes and make them easier to process and analyze and keep them up to date. It provides:

• 📂 Automated data download via a recursive web scraper
• 🛠 Custom Parser Definition using the config (irradiance, solar, temperature, …)
• 🧩 JSON-based configuration for easy scaling to new datasets
• 🔍 Focus on clarity and simplicity rather than long-term feature development

⚠️ Note: This repository is archived and has no ongoing development goals. However, community contributions are welcome—especially new or generalized parsers for the different dataset types.

---

📦 Features

• 🔽 Download entire dataset directories from Open Data Server of the German Meteorological Service (DWD)
• 📝 Concatenate raw text data into structured CSV & Feather files
• 🧮 Apply data cleaning & timestamp corrections (WOZ/MEZ logic) using custom parsers
• 🔗 Simple and scriptable for reproducible research workflows

⚠️ Note: This repository is written as flexible as possible while keeping the code simple. The structure is mainly focused on the datasets used in the underlying research (irradiance, temperature, solar). Due to the variety of dataset formats, the current workflow approach is may not fully applicable for all DWD datasets.

---

🚀 Quickstart

1. Clone the repository

git clone https://github.com/username/dwd-data-parser.git
cd dwd-data-parser

2. Install dependencies

pip install -r requirements.txt

3. Run all parsers

python main.py

This will:

1. Recursively download all datasets listed in dataset_conf.json
2. Concatenate raw text data into .csv and .feather files
3. Run dataset-specific parsers (if configured)

---

🔧 Configuration

Datasets are defined in dataset_conf.json. Example:

{
  "radiation": {
    "url": "https://opendata.dwd.de/climate_environment/CDC/derived_germany/climate/hourly/duett/",
    "raw_path_pattern": "duett/radiation_global/recent/stundenwerte_duett_FG_*/produkt_fg_duett*",
    "output_csv": "radiation_all.csv",
    "output_feather": "radiation_all.feather",
    "parser": "datasets.radiation_parser.parse",
    "seperator": ";",
    "keep_source": true
  }
}

---

🧩 Writing Your Own Parser

To add a new dataset:

1. Write a my_dataset_parser.py in datasets/
2. Implement a parse() function
3. Add a new block in dataset_conf.json

Example parser snippet:

def parse():
    data = pd.read_csv("my_dataset.csv", sep=";")
    # Clean, enrich, or transform
    data.to_feather("my_dataset_parsed.feather")
    return data

---

📖 Citation

Bibtex:

@mastersthesis{nerlich2025pv,
  author = {Finn Nerlich},
  title = {Efficiency Analysis of a Photovoltaic System – Development and Application of a Measurement Method for Determining the Efficiency of PV-Systems under Real Conditions},
  school = {University of Bremen},
  year = {2025}
}

---

🤝 Contributing

This project is archived and has no roadmap for ongoing development. However:

• ✅ Bug fixes, refactoring, or generalized parsers are welcome
• 📝 Open a pull request and describe your contribution
• 🗂 I’ll merge useful changes to make this repository a better resource
• 📥 Custom Parsers are especially welcome for other DWD datasets

---

📜 License

This project is licensed under the GPL v3 – see the LICENSE file for details.

---

❤️ Acknowledgements

• DWD Climate Data Center for free and open weather datasets