Road Safety Data Pipeline for New Zealand

Overview

This project implements an end-to-end ETL pipeline to process New Zealand road crash data using PySpark and Google Cloud Platform.

Tech Stack

• Python
• PySpark
• Apache Beam
• Google Cloud Storage
• BigQuery
• Dataflow
• Cloud Composer
• SQL
• Looker Studio

Features

• Automated data ingestion
• ETL pipeline
• Data cleaning and transformation
• Data loading into BigQuery
• Dashboard for crash analysis

Project Structure

road-safety-data-pipeline/ │── Bigquery/ │── beam_PipelineDataFlow.py │── gcsToPubSubDag.py │── readFromPubSub.py │── upload_csv_periodic.py

Architecture

CSV Dataset │ ▼ Google Cloud Storage │ ▼ Pub/Sub │ ▼ Apache Beam / Dataflow │ ▼ BigQuery │ ▼ Looker Studio Dashboard

Dataset

New Zealand Road Crash Dataset containing information such as:

• Crash severity
• Fatalities
• Injuries
• Road conditions
• Vehicle details
• Location data

Team Project

This project was developed collaboratively as part of the Revature Data Engineering training program.

My Contributions:

• Developed ETL pipeline components.
• Worked with Google Cloud Storage and BigQuery.
• Implemented data ingestion and processing scripts.