Optimizing Digital Marketing ROI: An A/B Testing Analysis of Facebook vs. AdWords Campaigns

Project Overview

This project conducts a comprehensive analysis of digital marketing campaigns run on two major platforms: Facebook and Google AdWords. The primary objective is to determine which platform provides a better return on investment (ROI) and to identify the key drivers of successful conversions for each. The analysis utilizes statistical A/B testing, Key Performance Indicator (KPI) evaluation, and predictive modeling to provide actionable, data-driven insights for optimizing future marketing spend.

---

Data Source

The analysis is based on the abdata.csv dataset, which contains daily performance metrics for campaigns conducted on both Facebook and AdWords.

Columns include:

• date_of_campaign
• facebook_ad_campaign, facebook_ad_views, facebook_ad_clicks, facebook_ad_conversions, facebook_cost_per_ad
• adword_ad_campaign, adword_ad_views, adword_ad_clicks, adword_ad_conversions, adword_cost_per_ad

---

Methodology

The analysis follows a structured, multi-step approach:

1. Data Cleaning and Feature Engineering

• The raw data was loaded, checked for duplicates and null values, and data types were corrected.
• New, essential marketing metrics were engineered from the base data to facilitate deeper analysis:
  • Click-Through Rate (CTR): clicks / views
  • Conversion Rate: conversions / clicks
  • Cost Per Click (CPC): cost / clicks
  • Cost Per Acquisition (CPA): cost / conversions

2. A/B Testing (Statistical Significance)

• An independent samples t-test was performed to compare the mean performance of Facebook and AdWords across the engineered metrics (CTR, Conversion Rate, CPC, CPA).
• This statistical test determines whether the observed differences between the two platforms are significant or simply due to random chance.

3. Key Performance Indicator (KPI) Analysis

• To align the analysis with business objectives, several financial KPIs were calculated by assuming an average order value of $50 per conversion:
  • Return on Ad Spend (ROAS): (Revenue - Cost) / Cost
  • Return on Investment (ROI)
• These KPIs were visualized to provide a clear comparison of the financial efficiency of each platform.

4. Predictive Modeling (Logistic Regression)

• To identify the factors that predict a high-conversion campaign, two separate logistic regression models were built—one for Facebook and one for AdWords.
• Target Variable: A binary converted flag (1 for high, 0 for low) was created based on whether the number of daily conversions was above or below the median for that platform. This balanced the classes for effective modeling.
• Feature Importance: The model coefficients were analyzed to understand which features (e.g., views, clicks, CPC) are the most influential predictors of a successful campaign on each platform.
• Important Note: Features that would cause target leakage (like conversion_rate and cpa, which are directly calculated from the target conversions) were correctly excluded from the model to ensure realistic and predictive results.

---

Key Findings & Business Insights

A/B Testing Results

• CTR & Conversion Rate: Facebook showed a statistically significant higher performance in both Click-Through Rate and Conversion Rate. This indicates that Facebook ads were more effective at both capturing user attention and driving them to convert.
• CPC & CPA: AdWords was the statistically significant winner for cost-efficiency, demonstrating a lower Cost Per Click and Cost Per Acquisition.
• Overall Performance: Despite higher individual costs, Facebook's superior conversion ability resulted in a significantly lower overall Cost Per Acquisition (CPA), making it the more efficient platform for acquiring customers.

KPI & ROI Insights

• The KPI analysis confirmed the findings from the A/B test. Facebook delivered a significantly higher Return on Ad Spend (ROAS) and overall ROI, driven by its strong conversion performance.
• Conclusion: For the primary goal of driving revenue and maximizing return, Facebook is the superior platform. While AdWords clicks are cheaper, they do not convert as effectively.

Predictive Modeling Insights

• After correcting for target leakage, the logistic regression models provided realistic insights into conversion drivers:
  • For Facebook: The most influential feature predicting a successful campaign was a low Cost Per Click (CPC). This suggests that while overall ad spend is necessary, efficient, low-cost clicks are a key indicator of high-converting campaigns on Facebook.
  • For AdWords: Cost Per Click (CPC) and Click-Through Rate (CTR) were the most important predictors.

---

Visualizations Included

The notebook contains several visualizations to support the analysis:

• Time-Series Plot: Comparing the CTR of both platforms over time.
• Boxplots: Showing the distribution and statistical differences for each core metric.
• Bar Charts:
  • A KPI dashboard comparing ROAS, CPA, and ROI.
  • A marketing funnel comparison visualizing the drop-off rate at each stage.
• Feature Importance Plots: Highlighting the logistic regression coefficients for each platform.

---

How to Run the Project

1. Dependencies: Ensure you have Python installed with the following libraries:
   • pandas
   • numpy
   • matplotlib
   • seaborn
   • scipy
   • scikit-learn
2. Dataset: Place the abdata.csv file in the same directory as the Jupyter Notebook.
3. Execution: Open ab_testing.ipynb in a Jupyter environment and run the cells sequentially.