basic-stats-u2-concerts.qmd

---
title: "Basic Statistics U2 Concerts"
author: "Hans van Leeuwen"
---

In this section we look at some basic statistics of the U2 shows and compare this with the statistics listed at <https://www.u2gigs.com>.

## Load custom concertData package

First we load my custom concertData package where I organized all the functions to analyze the U2 concert data. This R package is available on my GitHub page at: <https://github.com/vanleeuwen-hans/concertData>.

```{r load custom concertData package, message = FALSE, warning = FALSE}
library(devtools)
devtools::install_github("vanleeuwen-hans/concertData")
library(concertData)
```

```{r basic stats - read data, message = FALSE, warning = FALSE}

# read the following file where the Rosemont duplication was not fixed yet
# this is explained further down on this report
u2data <- read_concertData_csv('u2data/archive/u2data_all_shows_clean_final_rosemontdup.csv')
```

## Longest set lists

List the shows with the longest set lists, i.e. where U2 played most songs in a concert. The list corresponds to the statistics at <https://www.u2gigs.com/tourdb-stats.html>.

```{r basic stats - longest set lists, message = FALSE, warning = FALSE}

# longest set lists
longest_setlists <- find_longest_setlists(u2data)
print(longest_setlists)
```

The info at <https://www.u2gigs.com/tourdb-stats.html>:

![Longest U2 set lists - source u2gigs.com - October 2024.](u2gigs/u2_longest_setlists-u2gigscom.png)

One of these shows, with 27 songs played, was in Cologne on October 18th, 2015. I was there with my wife, and before the show we had the chance to meet and greet with Adam Clayton, U2's bass player. He signed my copy of the 1991 Achtung Baby album. I have included some pictures of that moment below.

![Meeting Adam Clayton in Cologne - source Hans van Leeuwen - 18 October 2015.](u2hvl/AdamClaytonCologne.JPG){width="208"} ![Meeting Adam Clayton in Cologne - source Hans van Leeuwen - 18 October 2015.](u2hvl/HvLmeetandgreetAdamClayton.JPG){width="246"} ![Meeting Adam Clayton in Cologne - source Hans van Leeuwen - 18 October 2015.](u2hvl/AchtungBaby_signed.JPG){width="162"}

## Shows with most snippets

A snippet is a short piece of song (either from another artist or from U2 themselves) played as part of a regular song in the show.

```{r basic stats - shows with most snippets, message = FALSE, warning = FALSE}

# Get shows with most snippets
snippet_summary <- find_shows_with_most_snippets(u2data)
head(snippet_summary, n=10)  # Show top 10 shows

```

The info at <https://www.u2gigs.com/tourdb-stats.html>:

![U2 shows with most snippets played - source u2gigs.com - October 2024.](u2gigs/u2_shows_most_snippets-u2gigscom.png)

I noticed a discrepancy in number of snippets for some of these shows when comparing with the statistics at <https://www.u2gigs.com/tourdb-stats.html>, for example the London show on November 3rd, 2015, where my data calculates the number of snippets played as 20, while the info at <https://www.u2gigs.com/show1957.html> states 19. I first checked in my data which song was incorrectly identified as snippet. That turned out to be 'Even Better Than The Real Thing'.

```{r basic stats - check snippet discrepancy, message = FALSE, warning = FALSE}

# check london show 2015-11-03
london_show <- subset(u2data, date == '2015-11-03' & snippet == TRUE, select = c("showID", "date","song_title", "snippet"))
print(london_show, n = max())
```

The issue seems to be in the different HTML coding for one snippet on that page: ![London snippet count - source u2gigs.com - 3 November 2015.](u2gigs/london_snippet_count_u2gigscom.png)

If we check the HTML code for the 'Fish Out of Water Mix' we see that it has a 'snippet' tag:

```         
<li><a href="/Even_Better_Than_The_Real_Thing-s38.html">Even Better Than The Real Thing</a> <em class="snippet">Fish Out of Water Mix</em></li>
```

Now compare this to what the HTML looks like normally for snippets at u2gigs.com:

```         
<li><a href="/Mysterious_Ways-s26.html">Mysterious Ways</a> / <a class="snippet" href="/Burning_Down_The_House-s878.html">Burning Down The House</a> <span class="snippet">(snippet)</span> / <a class="snippet" href="/Young_Americans-s30.html">Young Americans</a> <span class="snippet">(snippet)</span></li>
```

I'm not sure what u2gigs.com intended to do here, and whether this is a snippet or not. It does explain the different numbers seen earlier as my scraping code has counted songs preceding songs with those tags as snippet incorrectly.

## Cities with most shows

We used this metric for data curation already, but here is the top 11 (instead of Top 10 so that Amsterdam is displayed 😀 ) of cities where U2 played most often. This matches exactly the statistics listed at <https://www.u2gigs.com/tourdb-stats.html>.

```{r basic stats - cities with most shows, message = FALSE, warning = FALSE}

# cities with most shows played
top_cities <- find_most_played_cities(u2data, n = 13)
print(top_cities)
```

The info at <https://www.u2gigs.com/tourdb-stats.html>:

![Cities with most shows played - source u2gigs.com - October 2024.](u2gigs/u2_top_played_cities-u2gigscom.png){fig-align="left"}

## Venues with most shows

Here is the top 10 of venues where U2 played most often. This matches exactly the statistics listed at <https://www.u2gigs.com/tourdb-stats.html>.

```{r basic stats - venues with most shows, message = FALSE, warning = FALSE}

# Get top 10 venues with most shows played
top_venues <- find_most_played_venues(u2data, n = 10)
print(top_venues)
```

The info at <https://www.u2gigs.com/tourdb-stats.html>:

![Venues with most shows played - source u2gigs.com - October 2024.](u2gigs/u2_top_played_venues-u2gigscom.png){fig-align="left"}

## Countries with most shows

The top 10 of countries where U2 played most often can be seen below. This matches exactly the statistics listed at <https://www.u2gigs.com/tourdb-stats.html>.

```{r basic stats - countries with most shows, message = FALSE, warning = FALSE}

# countries with most shows played
# Get top 10 countries with most shows played
top_countries <- find_most_played_countries(u2data, n = 10)
print(top_countries)
```

The info at <https://www.u2gigs.com/tourdb-stats.html>:

![Countries with most shows played - source u2gigs.com - October 2024.](u2gigs/u2_top_played_countries-u2gigscom.png){fig-align="left"}

## Top 10 most played songs

Before we look at the top 10, let's see how many different songs U2 has played live.

```{r basic stats - number of unique songs played, message = FALSE, warning = FALSE}

# number of unique songs  played
nrow(unique(u2data[,c('song_title')]))
```

This does not match the info at <https://www.u2gigs.com/all-songnames.html> as there it is stated:

> We have 1011 different songs in our database.

However, at at a different page (<https://www.u2gigs.com/most-played-songs.html>) it is stated:

> We found 994 different songs in our database.

And that number coincides with the number found in my data.

The top 10 of songs that U2 played most often can be seen below. This does not match completely with the statistics listed at <https://www.u2gigs.com/most-played-songs.html>. There are several songs that have a count that is 1 higher in my data (Pride, Streets, One, Bullet, Still Haven't Found).

```{r visual exploration - songs most played, message = FALSE, warning = FALSE}

# songs most played
top_played_songs <- find_most_played_songs(u2data, n = 15)
print(top_played_songs)
```

The info at <https://www.u2gigs.com/tourdb-stats.html>:

![Most played songs - source u2gigs.com - October 2024.](u2gigs/U2_shows_top_played_songs-u2gigscom.png){fig-align="left"}

I took a closer look at the song Pride to see where the difference of 1 time played is between my data and the data at <https://www.u2gigs.com/Pride_(In_the_Name_of_Love)-s40.html>. I copied all the times the song was played from that page and pasted in Excel. With the code below I obtained the list from my data and compared that side by side in Excel.

```{r visual exploration - check Pride number of times played, message = FALSE, warning = FALSE}

# Pride - number of times played
pride_data <- subset(u2data, song_title == 'Pride (In the Name of Love)', select = c("date","city","tour", "song_title", "showID"))
#write.csv(pride_data, "pride_data.csv")
```

I discovered that the issue in my data is at the ZOO TV show on March 31, 1992 in Rosemont: <https://www.u2gigs.com/show235.html>. Pride was played one time at that show but appears in my data as it was played 2 times during that show:

```{r visual exploration - check set list Rosemont show 1992, message = FALSE, warning = FALSE}

# Rosemont show 1992 set list
rosemont_data <- subset(u2data, showID == 235, select = c("date", "city", "song_position","song_title","snippet", "encore"))
print(rosemont_data)
```

With this we can see that there is duplicated data for this show in my data. I went back to my original scraped files and indeed I saw that the duplication happened where the scraping process had been interrupted and later restarted manually. Let's check to what extent there are duplicated rows in my data:

```{r data curation - remove duplicates, message = FALSE, warning = FALSE}

# show duplicate rows 
duplicated_rows <- u2data[duplicated(u2data), ]
print(duplicated_rows, n=max())
```

Here we see the Rosemont show (show ID 235) again. I manually checked the others:

-   El Pueblo Vencerá at show with showID 1056, and also at show with showID 1060
    -   this turned out to be correct as the snippet was played two times in the same regular song (Mothers of the Disappeared), so it looks like a duplicate row but is correct.
-   Two Shots of Happy, One Shot of Sad at show with showID 1133
    -   this also turned out to be correct as the snippet was played two times in the same regular song (Stay), so it looks like a duplicate row but is correct.

This means I only have to remove the Rosemont show duplicate rows, as follows:

```{r data curation - remove duplicates Rosemont show (showID 235), message = FALSE, warning = FALSE}

# retrieve row indices of all duplicated rows 
which(duplicated(u2data) | duplicated(u2data, fromLast = TRUE))
# the first 21 are the ones that should be deleted

# check rows 20, 21 and 22 with indices 2959, 2979 and 2980, respectively
u2data[2959,]
u2data[2979,]
u2data[2980,]
# correct, these two rows correspond to the first one to delete (Zoo Station), the last one to delete (Desire) and the first one to keep (Zoo Station), respectively

# thus, we need to remove the rows with indices:
# 2959  2960  2961  2962  2963  2964  2965  2966  2967  2968  2969  2970  2971  2972  2973  2974  2975
# 2976  2977  2978  2979

# check number of rows before deletion
nrow(u2data)
# result: 39674

# delete duplicated rows from Rosemont show
u2data <- u2data[-(2959:2979),]

# check number of rows after deletion
nrow(u2data)
#result: 39653

# difference: 39674 - 39653
print(39674 - 39653)
# result: 21
#correct

# now save corrected data to csv file
# commented out because this has already been done
#write.csv(u2data, file = "u2data/u2data_all_shows_clean_final.csv", row.names = FALSE)
```

Now that those duplicated rows were removed, let's look at the songs by number of times played again and compare again to the info at <https://www.u2gigs.com/most-played-songs.html>.

```{r visual exploration - songs most played - second check, message = FALSE, warning = FALSE}

# read the u2 concertData
u2data <- read_concertData_csv('u2data/u2data_all_shows_clean_final.csv')

# songs most played
top_played_songs <- find_most_played_songs(u2data, n = 15)
print(top_played_songs)
```

Yes, total times the songs were played match now. There is still discrepancies in the times it was played as regular song or snippet.

![Top played songs - u2gigs.com - October 2024.](u2gigs/U2_shows_top_played_songs-u2gigscom.png){fig-align="left"}

Let's check Where The Streets Have No Name. In the u2gigs.com it has a total play count of 976, all regular performances, no snippets. In my data it states 974 regular performances and 2 times as a snippet.

```{r visual exploration - songs most played - snippet/regular check WTSHNN, message = FALSE, warning = FALSE}

WTSHNN_regular_play_count <- subset(u2data, song_title == 'Where the Streets Have No Name' & snippet == FALSE, select = c("showID", "date", "city", "song_position","song_title","snippet", "encore"))
print(WTSHNN_regular_play_count)

WTSHNN_snippet_play_count <- subset(u2data, song_title == 'Where the Streets Have No Name' & snippet == TRUE, select = c("showID", "date", "city", "song_position","song_title","snippet", "encore"))
print(WTSHNN_snippet_play_count)
```

In the two shows where my scraping code had counted WTSHNN as a snippet, we see the format difference:

```         
<li><a href='/Where_the_Streets_Have_No_Name-s17.html'>Where the Streets Have No Name</a> <span class='guestartist'>(with Bruce Springsteen)</span></li>

<li><a href='/Where_the_Streets_Have_No_Name-s17.html'>Where the Streets Have No Name</a> <em class='snippet'>partial chords</em></li>
```

Compared to a more normal coding for regular songs on the u2gigs.com site:

```         
<li><a href='/Where_the_Streets_Have_No_Name-s17.html'>Where the Streets Have No Name</a></li>
```

Thus, my scraping code had not identified these two performances of WTSHNN as regular but incorrectly as snippets. As my project was ending I did not follow up to correct my u2data set with these relativelz few errors in the snippet flag.

## Top 10 opening songs

Which were the songs U2 played most to open the shows? This can be seen at <https://www.u2gigs.com/tourdb-stats.html>.

![Top 10 opening songs - u2gigs.com - October 2024.](u2gigs/u2_top_opening_songs-u2gigscom.png){fig-align="left"}

In my data the numbers are like shown below:

```{r visual exploration - opening songs, message = FALSE, warning = FALSE}

# Get top 10 opening songs
top_10_openers <- find_most_played_opening_songs(u2data, n = 10)
print(top_10_openers)
```

The numbers did not match the information at <https://www.u2gigs.com/tourdb-stats.html>. I checked a few songs and observed a few things:

First, the opening songs counts listed at <https://www.u2gigs.com/tourdb-stats.html> seem to be outdated. When you click on one the songs in that list it shows a different number on the song page. For example, Zoo Station has the opening song count of 196 at <https://www.u2gigs.com/tourdb-stats.html> (see earlier screen shot), but 197 at <https://www.u2gigs.com/openingsong141.html>:

![Zoo Station opening song - u2gigs.com - October 2024.](u2gigs/u2_zoo_station_opening.png){fig-align="left"}

Thus, it looks like u2gigs.com needs to refresh the 'Top 25 show opening songs' list at <https://www.u2gigs.com/tourdb-stats.html>.

Second, I observed a few rare cases on the u2gigs.com show pages which were not handled properly by my scraping code, or were not handled properly by u2gigs.com, for example:

*Example 1:*

Oct 23rd, 2000 set list, <https://www.u2gigs.com/show1147.html>: ![Rare setlist format on show page.](u2gigs/u2_2000-10-23_setlist-u2gigscom.png)

My code and data did count Elevation as opening song for this 'show' while u2gigs.com did not count it, probably because it was preceded by an interview.

The same issue occurred with this show: <https://www.u2gigs.com/show1149.html>, and potentially with other shows as I only checked shows that opened with Elevation. u2gigs.com may want to check how they count opening songs in such case.

*Example 2*

2001-09-06 set list, <https://www.u2gigs.com/show1473.html>. In this case the set list started with a medley, with two songs as one entry. This is different than a regular song plus a snippet. My scraping code could apparently not handle this.

![Rare setlist format on show page.](u2gigs/u2_2001-09-06_setlist-u2gigscom.png){fig-align="left"}

I decided not to invest more time in trying to detect and fix these rare issues.
\
\
\

::: {style="text-align:center; font-size: 11px;"}
{{< fa solid copyright >}} Hans van Leeuwen 2024 \| {{< fa solid envelope >}} [E-mail](mailto:hans.data.universe@gmail.com) \| {{< fa brands linkedin >}} [LinkedIn](https://www.linkedin.com/in/vanleeuwenhans/) \| {{< fa brands x-twitter >}} [X-Twitter](https://x.com/hans444)
:::