R4DS Assignment 2: Wrangle

BDS Toolbox

Source: Abe Hofman / R for Data Science; Wickham & Grolemund Work on these assignments individually. You can find the deadline in the course syllabus. After the deadline my own solutions will be posted on Canvas. This assignment won’t be graded, because a lot of materials are available online. But you do need to hand it in through Canvas. Use these online materials wisely! Make the assignment in a .R or Rmarkdown file. If you work in a .R file, use comments (for us or yourself) to keep thing readable and make sure you document runs without errors. If you use an Markdown file, Knit it to an HTML file and hand that in through Canvas. Make sure that it includes your code. We will give a short Markdown demo during one of the tutorials (see course syllabus). The assignment will cover Ch 9-13, 15, 16 and includes a subset of the questions of these chapters [you can skip 14].

10.5

1. How can you tell if an object is a tibble? (Hint: try printing mtcars, which is a regular data frame).

mtcars # the rows are named

mtcars_tibble <- as_tibble(mtcars)
mtcars_tibble

class(mtcars)
#> [1] "data.frame"
class(mtcars_tibble)
#> [1] "tbl_df"     "tbl"        "data.frame"

2. Compare and contrast the following operations on a data.frame and equivalent tibble. What is different? Why might the default data frame behaviours cause you frustration?

From 10.5:

df <- data.frame(abc = 1, xyz = "a")
df$x
#> [1] "a"
df[, "xyz"]
#> [1] "a"
df[, c("abc", "xyz")]

# tibble equivalent
tb <- tibble(abc = 1, xyz = "a")
tb$x
#> Warning: Unknown or uninitialised column: `x`.
#> NULL
tb[, "xyz"]

tb[, c("abc", "xyz")]

The data frame returns a vector when selecting a single column, and a data frame when selecting multiple columns. So, the output type is inconsistent. In contrast, tibble returns a tibble either way.

Also, the data frame accepts partial matching of column names. This can lead to weird results if, for example, there’re typos in column names. In contrast, tibble throws a warning and returns NULL if a column name is not exactly matched.

11.2.2

1. What function would you use to read a file where fields were separated with “|”?

read_delim(I("a|b|c\n1|2|3"), delim = "|")

2. What are the most important arguments to read_fwf()?

From docs, it’s col_positions (besides the file of course). It’s used to specify the column positions since fixed width files don’t have a delimiter.

test_fwf <- I("AAAAAA\nBBBBBB")
read_fwf(test_fwf, fwf_widths(c(1, 2, 3)))

read_fwf(test_fwf)

3. Identify what is wrong with each of the following inline CSV files. What happens when you run the code?

read_csv("a,b\n1,2,3\n4,5,6")
#> Warning: The `file` argument of `read_csv()` should use `I()` for literal data as of readr 2.2.0.
#>   
#>   # Bad (for example):
#>   read_csv("x,y\n1,2")
#>   
#>   # Good:
#>   read_csv(I("x,y\n1,2"))
#> This warning is displayed once per session.
#> Call `lifecycle::last_lifecycle_warnings()` to see where this warning was generated.
#> Warning: One or more parsing issues, call `problems()` on your data frame for details, e.g.:
#>   dat <- vroom(...)
#>   problems(dat)

read_csv("a,b,c\n1,2\n1,2,3,4")
#> Warning: One or more parsing issues, call `problems()` on your data frame for details, e.g.:
#>   dat <- vroom(...)
#>   problems(dat)

read_csv("a,b\n\"1")
#> Warning: One or more parsing issues, call `problems()` on your data frame for details, e.g.:
#>   dat <- vroom(...)
#>   problems(dat)

read_csv("a,b\n1,2\na,b")

read_csv("a;b\n1;3")

# e.g. this works:
read_csv("a,b,c\n1,2,3\n4,5,6")

1. Uneven number of cols on line 1 vs others. Causes the last 2 cols to be squashed in the last col.
2. Uneven number of cols on all lines. Causes the values in the last col to be squashed when present, and NA if not.
3. Unclosed quote. The parser inserts a newline char at the end of literal strings. So, this causes the parser to read the next line as part of the character value, including \n.
4. I don’t see a problem besides line duplication. But it could be an issue that the column types are char?
5. Wrong delimiter. Causes the parser to read the whole line as a single value.
6. Indeed works well.

11.3.5.7

d1 <- "January 1, 2010"
d2 <- "2015-Mar-07"
d3 <- "06-Jun-2017"
d4 <- c("August 19 (2015)", "July 1 (2015)")
d5 <- "12/30/14" # Dec 30, 2014
t1 <- "1705"
t2 <- "11:15:10.12 PM"

Generate the correct format string to parse each of the following dates and times:

parse_date(d1, "%B %d, %Y")
#> [1] "2010-01-01"
parse_date(d2, "%Y-%b-%d")
#> [1] "2015-03-07"
parse_date(d3, "%d-%b-%Y")
#> [1] "2017-06-06"
parse_date(d4, "%B %d (%Y)")
#> [1] "2015-08-19" "2015-07-01"
parse_date(d5, "%m/%d/%y")
#> [1] "2014-12-30"
parse_time(t1, "%H%M")
#> 17:05:00
parse_time(t2, "%I:%M:%OS %p")
#> 23:15:10.12

12.2.1: 3

Recreate the plot showing change in cases over time using table2 instead of table1. What do you need to do first?

glimpse(table1)
#> Rows: 6
#> Columns: 4
#> $ country    <chr> "Afghanistan", "Afghanistan", "Brazil", "Brazil", "China", …
#> $ year       <dbl> 1999, 2000, 1999, 2000, 1999, 2000
#> $ cases      <dbl> 745, 2666, 37737, 80488, 212258, 213766
#> $ population <dbl> 19987071, 20595360, 172006362, 174504898, 1272915272, 12804…
glimpse(table2)
#> Rows: 12
#> Columns: 4
#> $ country <chr> "Afghanistan", "Afghanistan", "Afghanistan", "Afghanistan", "B…
#> $ year    <dbl> 1999, 1999, 2000, 2000, 1999, 1999, 2000, 2000, 1999, 1999, 20…
#> $ type    <chr> "cases", "population", "cases", "population", "cases", "popula…
#> $ count   <dbl> 745, 19987071, 2666, 20595360, 37737, 172006362, 80488, 174504…

table2_cases <- table2 %>%
  filter(type == "cases") %>%
  rename(cases = count) %>%
  select(-type)

glimpse(table2_cases)
#> Rows: 6
#> Columns: 3
#> $ country <chr> "Afghanistan", "Afghanistan", "Brazil", "Brazil", "China", "Ch…
#> $ year    <dbl> 1999, 2000, 1999, 2000, 1999, 2000
#> $ cases   <dbl> 745, 2666, 37737, 80488, 212258, 213766

# og
ggplot(table1, aes(year, cases)) +
  geom_line(aes(group = country), colour = "grey50") +
  geom_point(aes(colour = country))

# same code with table2
ggplot(table2_cases, aes(year, cases)) +
  geom_line(aes(group = country), colour = "grey50") +
  geom_point(aes(colour = country))

12.3.3: 3

What would happen if you widen this table? Why? How could you add a new column to uniquely identify each value?

From 12.3:

people <- tribble(
  ~name, ~names, ~values,
  #-----------------|--------|------
  "Phillip Woods", "age", 45,
  "Phillip Woods", "height", 186,
  "Phillip Woods", "age", 50,
  "Jessica Cordero", "age", 37,
  "Jessica Cordero", "height", 156
)

people %>%
  pivot_wider(names_from = names, values_from = values)
#> Warning: Values from `values` are not uniquely identified; output will contain list-cols.
#> • Use `values_fn = list` to suppress this warning.
#> • Use `values_fn = {summary_fun}` to summarise duplicates.
#> • Use the following dplyr code to identify duplicates.
#>   {data} |>
#>   dplyr::summarise(n = dplyr::n(), .by = c(name, names)) |>
#>   dplyr::filter(n > 1L)

It gives a warning because there are multiple values for age in Phillip Woods, and there are no identifiers to distinguish between them.

people %>%
  group_by(name, names) %>%
  mutate(id = row_number()) %>%
  ungroup() %>%
  pivot_wider(names_from = names, values_from = values)

We added an id column to identify different values for the same name and names combination. However, this is still ambiguous because if those are 2 different people with the same name, we don’t know which person the `heightz entry correspond to.

12.4.3: 1

What do the extra and fill arguments do in separate()? Experiment with the various options for the following two toy datasets.

tibble(x = c("a,b,c", "d,e,f,g", "h,i,j")) %>%
  separate(x, c("one", "two", "three"))
#> Warning: Expected 3 pieces. Additional pieces discarded in 1 rows [2].

tibble(x = c("a,b,c", "d,e", "f,g,i")) %>%
  separate(x, c("one", "two", "three"))
#> Warning: Expected 3 pieces. Missing pieces filled with `NA` in 1 rows [2].

From docs, the (not combined) options are:

test_d <- tibble(x = c("a,b,c", "d,e,f,g", "h,i,j"))

# extra
test_d %>% separate(x, c("one", "two", "three"), extra = "warn")
#> Warning: Expected 3 pieces. Additional pieces discarded in 1 rows [2].

test_d %>% separate(x, c("one", "two", "three"), extra = "drop")

test_d %>% separate(x, c("one", "two", "three"), extra = "merge")

# fill
test_d %>% separate(x, c("one", "two", "three"), fill = "warn")
#> Warning: Expected 3 pieces. Additional pieces discarded in 1 rows [2].

test_d %>% separate(x, c("one", "two", "three"), fill = "right")
#> Warning: Expected 3 pieces. Additional pieces discarded in 1 rows [2].

test_d %>% separate(x, c("one", "two", "three"), fill = "left")
#> Warning: Expected 3 pieces. Additional pieces discarded in 1 rows [2].

13.4.6: 2

Add the location of the origin and destination (i.e. the lat and lon) to flights.

airports_locations <- airports %>% select(faa, lat, lon)

flights %>%
  left_join(airports_locations, by = c("origin" = "faa")) %>%
  rename(origin_lat = lat, origin_lon = lon) %>%
  left_join(airports_locations, by = c("dest" = "faa")) %>%
  rename(dest_lat = lat, dest_lon = lon)

13.5.1: 2

Filter flights to only show flights with planes that have flown at least 100 flights.

flights %>%
  filter(!is.na(tailnum)) %>%
  group_by(tailnum) %>%
  filter(n() >= 100) %>%
  ungroup()

# what i think you want here but unnecessary because we're using the sama table
planes_100flights <- flights %>%
  filter(!is.na(tailnum)) %>%
  count(tailnum) %>%
  filter(n >= 100)

flights %>%
  semi_join(planes_100flights, by = "tailnum")

15.3.1

The following questions are about the forcats::gss_cat dataset

glimpse(gss_cat)
#> Rows: 21,483
#> Columns: 9
#> $ year    <int> 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 20…
#> $ marital <fct> Never married, Divorced, Widowed, Never married, Divorced, Mar…
#> $ age     <int> 26, 48, 67, 39, 25, 25, 36, 44, 44, 47, 53, 52, 52, 51, 52, 40…
#> $ race    <fct> White, White, White, White, White, White, White, White, White,…
#> $ rincome <fct> $8000 to 9999, $8000 to 9999, Not applicable, Not applicable, …
#> $ partyid <fct> "Ind,near rep", "Not str republican", "Independent", "Ind,near…
#> $ relig   <fct> Protestant, Protestant, Protestant, Orthodox-christian, None, …
#> $ denom   <fct> "Southern baptist", "Baptist-dk which", "No denomination", "No…
#> $ tvhours <int> 12, NA, 2, 4, 1, NA, 3, NA, 0, 3, 2, NA, 1, NA, 1, 7, NA, 3, 3…

2. Explore the distribution of rincome (reported income). What makes the default bar chart hard to understand? How could you improve the plot?

ggplot(gss_cat, aes(rincome)) +
  geom_bar()

ggplot(gss_cat, aes(rincome)) +
  geom_bar() +
  coord_flip()

The x labels are overlapping, so we can flip the coordinates, for example.

3. What is the most common relig in this survey? What’s the most common partyid?

most_common_relig <- gss_cat %>%
  count(relig) %>%
  arrange(desc(n)) %>%
  slice(1)

cat(
  "Most common relig:", as.character(most_common_relig$relig),
  "with", most_common_relig$n, "occurrences."
)
#> Most common relig: Protestant with 10846 occurrences.

most_common_partyid <- gss_cat %>%
  count(partyid) %>%
  arrange(desc(n)) %>%
  slice(1)

cat(
  "Most common partyid:", as.character(most_common_partyid$partyid),
  "with", most_common_partyid$n, "occurrences."
)
#> Most common partyid: Independent with 4119 occurrences.

15.5.1: 1

How have the proportions of people identifying as Democrat, Republican, and Independent changed over time?

glimpse(gss_cat)
#> Rows: 21,483
#> Columns: 9
#> $ year    <int> 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 2000, 20…
#> $ marital <fct> Never married, Divorced, Widowed, Never married, Divorced, Mar…
#> $ age     <int> 26, 48, 67, 39, 25, 25, 36, 44, 44, 47, 53, 52, 52, 51, 52, 40…
#> $ race    <fct> White, White, White, White, White, White, White, White, White,…
#> $ rincome <fct> $8000 to 9999, $8000 to 9999, Not applicable, Not applicable, …
#> $ partyid <fct> "Ind,near rep", "Not str republican", "Independent", "Ind,near…
#> $ relig   <fct> Protestant, Protestant, Protestant, Orthodox-christian, None, …
#> $ denom   <fct> "Southern baptist", "Baptist-dk which", "No denomination", "No…
#> $ tvhours <int> 12, NA, 2, 4, 1, NA, 3, NA, 0, 3, 2, NA, 1, NA, 1, 7, NA, 3, 3…

get_partyid_options <- function(x, excl = NA) {
  x_lower <- tolower(x)
  options <- gss_cat %>%
    select(partyid) %>%
    filter(!is.na(partyid)) %>%
    mutate(partyid_lower = tolower(partyid)) %>%
    filter(str_detect(partyid_lower, x_lower))

  if (!is.na(excl)) {
    excl_lower <- tolower(excl)
    options <- options %>%
      filter(!str_detect(partyid_lower, excl_lower))
  }

  options %>%
    distinct() %>%
    pull(partyid)
}

democrat_options <- get_partyid_options("dem", excl = "ind")
democrat_options
#> [1] Not str democrat Strong democrat 
#> 10 Levels: No answer Don't know Other party ... Strong democrat

republican_options <- get_partyid_options("rep", excl = "ind")
republican_options
#> [1] Not str republican Strong republican 
#> 10 Levels: No answer Don't know Other party ... Strong democrat

independent_options <- get_partyid_options("ind")
independent_options
#> [1] Ind,near rep Independent  Ind,near dem
#> 10 Levels: No answer Don't know Other party ... Strong democrat

gss_cat_props <- gss_cat %>%
  mutate(partyid = fct_collapse(
    partyid,
    "Democrat" = democrat_options,
    "Republican" = republican_options,
    "Independent" = independent_options
  )) %>%
  filter(partyid %in% c("Democrat", "Republican", "Independent")) %>%
  count(year, partyid) %>%
  group_by(year) %>%
  mutate(prop = n / sum(n)) %>%
  ungroup()

glimpse(gss_cat_props)
#> Rows: 24
#> Columns: 4
#> $ year    <int> 2000, 2000, 2000, 2002, 2002, 2002, 2004, 2004, 2004, 2006, 20…
#> $ partyid <fct> Republican, Independent, Democrat, Republican, Independent, De…
#> $ n       <int> 684, 1152, 921, 764, 994, 923, 821, 991, 959, 1132, 1851, 1436…
#> $ prop    <dbl> 0.2480958, 0.4178455, 0.3340588, 0.2849683, 0.3707572, 0.34427…

ggplot(gss_cat_props, aes(year, prop, colour = partyid)) +
  geom_line() +
  geom_point()

16.2.4: 3

d1 <- "January 1, 2010"
d2 <- "2015-Mar-07"
d3 <- "06-Jun-2017"
d4 <- c("August 19 (2015)", "July 1 (2015)")
d5 <- "12/30/14" # Dec 30, 2014

Use the appropriate lubridate function to parse each of the following dates:

mdy(d1)
#> [1] "2010-01-01"
ymd(d2)
#> [1] "2015-03-07"
dmy(d3)
#> [1] "2017-06-06"
mdy(d4)
#> [1] "2015-08-19" "2015-07-01"
mdy(d5)
#> [1] "2014-12-30"

16.3.4: 5

About the flights dataset. On what day of the week should you leave if you want to minimise the chance of a delay?

least_delay <- flights %>%
  filter(!is.na(time_hour)) %>%
  mutate(week_day = wday(time_hour, label = TRUE)) %>%
  group_by(week_day) %>%
  summarise(
    prop_arr_delay = mean(arr_delay > 0, na.rm = TRUE)
  ) %>%
  arrange(prop_arr_delay) %>%
  slice(1)

cat(
  "The least delays occur on", as.character(least_delay$week_day),
  "with a proportion of", least_delay$prop_arr_delay, "of flights delayed."
)
#> The least delays occur on Sat with a proportion of 0.3189924 of flights delayed.

16.4.5: 4

Write a function that given your birthday (as a date), returns how old you are in years.

age <- function(x) {
  (x %--% today()) %/% years(1)
}

birthday <- today() %>% update(year = 1995)

cat(
  "If you were born on", as.character(birthday),
  "you are", age(birthday), "years old."
)
#> If you were born on 1995-09-13 you are 31 years old.