The social weather of rOpenSci onboarding system

Our onboarding process ensures that packages contributed by the community undergo a transparent, constructive, non adversarial and open review process. Before even submitting my first R package to rOpenSci onboarding system in December 2015, I spent a fair amount of time reading through previous issue threads in order to assess whether onboarding was a friendly place for me: a newbie, very motivated to learn more but a newbie nonetheless. I soon got the feeling that yes, onboarding would help me make my package better without ever making me feel inadequate.

More recently, I read Anne Gentle’s essay in Open Advice where she mentions the concept of the social weather of an online community. By listening before I even posted, I was simply trying to get a good idea of the social weather of onboarding – as a side-note, how wonderful is it that onboarding’s being open makes it possible to listen?!

In the meantime, I’ve now only submitted and reviewed a few packages but also become an associate editor. In general, when one of us editors talks about onboarding, we like to use examples illustrating the system in a good light: often excerpts from review processes, or quotes of tweets by authors or reviewers. Would there be a more quantitative way for us to support our promotion efforts? In this blog post, I shall explore how a tidytext analysis of review processes (via their GitHub threads) might help us characterize the social weather of onboarding.

🔗 Preparing the data

🔗 A note about text mining

In this blog post, I’ll leverage the tidytext package, with the help of its accompanying book “Tidy text mining”. The authors, Julia Silge and David Robinson, actually met and started working on this project, at rOpenSci unconf in 2016!

🔗 The “tidy text” of the analysis

I’ve described in the first post of this series how I got all onboarding issue threads. Now, I’ll explain how I cleaned their text. Why does it need to be cleaned? Well, this text contains elements that I wished to remove: code chunks, as well as parts from our submission and review templates.

Edit: Find a cleaner approach to Markdown text extraction.

My biggest worry was the removal of templates from issues. I was already picturing my spending hours writing regular expressions to remove these lines… and then I realized that the word “lines” was the key! I could go split all issue comments into lines, which is called tokenization in proper text mining vocabulary, and then remove duplicates! This way, I didn’t even have to worry about the templates having changed a bit over time, since each version was used at least twice. A tricky part that remained was the removal of code chunks since I only wanted to keep human conversation. In theory, it was easy: code chunks are lines located between two lines containing “```”… I’m still not sure I solved this in the easiest possible way.

library("magrittr")
threads <- readr::read_csv("data/clean_data.csv")

# to remove code lines between ```
range <- function(x1, x2){
  x1:x2
}

# I need the indices of lines between ```
split_in_indices <- function(x){
  lengthx <- length(x)
  if(length(x) == 0){
    return(0)
  }else{
    if(lengthx > 2){
      
      limits1 <- x[seq(from = 1, to = (lengthx - 1), by = 2)]
      limits2 <- x[seq(from = 2, to = lengthx, by = 2)]
      purrr::map2(limits1, limits2, range) %>%
        unlist() %>%
        c()
    }else{
      x[1]:x[2]
    }
  }
}

# tokenize by line
threads <- tidytext::unnest_tokens(threads, line, body, token = "lines")

# remove white space
threads <- dplyr::mutate(threads, line = trimws(line))

# remove citations lines
threads <- dplyr::filter(threads, !stringr::str_detect(line, "^\\>"))

# remove the line from the template that has ``` that used to bother me
threads <- dplyr::filter(threads, !stringr::str_detect(line, "bounded by ```"))

# correct one line
threads <- dplyr::mutate(threads, line = stringr::str_replace_all(line, "`` `", "```"))

# group by comment 
threads <- dplyr::group_by(threads, title, created_at, user, issue)

# get indices
threads <- dplyr::mutate(threads, index = 1:n())

# get lines limiting chunks
threads <- dplyr::mutate(threads, chunk_limit = stringr::str_detect(line, "```")&stringr::str_count(line, "`") %in% c(3, 4))

# special treatment
threads <- dplyr::mutate(threads,
                     chunk_limit = ifelse(user == "MarkEdmondson1234" & issue == 127 & index == 33,
                                          FALSE, chunk_limit))
threads <- dplyr::mutate(threads, which_limit = list(split_in_indices(which(chunk_limit))))

# weird code probably to get indices of code lines
threads <- dplyr::mutate(threads, code = index %in% which_limit[[1]])
threads <- dplyr::ungroup(threads)

Let’s look at what this does in practice, with comments from gutenbergr submission as example. I chose this submission because the package author, David Robinson, is one of the two tidytext creators, and because I was the reviewer, so it’s all very meta, isn’t it?

In the excerpt below, we see the most important variable, the binary code indicating whether the line is a code line. This excerpt also shows variables created to help compute code: index is the index of the line withing a comment, chunk_limit indicates whether the line is a chunk limit, which_limit indicates which indices in the comment indicate lines of code.

dplyr::filter(threads, package == "gutenbergr", 
              user == "sckott", 
              !stringr::str_detect(line, "ropensci..footer")) %>%
  dplyr::mutate(created_at = as.character(created_at)) %>%
  dplyr::select(created_at, line, code, index, chunk_limit, which_limit) %>%
  knitr::kable()

So as you see now getting rid of chunks is straightforward: the lines with code == TRUE have to be deleted.

# remove them and get rid of now useless columns
threads <- dplyr::filter(threads, !code)
threads <- dplyr::select(threads, - code, - which_limit, - index, - chunk_limit)

Now on to removing template parts… I noticed that removing duplicates was a bit too drastic because sometimes duplicates were poorly formatted citations, e.g. an author answering a reviewer’s question by copy-pasting it without Markdown blockquotes, in which case we definitely want to keep the first occurrence. Besides, duplicates were sometimes very short sentences such as “great!” that are not templates, that we therefore should keep. Therefore, for each line, I counted how many times it occurred overall (no_total_occ), and in how many issues it occurred (no_issues).

Let’s look at Joseph Stachelek’s review of rrricanes for instance.

dplyr::filter(threads, user == "jsta", is_review) %>%
  dplyr::select(line) %>%
  head() %>%
  knitr::kable()

Now if we clean up a bit…

threads <- dplyr::group_by(threads, line)
threads <- dplyr::mutate(threads, no_total_occ = n(),
                     no_issues = length(unique(issue)), 
                     size = stringr::str_length(line))
threads <- dplyr::ungroup(threads)
threads <- dplyr::group_by(threads, issue, line)
threads <- dplyr::arrange(threads, created_at)
threads <- dplyr::filter(threads, no_total_occ <= 2,
                     # for repetitions in total keep the short ones 
                     # bc they are stuff like "thanks" so not template
                     # yes 10 is arbitrary
                     no_issues <= 1 | size < 10)
# when there's a duplicate in one issue
# it's probably citation
# so keep the first occurrence
get_first <- function(x){
  x[1]
}
threads <- dplyr::group_by(threads, issue, line)
threads <- dplyr::summarise_all(threads, get_first)
threads <- dplyr::select(threads, - no_total_occ, - size, - no_issues)

threads <- dplyr::mutate(threads, # let code words now be real words
                     line = stringr::str_replace_all(line, "`", ""),
                     # only keep text from links, not the links themselves
                     line = stringr::str_replace_all(line, "\\]\\(.*\\)", ""),
                     line = stringr::str_replace_all(line, "\\[", ""),
                     line = stringr::str_replace_all(line, "blob\\/master", ""), 
                     # ’
                     line = stringr::str_replace_all(line, "’", ""), 
                     # remove some other links
                     line = stringr::str_replace_all(line, "https\\:\\/\\/github\\.com\\/", ""))

threads <- dplyr::filter(threads, !stringr::str_detect(line, "estimated hours spent reviewing"))
threads <- dplyr::filter(threads, !stringr::str_detect(line, "notifications@github\\.com"))
threads <- dplyr::filter(threads, !stringr::str_detect(line, "reply to this email directly, view it on"))
threads <- dplyr::ungroup(threads)

Here is what we get from the same review.

dplyr::filter(threads, user == "jsta", is_review)  %>%
  dplyr::select(line) %>%
  head() %>%
  knitr::kable()

So now, we mostly got the interesting human and original language.

This got me “tidy enough” text. Let’s not mention this package author who found a way to poorly format their submission right under a guideline explaining how to copy the DESCRIPTION… Yep, that’s younger me. Oh well.

🔗 Computing sentiment

Another data preparation part was to compute the sentiment score of each line via the sentimentr package by Tyler Rinker, which computes a score for sentences, not for single words.

sentiment <- all %>%
  dplyr::group_by(line, created_at, user, role, issue) %>%
  dplyr::mutate(sentiment = median(sentimentr::sentiment(line)$sentiment)) %>%
  dplyr::ungroup() %>%
  dplyr::select(line, created_at, user, role, issue, sentiment)

This dataset of sentiment will be used later in the post.

🔗 Tidy text analysis of onboarding social weather

🔗 What do reviews talk about?

To find out what reviews deal with as if I didn’t know about our guidelines, I’ll compute the frequency of words and bigrams, and the pairwise correlation of words within issue comments.

My using lollipops below was inspired by this fascinating blog post of Tony ElHabr’s about his Google search history.

library("ggplot2")
library("ggalt")
library("hrbrthemes")

stopwords <- rcorpora::corpora("words/stopwords/en")$stopWords

word_counts <- threads %>%
  tidytext::unnest_tokens(word, line) %>%
  dplyr::filter(!word %in% stopwords) %>%
  dplyr::count(word, sort = TRUE) %>%
  dplyr::mutate(word = reorder(word, n))   

ggplot(word_counts[1:15,]) +
  geom_lollipop(aes(word, n),
                size = 2, col = "salmon") +
  hrbrthemes::theme_ipsum(base_size = 16,
                          axis_title_size = 16) +
  coord_flip()

bigrams_counts <- threads %>%
  tidytext::unnest_tokens(bigram, line, token = "ngrams", n = 2) %>%
  tidyr::separate(bigram, c("word1", "word2"), sep = " ",
                  remove = FALSE) %>%
  dplyr::filter(!word1 %in% stopwords) %>%
  dplyr::filter(!word2 %in% stopwords) %>%
  dplyr::count(bigram, sort = TRUE) %>%
  dplyr::mutate(bigram = reorder(bigram, n))   

ggplot(bigrams_counts[2:15,]) +
  geom_lollipop(aes(bigram, n),
                size = 2, col = "salmon") +
  hrbrthemes::theme_ipsum(base_size = 16,
                          axis_title_size = 16) +
  coord_flip()

I’m not showing the first bigram that basically shows I’ve an encoding issue to solve with a variation of “´”. In any case, both figures show what we care about, like our guidelines that are mentioned often, and documentation. I think words absent from the figures such as performance or speed also highlight what we care less about, following Jeff Leek’s philosophy.

Now, let’s move on to a bit more complex visualization of pairwise correlations between words within lines. First, let’s prepare the table of words in lines. Compared with the book tutorial, we add a condition for eliminating words mentioned in only one submission, often function names.

users <- unique(threads$user)
onboarding_line_words <- threads %>%
  dplyr::group_by(user, issue, created_at, package, line) %>%
  dplyr::mutate(line_id = paste(package, user, created_at, line)) %>%
  dplyr::ungroup() %>%
  tidytext::unnest_tokens(word, line) %>%
  dplyr::filter( word != package, !word %in% users,
                 is.na(as.numeric(word)),
                 word != "ldecicco",
                 word != "usgs") %>%
  dplyr::group_by(word) %>%
  dplyr::filter(length(unique(issue)) > 1) %>%
  dplyr::select(line_id, word)

onboarding_line_words %>%
  head() %>%
  knitr::kable()

Then, we can compute the correlation.

word_cors <- onboarding_line_words %>%
  dplyr::group_by(word) %>%
  dplyr::filter(!word %in% stopwords) %>%
  dplyr::filter(n() >= 20) %>%
  widyr::pairwise_cor(word, line_id, sort = TRUE)

For instance, what often goes in the same line as vignette?

dplyr::filter(word_cors, item1 == "vignette")

## # A tibble: 853 x 3
##    item1    item2     correlation
##    <chr>    <chr>           <dbl>
##  1 vignette readme         0.176 
##  2 vignette vignettes      0.174 
##  3 vignette chunk          0.145 
##  4 vignette eval           0.120 
##  5 vignette examples       0.108 
##  6 vignette overview       0.0933
##  7 vignette building       0.0914
##  8 vignette link           0.0863
##  9 vignette maps           0.0840
## 10 vignette package        0.0831
## # ... with 843 more rows

Now let’s plot the network of these relationships between words, using the igraph package by Gábor Csárdi and Támas Nepusz and ggraph package by Thomas Lin Pedersen.

library("igraph")
library("ggraph")

set.seed(2016)

word_cors %>%
  dplyr::filter(correlation > .35) %>%
  graph_from_data_frame() %>%
  ggraph(layout = "fr") +
  geom_edge_link(aes(edge_alpha = correlation), show.legend = FALSE) +
  geom_node_point(color = "lightblue", size = 5) +
  geom_node_text(aes(label = name), repel = TRUE) +
  theme_void()

This figure gives a good sample of things discussed in reviews. Despite our efforts filtering words specific to issues, some of them remain very specific, such as country/city/location that are very frequent in ropenaq review.

🔗 How positive is onboarding?

Using sentiment analysis, we can look at how positive comments are.

sentiments %>%
  dplyr::group_by(role) %>%
  skimr::skim(sentiment)

## Skim summary statistics
##  n obs: 11553 
##  n variables: 6 
##  group variables: role 
## 
## Variable type: numeric 
##               role  variable missing complete    n  mean   sd   min p25
##             author sentiment       0     4823 4823 0.07  0.21 -1.2    0
##  community_manager sentiment       0       97   97 0.13  0.21 -0.41   0
##             editor sentiment       0     1521 1521 0.13  0.22 -1.63   0
##              other sentiment       0      344  344 0.073 0.2  -0.6    0
##           reviewer sentiment       0     4768 4768 0.073 0.21 -1      0
##  median  p75  max     hist
##   0     0.17 1.84 <U+2581><U+2581><U+2582><U+2587><U+2581><U+2581><U+2581><U+2581>
##   0.071 0.23 1    <U+2581><U+2581><U+2587><U+2585><U+2582><U+2581><U+2581><U+2581>
##   0.075 0.25 1.13 <U+2581><U+2581><U+2581><U+2581><U+2587><U+2586><U+2581><U+2581>
##   0     0.2  0.81 <U+2581><U+2581><U+2582><U+2587><U+2582><U+2582><U+2581><U+2581>
##   0     0.17 1.73 <U+2581><U+2581><U+2587><U+2585><U+2581><U+2581><U+2581><U+2581>

summary(sentiments$sentiment)

##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
## -1.63200  0.00000  0.00000  0.07961  0.18353  1.84223

sentiments %>%
  dplyr::filter(!role %in% c("other", "community_manager")) %>%
  ggplot(aes(role, sentiment)) +
  geom_boxplot(fill = "salmon") +
  hrbrthemes::theme_ipsum(base_size = 16,
                          axis_title_size = 16,
                          strip_text_size = 16)

These boxplots seem to indicate that lines are generally positive (positive mean, zero 25th-quantile), although it’d be better to be able to compare them with text from traditional review processes of scientific manuscripts in order to get a better feeling for the meaning of these numbers.

On these boxplots we also see that we do get lines with a negative sentiment value… about what? Here are the most common words in negative lines.

sentiments %>%
  dplyr::filter(sentiment < 0) %>%
  tidytext::unnest_tokens(word, line) %>%
  dplyr::filter(!word %in% stopwords) %>%
  dplyr::count(word, sort = TRUE) %>%
  dplyr::mutate(word = reorder(word, n)) %>%
  dplyr::filter(n > 100) %>%
ggplot() +
  geom_lollipop(aes(word, n),
                size = 2, col = "salmon") +
  hrbrthemes::theme_ipsum(base_size = 16,
                          axis_title_size = 16) +
  coord_flip()

And looking at a sample…

sentiments %>%
  dplyr::arrange(sentiment) %>%
  dplyr::select(line, sentiment) %>%
  head(n = 15) %>%
  knitr::kable()

It seems that negative lines are mostly people discussing bugs and problems in code, and GitHub issues, and trying to solve them. The kind of negative lines we’re happy to see in our process, since once solved, they mean the software got more robust!

Last but not least, I mentioned our using particular cases as examples of how happy everyone seems to be in the process. To find such examples, we rely on memory, but what about picking heart-warming lines using their sentiment score?

sentiments %>%
  dplyr::arrange(- sentiment) %>%
  dplyr::select(line, sentiment) %>%
  head(n = 15) %>%
  knitr::kable()

As you can imagine, these sentences make the whole team very happy! And we hope they’ll encourage you to contribute to rOpenSci onboarding.

🔗 Outlook

This first try at text analysis of onboarding issue threads is quite promising: we were able to retrieve text and to use natural language processing to extract most common words and bigrams, and sentiment. This allowed us to describe the social weather of onboarding: we could see that this system is about software, and that negative sentiment was often due to bugs being discussed and solved; and we could extract the most positive lines where volunteers praised the review system or the piece of software under review.

One could expand this analysis with a study of emoji use, in the text using an emoji dictionary as in this blog post and around the text (so-called emoji reactions, present in the API and used in e.g ghrecipes::get_issues_thumbs). Another aspect of social weather is maybe the timeliness that’s expected or implemented at the different parts of the process, but it’d be covered by other data such as the labelling history of the issues, which could get extracted from GitHub V4 API as well.

This is the final post of the “Our package review system in review” series. The first post presented data collection from GitHub, the second post aimed at quantifying the work represented by onboarding. The posts motivated us to keep using data to illustrate and assess the system. Brace yourself for more onboarding data analyses in the future!