⇠ Day 2 || Day 4 ⇢

Download course materials (.zip file) from here after 4th June.

bit.ly/20190605-updates

You will need these datasets today:

  1. binomial-data.csv
  2. simulated-data.csv

You may also need to install the packages ggbeeswarm and yarrr (which comes with a lovely tutorial):

install.packages("ggbeeswarm")
install.packages("yarrr")

Load in today’s libraries.

# https://bookdown.org/ndphillips/YaRrr/pirateplot.html
library(tidyverse)
library(ggbeeswarm)
library(yarrr)

1 Why dataviz?

If you don’t know the shape of your data, you might not draw appropriate conclusions about the statistical tests you perform.

Check out the Datasaurus Dozen, which all have the same x/y means and standard deviations:

Dataviz is also engaging, communicative, creative, efficient, and fun.

2 Plotting in base R

I think it’s good to know how to do it, but for most purposes, ggplot2 will be easier and more powerful.

2.1 Scatter plots

Plot the geocoordinates of quakes:

# plot( y ~ x )
# plot( x, y )

Change the colour and point character (pch):

Change colour by longitude with a split at 175˚S:

plot(••• ~ •••, 
     col=•••, 
     pch=•••, 
     xlim=c(160,190), 
     ylim=c(-40,-10))
points(••• ~ •••, 
       col=•••, 
       pch=•••)

Change axis labels and add a vertical line to show the split:

•••
abline(v=•••,col=•••,lty="dashed")

2.2 Box plots

Here we can visualise the distribution of sleep results for groups 1 and 2. We can use the same extra parameters as before to clean it up, too.

boxplot(••• ~ •••, 
        col=c("red","blue"),
        xlab="group",
        ylab="extra hours of sleep",
        main="`sleep` dataset")

We can add the parameter notch to indicate where the bulk of the data are located, and are intended to give a visual estimate if the distributions overlap little enough to estimate that they are distinct populations.

But how many data points go into these graphs? Not enough to prevent “hinges” from being larger than the actual interquartile ranges. And where are the points?

So we get a decent view of how these groups are distributed, but we can do better.

2.3 When to use base R graphics

The only times I personally use base R to plot things now is when I only want to take a very quick peek at a scatter plot or if I want to use one of the built-in visualisations of regressions (e.g., qqplot, which you should now be able to explore on your own!)

3 Components of a ggplot

We build up plots in ggplot2 starting with a base and slowly adding different geometries and properties, using a similar logic to how we added the annotation line to a scatterplot in base R. However, ggplot gives us a much more powerful interface because it allows us to do calculations during plotting and remember certain parameters between different layers of the plot.

3.1 Base plot

The base plot by itself doesn’t look like much, but it already shows us that there’s a lot going on.

Plus it just looks sleeker. (And don’t underestimate the power of pleasant-looking graphics!)

One of the parameters of the base plot is aes(), which stands for “aesthetics”. Aesthetics of the base plot are the attributes of the plot that should be inherited for all layers. This typically includes things like the x-axis, y-axis, any colour-coding by groups, etc. We’ll come back to this in a few minutes.

3.2 Geometries

In order to actually see our data, we have to tell R what kind of plot we want to make. Let’s start with a scatterplot, i.e., points.

Notice how we didn’t have to specify anything within the parameters for geom_point(). This is because everything that it requires has been inherited from the base plot. We could also use different geometries.

geom_hex can clearly demonstrate how ggplot does “calculations” during plotting:

We can try a few others out, but these plots aren’t really designed to be informative for these kinds of data.

Some plots don’t need y aesthetics because they calculate y based on frequency in the input:

And some plots need (at least) one axis to not be continuous data, like boxplot.

3.2.1 Refresher

How would you use piping to plot only one island’s worth of data?

3.3 Aesthetics

Now let’s look at how we can use aes() to make our visualisations more informative and engaging.

sleep has two groups, so let’s plot a boxplot of extra with the two groups separated by colour.

We could also use colour to differentiate between individuals in an interaction plot. Try it with and without colour.

We can see that actually having colour specified as a distinct aesthetic separates out each of the IDs into pairs of extra values rather than treating them as points that all must be connected to each other. Moreover, we don’t need to specify what colours to use (although we could).

3.4 Axes

We can add layers to change the axes of the plot, add and change the spacing of the ticks, and label things.

What else would you like to do with this graph?

# ?

3.5 Themes

What if you don’t like having a grey background or you’re going to use the plot on a slide with a black background and you want it to look slick?

Try out different themes! (If we put + NULL at the end, we can comment out any line of the graph without having to worry what comes next)

quakes %>% 
  ••• +
  #theme_bw() +
  #theme_classic() +
  #theme_dark() +
  #theme_light() +
  #theme_minimal() +
  #theme_test() +
  #theme_void() +
  #theme() +
  NULL

3.6 Error bars

If we generate summaries of our data, we can calculate standard errors and plot error bars.

Let’s put error bars on quakes after calculating the means and standard errors. Then let’s try to make the ribbon defined by the standard deviation.

3.7 Bar plots

For bar plots, I recommend using summarised data. R can calculate certain things on its own (and there are ways to do any calculation you need using ggplot2), but for now this is my recommendation.

Plot means and standard errors of stations reporting for the range of magnitudes, and fill the bars with a continuous colour scale to indicate the number of observations per bar.

4 Beyond the tidyverse

There are lots of really fantastic packages for plotting that aren’t part of tidyverse or are add-ons that are part of the default offerings of ggplot2.

4.1 Beeswarm vs Violin

Beeswarm plots are good for categorical data when you want to see the actual values of the points in each category.

4.2 Pirate plots

Pirate plots do this sort of graph natively.

install.packages("yarrr")
library(yarrr)

We can use code copied from the tutorial.

pirateplot(formula = stations ~ mag, # dv is stations, iv is magnitude
           data = quakes,
           main = "Pirateplot of stations reporting earthquakes",
           xlab = "magnitude",
           ylab = "stations",
           theme = 1)

4.3 Raincloud plots

Originally from here.

source("https://gist.githubusercontent.com/benmarwick/2a1bb0133ff568cbe28d/raw/fb53bd97121f7f9ce947837ef1a4c65a73bffb3f/geom_flat_violin.R")
raincloud_theme <- theme(
  text = element_text(size = 10),
  axis.title.x = element_text(size = 16),
  axis.title.y = element_text(size = 16),
  axis.text = element_text(size = 14),
  axis.text.x = element_text(angle = 45, vjust = 0.5),
  legend.title = element_text(size = 16),
  legend.text = element_text(size = 16),
  legend.position = "right",
  plot.title = element_text(lineheight = .8, face = "bold", size = 16),
  panel.border = element_blank(),
  panel.grid.minor = element_blank(),
  panel.grid.major = element_blank(),
  axis.line.x = element_line(colour = "black", size = 0.5, linetype = "solid"),
  axis.line.y = element_line(colour = "black", size = 0.5, linetype = "solid"))

Adapted from Tidy Raincloud Plot tutorial:

5 Exploration

Dataset simulated-data.csv

5.1 Structure of the dataset

Using str, examine the dataset.

Check out the first few rows using head.

5.2 Plot the data

Plot boxplots and violin plots for the ratings. Subset by participant.

Group by region, word, frequency, and grammaticality. Summarise mean and standard error.

5.3 Bonus: Sandbox

Dataset iris

The function ggplot layers different geometries and aesthetics to build up a plot:

iris %>%
  mutate(Sepal.Width = Sepal.Width+rnorm(length(Sepal.Width),0,.1))%>%
  ggplot(aes(x=Species,y=Sepal.Width,fill=Species)) +
  geom_violin(lty=0,alpha=.5)+
  geom_boxplot(alpha=0.5,lwd=.5) +
  geom_quasirandom(dodge.width=1, alpha=.5)

In what ways might we change this plot?

5.3.1 Inheritance

We can layer two different plots in the same graph by keeping some of the aesthetics undefined in the base plot.

Going back to quakes, let’s pipe a summary table into ggplot that provides means, standard deviations, and standard errors for both depth and stations reporting, grouped by magnitude. In the graph, plot the means as a path (and points), the standard error as a more opaque ribbon and standard deviation as a more transparent ribbon:

quakes %>%
  group_by(•••) %>%
  summarise(n=•••,
            sta=•••,
            staSD=•••,
            staSE=•••,
            dep=•••,
            depSD=•••,
            depSE=•••) %>%
  ggplot(aes(x=•••)) +
  geom_point(aes(y=•••),colour="red")+
  geom_path(aes(y=•••),colour="red")+
  geom_ribbon(aes(ymin=•••-•••,ymax=•••+•••),fill="red",alpha=•••) +
  geom_ribbon(aes(ymin=•••-•••,ymax=•••+•••),fill="red",alpha=•••) +
  geom_point(aes(y=•••),colour="blue")+
  geom_path(aes(y=•••),colour="blue")+
  geom_ribbon(aes(ymin=•••-•••,ymax=•••+•••),fill="blue",alpha=•••) +
  geom_ribbon(aes(ymin=•••-•••,ymax=•••+•••),fill="blue",alpha=•••) +
  theme_bw() + ylab("depth OR number of stations reporting")

5.3.2 Your pick

What kind of graphs do you use most in your research? Can we simulate data as a group and learn to plot it in a few different ways?

LS0tCnRpdGxlOiAiKipEYXRhdml6KioiCmF1dGhvcjogIkRyIExhdXJlbiBNIEFja2VybWFuIgpkYXRlOiAiMDUgSlVOIDIwMTkiCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZGVwdGg6IDMKICAgIG51bWJlcl9zZWN0aW9uczogdHJ1ZQogICAgdG9jX2Zsb2F0OiAKICAgICAgc21vb3RoX3Njcm9sbDogdHJ1ZQogICAgICBjb2xsYXBzZWQ6IGZhbHNlCiAgICB0aGVtZTogc2FuZHN0b25lCiAgICBoaWdobGlnaHQ6IGRlZmF1bHQKLS0tCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IFRSVUUpCgoKIyAxLiBSIGFuZCBSU3R1ZGlvIGludGVyZmFjZXMsIFJNYXJrZG93biwgYW5kIGJlc3QgcHJhY3RpY2VzCiMgICAgLSBCYXNpYyBmdW5jdGlvbmFsaXR5LCBzaG9ydGN1dHMsIHdyaXRpbmcgc2NyaXB0cyBhbmQgbm90ZWJvb2tzLCBvcmdhbmlzaW5nIGZpbGVzLCBpbnRlcmZhY2luZyB3aXRoIEdpdEh1YgojIDIuIERhdGEgZnJhbWUgbWFuaXB1bGF0aW9uIHVzaW5nIFRpZHl2ZXJzZQojICAgIC0gUGlwaW5nIHN5bnRheCBhbmQgY29kZSByZXBsaWNhYmlsaXR5CiMgICAgLSBUaWR5ciwgRHBseXIsIHJlbGF0ZWQgcGFja2FnZXMKIyAgICAtIENsZWFuaW5nLCBjb21iaW5pbmcsIGFuZCByZWFycmFuZ2luZyBkYXRhIGZyYW1lcwojIDMuIERhdGEgdmlzdWFsaXNhdGlvbiB1c2luZyBnZ3Bsb3QgYW5kIGJlc3QgcHJhY3RpY2VzCiMgICAgLSBTdHJ1Y3R1cmUgYW5kIHN5bnRheCBvZiBnZ3Bsb3QgYW5kIGdlb20KIyAgICAtIEN1c3RvbWlzaW5nIGFuZCBjb21iaW5pbmcgcGxvdHMKIyAgICAtIERldGVybWluaW5nIHdoYXQgcGxvdCBpcyBiZXN0IGZvciB5b3VyIGRhdGEKIyA0LiBCYXNpYyBsaW5lYXIgbW9kZWxzIHdpdGhvdXQgYW5kIHdpdGggbWl4ZWQgZWZmZWN0cyAoaW5jbHVkaW5nIGdhdXNzaWFuLCBiaW5vbWlhbCwgYW5kIG9yZGluYWwpCiMgICAgLSBidWlsZGluZy9zZWxlY3RpbmcgYW4gYXBwcm9wcmlhdGUgbW9kZWwsIGluY2x1ZGluZyByYW5kb20gZWZmZWN0IHN0cnVjdHVyZXMKIyAgICAtIG1heGltYWwgdnMgcGFyc2ltb25pb3VzIG1vZGVscwojICAgIC0gcHJhY3RpY2FsIHVzZSBvZiBsbSgpLCBsbWVyKCksIGdsbSgpLCBnbG1lcigpLCBjbG1tKCkKIyAgICAtIGR1bW15IGNvZGluZyB2cyBjb250cmFzdCBjb2RpbmcKIyAgICAtIGludGVycHJldGluZyB0aGUgb3V0cHV0CmBgYApb4oegIERheSAyXSgyMDE5MDYwNC10aWR5dmVyc2UuaHRtbCkgfHwgW0RheSA0IOKHol0oMjAxOTA2MDYtc3RhdHMuaHRtbCkKCkRvd25sb2FkIGNvdXJzZSBtYXRlcmlhbHMgKC56aXAgZmlsZSkgZnJvbSBbaGVyZV0oaHR0cDovL2JpdC5seS8yMDE5MDYwNS11cGRhdGVzKSAqYWZ0ZXIgNHRoIEp1bmUqLgoKWyoqYml0Lmx5LzIwMTkwNjA1LXVwZGF0ZXMqKl0oaHR0cDovL2JpdC5seS8yMDE5MDYwNS11cGRhdGVzKQoKWW91IHdpbGwgbmVlZCB0aGVzZSBkYXRhc2V0cyB0b2RheToKCjEuIFtiaW5vbWlhbC1kYXRhLmNzdl0oLi4vZGF0YS9iaW5vbWlhbC1kYXRhLmNzdikKMi4gW3NpbXVsYXRlZC1kYXRhLmNzdl0oLi4vZGF0YS9zaW11bGF0ZWQtZGF0YS5jc3YpCgpZb3UgbWF5IGFsc28gbmVlZCB0byBpbnN0YWxsIHRoZSBwYWNrYWdlcyBgZ2diZWVzd2FybWAgYW5kIGB5YXJycmAgKHdoaWNoIGNvbWVzIHdpdGggYSBbbG92ZWx5IHR1dG9yaWFsXShodHRwczovL2Jvb2tkb3duLm9yZy9uZHBoaWxsaXBzL1lhUnJyLykpOgpgYGB7cn0KaW5zdGFsbC5wYWNrYWdlcygiZ2diZWVzd2FybSIpCmluc3RhbGwucGFja2FnZXMoInlhcnJyIikKYGBgCgpMb2FkIGluIHRvZGF5J3MgbGlicmFyaWVzLgpgYGB7cn0KIyBodHRwczovL2Jvb2tkb3duLm9yZy9uZHBoaWxsaXBzL1lhUnJyL3BpcmF0ZXBsb3QuaHRtbApsaWJyYXJ5KHRpZHl2ZXJzZSkKbGlicmFyeShnZ2JlZXN3YXJtKQpsaWJyYXJ5KHlhcnJyKQpgYGAKCiMgV2h5IGRhdGF2aXo/CgpJZiB5b3UgZG9uJ3Qga25vdyB0aGUgc2hhcGUgb2YgeW91ciBkYXRhLCB5b3UgbWlnaHQgbm90IGRyYXcgYXBwcm9wcmlhdGUgY29uY2x1c2lvbnMgYWJvdXQgdGhlIHN0YXRpc3RpY2FsIHRlc3RzIHlvdSBwZXJmb3JtLgoKQ2hlY2sgb3V0IHRoZSBbRGF0YXNhdXJ1cyBEb3plbl0oaHR0cHM6Ly93d3cuYXV0b2Rlc2tyZXNlYXJjaC5jb20vcHVibGljYXRpb25zL3NhbWVzdGF0cyksIHdoaWNoIGFsbCBoYXZlIHRoZSBzYW1lIHgveSBtZWFucyBhbmQgc3RhbmRhcmQgZGV2aWF0aW9uczoKIVtdKC4uL2ltYWdlcy9kYXRhc2F1cnVzLmdpZikKCkRhdGF2aXogaXMgYWxzbyBlbmdhZ2luZywgY29tbXVuaWNhdGl2ZSwgY3JlYXRpdmUsIGVmZmljaWVudCwgYW5kIGZ1bi4KCiMgUGxvdHRpbmcgaW4gYmFzZSBSCgpJIHRoaW5rIGl0J3MgZ29vZCB0byBrbm93IGhvdyB0byBkbyBpdCwgYnV0IGZvciBtb3N0IHB1cnBvc2VzLCBgZ2dwbG90MmAgd2lsbCBiZSBlYXNpZXIgYW5kIG1vcmUgcG93ZXJmdWwuCgojIyBTY2F0dGVyIHBsb3RzCgpQbG90IHRoZSBnZW9jb29yZGluYXRlcyBvZiBgcXVha2VzYDoKYGBge3J9CiMgcGxvdCggeSB+IHggKQojIHBsb3QoIHgsIHkgKQpgYGAKCkNoYW5nZSB0aGUgY29sb3VyIGFuZCBwb2ludCBjaGFyYWN0ZXIgKHBjaCk6CmBgYHtyfQoKYGBgCgpDaGFuZ2UgY29sb3VyIGJ5IGxvbmdpdHVkZSB3aXRoIGEgc3BsaXQgYXQgMTc1y5pTOgpgYGB7cn0KcGxvdCjigKLigKLigKIgfiDigKLigKLigKIsIAogICAgIGNvbD3igKLigKLigKIsIAogICAgIHBjaD3igKLigKLigKIsIAogICAgIHhsaW09YygxNjAsMTkwKSwgCiAgICAgeWxpbT1jKC00MCwtMTApKQpwb2ludHMo4oCi4oCi4oCiIH4g4oCi4oCi4oCiLCAKICAgICAgIGNvbD3igKLigKLigKIsIAogICAgICAgcGNoPeKAouKAouKAoikKYGBgCgpDaGFuZ2UgYXhpcyBsYWJlbHMgYW5kIGFkZCBhIHZlcnRpY2FsIGxpbmUgdG8gc2hvdyB0aGUgc3BsaXQ6CmBgYHtyfQrigKLigKLigKIKYWJsaW5lKHY94oCi4oCi4oCiLGNvbD3igKLigKLigKIsbHR5PSJkYXNoZWQiKQpgYGAKCiMjIEJveCBwbG90cwoKSGVyZSB3ZSBjYW4gdmlzdWFsaXNlIHRoZSBkaXN0cmlidXRpb24gb2YgYHNsZWVwYCByZXN1bHRzIGZvciBncm91cHMgMSBhbmQgMi4gV2UgY2FuIHVzZSB0aGUgc2FtZSBleHRyYSBwYXJhbWV0ZXJzIGFzIGJlZm9yZSB0byBjbGVhbiBpdCB1cCwgdG9vLgpgYGB7cn0KYm94cGxvdCjigKLigKLigKIgfiDigKLigKLigKIsIAogICAgICAgIGNvbD1jKCJyZWQiLCJibHVlIiksCiAgICAgICAgeGxhYj0iZ3JvdXAiLAogICAgICAgIHlsYWI9ImV4dHJhIGhvdXJzIG9mIHNsZWVwIiwKICAgICAgICBtYWluPSJgc2xlZXBgIGRhdGFzZXQiKQpgYGAKCldlIGNhbiBhZGQgdGhlIHBhcmFtZXRlciBgbm90Y2hgIHRvIGluZGljYXRlIHdoZXJlIHRoZSBidWxrIG9mIHRoZSBkYXRhIGFyZSBsb2NhdGVkLCBhbmQgYXJlIGludGVuZGVkIHRvIGdpdmUgYSB2aXN1YWwgZXN0aW1hdGUgaWYgdGhlIGRpc3RyaWJ1dGlvbnMgb3ZlcmxhcCBsaXR0bGUgZW5vdWdoIHRvIGVzdGltYXRlIHRoYXQgdGhleSBhcmUgZGlzdGluY3QgcG9wdWxhdGlvbnMuCmBgYHtyfQoKYGBgCgpCdXQgaG93IG1hbnkgZGF0YSBwb2ludHMgZ28gaW50byB0aGVzZSBncmFwaHM/IE5vdCBlbm91Z2ggdG8gcHJldmVudCAiaGluZ2VzIiBmcm9tIGJlaW5nIGxhcmdlciB0aGFuIHRoZSBhY3R1YWwgaW50ZXJxdWFydGlsZSByYW5nZXMuIEFuZCB3aGVyZSAqYXJlKiB0aGUgcG9pbnRzPwoKYGBge3J9CgpgYGAKClNvIHdlIGdldCBhIGRlY2VudCB2aWV3IG9mIGhvdyB0aGVzZSBncm91cHMgYXJlIGRpc3RyaWJ1dGVkLCBidXQgd2UgY2FuIGRvIGJldHRlci4KCiMjIFdoZW4gdG8gdXNlIGJhc2UgUiBncmFwaGljcwoKVGhlIG9ubHkgdGltZXMgSSBwZXJzb25hbGx5IHVzZSBiYXNlIFIgdG8gcGxvdCB0aGluZ3Mgbm93IGlzIHdoZW4gSSBvbmx5IHdhbnQgdG8gdGFrZSBhIHZlcnkgcXVpY2sgcGVlayBhdCBhIHNjYXR0ZXIgcGxvdCBvciBpZiBJIHdhbnQgdG8gdXNlIG9uZSBvZiB0aGUgYnVpbHQtaW4gdmlzdWFsaXNhdGlvbnMgb2YgcmVncmVzc2lvbnMgKGUuZy4sIGBxcXBsb3RgLCB3aGljaCB5b3Ugc2hvdWxkIG5vdyBiZSBhYmxlIHRvIGV4cGxvcmUgb24geW91ciBvd24hKQoKIyBDb21wb25lbnRzIG9mIGEgZ2dwbG90CgpXZSBidWlsZCB1cCBwbG90cyBpbiBgZ2dwbG90MmAgc3RhcnRpbmcgd2l0aCBhIGJhc2UgYW5kIHNsb3dseSBhZGRpbmcgZGlmZmVyZW50IGdlb21ldHJpZXMgYW5kIHByb3BlcnRpZXMsIHVzaW5nIGEgc2ltaWxhciBsb2dpYyB0byBob3cgd2UgYWRkZWQgdGhlIGFubm90YXRpb24gbGluZSB0byBhIHNjYXR0ZXJwbG90IGluIGJhc2UgUi4gSG93ZXZlciwgYGdncGxvdGAgZ2l2ZXMgdXMgYSBtdWNoIG1vcmUgcG93ZXJmdWwgaW50ZXJmYWNlIGJlY2F1c2UgaXQgYWxsb3dzIHVzIHRvIGRvIGNhbGN1bGF0aW9ucyBkdXJpbmcgcGxvdHRpbmcgYW5kIHJlbWVtYmVyIGNlcnRhaW4gcGFyYW1ldGVycyAqYmV0d2VlbiogZGlmZmVyZW50IGxheWVycyBvZiB0aGUgcGxvdC4KCiMjIEJhc2UgcGxvdAoKVGhlIGJhc2UgcGxvdCBieSBpdHNlbGYgZG9lc24ndCBsb29rIGxpa2UgbXVjaCwgYnV0IGl0IGFscmVhZHkgc2hvd3MgdXMgdGhhdCB0aGVyZSdzIGEgbG90IGdvaW5nIG9uLgpgYGB7cn0KCmBgYAoKUGx1cyBpdCBqdXN0IGxvb2tzIHNsZWVrZXIuIChBbmQgZG9uJ3QgdW5kZXJlc3RpbWF0ZSB0aGUgcG93ZXIgb2YgcGxlYXNhbnQtbG9va2luZyBncmFwaGljcyEpCgpPbmUgb2YgdGhlIHBhcmFtZXRlcnMgb2YgdGhlIGJhc2UgcGxvdCBpcyBgYWVzKClgLCB3aGljaCBzdGFuZHMgZm9yICJhZXN0aGV0aWNzIi4gQWVzdGhldGljcyBvZiB0aGUgYmFzZSBwbG90IGFyZSB0aGUgYXR0cmlidXRlcyBvZiB0aGUgcGxvdCB0aGF0IHNob3VsZCBiZSBpbmhlcml0ZWQgZm9yIGFsbCBsYXllcnMuIFRoaXMgdHlwaWNhbGx5IGluY2x1ZGVzIHRoaW5ncyBsaWtlIHRoZSB4LWF4aXMsIHktYXhpcywgYW55IGNvbG91ci1jb2RpbmcgYnkgZ3JvdXBzLCBldGMuIFdlJ2xsIGNvbWUgYmFjayB0byB0aGlzIGluIGEgZmV3IG1pbnV0ZXMuCgojIyBHZW9tZXRyaWVzCgpJbiBvcmRlciB0byBhY3R1YWxseSBzZWUgb3VyIGRhdGEsIHdlIGhhdmUgdG8gdGVsbCBSIHdoYXQga2luZCBvZiBwbG90IHdlIHdhbnQgdG8gbWFrZS4gTGV0J3Mgc3RhcnQgd2l0aCBhIHNjYXR0ZXJwbG90LCBpLmUuLCBwb2ludHMuCmBgYHtyfQoKYGBgCgpOb3RpY2UgaG93IHdlIGRpZG4ndCBoYXZlIHRvIHNwZWNpZnkgYW55dGhpbmcgd2l0aGluIHRoZSBwYXJhbWV0ZXJzIGZvciBgZ2VvbV9wb2ludCgpYC4gVGhpcyBpcyBiZWNhdXNlIGV2ZXJ5dGhpbmcgdGhhdCBpdCByZXF1aXJlcyBoYXMgYmVlbiBpbmhlcml0ZWQgZnJvbSB0aGUgYmFzZSBwbG90LiBXZSBjb3VsZCBhbHNvIHVzZSBkaWZmZXJlbnQgZ2VvbWV0cmllcy4KCmBnZW9tX2hleGAgY2FuIGNsZWFybHkgZGVtb25zdHJhdGUgaG93IGdncGxvdCBkb2VzICJjYWxjdWxhdGlvbnMiIGR1cmluZyBwbG90dGluZzoKYGBge3J9CgpgYGAKCldlIGNhbiB0cnkgYSBmZXcgb3RoZXJzIG91dCwgYnV0IHRoZXNlIHBsb3RzIGFyZW4ndCByZWFsbHkgZGVzaWduZWQgdG8gYmUgaW5mb3JtYXRpdmUgZm9yIHRoZXNlIGtpbmRzIG9mIGRhdGEuCmBgYHtyfQoKYGBgCgpTb21lIHBsb3RzIGRvbid0IG5lZWQgeSBhZXN0aGV0aWNzIGJlY2F1c2UgdGhleSBjYWxjdWxhdGUgeSBiYXNlZCBvbiBmcmVxdWVuY3kgaW4gdGhlIGlucHV0OgpgYGB7cn0KCmBgYAoKQW5kIHNvbWUgcGxvdHMgbmVlZCAoYXQgbGVhc3QpIG9uZSBheGlzIHRvIG5vdCBiZSBjb250aW51b3VzIGRhdGEsIGxpa2UgYGJveHBsb3RgLgpgYGB7cn0KCmBgYAoKIyMjIFJlZnJlc2hlcgoKSG93IHdvdWxkIHlvdSB1c2UgcGlwaW5nIHRvIHBsb3Qgb25seSBvbmUgaXNsYW5kJ3Mgd29ydGggb2YgZGF0YT8KYGBge3J9CgpgYGAKCiMjIEFlc3RoZXRpY3MKCk5vdyBsZXQncyBsb29rIGF0IGhvdyB3ZSBjYW4gdXNlIGBhZXMoKWAgdG8gbWFrZSBvdXIgdmlzdWFsaXNhdGlvbnMgbW9yZSBpbmZvcm1hdGl2ZSBhbmQgZW5nYWdpbmcuCgpgc2xlZXBgIGhhcyB0d28gZ3JvdXBzLCBzbyBsZXQncyBwbG90IGEgYm94cGxvdCBvZiBgZXh0cmFgIHdpdGggdGhlIHR3byBncm91cHMgc2VwYXJhdGVkIGJ5IGNvbG91ci4KYGBge3J9CgpgYGAKCldlIGNvdWxkIGFsc28gdXNlIGNvbG91ciB0byBkaWZmZXJlbnRpYXRlIGJldHdlZW4gaW5kaXZpZHVhbHMgaW4gYW4gaW50ZXJhY3Rpb24gcGxvdC4gVHJ5IGl0IHdpdGggYW5kIHdpdGhvdXQgY29sb3VyLgpgYGB7cn0KCmBgYAoKV2UgY2FuIHNlZSB0aGF0IGFjdHVhbGx5IGhhdmluZyBjb2xvdXIgc3BlY2lmaWVkIGFzIGEgZGlzdGluY3QgYWVzdGhldGljIHNlcGFyYXRlcyBvdXQgZWFjaCBvZiB0aGUgSURzIGludG8gcGFpcnMgb2YgYGV4dHJhYCB2YWx1ZXMgcmF0aGVyIHRoYW4gdHJlYXRpbmcgdGhlbSBhcyBwb2ludHMgdGhhdCBhbGwgbXVzdCBiZSBjb25uZWN0ZWQgdG8gZWFjaCBvdGhlci4gTW9yZW92ZXIsIHdlIGRvbid0ICpuZWVkKiB0byBzcGVjaWZ5IHdoYXQgY29sb3VycyB0byB1c2UgKGFsdGhvdWdoIHdlIGNvdWxkKS4KCiMjIEF4ZXMKCldlIGNhbiBhZGQgbGF5ZXJzIHRvIGNoYW5nZSB0aGUgYXhlcyBvZiB0aGUgcGxvdCwgYWRkIGFuZCBjaGFuZ2UgdGhlIHNwYWNpbmcgb2YgdGhlIHRpY2tzLCBhbmQgbGFiZWwgdGhpbmdzLgpgYGB7cn0KCmBgYAoKV2hhdCBlbHNlIHdvdWxkIHlvdSBsaWtlIHRvIGRvIHdpdGggdGhpcyBncmFwaD8KYGBge3J9CiMgPwpgYGAKCiMjIFRoZW1lcwoKV2hhdCBpZiB5b3UgZG9uJ3QgbGlrZSBoYXZpbmcgYSBncmV5IGJhY2tncm91bmQgb3IgeW91J3JlIGdvaW5nIHRvIHVzZSB0aGUgcGxvdCBvbiBhIHNsaWRlIHdpdGggYSBibGFjayBiYWNrZ3JvdW5kIGFuZCB5b3Ugd2FudCBpdCB0byBsb29rIHNsaWNrPwoKVHJ5IG91dCBkaWZmZXJlbnQgdGhlbWVzISAoSWYgd2UgcHV0IGArIE5VTExgIGF0IHRoZSBlbmQsIHdlIGNhbiBjb21tZW50IG91dCBhbnkgbGluZSBvZiB0aGUgZ3JhcGggd2l0aG91dCBoYXZpbmcgdG8gd29ycnkgd2hhdCBjb21lcyBuZXh0KQpgYGB7cn0KcXVha2VzICU+JSAKICDigKLigKLigKIgKwogICN0aGVtZV9idygpICsKICAjdGhlbWVfY2xhc3NpYygpICsKICAjdGhlbWVfZGFyaygpICsKICAjdGhlbWVfbGlnaHQoKSArCiAgI3RoZW1lX21pbmltYWwoKSArCiAgI3RoZW1lX3Rlc3QoKSArCiAgI3RoZW1lX3ZvaWQoKSArCiAgI3RoZW1lKCkgKwogIE5VTEwKYGBgCgojIyBFcnJvciBiYXJzIAoKSWYgd2UgZ2VuZXJhdGUgc3VtbWFyaWVzIG9mIG91ciBkYXRhLCB3ZSBjYW4gY2FsY3VsYXRlIHN0YW5kYXJkIGVycm9ycyBhbmQgcGxvdCBlcnJvciBiYXJzLgoKTGV0J3MgcHV0IGVycm9yIGJhcnMgb24gYHF1YWtlc2AgYWZ0ZXIgY2FsY3VsYXRpbmcgdGhlIG1lYW5zIGFuZCBzdGFuZGFyZCBlcnJvcnMuIFRoZW4gbGV0J3MgdHJ5IHRvIG1ha2UgdGhlIHJpYmJvbiBkZWZpbmVkIGJ5IHRoZSBzdGFuZGFyZCBkZXZpYXRpb24uCmBgYHtyfQoKYGBgCgojIyBCYXIgcGxvdHMKCkZvciBiYXIgcGxvdHMsIEkgcmVjb21tZW5kIHVzaW5nIHN1bW1hcmlzZWQgZGF0YS4gUiBjYW4gY2FsY3VsYXRlIGNlcnRhaW4gdGhpbmdzIG9uIGl0cyBvd24gKGFuZCB0aGVyZSBhcmUgd2F5cyB0byBkbyBhbnkgY2FsY3VsYXRpb24geW91IG5lZWQgdXNpbmcgYGdncGxvdDJgKSwgYnV0IGZvciBub3cgdGhpcyBpcyBteSByZWNvbW1lbmRhdGlvbi4KClBsb3QgbWVhbnMgYW5kIHN0YW5kYXJkIGVycm9ycyBvZiBzdGF0aW9ucyByZXBvcnRpbmcgZm9yIHRoZSByYW5nZSBvZiBtYWduaXR1ZGVzLCBhbmQgZmlsbCB0aGUgYmFycyB3aXRoIGEgY29udGludW91cyBjb2xvdXIgc2NhbGUgdG8gaW5kaWNhdGUgdGhlIG51bWJlciBvZiBvYnNlcnZhdGlvbnMgcGVyIGJhci4KYGBge3J9CgpgYGAKCiMgQmV5b25kIHRoZSBgdGlkeXZlcnNlYAoKVGhlcmUgYXJlIGxvdHMgb2YgcmVhbGx5IGZhbnRhc3RpYyBwYWNrYWdlcyBmb3IgcGxvdHRpbmcgdGhhdCBhcmVuJ3QgcGFydCBvZiB0aWR5dmVyc2Ugb3IgYXJlIGFkZC1vbnMgdGhhdCBhcmUgcGFydCBvZiB0aGUgZGVmYXVsdCBvZmZlcmluZ3Mgb2YgYGdncGxvdDJgLgoKIyMgQmVlc3dhcm0gdnMgVmlvbGluCgpCZWVzd2FybSBwbG90cyBhcmUgZ29vZCBmb3IgY2F0ZWdvcmljYWwgZGF0YSB3aGVuIHlvdSB3YW50IHRvIHNlZSB0aGUgYWN0dWFsIHZhbHVlcyBvZiB0aGUgcG9pbnRzIGluIGVhY2ggY2F0ZWdvcnkuCgpgYGB7cn0KCmBgYAoKIyMgUGlyYXRlIHBsb3RzCgpQaXJhdGUgcGxvdHMgZG8gdGhpcyBzb3J0IG9mIGdyYXBoIG5hdGl2ZWx5LgpgYGB7cn0KaW5zdGFsbC5wYWNrYWdlcygieWFycnIiKQpsaWJyYXJ5KHlhcnJyKQpgYGAKCldlIGNhbiB1c2UgY29kZSBjb3BpZWQgZnJvbSB0aGUgdHV0b3JpYWwuCmBgYHtyfQpwaXJhdGVwbG90KGZvcm11bGEgPSBzdGF0aW9ucyB+IG1hZywgIyBkdiBpcyBzdGF0aW9ucywgaXYgaXMgbWFnbml0dWRlCiAgICAgICAgICAgZGF0YSA9IHF1YWtlcywKICAgICAgICAgICBtYWluID0gIlBpcmF0ZXBsb3Qgb2Ygc3RhdGlvbnMgcmVwb3J0aW5nIGVhcnRocXVha2VzIiwKICAgICAgICAgICB4bGFiID0gIm1hZ25pdHVkZSIsCiAgICAgICAgICAgeWxhYiA9ICJzdGF0aW9ucyIsCiAgICAgICAgICAgdGhlbWUgPSAxKQpgYGAKCiMjIFJhaW5jbG91ZCBwbG90cwoKT3JpZ2luYWxseSBmcm9tIFtoZXJlXShodHRwczovL21pY2FoYWxsZW4ub3JnLzIwMTgvMDMvMTUvaW50cm9kdWNpbmctcmFpbmNsb3VkLXBsb3RzLykuCmBgYHtyfQpzb3VyY2UoImh0dHBzOi8vZ2lzdC5naXRodWJ1c2VyY29udGVudC5jb20vYmVubWFyd2ljay8yYTFiYjAxMzNmZjU2OGNiZTI4ZC9yYXcvZmI1M2JkOTcxMjFmN2Y5Y2U5NDc4MzdlZjFhNGM2NWE3M2JmZmIzZi9nZW9tX2ZsYXRfdmlvbGluLlIiKQpyYWluY2xvdWRfdGhlbWUgPC0gdGhlbWUoCiAgdGV4dCA9IGVsZW1lbnRfdGV4dChzaXplID0gMTApLAogIGF4aXMudGl0bGUueCA9IGVsZW1lbnRfdGV4dChzaXplID0gMTYpLAogIGF4aXMudGl0bGUueSA9IGVsZW1lbnRfdGV4dChzaXplID0gMTYpLAogIGF4aXMudGV4dCA9IGVsZW1lbnRfdGV4dChzaXplID0gMTQpLAogIGF4aXMudGV4dC54ID0gZWxlbWVudF90ZXh0KGFuZ2xlID0gNDUsIHZqdXN0ID0gMC41KSwKICBsZWdlbmQudGl0bGUgPSBlbGVtZW50X3RleHQoc2l6ZSA9IDE2KSwKICBsZWdlbmQudGV4dCA9IGVsZW1lbnRfdGV4dChzaXplID0gMTYpLAogIGxlZ2VuZC5wb3NpdGlvbiA9ICJyaWdodCIsCiAgcGxvdC50aXRsZSA9IGVsZW1lbnRfdGV4dChsaW5laGVpZ2h0ID0gLjgsIGZhY2UgPSAiYm9sZCIsIHNpemUgPSAxNiksCiAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9ibGFuaygpLAogIHBhbmVsLmdyaWQubWlub3IgPSBlbGVtZW50X2JsYW5rKCksCiAgcGFuZWwuZ3JpZC5tYWpvciA9IGVsZW1lbnRfYmxhbmsoKSwKICBheGlzLmxpbmUueCA9IGVsZW1lbnRfbGluZShjb2xvdXIgPSAiYmxhY2siLCBzaXplID0gMC41LCBsaW5ldHlwZSA9ICJzb2xpZCIpLAogIGF4aXMubGluZS55ID0gZWxlbWVudF9saW5lKGNvbG91ciA9ICJibGFjayIsIHNpemUgPSAwLjUsIGxpbmV0eXBlID0gInNvbGlkIikpCmBgYAoKQWRhcHRlZCBmcm9tIFtUaWR5IFJhaW5jbG91ZCBQbG90XShodHRwczovL29yY2hpZDAwLmdpdGh1Yi5pby90aWR5X3JhaW5jbG91ZHBsb3QpIHR1dG9yaWFsOgpgYGB7cn0KCmBgYAoKCiMgRXhwbG9yYXRpb24KCj4gRGF0YXNldCBgc2ltdWxhdGVkLWRhdGEuY3N2YAoKIyMgU3RydWN0dXJlIG9mIHRoZSBkYXRhc2V0CgpVc2luZyBgc3RyYCwgZXhhbWluZSB0aGUgZGF0YXNldC4KYGBge3J9CgpgYGAKCkNoZWNrIG91dCB0aGUgZmlyc3QgZmV3IHJvd3MgdXNpbmcgYGhlYWRgLgpgYGB7cn0KCmBgYAoKIyMgUGxvdCB0aGUgZGF0YQoKUGxvdCBib3hwbG90cyBhbmQgdmlvbGluIHBsb3RzIGZvciB0aGUgcmF0aW5ncy4gU3Vic2V0IGJ5IHBhcnRpY2lwYW50LgpgYGB7cn0KCmBgYAoKR3JvdXAgYnkgcmVnaW9uLCB3b3JkLCBmcmVxdWVuY3ksIGFuZCBncmFtbWF0aWNhbGl0eS4gU3VtbWFyaXNlIG1lYW4gYW5kIHN0YW5kYXJkIGVycm9yLgpgYGB7cn0KCmBgYAoKCiMjIEJvbnVzOiBTYW5kYm94Cgo+IERhdGFzZXQgYGlyaXNgCgpUaGUgZnVuY3Rpb24gYGdncGxvdGAgbGF5ZXJzIGRpZmZlcmVudCBnZW9tZXRyaWVzIGFuZCBhZXN0aGV0aWNzIHRvIGJ1aWxkIHVwIGEgcGxvdDoKYGBge3J9CmlyaXMgJT4lCiAgbXV0YXRlKFNlcGFsLldpZHRoID0gU2VwYWwuV2lkdGgrcm5vcm0obGVuZ3RoKFNlcGFsLldpZHRoKSwwLC4xKSklPiUKICBnZ3Bsb3QoYWVzKHg9U3BlY2llcyx5PVNlcGFsLldpZHRoLGZpbGw9U3BlY2llcykpICsKICBnZW9tX3Zpb2xpbihsdHk9MCxhbHBoYT0uNSkrCiAgZ2VvbV9ib3hwbG90KGFscGhhPTAuNSxsd2Q9LjUpICsKICBnZW9tX3F1YXNpcmFuZG9tKGRvZGdlLndpZHRoPTEsIGFscGhhPS41KQpgYGAKCkluIHdoYXQgd2F5cyBtaWdodCB3ZSBjaGFuZ2UgdGhpcyBwbG90PwpgYGB7cn0KCmBgYAoKIyMjIEluaGVyaXRhbmNlCgpXZSBjYW4gbGF5ZXIgdHdvIGRpZmZlcmVudCBwbG90cyBpbiB0aGUgc2FtZSBncmFwaCBieSBrZWVwaW5nIHNvbWUgb2YgdGhlIGFlc3RoZXRpY3MgdW5kZWZpbmVkIGluIHRoZSBiYXNlIHBsb3QuCgpHb2luZyBiYWNrIHRvIGBxdWFrZXNgLCBsZXQncyBwaXBlIGEgc3VtbWFyeSB0YWJsZSBpbnRvIGdncGxvdCB0aGF0IHByb3ZpZGVzIG1lYW5zLCBzdGFuZGFyZCBkZXZpYXRpb25zLCBhbmQgc3RhbmRhcmQgZXJyb3JzIGZvciBib3RoIGRlcHRoIGFuZCBzdGF0aW9ucyByZXBvcnRpbmcsIGdyb3VwZWQgYnkgbWFnbml0dWRlLiBJbiB0aGUgZ3JhcGgsIHBsb3QgdGhlIG1lYW5zIGFzIGEgcGF0aCAoYW5kIHBvaW50cyksIHRoZSBzdGFuZGFyZCBlcnJvciBhcyBhIG1vcmUgb3BhcXVlIHJpYmJvbiBhbmQgc3RhbmRhcmQgZGV2aWF0aW9uIGFzIGEgbW9yZSB0cmFuc3BhcmVudCByaWJib246CmBgYHtyfQpxdWFrZXMgJT4lCiAgZ3JvdXBfYnko4oCi4oCi4oCiKSAlPiUKICBzdW1tYXJpc2Uobj3igKLigKLigKIsCiAgICAgICAgICAgIHN0YT3igKLigKLigKIsCiAgICAgICAgICAgIHN0YVNEPeKAouKAouKAoiwKICAgICAgICAgICAgc3RhU0U94oCi4oCi4oCiLAogICAgICAgICAgICBkZXA94oCi4oCi4oCiLAogICAgICAgICAgICBkZXBTRD3igKLigKLigKIsCiAgICAgICAgICAgIGRlcFNFPeKAouKAouKAoikgJT4lCiAgZ2dwbG90KGFlcyh4PeKAouKAouKAoikpICsKICBnZW9tX3BvaW50KGFlcyh5PeKAouKAouKAoiksY29sb3VyPSJyZWQiKSsKICBnZW9tX3BhdGgoYWVzKHk94oCi4oCi4oCiKSxjb2xvdXI9InJlZCIpKwogIGdlb21fcmliYm9uKGFlcyh5bWluPeKAouKAouKAoi3igKLigKLigKIseW1heD3igKLigKLigKIr4oCi4oCi4oCiKSxmaWxsPSJyZWQiLGFscGhhPeKAouKAouKAoikgKwogIGdlb21fcmliYm9uKGFlcyh5bWluPeKAouKAouKAoi3igKLigKLigKIseW1heD3igKLigKLigKIr4oCi4oCi4oCiKSxmaWxsPSJyZWQiLGFscGhhPeKAouKAouKAoikgKwogIGdlb21fcG9pbnQoYWVzKHk94oCi4oCi4oCiKSxjb2xvdXI9ImJsdWUiKSsKICBnZW9tX3BhdGgoYWVzKHk94oCi4oCi4oCiKSxjb2xvdXI9ImJsdWUiKSsKICBnZW9tX3JpYmJvbihhZXMoeW1pbj3igKLigKLigKIt4oCi4oCi4oCiLHltYXg94oCi4oCi4oCiK+KAouKAouKAoiksZmlsbD0iYmx1ZSIsYWxwaGE94oCi4oCi4oCiKSArCiAgZ2VvbV9yaWJib24oYWVzKHltaW494oCi4oCi4oCiLeKAouKAouKAoix5bWF4PeKAouKAouKAoivigKLigKLigKIpLGZpbGw9ImJsdWUiLGFscGhhPeKAouKAouKAoikgKwogIHRoZW1lX2J3KCkgKyB5bGFiKCJkZXB0aCBPUiBudW1iZXIgb2Ygc3RhdGlvbnMgcmVwb3J0aW5nIikKYGBgCgojIyMgWW91ciBwaWNrCgpXaGF0IGtpbmQgb2YgZ3JhcGhzIGRvIHlvdSB1c2UgbW9zdCBpbiB5b3VyIHJlc2VhcmNoPyBDYW4gd2Ugc2ltdWxhdGUgZGF0YSBhcyBhIGdyb3VwIGFuZCBsZWFybiB0byBwbG90IGl0IGluIGEgZmV3IGRpZmZlcmVudCB3YXlzPw==