Maëlle's R blog on Maëlle Salmon's personal website

More crochet/programming thoughts

Tue, 19 May 2026 00:00:00 +0000

Here I am again, writing about crochet and programming! I’ve continued creating ~~my tons of shitty stitches~~ cute creatures.

New Git analogy! The crochet lifeline

First Git/crochet analogy.

I read a great crochet book about creating your own amigurumi patterns: Créer ses propres modèles d’amigurumis au crochet by Clotilde Massot and Lise Grandjonc. One of its authors (Clotilde Massot) is a software developer who, among other things, published an octocat pattern that I bought and used… Anyway in the book they explain that when you create a pattern, you will probably have to undo your work several times. Undoing a round is easy if you have a stitch marker in the first stitch of the current/last round: you undo until you hit that stitch marker. But what about undoing several rounds? In that case, you’ll be better off if instead of using a stitch marker at the beginning of the current round, you use contrasting yarn stuck under the first stitch of each round. That yarn creates what the authors of the book call your lifeline!

Now, if that’s not a great analogy for commits and the ability to reset…

Communities of practice

If you’ve had the opportunity to attend or watch the wonderful useR! 2025 keynote talk of my rOpenSci colleague Yanina Bellini Saibene, you’ve heard of communities of practice. In her talk, Yani quoted Etienne Wenger who defined communities of practice as “groups of people who share a passion for something that they know how to do, and who interact regularly in order to learn how to do it better“. Yani mentioned her swimming team and English conversation club. Well, I found a community of practice for crochet: a stitch club at a local café! Participants meet up to crochet side by side, talking a lot about crochet: comparing yarns, exchanging tips, etc. The first time I went, we even started with a round of introductions where we said what each of us would work on during that meeting, which reminded me of rOpenSci coworking sessions where participants do exactly that.

Usefulness of seeing others’ work

I’ve worked on some patterns by Yan Schenkel a.k.a. Pica Pau. One cool aspect of the patterns is that they include a link to a gallery where anyone can upload pics of their take on each creature. So you can look at them, maybe seeing a crucial (to you) angle that’s absent from the pattern, noticing whether some “flaw” of your own project is present in others’ projects, comparing variations and picking what you prefer before you start, etc. For instance I stared at many pics of Alberto Seagull before making the legs for mine.

The usefulness of seeing others’ crocheted animals reminds me of how reading and reviewing open-source, or our colleagues’, code helps us (and LLMs, I suppose) learn how to do, or not to do, some things, how it help us refine our taste! Ironically, I do publish most of my code, but I haven’t had interest in doing that for my crocheting yet. 😸

Conclusion

In summary, I keep finding excuses to talk about crochet. My pink octocat approves!

Git worktrees are now cool

Thu, 23 Apr 2026 00:00:00 +0000

I’m not mad when current AI tools or tips¹ make Git look cool and important. Something old and stable to cling to! 🤗 In particular, it’s now trendy to have agents work on different tickets in the same project in parallel by using Git worktrees. Git worktrees are even useful for us humans, even if we should probably not work on a dozen things at the same time.

I had learnt about Git worktrees a while ago, but only viewed them as a way to open a read-only view of a past version of a repository: I have written about loading different R package versions at once with git worktree. The exercise about worktrees I added to the saperlipopette R package is very much about reading too.

Let me sum up what I learnt about Git worktrees in the last weeks…

Git worktrees are for any work, not only reading

Let me first admit I have trouble getting used to the idea of a “working tree”. What Git calls a “working tree” I call “my files” (my precious files). But knowing what a “working tree” is makes it easier to get used to a “worktree”, which is a working tree with some metadata.

You can init/clone your Git repository once and then from there sprout as many working trees as you need. Only the main one has a fully-fledged .git folder, the others have a .git file with a line like “gitdir: /path-to-the-main-worktree/.git/worktrees/current-worktree” in it.² But from any worktrees you can run any Git commands. You can for instance:

have a worktree where you fix a bug and another worktree where you write docs.
create commits and push from either one.

From reading Julia Evans’ bonus comic about Git worktrees, I learnt that an advantage of using worktrees rather than cloning the same repo several times is that it’s faster, because of the worktrees’ sharing a .git folder.

Creating a worktree for a brand-new branch is easy

If you want to create a new worktree for a new branch feat-bla, do not run

git branch -c feat-bla
git worktree add ../feat-bla feat-bla

You can simply run:

git worktree add ../feat-bla

I learnt that from reading the manual page for git worktrees, who’d have thought reading docs was useful? 🫠

Conclusion

Git worktrees are cool, and they’re not even fancy, even if your AI service or orchestrator might manage them for you. They’re sort of a… low-hanging fruit I guess. 🙊

For instance, the post “Becoming an AI-proof software engineer” by Iris Meredith states “Being an effective user of git makes software development much less painful and importantly allows you to collaborate with other developers”. It also recommends writing. 😸 ↩︎
I am very proud of having gone inside .git to prep this post. 🫡 ↩︎

Better Git diff with difftastic

Mon, 30 Mar 2026 00:00:00 +0000

I’m currently on a quest to better know and understand treesitter-based tooling for R. To make it short, treesitter is a tool for parsing code, for instance recognizing what is a function, an argument, a logical in a string of code. With tools built upon treesitter you can search, reformat, lint and fix, etc. your code. Exciting stuff, running locally and deterministically on your machine.

Speaking of “etc.”, Etienne Bacher helpfully suggested I also look at treesitter-based tooling for other languages to see what’s still missing in our ecosystem. This is how I stumbled upon difftastic by Wilfred Hughes, “a structural diff tool that understands syntax”. ✨ This means that difftastic doesn’t only compare line or “words” but actual syntax by looking at lines around the lines that changed (by default, 3), Even better, it understands R out of the box¹.

Many thanks to Etienne Bacher not only for making me discover difftastic but also for useful feedback on this post!

Installing difftastic

To install difftastic I downloaded a binary file for my system from the releases of the GitHub repository, as documented in the manual.

difftastic on two files

You can run difftastic on two files, a bit like you would use the waldo R package on two objects.

Let’s compare:

a <- gsub("bad", "good", x)

a <- stringr::str_replace(x, "bad", "good")

respectedly saved in old.R and new.R. The CLI is called difft not difftastic. I use the “inline” display rather than the two columns default in order to save horizontal space.

difft old.R new.R --display inline

We’d get to this nice looking diff:

The parentheses and "bad" and "good" arguments are ignored.

We can also get the JSON version of this diff, which is an unstable feature which usage requires setting an environment variable:

export DFT_UNSTABLE=yes
difft old.R new.R --display json

This gets us

{"aligned_lines":[[0,0],[1,1]],"chunks":[[{"lhs":{"line_number":0,"changes":[{"start":5,"end":9,"content":"gsub","highlight":"normal"},{"start":23,"end":24,"content":",","highlight":"normal"},{"start":25,"end":26,"content":"x","highlight":"normal"}]},"rhs":{"line_number":0,"changes":[{"start":5,"end":12,"content":"stringr","highlight":"normal"},{"start":12,"end":14,"content":"::","highlight":"keyword"},{"start":14,"end":25,"content":"str_replace","highlight":"normal"},{"start":26,"end":27,"content":"x","highlight":"normal"},{"start":27,"end":28,"content":",","highlight":"normal"}]}}]],"language":"R","path":"content/post/2026-03-26-difftastic/new.R","status":"changed"}

Now, none of this isn’t very useful because I would never compare files in this way… I use version control!

difftastic with Git

We can set difftastic as the external diff tool for Git globally or for the current project.

For instance with the gert R package, to set it locally:

gert::git_config_set("diff.external", "difft")

If I want to use the inline display I’d set:

gert::git_config_set("diff.external", "difft --display inline")

Then git diff will by default use difftastic. Most interestingly for me, git show --ext-diff will use difftastic. I never use git diff directly but I do look at more or less recent commits a lot.

Say I am interested in the commit that removed roxygen2’s dependency on stringi, I’ll run:

git show 7a1dd39866699a2b0a034bb15244c07698a1e2e7 --ext-diff

and get:

This isn’t spectacular because this is a small diff, but I enjoy the highlighting of the parentheses of the removed nested call, and of the logical.

Cool features of difftastic

Building on two examples of the difftastic homepage…

Ignoring formatting changes

Since formatters can so helpfully apply your formatting preferences, reviewing formatting changes in a patch that’s about something else entirely is useless and annoying. Imagine having a function definition that fits on a single line, then adding one argument to it.

Going from

f <- function(myarg1 = foo, myarg2 = bar) {}

f <- function(
  myarg1 = foo,
  myarg2 = bar,
  myarg3 = baz
) {}

Because the definition is now longer than 80 characters, your formatter might switch the definition to be on multiple lines. But the actually interesting change is the addition of one argument.

Native Git diff² would show:

Git with difftastic would show:

The matching of delimiters is why I found the difftastic’s display of the roxygen2 commit more pleasing.

Matching delimiters in wrappers

The Git diff can look a bit ugly when you simply move code from one function to the other.

Say we go from

f <- function() {
  1 + 1
}

f <- function() {
  g()
}

g <- function() {
  1 + 1
}

Git diff would show:

Whereas Git with difftastic would show:

Will I use difftastic?

I really like the concept behind difftastic and the few Git commits I looked at with it rendered nicely. Now, what’s missing for me to use difftastic a lot is its integration with the tools where I actually use Git:

Positron including the GitLens extension;
GitHub Pull Request Files tab.

In any case, I’ll continue learning about tools based on treesitter, some of which like Air and Jarl I can already use directly from my IDE. 😸

It’s not every day we R developers look at the homepage of a tool and see the R logo among the logos of other languages! ↩︎
To get the diff that Git would show me I ran git diff --no-index old-args.R new-args.R --no-ext-diff, cool trick I didn’t know about! Very glad I didn’t have to create a fake Git repo just for this. (--no-ext-diff because my diff in this repo would use difftastic by default!) ↩︎

Zut, Git ! Porting {saperlipopette} to the terminal with Claude

Fri, 27 Feb 2026 00:00:00 +0000

A while ago, I decided to learn some Rust. I bought a book, opened it months later and started a small side-project: porting my R package saperlipopette to the terminal by writing a CLI with Rust. I then lost steam and discarded the project. Now, teaching more Git with saperlipopette has me itching to do the same outside of R, to reach more learners. And in the meantime, even though I have conflicting feelings about it, I realized that using an LLM, specifically Claude Code (Opus 4.6), would help me take my CLI over the finishing line!

What my CLI already had

The CLI project already had a few exercises, all the “Oh shit, Git!” exercises from saperlipopette. It had a bad name: ohcrabgit, a pun based on “Oh crap, Git!” and on Rust’s logo being a crab. Definitely not something you want to be typing in a terminal again and again.

Its interface was the following:

You’d create an exercise in a folder by typing a command in the terminal.
You’d then open that folder, and read the file instructions.txt to know what to do, then the file tip.txt in case you needed hints.

That interface I wanted to keep, because it’s fairly simple and would work no matter what people usually use to work with Git. Surely they use Git somewhere where they can read text files.

It had some internal infrastructure: for instance a file called git.rs that had some utility functions such as init_playground or create_branch. It used the Rust crate git2 in particular, for the Git aspects.

Each line had been painstakingly written by me, with the help of a search engine and of the helpful error messages from the compiler. 😅

What I wanted for the CLI

I wasn’t so interested any more in writing Rust code myself. Maybe one day that’ll be relevant for my work, but it isn’t right now. I wanted to grow the CLI for it to match the features of saperlipopette: more exercises, and translations of the instructions/tips in French and Spanish. With that end product, I should be able to teach the same workshops as before, but using the CLI instead of saperlipopette if my audience were not R users only.

I also wanted to improve the formatting of the help, and the installation instructions.

The project went well! I have a Claude Code subscription, I call Claude from the terminal, from within Positron IDE. Note that I reached the rate limit of a session right before it was time for my Pilates class so it wasn’t too bad to just wait. 🧘

Here’s what was produced in zut source code, and reviewed by me the Rust newbie, therefore not extremely deeply:

Improvements of the help formatting. I pushed back on ANSI strings in the code, and asked for more accessible colors (not sure they’re the best choice). Commit.
I chose a better name for the CLI, the French curseword “zut” (“darn”). Claude did the renaming everywhere, which was especially lazy from me. Commit.
The installation instructions are now to run cargo install --git https://github.com/maelle/zut, suggestion by Claude, since the audience for my tool should mostly be developers. I guess I’ll see what the pain points are when people do try to install it on different platforms… Commit.
The CLI now outputs instructions and tips in different languages based on the system locale, an environment variable, or a flag. I asked for the system locale to be the default. In that same stream of work, Claude added more unit tests, which I didn’t even request, and integration tests, which I did request after seeing the tests folder was empty. Commit
All new exercises from saperlipopette were ported to zut in French, English and Spanish, and they were categorized in the help output as well as the docs (which at the moment is… the README). I tried running them, which lead to small tweaks (empty lines at the end of files) and bigger tweaks (for instance the git bisect exercise in saperlipopette has you run an R script, but it needed to become a shell script for it to make sense in zut). Commit.
A bug fix, since zut only worked… from the folder containing its source. 🤦 Commit.

The state of zut

To install zut:

cargo install --git https://github.com/maelle/zut

To view help:

zut --help

whose output is:

Create example Git messes to solve, inspired by https://ohshitgit.com/

Usage: zut <EXO>[TARGET]. In the exercise folder, open instructions.txt.

Arguments:
  <EXO>
          Name of the exercise

          Possible values:
          - time-machine:       Oh shit, I did something terribly wrong, please tell me git has a magic time machine!?!
          - small-change:       Oh shit, I committed and immediately realized I need to make one small change!
          - latest-message:     Oh shit, I need to change the message on my last commit!
          - committed-to-main:  Oh shit, I accidentally committed something to main that should have been on a brand new branch!
          - committed-to-wrong: Oh shit, I accidentally committed to the wrong branch!
          - undo-commit:        Oh shit, I need to undo a commit from like 5 commits ago!
          - undo-file:          Oh shit, I need to undo my changes to a file!
          - split-changes:      Hey I'd like to split these changes to the same file into several commits!
          - clean-dir:          Hey, how do I remove all my debugging left-over stuff at once?
          - conflict:           Hey I'd like to see what merge conflicts look like!
          - rebase-i:           Hey I'd like to make my commits in a branch look informative and smart! (interactive rebase)
          - reset:              Hey I'd like to restart from scratch and reorganize my commits! (git reset --mixed)
          - bisect:             Hey I'd like to find which commit introduced a bug!
          - log-deleted-file:   I want to find which commit deleted a file!
          - log-deleted-line:   I want to find which commit deleted a line!
          - revparse:           I want to understand ancestry references like HEAD~5 and HEAD^^!
          - blame:              I want to find who added a specific line and when!
          - worktree:           I need to see what the project looked like at a certain version!

  [TARGET]
          Where to create the exercise directory. Default: temporary directory
          
          [default: tempdir]

Options:
      --lang <LANG>
          Language (default: auto-detected from system locale)
          
          [possible values: en, fr, es]

  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Examples:

  zut small-change  creates the small-change exercise folder in a temporary folder.
  zut latest-message ..  creates the latest-message exercise folder in the parent of the current folder.

Categories:

  Oh shit, Git!:
    time-machine, small-change, latest-message, committed-to-main,
    committed-to-wrong, undo-commit, undo-file

  Clean history:
    split-changes, clean-dir, conflict, rebase-i, reset

  Use history:
    bisect, log-deleted-file, log-deleted-line, revparse, blame, worktree

To create the time machine exercise in a temporary folder:

zut time-machine

The path to the folder is printed to screen so you can open it, read instructions.txt there and get working: in that challenge a git reset --hard too many was run so you need git reflog to rescue an important commit.

To create the blame exercise in a folder above the current one, in French:

zut blame .. --lang fr

In that challenge, you need to find who committed a given line in a script. Dr Jekyll or Mr Hyde? git blame will help!

Last but not least, if you try creating an exercise somewhere where it exists:

zut blame .. --lang fr

you get:

Exercise folder already exists: ../exo-blame — delete it or choose a different target.

Conclusion: future work

Using Claude Code helped me port the features of the saperlipopette R package to the zut CLI. The next steps are for me to actually use zut more beyond the tests I did, to catch further problems. Say I need to teach “Painlessly Improve Your Git History” again, I’d do all the exercises again in a row. I would do that in the human language of the workshop.

I could also improve the code of the CLI based on user’s feedback and on, say, feedback from Claude Code using the posit-dev/critical-code-reviewer skill (thanks to Mo for telling me about it!). The documentation could also use some work, compared to saperlipopette, but as a companion to workshops it might be ok.

The saperlipopette package itself has been submitted to rOpenSci software peer-review so I expect it to change for the better. When I need to port over those changes, depending on what they are I might resort to Claude Code again.

Git commits: please mark your stitches!

Sun, 15 Feb 2026 00:00:00 +0000

In which I share a crochet analogy for Git commits…

This post was featured on the R Weekly highlights podcast hosted by Eric Nantz and Mike Thomas.

Don’t forget your stitch markers!

I am happily continuing my crochet learning journey. One lesson I have now internalised is that I should never ever forget or neglect to use stitch markers. Stitch markers are sort of small, not spiky, safety pins.

A crochet project featuring several stitch markers (beginning of a Snorlax from Lee Sartori’s Pokemon crochet book)

Stitch markers should be used:

at the beginning of a round to not forget where it started – in amigurumi you often crochet in a spiral so it’s hard to know where a round started,
every multiple of stitches to not have to count stitches above 12 even if you happen to have a PhD in counting¹. Many crochet patterns will have you, say, “increase by two stitches every 8th stitch”, then “increase by two stitches every 9th stitch” so having your markers every 9 then 10 stitches will be most helpful.

I have sometimes forgotten to place my stitch marker at the beginning of a round, and really regretted my lack of concentration. Furthermore, I used to think that adding more stitch markers was too much work (Opening the marker! Putting it under a stitch! Closing the marker! Could I be any lazier?!) which is absolutely wrong. Stitch markers bring peace of mind. They spark joy.

Don’t forget your Git commits!

Well, Git commits are just like stitch markers. Putting them regularly along your strand of work will only help you in the future. It is well worth the small annoyance of making them. With good commits, you can undo your work up to a certain point. You can understand what was done when and why.

In the saperlipopette R package I’ve recently added exercises related to using history:

“I want to find which commit deleted a file!" (git log -- <path>)
“I want to find which commit deleted a line!" (git log -S<string>, git log -G<regex>)
“I want to understand ancestry references!" (git show, git rev-parse)
“I want to find who added a specific line!" (git blame)
“I need to see what the project looked like at a certain version!" (git worktree)
An older exercise: “Hey I’d like to find which commit introduced a bug!" (git bisect)

Most of those exercises teach you the skill of obtaining a Git commit ID in answer to a question. How useful that is depends on how good the commit history is: big commits with uninformative messages are not great. If the exercises related to using the history seem relevant to you, try improving your commit history.

Note that you use conventional commits, you might even get nice colours for them, just as if you were using stitch markers! The above is due to RefinedGitHub, a browser extension for GitHub, but I’m sure similar functionality exist for other Git platforms and in IDEs.

Conclusion

In conclusion, I firmly believe good Git commits can change your life for the better, just like stitch markers. Don’t learn the lesson the hard way!

Besides, if you’re interested in learning how to recover from common Git mistakes such as a commit made on the wrong branch, consider registering to my upcoming workshop at Workshops for Ukraine: “Oops, Git! How to recover from common mistakes”, Thursday, March 19th, 14:00 - 16:00 CET (Rome, Berlin, Paris timezone). All proceeds going to support Ukraine. (After the workshop, making a donation gets you access to the recording)

Also called PhD in statistics! ↩︎

Parallel strands between learning crochet and learning programming

Mon, 26 Jan 2026 00:00:00 +0000

Thanks to Athanasia Mo Mowinckel for reviewing this post!

One goal of mine this year is to learn how to crochet amigurumi, that is to say, cute creatures. When working on my first project, a cat toy made out of yarn left-over from knitting projects, I thought of some similarities between this activity and being a beginner in programming. Here they are!

This poor bee has heart wings… mounted the wrong way so won’t be able to flee the cat it was gifted to. Unrelated excellent book in the background.

Importance of instruction quality

I bought an amigurumi kit that came with horrible instructions. Probably a bad translation to English but even assuming that, I can’t even think what crappy explanations it was translated from. 😅 I’m so grateful for the possibility to search for tutorials or videos that “click”! Even among good explanations, some of them didn’t work for me, so I guess writing yet another tutorial on something isn’t necessarily a waste of time as your own voice might unstuck someone.

Frustration is part of the process

I’ve seen a reddit thread where someone said learning how to make a magic ring, the very first step of many (all?) amigurumi patterns, made them cry. Well this intersects with finding instructions that resonate, but even so, having to frog (undo¹) your project over and over again is normal. Just like you might have to try and try again something in programming… Even beyond one’s beginner’s days actually. We have to work on our endurance.

Thanks for tech

Very much like learning can be helped by technical tools (software for webinars!), I am thankful in general for the internet and computers and cameras and in particular for tools that help creators mirror their videos or photos so that I, a left-handed crochet learner, don’t need to add to my overwhelm by having to mirror things in my head.

The importance of practice

I am getting better at crochet because of practice. There’s no magical shortcut. Just like this quote of Hadley Wickham’s:

“Periodic reminder: The only way to write good code is to write tons of shitty code first. Feeling shame about bad code stops you from getting to good code.”

I am working on my tons of shitty stitches. 😸

The importance of scrap yarn

My first kit is cheap. I even started with left-over yarn, which is even less valuable². This gave me so much freedom! In programming, I also believe in the importance of low pressure to learn. If you practice Git with my saperlipopette package, you can do so in temporary folders that are very much like scrap yarn: good for learning, fine to throw away.

Silly conventions

R starts indexing from 1, Python from 0. Well in crochet, a French maille serrée is single crochet in US patterns and a double crochet in UK patterns. 😵‍💫

So many pitfalls

Even beyond those silly conventions, there are so many pitfalls as a beginner! For some of the things I regularly do at work, I have internalized many rules and risks thanks to practice, but in crochet, I can get so overwhelmed. You have to count (harder than it sounds), you’ll be in hell if you forget to place your stitch marker at the beginning of a row, the tension is important (not too loose, not too tight), it’s hard to see what you’re doing with dark yarn, set up (the magic ring that doesn’t magically appear!) is a chore and difficult, etc. The list goes on and on. I guess they don’t say for nothing that it’s good to be a beginner: it will increase my empathy as an instructor!

Conclusion

In this post I summarized some common points between being a beginner in crochet and programming. I’d better follow my own advice and continue practicing crochet regularly. 🙂

Called frogging apparently because the sound of undoing your stitches resembles the sound of a frog? ↩︎
Even if I love that yarn because it is not made of several strands, it is a sturdy chain I can’t undo! So handy when you’re bad at orientating your crochet! ↩︎

How to open files, folders, websites in R

Wed, 25 Jun 2025 00:00:00 +0000

Coming to you from France, a post about Mise en place for R projects. In a less francophone phrasing: to get to work on something you have to open that thing, be it a script or a project or a website. The easier that is, the faster you get to work. In this post I’ll show a roundup of R functions and related tools for opening scripts, projects and websites for yourself or on behalf of the users of your code.

Many thanks to Hannah Frick for providing inspiration for some items of this post, and for reviewing it!

Open any file in the editor: `utils::file.edit()` or styling with `{cli}`

If you write code that creates a file at path and then is supposed to open it for the user, there is no need for you to use, say, rstudioapi::documentOpen(path) that only works in RStudio IDE. You can use a base R function, utils::file.edit()! It will open the path in the default editor. Without me setting up anything, that is the RStudio IDE when I’m already in RStudio and Positron when I’m already in Positron. From an R session in the terminal¹, that is the default editor of my system².

I was surprised to see that usethis::edit_file() uses an RStudio specific function when available:

# https://github.com/r-lib/usethis/blob/4aa55e72ccca131df2d98fcd84fff66724d6250a/R/edit.R#L32C1-L36C4
if (rstudio_available() && rstudioapi::hasFun("navigateToFile")) {
  rstudioapi::navigateToFile(path)
} else {
  utils::file.edit(path)
}

According to the commit that added this logic more than 7 years ago, “utils::file.edit opens in dialog” which I do not understand. Maybe it’s different depending on the OS? Maybe RStudio changed in the meantime? Please tell me if you have any more information on this. 🙏

In any case, I really enjoy file.edit() in code bases. Interactively, I do not use is as often as, say, Positron’s shortcut for navigating to files (Ctrl+P on my machine).

Last but not least, if you want to make it easy for the user to open a file, without opening it on their behalf, in messages emitted through the cli package you can use the file class:

cli::cli_alert_success("Hey go edit {.file config.toml} please!")

“If the terminal supports ANSI hyperlinks (e.g. RStudio, iTerm2, etc.), then cli creates a clickable link that opens the file in RStudio or with the default app for the file type."

Open the script or test file of a script-test pair: `usethis::use_r()` and `usethis::use_test()`

Coming back to the wonderful usethis package, when you name your test files after your scripts as you should, running usethis::use_test() when focussed on an R script will open its test file, and conversely usethis::use_r() when focussed on a test file. This is still how I switch between the two even if inside Positron.

Open a project: `positron`, project launcher, {usethis}

This is all very good when within a project, but how do you enter your IDE in the first place? How sad would it be to lose momentum before even launching a project?

When I used the RStudio IDE, I would navigate to the folder of interest, then double-click on the .Rproj file. Hannah Frick inspired me, when demoing Positron to me, to think a bit harder about how much I click. 😅 I have adopted a more keyboard-based workflow since I switched to Positron, which I could have done with RStudio IDE.

One thing in particular that Hannah showed me that looks cool is the use of a project launcher.

Now, I myself still use the terminal. For instance if I clone a repository³, I’ll then open it with the positron command. If that command does not work for you yet on macOS or Windows, you might need to add Positron to your PATH.

Another way to open projects that I use a lot, in Positron and RStudio, is to click on the recent projects when the IDE is already up and running. Unfortunately, I created and currently often use the saperlipopette package that creates throwaway folders for practicing Git. This makes the list of recent projects utter rubbish. 🙃

Within saperlipopette itself, to create and open the exercise folder on behalf of the user, I use usethis::create_project(), which handily opens it in Positron when run in Positron and in RStudio when run in RStudio.

Open an URL in the browser: `browseURL()` or styling with {cli}

Sometimes you want the user of some code of yours to go admire or read a web page.

To open an URL in the default browser, you can use utils::browseURL(). In all of the usethis::browse_ functions, utils::browseURL() is what’s used under the hood. For instance:

browseURL("https://masalmon.eu/post")

To make it easy to open an URL from a message, you can use the URL class of the cli package. For instance:

cli::cli_alert_info("Go read that blog! {.url https://masalmon.eu/post}!")

When getting that message, the user will simply have to click on the link to follow it (after trusting the domain in Positron).

Open only in interactive sessions: `interactive()`, `rlang::is_interactive()`

If you use utils::file.edit() or utils::browseURL() in your code, you need to ensure the session is interactive.

Either

if (interactive()) {
  file.edit("config.toml")
}

if (rlang::is_interactive()) {
  file.edit("config.toml")
}

Compared to interactive(), rlang::is_interactive() also checks whether knitr or testthat are in progress, and provides an escape hatch through the rlang_interactive option.

All of the usethis functions that try to open or browse something on behalf of the user behave differently based on rlang::is_interactive().

Conclusion

In this post I summarized some tools for opening scripts, projects, and URLs - for yourself or on behalf of your user. I’d like to add some words of conclusion beyond the summary table below, but after so much talk of opening, I have trouble closing this post. 😁

Target to open	Tool	Audience
File	`utils::file.edit()`	Users of your code
-	IDE’s shortcut	You – get to know your IDE
-	`.file` class of cli message	Users of your code
R script from test file	`usethis::use_r()`	You
-	IDE’s shortcut	You – get to know your IDE
Test file from R script	`usethis::use_test()`	You
-	IDE’s shortcut	You – get to know your IDE
Project (folder)	`.Rproj` file	RStudio IDE users who like clicking
-	List of recent projects within the IDE	You when the IDE is already launched
-	`rstudio`, `positron`	Terminal dwellers
-	Project launcher	Positron users on macOS
-	`usethis::create_project()`	Users of your code that created a project
URL	`utils::browseURL()`	Users of your code
-	`.url` class of cli message	Users of your code

Look how adventurous I was when preparing this post! Running R from a terminal! ↩︎
Which is… Atom, meaning I live in the past since its end-of-life occurred a few years ago. 👻 I actually rarely use it. ↩︎
I know I could use usethis::create_from_github() and sometimes do but not always… Maybe the message of this post is that my workflows are a big mess. ↩︎

Cover and modify, some tips for R package development

Tue, 24 Sep 2024 00:00:00 +0000

I’ve recently been dealing with legacy code refactoring both in theory and in practice: while I’m continuing some work on the igraph R package, I’ve started reading Working Effectively with Legacy Code by Michael Feathers, that had been in my to-read pile for months. In this post, I’ll summarize some ideas from both the book and my work.

“Cover and modify” with “characterization tests”

When you start modifying your rusty code, how do you ensure you do not break existing and important behaviour inadvertently? In his book, Michael Feathers advocates for “cover and modify” (test, then amend) rather than “edit and pray” (that you’re not breaking anyone’s script or reverse dependency). Not necessarily brand-new ideas but nice phrases. 😁

In igraph for instance, when I have to add tests for a function that does not have any test yet, I tend to use code from examples as a starting point. I run the code in an unit test and add expectations that seem relevant to me, and to collaborators more knowledgeable about graphs. Part of this work is very mechanical and probably also very common: running the code to figure out its output, codify its output as expectation. For instance if f() returns a numeric of value 0.42, I’ll add something like testthat::expect_type(output, "double") and testthat::expect_equal(output, 0.42). It turns out such tests have a name in the book, which made me very happy: they are “characterization tests”.

How to characterize the output

Three tips on how to characterize the output:

You can use typeof() and sloop::otype() to help you find out which testthat type/class expectation you should use. The latter is linked from a chapter in Hadley Wickham’s Advanced R that in turn is linked from the testthat docs. I am mildly embarrassed about even needing the former. 😂
You can use constructive::construct() to record an expected value in a more readable way than with dput() – for use in snapshot tests or even directly in testthat::expect_equal().
Michael Feathers notes that characterization tests are a form of documentation, so “you have to think about what will be important to the reader”. I try to order expectations logically, and… I request a review on my pull requests so there might actually be another reader involved in the short term. 😇

How to check coverage

To know whether you’ve covered the code you’re about to modify, you have to assess your tests.

For some (historical I suppose) reason, in Michael Feathers’ book code coverage tooling seems absent. I’ve instead read about “sensing variables” and using the debugger to step through the code.

In R when using devtools we can rely on devtools::test_coverage_active_file() (including showing you an interactive report!).¹ It’s allowed me to see some branches my tests weren’t hitting yet. Running it again and again and seeing more lines in green is very satisfying.

The code coverage might be a part of the continuous integration workflows but in the case of igraph it only runs after all other checks (so not quickly).

Conclusion

In this post I’ve presented some ideas around “covering” R code with tests before you “modify” it. Do you have any insight to share that I haven’t… covered? 😅

I first used a few lines that did the same thing (but obviously not as well) because I had forgotten about this function. Don’t do like me! ↩︎

Create and use a custom roxygen2 tag

Tue, 03 Sep 2024 00:00:00 +0000

This post contains mistakes! No need for a roclet for custom tags! Use the packages load option. Refer to the roxygen2 vignette.

This post was featured on the R Weekly highlights podcast hosted by Eric Nantz and Mike Thomas.

You might know that it’s possible to extend roxygen2 to do all sorts of cool things including but not limited to:

documenting your internal functions for developers only (that’s devtag by my cynkra colleague Antoine Fabri),
recording your following statistical software standards (that’s srr by my rOpenSci colleague Mark Padgham),
writing tests from within R scripts (that’s roxytest by Mikkel Meyer Andersen).

I did something much more basic but still challenging to me: creating a custom tag that adds a new section to a manual page. In this post I’ll summarize how to go about that.

Context: need for a custom tag in the igraph R package

The igraph R package outsources a lot of its work to the igraph C library. Sometimes, the R package docs therefore have to try and catch up with the C library docs. One idea we’ve had when thinking about the problem is to link to relevant C docs from each R manual page: if a function called igraph::empty_graph() uses igraph_empty under the hood, then its manual page should link to https://igraph.org/c/doc/igraph-Basic.html#igraph_empty. It means the users would be able to easily browse the C docs, and as a side-effect they’d be aware of what part of the C library supports what functionality in the R package.

A workflow to make that happen is the following:

have a sitemap of the C docs at hand (we parse the index page of the C docs);
add @cdocs tags in the source of the R package, for instance @cdocs igraph_empty near the function definition (adding them semi-automatically will be a topic for another day 😅);
have some roxygen2 machinery parsing the @cdocs into links in the manual page when we call document().

This blog post is about the roxygen2 machinery as I had to piece together several things I saw online.

Minimal example

Create our minimal R package

R package hgb

The hgb package has one script whose docs use a custom @custom tag:

#' A function messaging a thing
#'
#' @custom Hello I am here
#'
#' @return Nothing, print thing
#' @export
#'
#' @examples
#' thing()
thing <- function() {
  message("thing")
}

Create a package holding the custom tag

R package myroxy

In a script I have the following lines that mostly come from an roxygen2 vignette, in which the creation of a custom tag is well documented (its usage less so 😅).

There’s a method roxy_tag_parse.roxy_tag_custom() telling roxygen2 how to parse the tag, a method roxy_tag_rd.roxy_tag_custom telling roxygen2 what to do with the value: a special “custom” section. Then there’s a method format.rd_section_custom() telling roxygen2 how to build the “custom” section. Here it’s just pasting. In my real use case I defined a helper function to identify the C docs link corresponding to a given tag value.

#' @importFrom roxygen2 roxy_tag_parse
#' @importFrom roxygen2 roxy_tag_rd
NULL

#' @export
# same as https://roxygen2.r-lib.org/articles/extending.html
roxy_tag_parse.roxy_tag_custom <- function(x) {
  roxygen2::tag_markdown(x)
}

#' @export
# same as https://roxygen2.r-lib.org/articles/extending.html
roxy_tag_rd.roxy_tag_custom <- function(x, base_path, env) {
  roxygen2::rd_section("custom", x$val)
}

#' @export
# same as https://roxygen2.r-lib.org/articles/extending.html
format.rd_section_custom <- function(x, ...) {
  paste0(
    "\\section{Custom section}{\n",
    "\\itemize{\n",
    paste0("  \\item ", x$value, "\n", collapse = ""),
    "}\n",
    "}\n"
  )
}

Even if all we do is creating a custom roxygen2 tag, we need to define a roclet as it’ll be the vessel by which another package can use the custom tag. You can have a custom tag defined inside a package without a roclet if you use the custom tag within the same package. But to cross package boundaries you need a roclet.

Further in the same script I have those lines:

#' @export
# https://github.com/shahronak47/informationtag
custom_roclet <- function() {
  roxygen2::roclet("custom")
}

#' @importFrom roxygen2 block_get_tags roclet_process
#' @method roclet_process roclet_custom
#' @export
# https://github.com/shahronak47/informationtag
roclet_process.roclet_custom <- function(x, blocks, env, base_path) {
  x
}


#' @export
#' @importFrom roxygen2 block_get_tags roclet_output
roclet_output.roclet_custom <- function(x, results, base_path, ...) {
  x
}

You’ll notice we defined something called “custom_roclet” but now the methods are for “roclet_custom” (inverted word order). This confused me at first. Furthermore, I had no idea what the process and output methods were supposed to do, so I used the demo by Ronak Shah. Lastly, the roxygen2 tags used in that demo didn’t work for me so I stole the ones used by Antoine Fabri in devtag.

Build the package holding the custom tag

By that I mean running document() and installing the myroxy package on my machine to try it.

Register the roclet in our minimal package

I added the following lines to the DESCRIPTION of the hgb package to indicate to roxygen2 that there’s another package to use when documenting my package:

Roxygen: list(markdown = TRUE, roclets = c("collate", "rd", "namespace",
    "myroxy::custom_roclet"), packages = "myroxy")

I also added these lines so that fellow developers might know where to get myroxy from¹:

Config/Needs/build: roxygen2, devtools, maelle/myroxy

Try it

If all goes well, running document() in the hgb package works without any error. The manual page for ?thing has the custom section.

Do I need an external package?

The roxygen2 extensions I mentioned in the introduction (devtag, srr, roxytag) are standalone packages that you can use when building your own package, because their use case is fairly general. Here, I am developing a custom tag for igraph only. So in theory, the infrastructure could live inside the R package itself.

I actually started by implementing the custom tag within the igraph package but I was not a fan of the clutter it created in data-raw, R, the NAMESPACE, etc. It was great to start this way as it was much easier (no roclet!), but I ended up factoring out the roxygen2 code in a distinct GitHub-only package called igraph.r2cdocs.

Sources

I am very thankful to the authors of roxygen2 obviously, and to the authors of the following resources:

roxygen2 vignette “Extending roxygen2” that says of itself that it’s “very rough”. For me the missing part was understanding what goes in what package (the package where you define the tags, the package where you use the tags); and I’d have obviously liked some explanations on roclets that are just there for a basic custom tag, as opposed to printing content / saving information in other files. I want docs just for me. 😉
Stack Overflow post
Video by Ronak Shah that I didn’t watch and related repositories that I did stare at a lot: informationtag and infotagdemo. This is how I understood what the “process” and “output” methods of my roclet had to look like.
devtag source which is where I got the necessary roxygen2 tags for the “process” and “output” methods of my roclet (the ones from the aforementioned demo didn’t work for me).

Conclusion

In this post I explained the creation and usage of a custom roxygen2 tag, with a minimal example. If you’re curious about the igraph use case, find the demo PR to igraph and the igraph.r2cdocs package. Have you ever extended roxygen2? What were the difficulties in your experience?

This is a custom… DESCRIPTION field. ↩︎

Does my test really validate a bug fix? Check it with git cherry-pick

Thu, 29 Aug 2024 00:00:00 +0000

Earlier this year I wrote a post about git worktree that allows you to load different versions of an R package at once on your computer. To keep with the “juggle between versions of a codebase with Git plant-related commands” theme, let me show you how I use cherry-pick to assess the quality of an unit test.

Scenario: you fix a bug in a branch

In a perfect world, the bug you’re working on is paired with a ✨ reprex ✨, and you even start your work by adding a test case for it. The test case fails at first, then you fix the bug in the code so that the test case succeeds.

You might even push the test in a first commit, so that on continuous integration you get a failure then a success in later commits on the branch. Note that merging the commits in this order without rewriting history means that debugging with git bisect later on might be problematic, as the codebase at the commit that added the test had, well, failing tests. Thanks Kirill for that useful reminder.

Problem: you add a test after the fix, not before the fix

Now, sometimes things are less than optimal and you add the test case after the fact. So you have a commit or several ones fixing the bug, then a commit adding a passing test. Maybe there was not even a reprex so you’re not really sure what the same code snippet returned before the bug fix. How can you ensure the new test was failing before you added the fix?

Without undoing the bug fix using code comments, that is. 😉

Solution: a throwaway branch and git cherry-pick

What I’ve done a few times now (so clearly an expert workflow 😅) is

getting the hash of the commit that added the test.
returning to the main branch, for instance with gert::git_branch_checkout("main").
creating a throwaway branch, for instance with gert::git_branch_create("deleteme").
putting the test on that branch with git cherry-pick, for instance with gert::git_cherry_pick("<commit-hash>").
running the test. It is so validating when it does fail on the main branch! It’s a good test! 🎉
returning to the main (or the bug-fix) branch, for instance with gert::git_branch_checkout("main") (or gert::git_branch_checkout("bug-fix")).
deleting the throwaway branch, for instance with gert::git_branch_delete("deleteme"). 😇

Conclusion

In this post I’ve explained a small git workflow I enjoy for checking a test of a bug fix is a good test for it: failing without the bug fix, succeeding with the bug fix. I find this process a nice use case for cherry-pick, beside moving a commit that was made on the wrong branch.

Get your codebase lint-free forever with lintr

Wed, 28 Aug 2024 00:00:00 +0000

This post was featured on the R Weekly highlights podcast hosted by Eric Nantz and Mike Thomas.

Writing good code is hard. Some aspects get easier with experience – although I observe that I consistently forget some things. 🙈 Other aspects can be tackled through code review – although your reviewer’s time will be better spent on design questions than on nitpicks. 💅 Static code analysis can help with code quality.

In this post I will show how I set up packages to benefit from the lintr R package for static code analysis.

Step 1: Add a lintr configuration file

I start by creating (and .Rbuildignoring) a .lintr file at the root of my package with contents indicating I want to run all available linters (there are so many wonderful linters!).

linters: lintr::linters_with_tags(tags = NULL)
encoding: "UTF-8"

Now the hard work can start…

Step 2: Run lintr, fix or skip or configure until done

I then run lintr::lint_package() and have a look at the output. It can be quite humbling. 😅

For each marker, I can…

Fix the problem

For instance if a line is too long, I might split it into two, going from

config_lines <- brio::read_lines(<super-long-path>)

config_path <- <super-long-path>
config_lines <- brio::read_lines(config_path)

I can also change function calls from long to wide, create helper functions, etc.

I might notice some linters come up especially often which makes me vow to try and remember about them, for instance writing explicit integers (2L) rathen than implicit ones (2).

Skip a linter for a line

For some marked lines that I view as non problematic or as a false positive, I will add a nolint: <name-of-the-linter> exclusion to the specific line.

Configure lintr

I might also decide to stop using some linters, like the complicated indentation one, or to exclude whole files, like tests/testthat.R.

For instance in a package the .lintr might end up looking like this:

linters: lintr::linters_with_tags(tags = NULL, indentation_linter = NULL)
encoding: "UTF-8"
exclusions: list(
    # excluded from all lints:
    "tests/testthat.R"
    )

Repeat until a clean output

After a few (or more, let’s be honest) cycles of running lintr and responding to part of the output, lintr has nothing left to complain about! The codebase is in a good shape. 🎉

Step 3: Set up lintr on continuous integration

At this point I call usethis::use_github_action("lint-changed-files") that will download a GitHub Actions workflow for me. This workflow tells GitHub Actions to run lintr in pull requests, only for files changed in the pull request.

Having this safeguard means that I’ll never have to run step 2 ever again, as problems in new code will be flagged (and hopefully addressed) continuously. I might only have to run step 2 again because of newly released linters.

An alternative approach might be to use pre-commit to run lintr before making any commit.

Conclusion

In this post I explained how I set up lintr safeguards in packages where I decide to get serious about code quality:

first creating a basic lintr configuration file,
then running lintr, editing my code and lintr configuration over and over again until lintr marks nothing,
adding a continuous integration workflow that will run lintr on files changed by every pull request.

Thanks to lintr maintainer Michael Chirico and to all other lintr authors and contributors. 🙏

How do you use lintr? Have you heard of flint by Etienne Bacher, that does not only flags problems but also fixes them, for a subset of lintr’s linters?

Extracting names of functions defined in a script with treesitter

Thu, 18 Jul 2024 00:00:00 +0000

Coming back from a conference, we might be excited to install and try out the cool things we have heard about. I, going against the stream 🐟, decided to experiment with a tool I have not heard about last week, as I unfortunately missed Davis Vaughan’s talk about treesitter. Nonetheless, I caught the idea that treesitter is a parser of code, R code in particular. The treesitter R package uses the tree-sitter C library. There are awesome applications of treesitter among which ✨ code search for R on GitHub ✨ but I learnt to know a bit of treesitter by solving a boring use case.

My use case: I won’t copy-paste these function names

Have you noticed this nice error message you get if you try to use dplyr::across() directly?

dplyr::across()
#> Error in `dplyr::across()`:
#> ! Must only be used inside data-masking verbs like `mutate()`,
#>   `filter()`, and `group_by()`.

Kirill Müller got the idea to offer a similar behaviour to some igraph functions that exist only for use only within the square operator. Therefore I had to find all functions defined within this function (whose body has been simplified here):

`[.igraph.vs` <- function(...) {
  # some code
  bla <- function(...) {
    # some code
  }
  blop <- function(...) {
    # some code
  }
}

On the script outline I get on the right of my script in RStudio IDE¹, I can see the function names (that would be bla() and blop() in the simplified chunk above) but not copy-paste from there.

I would know how to extract them with xmlparsedata but I thought it might be an opportunity for me to have a look at treesitter. I went through different emotions as a beginner, not all of them positive 😭, but I did get the function names in the end! 💪

Get the whole tree from the script

Below, I dutifully followed the example on the package homepage:

load the R language from the treesitter.r package, as far as I understand the only language available at this point for the R package.
load the parser for that language.
read in the text.
parse the text.

A last step is getting the tree root node as node, because I will query the whole script, and you can only query nodes, not trees.

language <- treesitter.r::language()
parser <- treesitter::parser(language)
text <- brio::read_lines("/home/maelle/Documents/rigraph/R/iterators.R") |>
  paste(collapse = "\n")
tree <- treesitter::parser_parse(parser, text)
node <- treesitter::tree_root_node(tree)

Find the node of the parent function of interest

The documentation page of treesitter::query() recommends reading the documentation of tree-sitter about the query syntax. That documentation features very useful examples.

Below I am looking for “binary_operator” whose left hand side is an identifier that I capture as “name”, and whose right hand side is a “function_definition” I capture as “def”. For some reason if I did not capture “def” then I got less information about the node. 🤷 I also use a predicate: the name of the function has to be equal to “[.igraph.vs”.

To find out I was after a “function_definition”, I parsed a few lines of made-up code to study the resulting tree.

square_bracket_thing <- '
(
((binary_operator
  lhs: (identifier) @name
  rhs: (function_definition)) @def
  (#eq? @name "`[.igraph.vs`"))
  
)
'
square_bracket_query <- treesitter::query(language, square_bracket_thing)

Then I executed the query and extracted the node. In reality this took a bit more trial and error.

square_bracket_thing_captures <- treesitter::query_captures(square_bracket_query, node)
square_bracket_thing_captures
#> $name
#> [1] "def"  "name"
#> 
#> $node
#> $node[[1]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> `[.igraph.vs` <- function(x, ..., na_ok = FALSE) {
#>   args <- lazy_dots(..., .follow_symbols = FALSE)
#> 
#>   ## If indexing has no argument at all, then we still get one,
#>   ## but it is "empty", a name that is  ""
#> 
#>   ## Special case, no argument (but we might get an artificial
#>   ## empty one
#>   if (length(args) < 1 ||
#>     (length(args) == 1 && inherits(args[[1]]$expr, "name") &&
#>       as.character(args[[1]]$expr) == "")) {
#>     return(x)
#>   }
#> 
#>   ## Special case: single numeric argument
#>   if (length(args) == 1 && inherits(args[[1]]$expr, "numeric")) {
#>     res <- simple_vs_index(x, args[[1]]$expr, na_ok)
#>     return(add_vses_graph_ref(res, get_vs_graph(x)))
#>   }
#> 
#>   ## Special case: single symbol argument, no such attribute
#>   if (length(args) == 1 && inherits(args[[1]]$expr, "name")) {
#>     graph <- get_vs_graph(x)
#>     if (!(as.character(args[[1]]$expr) %in% vertex_attr_names(graph))) {
#>       res <- simple_vs_index(x, lazy_eval(args[[1]]), na_ok)
#> <truncated>
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (binary_operator [(479, 0), (637, 1)]
#>   lhs: (identifier [(479, 0), (479, 13)])
#>   operator: "<-" [(479, 14), (479, 16)]
#>   rhs: (function_definition [(479, 17), (637, 1)]
#>     name: "function" [(479, 17), (479, 25)]
#>     parameters: (parameters [(479, 25), (479, 48)]
#>       open: "(" [(479, 25), (479, 26)]
#>       parameter: (parameter [(479, 26), (479, 27)]
#>         name: (identifier [(479, 26), (479, 27)])
#>       )
#>       (comma [(479, 27), (479, 28)])
#>       parameter: (parameter [(479, 29), (479, 32)]
#>         name: (dots [(479, 29), (479, 32)])
#>       )
#>       (comma [(479, 32), (479, 33)])
#>       parameter: (parameter [(479, 34), (479, 47)]
#>         name: (identifier [(479, 34), (479, 39)])
#>         "=" [(479, 40), (479, 41)]
#>         default: (false [(479, 42), (479, 47)])
#>       )
#>       close: ")" [(479, 47), (479, 48)]
#>     )
#>     body: (braced_expression [(479, 49), (637, 1)]
#>       open: "{" [(479, 49), (479, 50)]
#>       body: (binary_operator [(480, 2), (480, 49)]
#> <truncated>
#> 
#> $node[[2]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> `[.igraph.vs`
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(479, 0), (479, 13)])
square_fn <- square_bracket_thing_captures$node[[1]]

At this point I was already very proud of my tiny win but I did not have the “children functions” yet!

Find the functions defined within the parent

I first considered creating a complicated nested query but I found no example of that. I did see someone telling StackOverflow they did a recursive query and for some reason that gave me the idea of parsing the text of the node, then look for functions in that text.

The query was simpler: looking for function definitions, only capturing the names on the left hand side.

square_tree <- treesitter::parser_parse(parser, treesitter::node_text(square_fn))
square_node <- treesitter::tree_root_node(square_tree)
kiddos_source <- '
(
(binary_operator
  lhs: (identifier) @name
  rhs: (function_definition))
)
'
kiddos_query <- treesitter::query(language, kiddos_source)
square_bracket_thing_captures <- treesitter::query_captures(kiddos_query, square_node)
head(square_bracket_thing_captures)
#> $name
#>  [1] "name" "name" "name" "name" "name" "name" "name" "name" "name" "name"
#> [11] "name" "name" "name" "name"
#> 
#> $node
#> $node[[1]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> `[.igraph.vs`
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(0, 0), (0, 13)])
#> 
#> $node[[2]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> .nei
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(29, 2), (29, 6)])
#> 
#> $node[[3]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> nei
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(50, 2), (50, 5)])
#> 
#> $node[[4]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> .innei
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(53, 2), (53, 8)])
#> 
#> $node[[5]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> innei
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(56, 2), (56, 7)])
#> 
#> $node[[6]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> .outnei
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(59, 2), (59, 9)])
#> 
#> $node[[7]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> outnei
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(62, 2), (62, 8)])
#> 
#> $node[[8]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> .inc
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(65, 2), (65, 6)])
#> 
#> $node[[9]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> inc
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(78, 2), (78, 5)])
#> 
#> $node[[10]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> adj
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(81, 2), (81, 5)])
#> 
#> $node[[11]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> .from
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(84, 2), (84, 7)])
#> 
#> $node[[12]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> from
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(96, 2), (96, 6)])
#> 
#> $node[[13]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> .to
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(99, 2), (99, 5)])
#> 
#> $node[[14]]
#> <tree_sitter_node>
#> 
#> ── Text ────────────────────────────────────────────────────────────────────────
#> to
#> 
#> ── S-Expression ────────────────────────────────────────────────────────────────
#> (identifier [(111, 2), (111, 4)])

After that I was able to get the names of the children functions! I was actually only interested in those whose names start with a dot as all the other ones are deprecated anyway.

kiddos_functions <- purrr::map_chr(square_bracket_thing_captures$node, treesitter::node_text)
kiddos_functions[startsWith(kiddos_functions, ".")]
#> [1] ".nei"    ".innei"  ".outnei" ".inc"    ".from"   ".to"

Tada! Now I can go fix the issue I was tasked with.

Conclusion

In this post I report on my first encounter with the treesitter package for parsing R code. Copy-pasting the few function names would surely have been faster, but sometimes you’ve got to sit and learn something new. ☺️

No, I have not installed Positron yet. ↩︎

Extracting all links from my slidedeck

Tue, 16 Jul 2024 00:00:00 +0000

Last week after my useR! talk, someone I had met at the R-Ladies dinner asked me for a list of all the links in my slides. I said I’d prepare it, not because I’m a nice person, but because I knew it’d be an use case where the great tinkr package would shine! 😈

What is tinkr?

tinkr is an R package I created, and that its current maintainer Zhian Kamvar took much further that I’d ever would have. tinkr can transform Markdown into XML and back.

Under the hood, tinkr uses

commonmark for the Markdown-to-XML conversion. CommonMark, in the form of its cmark implementation, is the C library that GitHub for instance uses to display your Markdown comments as HTML. The commonmark package is also what powers Markdown support in roxygen2.
xslt for the XML-to-Markdown conversion. XSLT is a templating language for XML.

Anyway, enough said, let’s go back to today’s use case.

Extract and format links from `index.qmd`

With tinkr I can use XPath, the query language for XML or HTML, to extract links from my slidedeck source. Then I will format them as a list.

First, I create a yarn object from my slidedeck source.

talk_yarn <- tinkr::yarn$new("/home/maelle/Documents/conferences/user2024/index.qmd")
talk_yarn
#> <yarn>
#>   Public:
#>     add_md: function (md, where = 0L) 
#>     body: xml_document, xml_node
#>     clone: function (deep = FALSE) 
#>     get_protected: function (type = NULL) 
#>     head: function (n = 6L, stylesheet_path = stylesheet()) 
#>     initialize: function (path = NULL, encoding = "UTF-8", sourcepos = FALSE, 
#>     md_vec: function (xpath = NULL, stylesheet_path = stylesheet()) 
#>     ns: http://commonmark.org/xml/1.0
#>     path: /home/maelle/Documents/conferences/user2024/index.qmd
#>     protect_curly: function () 
#>     protect_math: function () 
#>     protect_unescaped: function () 
#>     reset: function () 
#>     show: function (lines = TRUE, stylesheet_path = stylesheet()) 
#>     tail: function (n = 6L, stylesheet_path = stylesheet()) 
#>     write: function (path = NULL, stylesheet_path = stylesheet()) 
#>     yaml: --- format:   revealjs:       highlight-style: a11y      ...
#>   Private:
#>     encoding: UTF-8
#>     md_lines: function (path = NULL, stylesheet = NULL) 
#>     sourcepos: FALSE

Then I extract all links.

links <- xml2::xml_find_all(
  talk_yarn$body, 
  xpath = ".//md:link",
  ns = talk_yarn$ns
)
head(links)
#> {xml_nodeset (6)}
#> [1] <link destination="https://user-maelle.netlify.app" title="">\n  <text xm ...
#> [2] <link destination="https://www.pexels.com/photo/old-cargo-ship-on-sea-207 ...
#> [3] <link destination="https://www.pexels.com/photo/the-word-louise-is-spelle ...
#> [4] <link destination="https://www.pexels.com/photo/gray-rotary-telephone-on- ...
#> [5] <link destination="https://www.pexels.com/photo/close-up-photography-of-y ...
#> [6] <link destination="https://www.r-consortium.org/all-projects/call-for-pro ...

I then throw away the links to the great website Pexels, because these are image credits rather than information useful to do R stuff.

links <- purrr::discard(
  links, 
  \(x) startsWith(xml2::xml_attr(x, "destination"), "https://www.pexels")
)
head(links)
#> {xml_nodeset (6)}
#> [1] <link destination="https://user-maelle.netlify.app" title="">\n  <text xm ...
#> [2] <link destination="https://www.r-consortium.org/all-projects/call-for-pro ...
#> [3] <link destination="https://www.r-consortium.org/all-projects/call-for-pro ...
#> [4] <link destination="https://www.heltweg.org/posts/who-wrote-this-shit/" ti ...
#> [5] <link destination="https://fosstodon.org/@hadleywickham/11202130903588421 ...
#> [6] <link destination="https://nostarch.com/kill-it-fire" title="">\n  <text  ...

After that I can format the links and display them here using an “asis” chunk. Yes, my slidedeck uses Quarto but this blog is still powered by R Markdown, hugodown to be precise.

I’m using the formatting as an opportunity to only keep distinct links: sometimes I had very similar slides in a row, with repeated information.

format_link <- function(link) {
  url <- xml2::xml_attr(link, "destination")
  text <- xml2::xml_text(link)
  sprintf("* [%s](%s)", text, url)
}

formatted_links <- purrr::map_chr(links, format_link)

formatted_links <- unique(formatted_links)

formatted_links |>
  paste(collapse = "\n") |>
  cat()

Conclusion

Using tinkr, XPath and sprintf(), I was able to create a list of all the links shared in my useR! slidedeck. Some of them have no text, meaning that the URL is used as text for the link; or text that only makes sense in the context of the paragraph they were a part of; others are slightly more informative; but at least none of them is a “click here” link. 😅

Hack your way to a good Git history

Tue, 11 Jun 2024 00:00:00 +0000

I’ve now explained on this blog why it’s important to have small, informative Git commits¹ and how I’ve realized that polishing history can happen in a second phase of work in a branch. However, I’ve more or less glossed over how to craft the history in a branch once you’re done with the work.

I’ve entitled this post “Hack your way to a good Git history” because writing the history after the fact can feel like cheating, but it’s not!

To branch or not to branch

With the second method of re-writing history I’ll present, you could get away with working on the default branch as long as you do not push. However, I’d really recommend creating a branch before you start working on whatever task you’ve assigned yourself, be it a bug fix, new feature or some refactoring. It feels safer, and it’s a good habit to take.

On the default branch, you can’t force push, whereas in your own branch, you can push regularly as your work, as a backup of sorts, then rewrite history, then force push.

Now let’s assume you did a bunch of changes in a branch: adding a bug fix to a script, as well as refactoring part of it, adding a test to ensure the bug never reappears, updating the changelog². You have a dozen of commits because actually fixing the commit took more than one try, and you first got your test wrong so it took you 3 commits.

“Squash and merge”: click the right GitHub/GitLab button

If you’re fine with all your changes being reduced to a single commit, you can squash merge all the changes from your branch into a single commit on the default branch.

--squash is an option of git merge but to be honest, I always merge branches through the GitHub pull requests interface. You can easily perform a “squash and merge” on GitHub and GitLab.

It’s really the easiest method to get a good Git history that does not include your desperate intermediary steps, but it does not allow for more granularity than one commit. It also means that if someone reviews your pull request, they do it without the benefit of a story in the form of an informative succession of Git commits.

“The repeated amend"™️: `git commit --amend`

Thanks to Jenny Bryan for describing this workflow and for pointing it to me.

This second method comes from the excellent Happy Git and GitHub for the useR by Jenny Bryan. The idea is than when fixing a bug in your script for instance, you make a first change, create a commit with the message “fix: fix indentation bug” or so, then do more changes and amend the commit with git commit --amend instead of creating a new one.

This way, you ‘build up a “good” commit gradually’.

You can use this method on the default branch, if you don’t push before the commit is really done. If you use this method on a branch of yours, you can force push. I really enjoy this method, when writing blog posts for instance!

This method does allow for granularity, as you can first “repeat amend” a refactoring commit, then “repeat amend” a bug fix commit, then “repeat amend” a new test commit, etc.

You can practice git commit --amend with those exercises of the saperlipopette package: “Oh shit, I committed and immediately realized I need to make one small change!" and “Oh shit, I need to change the message on my last commit!".

“Start from scratch”: `git reset --soft` + `git add` (`--patch`)

Thanks to Hugo Gruson for introducing me to this workflow.

So, you have all the changes in three places (R script, test file, NEWS.md) and a dozen commits, let’s say 12.

With this method you first reset the Git history to how it was before it started, without removing the changes to the files: git reset --soft HEAD~12. Now you will gradually create the 4 commits you want:

“refactor: stop using useless comments”
“fix: fix indentation bug”
“test: add test for indentation bug”
“docs: update changelog”

For the first commit, you run git add --patch R/script.R to only pick the refactoring changes you made to the script, then git commit -m "refactor: stop using useless comments". For the other commits, you should be able to use git add without the patch option.

Then, you force push to your branch (not the default branch!).

You can practice git add --patch with this exercise of the saperlipopette package: “Hey I’d like to split these changes to the same file into several commits!".

“Mix and match your commits”: `git rebase -i`

With this method, instead of throwing away the commits, you will be able to combine them into a commit (squash), change their message (reword), edit them (edit), delete them, change their order.

Say you start from

fix: fix indentation bug
try again
ok like this?!!
pff so hard
found the actual bug!
ah no
found it
refactor: stop using useless comments
test: add test for indentation bug
oops, forgot a quote
actually follow our own standards
docs: update changelog

git rebase -i will get you to something like:

pick <hash> fix: fix indentation bug
pick <hash> try again
pick <hash> ok like this?!!
pick <hash> pff so hard
pick <hash> found the actual bug!
pick <hash> ah no
pick <hash> found it
pick <hash> refactor: stop using useless comments
pick <hash> test: add test for indentation bug
pick <hash> oops, forgot a quote
pick <hash> actually follow our own standards
pick <hash> docs: update changelog

That you edit to:

pick <hash> refactor: stop using useless comments
pick <hash> fix: fix indentation bug
squash <hash> try again
squash <hash> ok like this?!!
squash <hash> pff so hard
squash <hash> found the actual bug!
squash <hash> ah no
squash <hash> found it
pick <hash> test: add test for indentation bug
squash <hash> oops, forgot a quote
squash <hash> actually follow our own standards
pick <hash> docs: update changelog

to get your history to:

refactor: stop using useless comments
fix: fix indentation bug
test: add test for indentation bug
docs: update changelog

I wrote git rebase -i but it actually needs an argument: the commit from which to rebase. Usually I run

git merge-base my-branch main

to get that commit and copy-paste it to my next command. In theory I could count commits and use something à la HEAD~n but for some reason it makes me more nervous.

After that, you can force push to your branch.

You can practice git rebase -i with this exercise of the saperlipopette package: “Hey I’d like to make my commits in a branch look informative and smart!".

Beside practicing, a really important thing is to configure your Git editor. I do not remember how I did it but now my Git editor is Atom, which I find easier to deal with than Vim³.

Do not miss Julia Evans’ rules for rebasing. One of them is to not do more than one thing (for instance not reordering and squashing like we did above 😅), you can instead do several git rebase -i in a row!

How to merge if you have >1 nice commits

If your branch features more than one nice commits and you don’t squash it, then, you can rebase and merge it so that no merge commits is created. This way, your default branch has a nice readable history.

Conclusion

In this post, I explained everything I know about creating a good Git history. Please do not look at the commits I end up pushing to default branches, I’m still learning. 😁 I’m doing my best to rewrite history as often as I can, to follow Martin Fowler’s principle “if it hurts, do it more often”.

I should also have added “atomic”, because if you can’t load your project at a given commit because of a missing parenthesis, then you can’t run Git bisect for instance. ↩︎
If you find updating the changelog manually is a bit of a faff, check out the fledge package. ↩︎
Although I suppose I could one day try and get better at Vim… Baby steps… ↩︎

Why you need small, informative Git commits

Mon, 03 Jun 2024 00:00:00 +0000

This post was featured on the R Weekly highlights podcast hosted by Eric Nantz and Mike Thomas.

“Make small Git commits with informative messages” is a piece of advice we hear a lot when learning Git. That’s why we might want to sometimes rewrite history in a branch. In this post, I’d like to underline three main (😉) reasons why you’ll be happy you, or someone else, made small and informative Git commits in a codebase.

A disclaimer: these three reasons are only valid if you do not write perfectly working and readable code all the time. If you do, you won’t need to use your Git log for debugging and undoing purposes, so who cares about your Git commits?

A mysterious line of code (Git blame)

Imagine you encounter a mysterious line of code in a script that either someone else or yourself wrote a while ago.

x <- x - 1

As luck will have it, there is no comment explaining the line around it. How can you guess the intent of that line?

A Git command might help you: Git blame! It’d better be called Git explain, as blaming is a bad idea. Anyway, it’s a tool for seeing the code with more context. For each line, you get to see when it was added or last modified, by whom, and when.

For instance, I added or modified the 2 lines at the top and the the two lines in the middle, that are highlighted in yellow, a few years ago. The line that is interesting to me, x <- x - 1, was added by ropenscibot in 2017.

Now, I can try and display more information about the Git commit that last modified, or added that mysterious line. The person who last touch the line chose our adventure…

If we’re unlucky, we get a Git commit whose message is ‘“Commit a bunch of files before lunch 🍝”’ and whose diff shows 145 changed files with 2,624 additions and 2,209 deletions. Not exactly helpful, we’re still stuck.
If we’re lucky, we get a Git commit whose message is ‘“fix: adapt code to tool’s 0-indexing”’ and whose diff shows 2 changed files with 3 additions and 2 deletions. Now we know why x was modified this way! We can add an explaining comment to make the code more readable for later, and continue with our work.

Git blame is a built-in Git thing, but for me, it’s first and foremost a button in the Github interface for any file. Use Git blame as you prefer.

A bad idea 7 commits ago (Git revert)

Oh no, that idea from 7 commits ago is bad! Do we

Manually remove the change;
Simply revert the commit that added the change?

Once again, the quality and granularity of the commits will decide for you. If the snapshot from 7 commits ago included many different unrelated changes, it will be easier to manually undo the changes yourself by going into the files and deleting or amending stuff. If the snapshot from 7 commits ago was small enough, you can use Git revert with that commit ID, that will create a new commit undoing that commit.

You can try out Git revert in an exercise of the saperlipopette package.

All was well 3 days ago, now my thing is broken

Imagine you were doing something with your somewhat complex R project 3 days ago, and that all was well.

Now, after a bunch of changes to your still complex R project, you set out to do the same thing, and… it no longer works.

To add insult to injury, your usual debugging tools aren’t helping you for some reasons. Is all hope lost? No!

Git has a tool to help you go over the Git history between now and then (3 days ago, 100 commits ago) in an optimal way: Git bisect. Git bisect will make you try some commits to help pinpoint which change, which commit exactly, introduced the bug you’re now observing.

Your Git history looks something like this:

a commit at which you know the thing worked,
a commit at which it no longer works,
with commits in between about which you have no information because you were not doing or testing the thing, so at some point you introduced a bug.

You tell Git bisect which commit was for sure good, and which commit is bad, and it starts putting your project in the state it was at a given commit, let’s say the third commit:

You have to tell Git bisect whether that commit is good or bad (no complex vocabulary for once). Let’s say your thing seems to work at that point.

Git bisect will assume all commits up to that commit were good.

It now makes you explore another commit.

You try your thing, it does not work. Therefore this commit, and the following ones, are bad.

You’re in luck, after only a few steps you get to the last commit to try out.

Again a bad one, your thing does not work for your project in that state.

It’s the first bad commit, so it’s the one that introduced the breaking change!

Now, again, the person who made that commit chose our adventure:

If we’re unlucky, we get a Git commit whose message is ‘“Commit a bunch of files before workout 💪”’ and whose diff shows 145 changed files with 2,624 additions and 2,209 deletions. Nooo, that’s still a ton of code to read.
If we’re lucky, we get a Git commit whose message is ‘“refactor: start using YAML”’ and whose diff shows 2 changed files with 3 additions and 2 deletions. Ah, YAML, what an idea! We can now search where we forgot to add quotes in a YAML file. 😉

You can practice Git bisect in an exercise of the saperlipopette package.

Two bonus reasons

Easier review

Small commits will be easier to review by another human. Ideally, it means the code that ends up in the codebase is better, so that there’ll be less future need of debugging and undoing. 😁

Better automatic changelog generation

If you use a tool like the R package fledge for generating your changelog or changelog draft, that uses Git commits as input data… you’ll get a better starting changelog if your commits are good.

Conclusion

In this post I showed three main reasons for making small and informative Git commits, all related to debugging or undoing some bad changes. So, creating small, informative Git commits will help someone in the future in trying times. I hope it can inspire myself in particular to pay attention to the Git history I create. 😁

The saperlipopette package contains a few exercises that can help write a better Git history, including: “Oh shit, I committed and immediately realized I need to make one small change!", “Hey I’d like to split these changes to the same file into several commits!", “Hey I’d like to make my commits in a branch look informative and smart!"

What I edit when refactoring a test file

Thu, 23 May 2024 00:00:00 +0000

This post was featured on the R Weekly highlights podcast hosted by Eric Nantz and Mike Thomas.

I’m currently refactoring test files in a package. Beside some automatic refactoring, I am also manually updating lines of code. Here are some tips (or pet peeves, based on how I look at it / how tired I am 😁)

Prequel: please read the R packages book

The new edition of the R Packages book by Hadley Wickham and Jenny Bryan features three chapters on testing, all well worth a read. The “High-level principles for testing” in particular are good to keep in mind.

Use informative variable or function names

If your test is

test_that("function_maelle_does_not_know_well() works", {
  q <- function_maelle_does_not_know_well(1:10)
  ab <- function(x) {
    y <- exp(x) + 42 - 7
    round(log(y / 50))
}
  expect_equal(q, ab(6))
})

you might want to rename q and ab to something more informative. Note that this is a fictional example, it does not make any mathematical sense.

test_that("function_maelle_does_not_know_well() works", {
  transformed_vector <- function_maelle_does_not_know_well(1:10)
  crude_transformation_approximation <- function(x) {
    y <- exp(x) + 42 - 7
    round(log(y / 50))
}
  expect_equal(transformed_vector, crude_transformation_approximation(6))
})

Imagine it will be me reading your test file, and I know nothing about your use case. More seriously, that’s one of the numerous aspects code review can help with.

Vary variable names within tests

To prevent copy-pasting mistakes and reading fatigue,

test_that("awesome_function() works", {
  x <- awesome_function(1:10, method = "easy")
  expect_typeof(x, "double")
  expect_gt(x, 5)
  
  x <- awesome_function(1:10, method = "medium")
  expect_typeof(x, "double")
  expect_gt(x, 15)
  
  x <- awesome_function(1:10, method = "hard")
  expect_typeof(x, "double")
  expect_gt(x, 25)
  
})

is in my opinion better as

test_that("awesome_function() works", {
  easy_output <- awesome_function(1:10, method = "easy")
  expect_typeof(easy_output, "double")
  expect_gt(easy_output, 5)
  
  medium_output <- awesome_function(1:10, method = "medium")
  expect_typeof(medium_output, "double")
  expect_gt(medium_output, 15)
  
  hard_output <- awesome_function(1:10, method = "hard")
  expect_typeof(hard_output, "double")
  expect_gt(hard_output, 25)
  
})

I do not hate it when several tests use the same variable names though.

Imagine I made the code chunk from above

test_that("awesome_function() works", {
  easy_output <- awesome_function(1:10, method = "easy")
  medium_output <- awesome_function(1:10, method = "medium")
  hard_output <- awesome_function(1:10, method = "hard")
  
  expect_typeof(easy_output, "double")
  expect_gt(easy_output, 5)
  expect_typeof(medium_output, "double")
  expect_gt(medium_output, 15)
  expect_typeof(hard_output, "double")
  expect_gt(hard_output, 25)
  
})

I much prefer the version

test_that("awesome_function() works", {
  easy_output <- awesome_function(1:10, method = "easy")
  expect_typeof(easy_output, "double")
  expect_gt(easy_output, 5)
  
  medium_output <- awesome_function(1:10, method = "medium")
  expect_typeof(medium_output, "double")
  expect_gt(medium_output, 15)
  
  hard_output <- awesome_function(1:10, method = "hard")
  expect_typeof(hard_output, "double")
  expect_gt(hard_output, 25)
  
})

because reading the expectations doesn’t require me to go too far back or to remember too much.

In some cases, if the object creation is simple, and it is used in one expectation only, why not put it inside the expectation rather than having to assign it to an object with an informative name?

the_log <- log(x)
expect_gt(the_log, 1)

vs.

expect_gt(log(x), 1)

Split the tests as needed, navigate them seamlessly

Splitting the tests (into several test_that()) means each one is more focused, easier to read, easier to investigate when failing (tests tend to fail).

In RStudio IDE, each test_that() calls is shown in the script outline (the table of contents on the right) so you can hop from one to the other at will.

Use a bug summary, not number, as test name

Instead of

test_that("Bug #666 is fixed", {
  ...
})

I prefer

test_that("awesome_function() does not remove the ordering", {
  # <link-to-bug-report>
  ...
})

because:

I can understand what the test is about without clicking the link;
If your repository moved for instance from a monorepo to a regular repository, the hyperlink added by RStudio IDE might no longer be valid.

Conclusion

In this post I summarized my testing preferences du jour. Do you have any tips for refactoring test scripts?

Automate code refactoring with {xmlparsedata} and {brio}

Wed, 15 May 2024 00:00:00 +0000

Once again a post praising XML. 😇 These are notes from a quite particular use case: what if you want to replace the usage of a function with another one in many scripts, without manual edits and without touching lines that do not contain a call to replace?

The real life example that inspired this post is the replacement of all calls to expect_that(..., equals(...)), like expect_that(a, equals(1)), in igraph tests with expect_equal(). If you’re a newer package developer who grew up with testthat’s third edition, you’ve probably never heard of that cutesy old-school testing style. 😉

Why automate? Where I subjectively justify my choice

As brilliantly explained by XKCD 1205, automation is not necessary worth the time. In the case that motivated this post, automation was worth it because there were many test files, and because being able to regenerate all edits meant we can recreate the changes after merging other edits to the main branch, without any conflict.

Parse the code to XML, detect problematic calls

For any path, we detect function calls to expect_that(). The code is parsed using the parse() function, digested into XML with {xmlparsedata}.

xml <- path |>
  parse(keep.source = TRUE) |>
  xmlparsedata::xml_parse_data(pretty = TRUE) |>
  xml2::read_xml()

deprecated <- xml2::xml_find_all(
  xml,
  ".//SYMBOL_FUNCTION_CALL[text()='expect_that']"
)

The deprecated object contains all the nodes we need to amend.

For info, here’s how code parsed to XML looks like (yes, it is big despite representing two short lines of code):

parse(text = "1+1\nsum(c(2,2))", keep.source = TRUE) |>
  xmlparsedata::xml_parse_data(pretty = TRUE) |>
  cat()
#> <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
#> <exprlist>
#>   <expr line1="1" col1="1" line2="1" col2="3" start="13" end="15">
#>     <expr line1="1" col1="1" line2="1" col2="1" start="13" end="13">
#>       <NUM_CONST line1="1" col1="1" line2="1" col2="1" start="13" end="13">1</NUM_CONST>
#>     </expr>
#>     <OP-PLUS line1="1" col1="2" line2="1" col2="2" start="14" end="14">+</OP-PLUS>
#>     <expr line1="1" col1="3" line2="1" col2="3" start="15" end="15">
#>       <NUM_CONST line1="1" col1="3" line2="1" col2="3" start="15" end="15">1</NUM_CONST>
#>     </expr>
#>   </expr>
#>   <expr line1="2" col1="1" line2="2" col2="11" start="25" end="35">
#>     <expr line1="2" col1="1" line2="2" col2="3" start="25" end="27">
#>       <SYMBOL_FUNCTION_CALL line1="2" col1="1" line2="2" col2="3" start="25" end="27">sum</SYMBOL_FUNCTION_CALL>
#>     </expr>
#>     <OP-LEFT-PAREN line1="2" col1="4" line2="2" col2="4" start="28" end="28">(</OP-LEFT-PAREN>
#>     <expr line1="2" col1="5" line2="2" col2="10" start="29" end="34">
#>       <expr line1="2" col1="5" line2="2" col2="5" start="29" end="29">
#>         <SYMBOL_FUNCTION_CALL line1="2" col1="5" line2="2" col2="5" start="29" end="29">c</SYMBOL_FUNCTION_CALL>
#>       </expr>
#>       <OP-LEFT-PAREN line1="2" col1="6" line2="2" col2="6" start="30" end="30">(</OP-LEFT-PAREN>
#>       <expr line1="2" col1="7" line2="2" col2="7" start="31" end="31">
#>         <NUM_CONST line1="2" col1="7" line2="2" col2="7" start="31" end="31">2</NUM_CONST>
#>       </expr>
#>       <OP-COMMA line1="2" col1="8" line2="2" col2="8" start="32" end="32">,</OP-COMMA>
#>       <expr line1="2" col1="9" line2="2" col2="9" start="33" end="33">
#>         <NUM_CONST line1="2" col1="9" line2="2" col2="9" start="33" end="33">2</NUM_CONST>
#>       </expr>
#>       <OP-RIGHT-PAREN line1="2" col1="10" line2="2" col2="10" start="34" end="34">)</OP-RIGHT-PAREN>
#>     </expr>
#>     <OP-RIGHT-PAREN line1="2" col1="11" line2="2" col2="11" start="35" end="35">)</OP-RIGHT-PAREN>
#>   </expr>
#> </exprlist>

Fix problematic calls in the XML representation

The treat_deprecated() function below tries to find a call to equals() inside the expect_equal(), since we only fix the calls to expect_that() that contain equals(). We return early for these other cutesy expectations, with a warning so we can go look at the scripts and get an idea of what the calls are. They will have to be fixed with another script, or manually, depending on how many of them there are.

For the calls to expect_that() that contain a call to equals(), we

replace expect_that() with expect_equal()
extract the text inside equals() to put it directly as second argument of expect_equal().

Thus expect_that(a, equals(1)) becomes expect_equals(a, 1).

treat_deprecated <- function(xml, path) {
  siblings <- xml2::xml_parent(xml) |> xml2::xml_siblings()
  equal <- siblings[grepl("equals\\(", xml2::xml_text(siblings))]
  if (length(equal) == 0) {
    cli::cli_alert_warning("WARNING AT {path}.")
    return()
  }
  xml2::xml_text(xml) <- "expect_equal"
  text <- xml2::xml_contents(equal)[[3]] |> xml2::xml_text()
  xml2::xml_remove(xml2::xml_contents(equal))
  xml2::xml_text(equal) <- text
}

Serialize XML to character, write back

We only modify the lines of the script that need to be modified, as it will avoid spurious changes but also avoid figuring out how to serialize the whole HTML.

For each call that was edited in XML, we identify the corresponding start and end lines in the original script. Below is again an example of parsing just to show that each expression has attributes called line1 and line2, the start and end lines.

parse(text = "1+1\n2+2", keep.source = TRUE) |>
    xmlparsedata::xml_parse_data(pretty = TRUE) |>
    xml2::read_xml()
#> {xml_document}
#> <exprlist>
#> [1] <expr line1="1" col1="1" line2="1" col2="3" start="5" end="7">\n  <expr l ...
#> [2] <expr line1="2" col1="1" line2="2" col2="3" start="9" end="11">\n  <expr  ...

There are two cases:

the start and end line is the same. We replace the corresponding line with the text of the grand-parent node.
the start and end lines are different. We loop over them, for each of them we replace the corresponding line with the text of parent and uncles/aunts nodes.

The choice of parents/siblings might seem a bit arbitrary. I made it work by putting a browser() in my code and figuring out what level of ancestry I had to deal with thanks to random tries. I do not have a particularly good mental model of R code as XML. 😉

For some reason I wrote for loops in the code below, probably because that’s what made sense to me at the time. 🤷

lines <- brio::read_lines(path)

# ...
# code identifying deprecated calls
# ...

purrr::walk(deprecated, treat_deprecated, path = path)

for (deprecated_call in deprecated) {
  
  parent <- xml2::xml_parent(xml2::xml_parent(deprecated_call))
  
  line1 <- as.numeric(xml2::xml_attr(parent, "line1"))
  line2 <- as.numeric(xml2::xml_attr(parent, "line2"))
  
  if (line1 == line2) {
    lines[line1] <- xml2::xml_text(parent)
  } else {
    for (line in line1:line2) {
      siblings <- xml2::xml_children(parent)
      lines[line] <- paste(
        xml2::xml_text(siblings[xml2::xml_attr(siblings, "line1") == line]),
        collapse = ""
      )
    }
  }
  
  
}

brio::write_lines(lines, path)

After all this, we write the lines, modified and not, to the original path. It’s important to first try this on a script and check the diff.

Put it all together

Here’s the all script, including automatic commit generation.

parse_script <- function(path) {

  cli::cli_alert_info("Refactoring {path}.")

  lines <- brio::read_lines(path)

  xml <- path |>
    parse(keep.source = TRUE) |>
    xmlparsedata::xml_parse_data(pretty = TRUE) |>
    xml2::read_xml()

  deprecated <- xml2::xml_find_all(
    xml,
    ".//SYMBOL_FUNCTION_CALL[text()='expect_that']"
  )

  purrr::walk(deprecated, treat_deprecated, path = path)

  for (deprecated_call in deprecated) {

    parent <- xml2::xml_parent(xml2::xml_parent(deprecated_call))

    line1 <- as.numeric(xml2::xml_attr(parent, "line1"))
    line2 <- as.numeric(xml2::xml_attr(parent, "line2"))

    if (line1 == line2) {
      lines[line1] <- xml2::xml_text(parent)
    } else {
      for (line in line1:line2) {
        siblings <- xml2::xml_children(parent)
        lines[line] <- paste(
          xml2::xml_text(siblings[xml2::xml_attr(siblings, "line1") == line]),
          collapse = ""
        )
      }
    }


  }

  brio::write_lines(lines, path)

  if (! (path %in% gert::git_status()[["file"]])) {
    return(invisible(TRUE))
  }

  styler::style_file(path)

  gert::git_add(path)
  gert::git_commit(
    sprintf("refactor: remove deprecated expect_that() from %s", fs::path_file(path))
  )
}

treat_deprecated <- function(xml, path) {
  siblings <- xml2::xml_parent(xml) |> xml2::xml_siblings()
  equal <- siblings[grepl("equals\\(", xml2::xml_text(siblings))]
  if (length(equal) == 0) {
    cli::cli_alert_warning("WARNING AT {path}.")
    return()
  }
  xml2::xml_text(xml) <- "expect_equal"
  text <- xml2::xml_contents(equal)[[3]] |> xml2::xml_text()
  xml2::xml_remove(xml2::xml_contents(equal))
  xml2::xml_text(equal) <- text
}

paths <- fs::dir_ls("tests/testthat", regex = "test-")

purrr::walk(paths, parse_script)

Example PR

Conclusion

In this post I presented a strategy that served me well when refactoring igraph’s test scripts: parsing code to XML, editing it as XML, then writing back the edited lines thanks to the attributes of XML nodes that indicate their original lines in the script.

Other possible approaches include styler’s parsing of code into a table and serialization of that table.

In a more similar approach, which means it might have been wise for me to explore this codebase sooner 😅, the codegrip package uses xmlparsedata and has helpers for finding which lines a node refers to.

Do you sometimes use automatic refactoring (styler, codegrip, etc.), or automatic diagnostics (lintr, pkgcheck, etc.)? Have you written any customization or standalone script to help you with that?

Reading notes on Producing open source software by Karl Fogel (First edition)

Thu, 08 Feb 2024 00:00:00 +0000

I recently re-read Nadia Eghbal’s Working in public. This time around, I noticed her mention of the book “Producing open source software” by Karl Fogel. It is a book about the people aspects of open-source projects, including money, and it reads like a sort of guide. Complying with my first-edition-curse, I did not notice there was a second edition online and soonish to be in print apparently, so I bought and read a second-hand first edition.

I really enjoyed it, and found myself referring to or thinking of parts of the book at work. Kind of impressive for a 2005 book about open-source to have aged so well in at least some chapters. Here are some of my highlights!

What didn’t age well

In 2005, Git was still in development (the book author is actually a Subversion contributor). Furthermore, mailing lists were apparently big.

Now, the R project still uses Subversion and has mailing lists…

Besides, it was nice to think of some things GitHub has, like discussions board, in general terms.

The Pompeii projects 🌋

In the introduction, the author says it’s hard to define failures of open-source projects because often the project quietly dies.

“There may be a moment when a final change is made to the project, but those who made it usually didn’t know at the time that it was the last one.”

When visiting old repositories, I regularly have the impression I am visiting a less dramatic version of Pompeii.

At rOpenSci, we strive to explicitly archive abandoned or retired packages.

About motivations for participating in open-source

To the questions “What will persuade all these people to stick together long enough to produce something useful?”, Karl Fogel answers:

“People should feel that their connection to a project, and influence over it, is directly proportional to their contributions.”

A further aspect mentioned in the book is the idea that participants have a direct interest in the project because they, or their organization, use(s) it.

Later in the book, Karl Fogel also writes this about participants’ motivation:

“The desire to produce good software may be among those motivations, along with the challenge and educational value of working on hard problems. But humans also have a built-in desire to work with other humans, and to give and earn respect through cooperative activities.”

❤️

Another quote I don’t want to forget:

“Credit is the primary currency of the free software world.”

(yes, there are different currencies of payment, but it is nice to underline that credit is important)

On code review

Also related to motivation…

“code review also serves an indirect purpose: it confirms to people that what they do matters, because one obviously wouldn’t take time to review a commit unless one cared about its effect.”

“Treat Every User as a Potential Volunteer”

The idea is that people you meet in the issue tracker are already more involved than most users. 😉

“Appearances matter”

The author makes the point that by providing expected information in expected places in a nice way, you help potential users and contributors feel their time will not be wasted.

To me as an R package developer, “appearances matter” means: please switch your pkgdown website to Bootstrap 5. 😸 The small font size of the former template doesn’t give me a good impression as I need to zoom! Also, creating a pkgdown website in the first place, and linking it from your Git(Hub) repository, takes very little time in general. By the way, in 2005 the author already recommended for docs to be available locally but also online for people to be able to read it without installing your thing.

“Hacktivation energy”

The “hacktivation energy” is “the amount of energy a newcomer must put before she starts getting something back”.

Obviously, that is something maintainers should strive to minimize. I like catchy phrases!

The case for sitrep (situation report)

The sentence

“somewhere near the beginning of the documentation, tell the user how to run some sort of diagnostic test or simple command to confirm that they’ve set things up correctly”

reminds me of “sitrep” functions in usethis and devtools.

Developer guidelines vs developer documentation

“Developer guidelines tell programmers how to get along with each other; developers documentation tells them how to get along with the code itself.”

About secrecy

The author writes about discussions in public vs in private several times. I like the sentence below, although I guess any project can operate on a different degree of openness based on its goals:

“Making important decision in private is like spraying contributions repellent on your project.”

In another place in the book, the author states that of course, every organization has secrets.

On issue trackers

Compared to closing off issue trackers,

“It’s better to accept that cleaning out invalid issues will always be part of the project’s routine maintenance, and to try to get as many people as possible to help.”

Project health

Two more catchy phrases or words! One is:

“Operational health is the project’s ongoing ability to incorporate new code contributions and new developers, and to be responsive to bug reports.”

And the other is a nicer synonym of “bus factor”:

“Survivability is the project’s ability to exist independently of any individual participant or sponsor.”

To put actual numbers on those, nowadays one can use CHAOSS.

Scale

I feel some ideas presented in the book, like voting systems, apply to bigger projects than what I am typically dealing with.

Still, these were interesting paragraphs:

“The price of success is heavy in the open source world. As your software gets more popular, the number of people who show up looking for information increases dramatically, while the number of people able to provide information increases much more slowly.”

and

“one side effect of continuing to use unscalable communications models as the project grows is that the average quality of both questions and answers tends to go down, which makes it look like new users are dumber that they used to be, when in fact they’re probably not. It’s just that the benefit/cost ratio of using those high-population forums goes down, so naturally those with the experience to do so start to look elsewhere for answers first.”

Terse, yes, but with “emoticlues”

Karl Fogel says that if you write a terse comment, which is acceptable,

“But at the end, sign off with something indicating that your terseness is not to be taken as coldness (…) A strategically placed smiley face or emoticlue can often be enough to reassure an interlocutor, too.”

I really like the made-up word “emoticlue”. 😁 Adding emojis in issue comments is something I consciously do.

Holy wars

“Each project usually has a holy war or two all its own, as well, which long-time developers will quickly become familiar with.”

A pint of tidyverse-vs-base-R, anyone? 🍻

Releases

“Stabilization” is the process of getting a release branch into a releasable state; that is, of deciding which changes will be in the release, which will not, and shaping the branch content accordingly.”

“(…) the process of stabilizing a release is mostly about creating mechanisms for saying ‘no’.”

Automation

“Multi-step, deterministic operations are exactly what computers were invented for, save your humans for more interesting things.”

Examples of this would be usethis and rOpenSci automation of software-review.

Conclusion

“Producing open-source software” was a nice read. I don’t want to read more things on screen given how I spend my days 😅, so I will wait for the second edition to be in print to read it, but I am glad to have already read the first. I feel it’d be a nice book to discuss with others. It reminded me of Tracy Teal’s posit conference workshop “It’s not just code: managing an open source project” that I didn’t attend, but whose existence I found cool; and of an rOpenSci community call I participated it in, “Set Up Your Package to Foster a Community”.

Load different R package versions at once with git worktree

Tue, 23 Jan 2024 00:00:00 +0000

This post was featured on the R Weekly highlights podcast hosted by Eric Nantz and Mike Thomas.

Do you ever see GitHub issue comments where someone posts the results of a reprex with a current package version, and then with an older one, to prove a regression? How would you go about preparing such a report? Today I learnt there is a clean way to have different versions of a codebase at once on your computer, thanks to the ever powerful Git.

Assumption: you want to load the package at different stages of its history

I’m not talking about installing different R package versions at once, this is not a post about renv or Docker. 😉 I mean having the ability to quickly run devtools::test() or devtools::test_active_file() on two different versions of an R package source.

Although I guess that for running an actual reprex::reprex() on your code, you’d need a quick installation in each version of the codebase.

Solution: git worktree!

The git worktree command tends to be introduced as an alternative to a quick git stash when switching from feature work to a hot fix (well, in all three posts I read/skimmed on the topic 😅). With git worktree you can create a new folder somewhere on your computer, that is at a different Git state than your current folder (other commit/branch/release – aren’t all of those the same thing anyway 🤪).

Say, in Documents/rigraph you are working on the main branch of the https://github.com/igraph/rigraph repository. Now, someone asks you “what did this code do with version 1.6.0”.

Past me would have diligently installed that version of the package in the same R session, probably from CRAN or with pak, run the code, re-started my R session, re-loaded the development version or something like that.

Current me would run git worktree add ../rigraphv1.6.0 v1.6.0 which means I now have a folder Documents/rigraphv1.6.0 with the rigraph repository as it was at the v1.6.0 release. Then inside that folder I can open my favorite IDE, load rigraph as it was at the time, and run the code I am curious about. I can switch between my two IDE windows to try things out with the two package versions.

Once I am done, from my original rigraph folder I can run git worktree remove rigraphv1.6.0 (or git worktree remove rigraphv1.6.0 --force if there are some changes in that folder). git worktree list shows me what worktrees (folders!) there are.

Conclusion: git worktree is great!

I’m glad I learnt about git worktree, “Manage multiple working trees” a.k.a (in my head) “Create multiple folders corresponding to a single Git repo at different states”. Do you use git worktree? Did I miss something in the around 10 minutes I’ve known about it? 😅

Reading notes on Pro Git by Scott Chacon

Fri, 19 Jan 2024 00:00:00 +0000

As mentioned about a million times on this blog, last year I read Git in practice by Mike McQuaid and it changed my life – not only giving me bragging rights about the reading itself. 😅 I decided to give Pro Git by Scott Chacon a go too. It is listed in the resources section of the excellent “Happy Git with R” by Jenny Bryan, Jim Hester and others. For unclear reasons I bought the first edition instead of the second one.

Git as an improved filesystem

In the Chapter 1 (Getting Started), I underlined:

“[a] mini filesystem with some incredibly powerful tools built on top of it”.

Awesome diagrams

One of my favorite parts of the book were the diagrams such as the one illustrating “Git stores data as snapshots of the project over time”.

A reminder of why we use Git

“after you commit a snapshot into Git, it is very difficult to lose, especially if you regularly push your database to another repository.”

The last chapter in the book, “Git internals”, include a “Data recovery” section about git reflog and git fsck.

One can negate patterns in `.gitignore`

I did not know about this pattern format. In hindsight it is not particularly surprising.

`git log` options

The format option lets one tell Git how to, well, format the log. I mostly interact with the Git log through a GUI or the gert R package, but that’s good to know.

The book also describes how to filter commits in the log (by date, author, committer). I also never do that with Git itself, but who knows when it might become useful.

A better understanding of branches

I remember reading “branches are cheap” years ago and accepting this as fact without questioning the reason behind the statement. Now thanks to reading the “Git Branching” chapter, but also Julia Evans’ blog post “git branches: intuition & reality”, I know they are cheap because they are just a pointer to a commit.

Likewise, the phrase “fast forward” makes more sense after reading “Git moves the pointer forward”.

The “Git Branching” chapter is also a place where diagrams really shine.

Is rebase better than merge

“(…) rebasing makes for a cleaner history. If you examine the log of a rebased branch, it looks like a linear history.”

“Rebasing replays changes from one line of work onto another in the order they were introduced, whereas merging takes the endpoints and merges them together.”

Reading this reminds me of the (newish?) option for merging pull requests on GitHub, “Rebase and merge your commits” – as opposed to merge or squash&merge.

Don’t rebase commits and force push to a shared branch

The opportunity to plug another blog post by Julia Evans, “git rebase: what can go wrong?".

The chance of getting the same SHA-1 twice in your repository

“A higher probability exists that every member of your programming team will be attacked and killed by wolves in unrelated incidents on the same night.”

Ancestry references

The name of references with “^” or “~” (or both!) is “ancestry references”. Both HEAD~3 and HEAD^^^ are “the first parent of the first parent of the first parent”.

Double dot and triple dot

I do not intend to try and learn this but…

The double-dot syntax “asks Git to resolve a range of commits that are reachable from one commit but aren’t reachable from another”.

The triple-dot syntax “specifies all the commits that are reachable by either of the two references but not by both of them”.

New-to-me aspects of `git rebase -i`

git rebase -i lists commits in the reverse order compared to git log. I am not sure why I did not make a note of this before.

I had not really realized one could edit single commits in git rebase -i. When writing “edit”, rebasing will stop at the commit one wants to edit. That strategy is actually featured in the GitHub blog post “Write Better Commits, Build Better Projects”.

Plumbing vs porcelaine

I had seen these terms before but never taken the time to look them up. Plumbing commands are the low-level commands, porcelain commands are the more user-friendly commands. At this stage, I do not think I need to be super familiar with plumbing commands, although I did click around in a .git folder out of curiosity.

Removing objects

There is a section in the “Git internals” chapter called “Removing objects”. I might come back to it if I ever need to do that… Or I’d use git obliterate from the git-extras utilities!

Conclusion

Pro Git was a good read, although I do wish I had bought the second edition. I probably missed good stuff because of this! My next (and last?) Git book purchase will be Julia Evans’ new Git zine when it’s finished. I can’t wait!

Introducing saperlipopette, a package to practice Git!

Thu, 18 Jan 2024 00:00:00 +0000

I got more confident with Git since reading Git in practice. This has resulted in a more enjoyable Git practice! I’m also more keen to sharing Git “tips” with others, but felt it was challenging to quickly come up with examples to demo some Git workflows. This is what motivated my creating saperlipopette, an R package containing small Git playgrounds to practice various Git commands and strategies!

What is saperlipopette?

The saperlipopette package creates Git messes, or playgrounds, that users need to solve. Part of the exercises are inspired by the wonderful website Oh Shit, Git!?! by Katie Sylor-Miller; others reflect commands that I’ve enjoyed using… or that I’d like to use one day, like git bisect, so the list is partly aspirational!

While saperlipopette itself makes good use of the gert package under the hood, the users can solve the Git messes any way they like, be it with some sort of Git interface, the command line, etc. The package provides no checking of solutions. You get a new folder with a Git mess inside, some tips, and you are the one defining success at the end of one try, by looking at your Git history. Because re-creating an exercise folder only demands your running a function, you can re-create the exercise as needed.

Why this name?

This package is intended to be a companion to https://ohshitgit.com/, so its name had to honour the exclamation. “saperlipopette” is an old-fashioned French exclamation. You can say “Saperlipopette, Git!”.

How to use saperlipopette?

Setup

You can install the development version of saperlipopette like so:

pak::pak("ropensci-training/saperlipopette")

You’ll also need

a Git installation, but if you made it here you probably already use Git at least a bit.
basic Git knowledge, in particular being able to examine the Git history, be it with git log or a tool in your IDE.
a directory where to store the exercises folder. In all examples we use a temporary directory but if you prefer, you could use a dedicated “scratch directory”.

Basic example

Let’s try the saperlipopette::exo_one_small_change() that goes with “Oh shit, I committed and immediately realized I need to make one small change!".

library("saperlipopette")
parent_path <- withr::local_tempdir()
path <- exo_one_small_change(parent_path)
#> ℹ Follow along in /tmp/RtmpoAeVTm/file152cb747520d9/one-small-change!
# what's in path
fs::dir_tree(path)
#> /tmp/RtmpoAeVTm/file152cb747520d9/one-small-change
#> ├── R
#> └── bla
# with Git in a command line: git log
# or the gert R package
gert::git_log(repo = path)
#> # A tibble: 2 × 6
#>   commit                          author time                files merge message
#> * <chr>                           <chr>  <dttm>              <int> <lgl> <chr>  
#> 1 2ff0d31f566e68ae0ee94b6028a3fa… Jane … 2023-12-15 16:25:00     1 FALSE "feat:…
#> 2 e227ecc55e421f70b6e30602e6a2ee… Jane … 2023-12-15 16:25:00     2 FALSE "First…

At this stage, the user would open the newly created R project and launch an R session, where messages would indicate them what to do, and which URL to follow, to find the corresponding ohshitgit entry if relevant. In practice here the user would change a file, then Git add it, then run git commit --amend --no-edit. The user would examine the Git history before and after this.

#> ✖ "Oh shit, I committed and immediately realized I need to make one small change!"
#> ✖ I wanted to list 3 things in my bla file, not only two!
#> ℹ See <https://ohshitgit.com/#change-last-commit>
#> ℹ For more help use `tip()`

If they need more instructions than what is initially provided, the user can run:

tip()
#> • Add 'thing 3' to the bla file and save it.
#> • Add 'bla' file to Git.
#> • `git commit --amend --no-edit`
#> • Examine Git history.

That interface relies on adding an (.gitignored!) .Rprofile to the newly created project, with instructions formatted with the cli package.

We’ve set the Git author, committer and date so that the automatic commits get the same hashes, which could be useful when teaching a group: everyone should be looking at the same hashes on their machine, except for those commits they create themselves.

Feedback welcome!

In this post I introduced the saperlipopette package whose aim is to help users practice their Git skills in a safe (because throw-away) environment! I am very grateful to Jim Gardner for useful feedback and would love to hear from more users, if saperlipopette is of any interest to you.

Reading notes on The Pragmatic Programmer by David Thomas and Andrew Hunt

Mon, 11 Dec 2023 00:00:00 +0000

In my quest to having reading notes on the tech books I read, and while waiting for code to run, I recently re-read The Pragmatic Programmer by David Thomas and Andrew Hunt. That book, whose second edition was published in 2019, offers an overview of many topics useful to programmers, from the idea of taking responsability, to testing, tooling, etc.

Not my favorite tone

I’m not the biggest fan of the tone used in the book, that feels a bit patronizing to me. One example that comes to mind is that the authors recommend reading non-tech books to learn about humans, because programming is also about humans. That’s not a piece of advice I’d give, even as a book lover, because one can learn about humans outside of books (not exactly breaking news, I know), and also because I’d like reading non-tech books to be a thing one can do for enjoyment unrelated to work.

I also disliked analogies that I found uselessly violent: car crashes, train wrecks, boiling frogs, not to forget broken windows. There’s enough violence in the world, so I prefer further violence to inhabit my non-tech reading rather than my tech reading. 😉

Now, to each their own, and I can imagine such writing being enjoyable to others.

An impressive overview

The Pragmatic Programmer is rather famous. I can see why when I think about its table of contents: even if the book would not be my first recommendation for any topic it covers, it’s the only one I know that covers so many topics: version control, testing, domain-specific languages, orthogonality, decoupling, etc. Seen this way, it’s quite unique and precious.

Furthermore, I can imagine productive book clubs using the questions in the book.

What value for me? Pieces of wisdom and catchy phrases

I did not read The Pragmatic Programmer as an absolute beginner, so I had at least vaguely heard of many (not all!) elements in the book. However, I find some value in being exposed to concepts I didn’t know, seeing new explanations or catchy descriptions for those I was more familiar with, and thinking differently about current work projects (the two latter aspects are often the value I find in so-called “productivity books”).

For most of the notes below I’m adding links to more or less related things, which would probably make me an annoying participant of a book club about The Pragmatic Programmer. Luckily this is my blog. 😇

ETC

The authors introduced the ETC value, as in “Easy to Change”. That made me think of “A Philosophy of Software Design”.

“Forgo following fads”

That nice alliteration reminded me of “Kill it with fire”.

Reversibility

The paragraphs on reversibility, i.e. flexible architectures, make me feel like sharing this excellent blog post by Vicki Boykis: “Commit to your lock-in” that I often think about, and that’s sort of about not thinking too much about reversibility. 😅

Encouragement to learn more about the shell

I should learn more shell commands. Incidentally, after reading about that in the book, I had to add a system2() command to a script of mine.

“Achieve Editor Fluency”

Also a good piece of advice. Have you read Nick Tierney’s great post “Get Good with R: Typing Skills and Shortcuts”? On the other hand I’m also reminded me of the video “What editor do you use”? 😁

Engineering daybook

The authors encourage the practice of writing down things you learn, challenges, every day. I most often have a piece of paper near me, including my sticky note for my posts about useful functions, but haven’t been able to really stick to maintaining an engineering daybook. Since I work from home I don’t observe many people at work, so have to resort to tech books to imagine people working 😉, and know that the protagonist of The Unicorn Project maintains one.¹

“Act locally”

The occasion for me to spread the word about the existence of the fantastic withr package for locally changing options, environment variables, languages, etc.; for instance in examples and tests.

Property-based testing

Do you know about Mark Padgham’s autotest package? Using it would allow one to try out property-based testing in one’s package, by feeding random input into functions. The book says one is often surprised by the results of property-based testing. 😅

“Build a Test Window”

I remember similar advice in Jenny Bryan’s fantastic talk “Object of type ‘closure’ is not subsettable” (slide 77).

“Delight your Users”

The link I’ll throw here is François Chollet’s post “User experience design for APIs”.

A last part on ethics

The book’s second edition ends with some thoughts on responsability, with the tip “Don’t enable scumbags”.

Conclusion

I have ambivalent thoughts about The Pragmatic Programmer: it’s an extensive reference for programmers, but not my favorite tech book. Have you read it? Do you know other similar reference books? Are you planning to write one and do you need readers of advanced copies? 😸

Have you ever read riveting kid books about some little character “who don’t want to share”, “who bakes a cake”, or “who discovers daycare”? The Unicorn Project, and The Phoenix Project are similar books, but for adults working with tech. ↩︎

The two phases of commits in a Git branch

Thu, 07 Dec 2023 00:00:00 +0000

I seem to have at last entered my Git era. 🎉 Reading and applying Git in practice was probably the best thing I did for my upskilling this year. One Git workflow aspect I’ve finally realized is that it’s fine to have two phases of work in a Git branch. I’ll explain it in this post.

Set up: create a branch for your work!

Ideally, in most cases, when adding a feature or fixing a bug or whatever, I’ll work in a branch.

gert::git_branch_create("feature")

I really like running the code above as I feel it puts me in the right mind space. I signal my intention to work on a given issue to the universe, it can’t hurt, right. 😁

Phase 1: do not think about a clean Git history nor your reputation, just commit all the time

Now, even if I might already open a draft PR to signal my working on a given thing, I just do the thing and try to not think about looking stupid with 1,000 commits “try to make R CMD check happy”. I absolutely do not want to lose my work so I commit (with RStudio IDE tab) and push (with gert::git_push() or that same tab) regularly. In this phase, Git is merely my backup.

Phase 2: clean up!

That’s the phase I used to never do¹ but that I’m trying to consciously practice. I guess blogging about this will force me to keep doing it.

In the comments of my reading notes on Git in practice, one of my Git models, Hugo Gruson², recommended the post “Write Better Commits, Build Better Projects”. The kind of posts I used to think were for other people since I did not understand half of the Git words in it. 😉 (Have I mentioned reading a Git book was life-changing? 😅)

Basically once you have your PR code ready, you have to try and create nice commits, in a nice order. It’s important both for the people that will have to review your code (which might even be just you in a few days for lack of collaborators), and for the people who might have to perform a Git bisect-ion or some other form of archeology on the code base later (again, this might be you).

Part of the work is thinking about what a good set of commits might be.

Then in practice I’ll use

Git rebase -i to squash, reorder or even drop commits and to improve commit messages;
Git reset --soft to remove changes from the Git history but not from the files, and then commit them incrementally in a smarter way than when I was in phase 1.
Git push -f which is fine since my branch is mine only until I mark the PR as ready to review. 😈

Conclusion

In this post I explained how I now think as work in a branch as happening in two phases, one phase where Git is my backup, and one phase where I try to think more about code reviewers (for the PR) and code archeologists (for later when the PR is merged). I’m still very much learning so don’t think you’ll get perfect PRs from me just yet!

And I was so embarrassed when a PR of mine was merged without being squashed. ↩︎
If you’ve ever reviewed a PR by Hugo you have to know what I mean! Such good story-telling in commits! ↩︎

Reading notes on Kill It with Fire by Marianne Bellotti

Mon, 20 Nov 2023 00:00:00 +0000

Another month, another long train trip enjoyed in the company of great books, among which a work-related one, Kill it with fire by Marianne Bellotti. As indicated by its subtitle “Manage Aging Computer Systems (and Future Proof Modern Ones)”, the book deals with handling legacy computer systems. The focus is on bigger projects (including serious internal politics) than what I usually deal with, but there’s some valuable lessons for any project size in there.

It was actually my second reading of the book, since I regretted not having taken notes the first time around! Some aspects had already stuck with me though.

Let’s dive into some highlights…

Dealing with legacy code is a honour

The number one thing I got out of this book is an enhanced appreciation for having to work with legacy codebases. I did not start out hating them, but still, I’m glad to now be able to refer to these two quotes for instance (although I do not work with very old code!):

“legacy technology exists only if it is successful. These old programs are perhaps less efficient than they were before, but technology that isn’t used doesn’t survive decades.”

And

“To most software engineers, legacy systems may seem like torturous dead-end work, but the reality is systems that are not used get turned off.”

Why are older systems bad

Chapter 3 (“Evaluating Your Architecture”) Marianne Bellotti explains three kinds of issues that might be the reason a modernization is wished for:

tech debt. It reminded me, among other things, of John Ousterhout’s “tactical vs strategic programming”;
performance issues;
stability issues (surprising failures, due to tight coupling and/or complexity).

In Chapter 10 (“Future Proofing”) the author mentions two ways in which systems age:

change of usage patterns (scale);
the resources that back them deteriorate.

That’s the chapter where the idea of legacy modernization being an anti-pattern is presented: if you maintain your system well over time, you won’t need a special operation. 😅

How people adopt new software

There’s a lot of content about how technology gets adopted or not, including these two nuggets of wisdom:

“familiar interfaces help speed up adoption” (it reminded me of Quarto vs R Markdown);
“people gain awareness of interfaces and technology through their networks, not necessarily by popularity”.

The latter is not the only time that the book underlines human aspects of computer systems, far from it.

Lessons for planning modernization efforts

Obviously the book is full of lessons about planning modernization efforts, but here are three of them:

“Migrating for Value, Not for Trends”. Don’t go for the next new shiny thing, pragmatically assess whether it’ll be best for the system (or just best for bragging about it to your peers 😅 See the next section of this post!)

It’s important to first learn how to understand, test (what does not fail?), monitor (what fails?) the codebase before one makes big changes to it.

The book includes many tips on how to build and keep momentum of a team or organization during a modernization effort (the whole chapter 5, “Building and Protecting Momentum”), as well as how to keep one’s own motivation (marking time, or I guess more generally tracking progress).

At several points, Marianne Bellotti underlines how the atmosphere at an organization will influence its code. For instance if there’s no well-defined career ladder at an organization, maybe people will make choices to get recognition from their peers outside of it, for instance by being the first to try out tool X for some application.

This quote would pair well with some more reading on psychological safety and the like, as that’s what it made me think of:

“Good technologists should focus on what brings the most benefit and highest probability of success to the table at the current moment, with the confidence of knowing they have nothing to prove.”

There’s also a few pages on how people get seen, and how that influences what gets built.¹

Use the codebase and its context but not too much

In my own words,

Retrieve context as much as possible.
Think of the tools available at the time the system was created as that might you help understand some constraints.
Respect that the system is working (otherwise it wouldn’t be used any more).
However don’t make the mistake of assuming all decisions were good ones.

The tools are not the most important thing

Marianne Bellotti encourages the reader to be realistic about what automatic tools (for instance for traducing code from one language to another) will give you.

“The tools themselves are not as important as the phases of excavating, understanding, documenting, and ultimately rewriting and replacing legacy systems. Tools will come and go.”

Turn things off and see what happens

I love this idea, apparently called “chaos testing”, for when someone encounters a thing one isn’t sure is used at all or where. You turn it off and wait for someone to complain. It’s in a book so it’s a legit strategy. 😸

Tips on designing technical conversations

I don’t want to reproduce what’s in the book here but just to remember there are good tips around conversations in this book with exercises such as “The Saboteur”.

Breaking changes and users’ procrastination

In “Communicating Failure”, in Chapter 8 (“Breaking Changes”), Marianne Bellotti explains that giving too much time to users to stop using a thing to be deprecated, just leads them to procrastinate more.

Conclusion

“Kill It with Fire” is a great book! I really enjoyed the pragmatism and the focus on human aspects of modernization.

I might re-read it when the stories of bigger projects have more value to me than being interesting stories, that is to say, if I ever participate in bigger modernization efforts!

Have you read the book? Any thoughts on it or on the general topic of modernizing old computer systems?

These are strong pages despite citing work by Dan Ariely. ↩︎

The current introduction to my package development workshops

Thu, 09 Nov 2023 00:00:00 +0000

I somewhat regularly teach about package development. One recent example was a workshop for rOpenSci champions. I am improving my teaching over time (thankfully 😅) but one thing I have down by now is the intro, which is mostly my throwing together my favorite quotes about R package development! Let me write it up.

Where I explain why people shouldn’t flee the workshop

After boasting a bit (a.k.a. sharing my package development creds to introduce myself), I answer three retorical questions:

Why develop an R package?

Developing an R package is the easiest way to share code/data/R Markdown templates… with

(future) you: a package for organizing your utility functions in a smart way;
the collaborators you know: a package for your team;
and the collaborators you don’t: a package for the whole wide world. Without needing to be a generalist tool à la ggplot2, your package could help other R users!

Why learn about package development?

This is a different question! I love Jon Calder’s wording about this. You should learn about package development

“- To share code (and data)";

Which is learning about package development to develop your packages.

“- To leverage existing tooling”;

Which could be learning about package development for using part of the infrastructure, like the one for testing, in your own projects.

“- To contribute to other packages”.

Which is learning about package development to improve packages owned by other people (in my words, being a package aunt/babysitter).

So you don’t need to even have an idea or need for your own R package for that learning to be relevant.

Who can learn about package development?

To answer this question I use the wise words of Susan Johnston.

“- Can you open R or RStudio?”

“- Can you install a package?”

“- Have you ever written a function in R?”

“- Could you learn how to write a function in R?”

“➡️ You can write a package in R!”

Side-note: where to learn about functions

Knowing vaguely what a function is a pre-requisite when I teach package development (the example functions I use are nothing fancy) but I’ve collated links for those who want to learn:

Materials from Stephanie Kirmer’s R-Ladies East Lansing tutorial
Write your own R functions, stat 545 course by Jenny Bryan and The STAT 545 TAs;
Chapter about functions in the book “R for Data Science” by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund;
Fun with Functions talk by Kaylea Haynes, R-Ladies Manchester.

Where I go more into details about packages

What is a package?

The author of this excellent quote (first reproduced in French) is Sébastien Rochette:

“Pour réduire ses craintes, il faut se dire que ce n’est ni plus ni moins qu’un dossier organisé d’une manière contrainte.”

“To be less afraid you have to tell yourself that it’s simply a folder organized in a constrained way.”

At this point what could remain scary is, how do I remember about these constraints and rules? So here comes…

Automation!

At this point I introduce usethis as an “actually useful Clippy”.

I don’t necessarily use, but think of, this great quote of Jenny Bryan’s:

“The usethis package implements this important principle: Automate that which can be automated. Your computer was literally born to implement rote-but-fussy stuff for you.”

Why automate?

Even in the intro I like to underline that automating package development tasks is useful not only for the regular work of package development, but also for teaching (😉) and for reproducing problems.

Goals of a beginner workshop

At least the latest one I taught had two goals:

For the learners to get to know the tools for package development that I use myself;
For the learners to learn that there is no magic, only practice and ✨ tips ✨.

Conclusion

In this post I wrote up the intro that I’ve now used a few times for my package development teaching. If you’re curious, to conclude such workshops I list what I think is actually difficult: writing good code, writing a good interface, writing good docs. But even that can be learnt (still learning myself 💪). In my teaching materials I share a list of resources.

Useful functions for dealing with object names

Mon, 06 Nov 2023 00:00:00 +0000

My sticky note filled up quickly after I only added setNames() on it, with related functions for dealing with object names, in base R and beyond!

(Un)Setting object names: `stats::setNames()`, `unname()` and `rlang::set_names()`

I noticed a function ending with something like this:

blop <- function() {
  
  # code creating the df data.frame
  # ...
  
  names(df) <- c("col1", "col2")
  df
}

It struck me as simplifiable by:

blop <- function() {
  
  # code creating the df data.frame
  # ...
  
  stats::setNames(df, c("col1", "col2"))
}

Interestingly the docs for setNames() sound as if it were created just for this use case!

“This is a convenience function that sets the names on an object and returns the object. It is most useful at the end of a function definition where one is creating the object to be returned and would prefer not to store it under a name just so the names can be assigned.”

For the opposite operation, removing the names of an object, we can use unname().

Unsurprisingly, the rlang package has its own special rlang::set_names(), that is more or less a stats::setNames() + unname() + generic janitor::clean_names() combo!

(df <- data.frame(c(1, 2), c(5, 7)))
#>   c.1..2. c.5..7.
#> 1       1       5
#> 2       2       7

rlang::set_names(df, c("a", "b")) # sets names!
#>   a b
#> 1 1 5
#> 2 2 7

rlang::set_names(df, NULL) # removes names
#>      
#> 1 1 5
#> 2 2 7

df2 <- rlang::set_names(df, c("SHOUTING", "LOUDLY"))
df2
#>   SHOUTING LOUDLY
#> 1        1      5
#> 2        2      7
rlang::set_names(df2, tolower) # using a function to transform names
#>   shouting loudly
#> 1        1      5
#> 2        2      7

Of note,

“set_names() is stable and exported in purrr”.

So if you prefer, you can use purrr::set_names() instead.

Last but not least, despite the examples above being on a data.frame, these functions can be used on lists, vectors…

Extracting names: `names()`, `rlang::names2()`

names() probably doesn’t need any introduction, but it’s interesting to mention it in contrast to rlang::names2() that " always returns a character vector, even when an object does not have a names attribute.”.

vector <- c(1, 2)
names(vector)
#> NULL
rlang::names2(vector)
#> [1] "" ""

rlang also has a replacement variant, rlang::names2<-, that “never adds NA names and instead fills unnamed vectors with "".”

I didn’t know all this before skimming the list of rlang functions dealing with object attributes.

Checking names

Is the object named: `rlang::is_named()`, `rlang::is_named2()`, `rlang::have_name()`

rlang::have_name() is the vectorized version of rlang::is_name(), not to be confused with rlang::has_name(). 😅

rlang::is_named(c(a = 1))
#> [1] TRUE

rlang::is_named(NULL)
#> [1] FALSE
rlang::is_named2(NULL)
#> [1] TRUE

rlang::have_name(c(a = 1, 2)) # the example from the manual page
#> [1]  TRUE FALSE

I do not know of a base R equivalent, I mean I’d find a way to express it, but not as directly. But maybe I am missing yet another base R gem!

Does the object have an element called X, does the object have elements called X, Y: `utils::hasName()`, `rlang::has_name()`

If you want to check that an object contains elements with certain names, you can use utils::hasName()

expected_names <- c("pof", "blop")

vector1 <- c(pof = 1, bla = 2)
vector2 <- c(pof = 1, bla = 2, blop = 3)

utils::hasName(vector1, expected_names)
#> [1]  TRUE FALSE
utils::hasName(vector2, expected_names)
#> [1] TRUE TRUE

The rlang package has a similar function, rlang::has_name().

expected_names <- c("pof", "blop")

vector1 <- c(pof = 1, bla = 2)
vector2 <- c(pof = 1, bla = 2, blop = 3)

rlang::has_name(vector1, expected_names)
#> [1]  TRUE FALSE
rlang::has_name(vector2, expected_names)
#> [1] TRUE TRUE

I’m honestly not sure whether there’s any difference between the utils and rlang versions, apart from the name!

In package tests: `testthat::expect_named()`

testthat::expect_named() allows you to check that an object returned by your code has the names you expect, or simply has names. You can ignore case and order based on the arguments ignore.case and ignore.order.

(object <- tibble::tibble(a = c(1, 2), b = c(3, 4)))
#> # A tibble: 2 × 2
#>       a     b
#>   <dbl> <dbl>
#> 1     1     3
#> 2     2     4

testthat::expect_named(object)

unnamed_object <- unname(object)
testthat::expect_named(unnamed_object)
#> Error: `unnamed_object` does not have names.

testthat::expect_named(object, c("b", "a"))
#> Error: Names of `object` ('a', 'b') don't match 'b', 'a'
testthat::expect_named(object, c("b", "a"), ignore.order = TRUE)

testthat::expect_named(object, c("A", "B"))
#> Error: Names of `object` ('a', 'b') don't match 'A', 'B'
testthat::expect_named(object, c("A", "B"), ignore.case = TRUE)

If you’re not used to using this expectation yet, lintr can help you a bit.

Conclusion

In this post I went through functions that deals with names: stats::setNames(), unname() and rlang::set_names() for (un)setting names (my initial motivation for this post!); names(), rlang::names2() for extracting names; rlang::is_named(), rlang::is_named2() and rlang::have_name() to find out whether an object is named; utils::hasName() and rlang::has_name() to find out whether an object contains elements of a given name; testthat::expect_named() for testing whether an object has names, or specific names. Any other related function that comes to mind?

Reading notes on Git in Practice by Mike McQuaid

Wed, 01 Nov 2023 00:00:00 +0000

While preparing materials for teaching Git a few months ago, I re-read Suzan Baert’s excellent post about Git and GitHub, where she mentioned having read “Git in Practice” by Mike McQuaid. I added the book to my Momox alerts, where it got available a few weeks later. The book source is on GitHub.

The book isn’t too heavy, so I took it with me on a long train journey! 🚋

Here are my reading notes, since I hope this habit of writing book learnings will stick. These notes are of limited generalizability, depending on what your own Git practice and knowledge are. Some chapter numbers are missing: they are those that were less relevant to me.

First impressions

Manning Publications book cover

This is the second book published by Manning that I own, the other being “The Programmer’s Brain” by Felienne Hermans. Both have covers featuring an ancient drawing of someone wearing a traditional dress, but different drawings of course, and different artists. For “Git in Practice” the cover is “A Kamtchadale in her full dress in 1768”. While the editor didn’t explain why a particular drawing was chosen, they have a general explanation

At a time when it is hard to tell one computer book from another, Manning celebrates the inventiveness and initiative of the computer business with book covers based on the right diversity of national costumes (…)

Good to know!

Not a book for beginners

The book is advertised for users at an “intermediate to advanced level”. One thing that clearly shows this is that the content on “Why you should use version control and how to convince others that doing so is a good idea” is an Appendix! Note that the content I’d actually use on the topic is the article “Excuse me, do you have a moment to talk about version control?” by Jenny Bryan.

An older book?

The book was published in 2015. I guess some Git things have been deprecated but I assume most of the book is still valid. What aged really bad: the screenshots from GitHub interface!

66 “techniques”

The book cover indicates it includes 66 techniques. The first techniques are git init, git add, git commit: calling them techniques is a bit of a stretch to me? But then the very predictable structure of the book content was actually great, so using that structure from the get go wasn’t bad at all.

Chapter 1: Local Git

Clearer understanding of Git add

“Rather than require all changes in the working tree to make up the next commit, Git allows files to be added incrementally to the index”

So, while I’ve been practicing this, I’m not sure I ever expressed the usefulness of git add.

I also underlined a sentence explaining that git add does two things:

“git add is used both to initially add a file to the Git repository and to request that changes to the fine be used in the next commit.”

Garbage in, garbage out

I liked that sentence:

“How useful the history is relies a great deal on the quality of data entered into it.”

Then the author mentions two reasons why small commits are better:

Better readability of the history;
Easier reversal of an unwanted change.

Commit messages are structured like emails

And apparently this comes from the history of Git, where commits could be sent as email. So the first line of a commit message is the subject and the rest, the body.

I guess there’s also a parallel between the conventional commits convention, where you add a prefix to the subject indicating its category: “fix:” for bug fixes, “feat:” for features, is like adding a prefix to an email subject: “[ACTION REQUIRED]”.

A motivation for rewriting history

“there’s no need for everyone to see the mistakes you made along the way”

Now let’s see if this, and other parts of the book, motivate me enough to rewrite the history of my Git branches before I mark a PR as ready for review.

A better understanding of Git diff

I didn’t know that one could call git diff without an argument, and that it’d show the diff between the working directory and the index staging area… but apparently only for files that were already added, which answers my question about how different that is from git status:

git_repo <- withr::local_tempdir()
gert::git_init(git_repo)

brio::write_lines("blop", path = file.path(git_repo, "bla.txt"))
gert::git_add("bla.txt", repo = git_repo)
#>      file status staged
#> 1 bla.txt    new   TRUE
first_commit <- gert::git_commit("first commit", repo = git_repo)

brio::write_lines("bip", path = file.path(git_repo, "other.txt"))

gert::git_diff("HEAD", repo = git_repo)
#>   status     old     new
#> 1      A bla.txt bla.txt
#>                                                                                                                                patch
#> 1 diff --git a/bla.txt b/bla.txt\nnew file mode 100644\nindex 0000000..35733a0\n--- /dev/null\n+++ b/bla.txt\n@@ -0,0 +1 @@\n+blop\n
gert::git_diff(first_commit, repo = git_repo)
#>   status     old     new
#> 1      A bla.txt bla.txt
#>                                                                                                                                patch
#> 1 diff --git a/bla.txt b/bla.txt\nnew file mode 100644\nindex 0000000..35733a0\n--- /dev/null\n+++ b/bla.txt\n@@ -0,0 +1 @@\n+blop\n
gert::git_diff(repo = git_repo)
#> [1] status old    new    patch 
#> <0 rows> (or 0-length row.names)
gert::git_status()
#>                                                              file  status
#> 2  content/post/2023-01-02-reading-notes-git-in-practice/index.md deleted
#> 1 content/post/2023-01-02-reading-notes-git-in-practice/index.Rmd deleted
#> 3          content/post/2023-11-01-reading-notes-git-in-practice/     new
#>   staged
#> 2  FALSE
#> 1  FALSE
#> 3  FALSE

brio::write_lines("bip", path = file.path(git_repo, "bla.txt"))
gert::git_diff(repo = git_repo)
#>   status     old     new
#> 1      M bla.txt bla.txt
#>                                                                                                                     patch
#> 1 diff --git a/bla.txt b/bla.txt\nindex 35733a0..94652b8 100644\n--- a/bla.txt\n+++ b/bla.txt\n@@ -1 +1 @@\n-blop\n+bip\n
gert::git_status()
#>                                                              file  status
#> 2  content/post/2023-01-02-reading-notes-git-in-practice/index.md deleted
#> 1 content/post/2023-01-02-reading-notes-git-in-practice/index.Rmd deleted
#> 3          content/post/2023-11-01-reading-notes-git-in-practice/     new
#>   staged
#> 2  FALSE
#> 1  FALSE
#> 3  FALSE

Smiling refs

I learnt that main^ is equivalent to main~1 and main^^ is equivalent to main~2. ^_^

Git rev-parse

So if you use say main^^ and want to get the actual SHA, you can use git rev-parse. In gert, that’s gert::git_commit_id().

git_repo <- withr::local_tempdir()
gert::git_init(git_repo)

brio::write_lines("blop", path = file.path(git_repo, "bla.txt"))
gert::git_add("bla.txt", repo = git_repo)
#>      file status staged
#> 1 bla.txt    new   TRUE
gert::git_commit("first commit", repo = git_repo)
#> [1] "c17a89d5682d3bab91e412aa935062c35dcdc1e2"

brio::write_lines("blip", path = file.path(git_repo, "bla.txt"))
gert::git_add("bla.txt", repo = git_repo)
#>      file   status staged
#> 1 bla.txt modified   TRUE
gert::git_commit("second commit", repo = git_repo)
#> [1] "9a02ae19cbcd1f03bd68aba9cd414b7d83250d3d"

brio::write_lines("blup", path = file.path(git_repo, "bla.txt"))
gert::git_add("bla.txt", repo = git_repo)
#>      file   status staged
#> 1 bla.txt modified   TRUE
gert::git_commit("third commit", repo = git_repo)
#> [1] "cd437aebe6e3f11223cdc4c4d62e3660bde56dbd"

gert::git_commit_id("main", repo = git_repo)
#> [1] "cd437aebe6e3f11223cdc4c4d62e3660bde56dbd"
gert::git_commit_id("main^", repo = git_repo)
#> [1] "9a02ae19cbcd1f03bd68aba9cd414b7d83250d3d"
gert::git_commit_id("main^^", repo = git_repo)
#> [1] "c17a89d5682d3bab91e412aa935062c35dcdc1e2"

Chapter 2: Remote Git

Interestingly that’s the chapter where branches are introduced.

Git as a distributed version control system

My first version control experience was with SVN, for a package developed on R-forge. I only remember two things: the cute tortoise of Tortoise-SVN, and the fact that instead of doing commit+push I’d only do commit. It makes sense now that I understand that Git is a distributed version control system whereas SVN is centralized, with a remote repo being the source of truth.

Why Git fetch?

To be honest, I’m still not sure when I would use git fetch! git fetch is for fetching changes from a remote without modifying local branches. I only use git pull (which is fetch+merge) or git pull --rebase when that fails, mostly. Note that the “Pull, but you have local work” chapter of happygitwithr by Jenny Bryan, the STAT 545 TAs, Jim Hester looks useful (I found it by searching for “fetch” in that book).

Mike McQuaid wrote a paragraph “Should you use pull or fetch?” where he explains he prefers git fetch because he can incorporate remote changes when it makes sense, using a method of his choice (merging, rebasing, resetting). He also says git fetch is better for working in situations with no internet connection, which was unclear to me.

Branch names

Mike McQuaid recommends creating a branch name that describes “the branch’s purpose in multiple words separated by hyphens”. That’s what I tend to do, probably imitating what I saw elsewhere. Sometimes my branch names are less informative, and this makes me cringe when I open the corresponding PR.

Pushing a local branch remotely

This section just made me very happy I push local branches with gert::git_push(), because it sets the upstream branch automatically by default.

Slowly, slowly understanding merge commits

I learnt that merge commits are commits with multiple parents when working on the fledge package. When reading “Git in practice”, I at least understood “fast-forward merge” which is something I had simply ignored when Git told me it had done one.

So, for my future reference… If I am merging a branch A into main, and Git tells me it did a fast-forward merge, it means no merge commit was needed so none was made. The branch A commits were added to the main branch without new commit before that.

Conflicts and merge commits

“When conflicts have been resolved, a merge commit can be made. This stores the two parent commits and the conflicts that were resolved so they can be inspected un the future. Unfortunately, sometimes people pick the wrong option or merge incorrectly, so it’s good to be able to later see what conflicts they had to resolve.”

This was new to me, certaintly not something I had in mind when resolving conflicts.

Chapter 3: Filesystem interactions

Git mv, Git rm

I was never aware of git mv, for renaming and moving a file. I think it’s because the RStudio IDE Git thingie was doing this for me? When I rename a file, I see the status becomes “R” once I’ve added both the file deletion (old name) and the file adding (new name).

The same goes for git rm for removing a file.

I should use Git reset more often

An use case for git reset that was described really resonated with me. 😅

“Perhaps you added debugging statements to files and have now commited a fix, so you want to reset all the files that haven’t been committed to their last committed state (on the current branch)”.

So instead of looking at the Git diff to remember where I added browser(), or looking for occurrences of a given debugging thing, I could use git reset, now that sounds smarter!

I should adopt Git clean

git clean will remove any untracked file from the filesystem. Like the previous idea, it sounds like something I could use when debugging something and creating files à la “blop.text”.

Now git clean can be dangerous so you can first run git clean -n or git clean --dry-run to see what would be removed.

Note that gert doesn’t seem to support git clean so I’d do this from a terminal.

The book also explains the -X argument of git clean for removing only ignored files, compared to the lower case -x argument for removing both ignored and untracked files.

In conclusion I need to practice using git clean, and to refer to its docs when doing so, but at least now I know it exists.

Could Git stash be useful to me

git stash save is a command for temporarily stashing some changes. git stash pop gets them back. git stadsh clear clears stashed commits.

I can’t think of an use case right now but maybe I will encounter one one day?

Git update-index `--assume-unchanged`

Again an use case that resonated with me (well I don’t use Rails but still)

“I’ve found myself in a situation in the past where I wanted to test a Rails configuration file change for a week or two while continuing to do my normal work. I didn’t want to commit it because I didn’t want it to apply to servers or my coworkers, but I did want to continue testing it while I made other commits rather than change to a particular branch each time.”

So in that case one can run Git update-index --assume-unchanged <path>. Then you undo this by running Git update-index --no-assume-unchanged <path>.

Chapter 4: History visualization

Honestly most of this chapter made me happy about the GitHub interface!

The other thing is that it has an explanation of git bisect that I need to come back to next time I need to find which commit caused a particular bug. I tend to be scared of git bisect, but the explanation made sense. Also I guess I need to focus on making small informative commits before bisect really shines in my use cases. 😇

I am especially curious about git bisect run for automating git bisect, I wonder how that would work with a testthat unit test.

Chapter 5: Advanced branching

Merge strategies

I learnt there’s a merge strategy, --ours, that will ignore all changes from the incoming branch but still indicate it in the history. That’s supposed to be useful for when you want to keep an experiment from an experimental branch in the history.

There’s another interesting merge strategy, --patience, that “uses a slightly more expensive git diff algorithm to try to decrease the chance of a merge conflict”. It sounds like a joke.

Not useful to me but fun name: git rerere

git rerere, Reuse Recorded Resolution, allows you to resolve each merge conflict only once.

I sort of understand the use case but it doesn’t sound useful to my work:

“You may find yourself in a situation where you have a long-running branch that you have to keep merging in another branch, and you get the same merge conflict every time.”

Yay git cherry-pick

git cherry-pick allows you to add a single commit to the current branch. I recently successfully used it!

A collaborator had mistakenly created a new feature branch from another feature branch instead of from the main branch. After that they added their feature commit but the PR was all broken. I created a new branch from the main branch, cherry-picked the commit and then opened a new PR. It’s a small step for humanity but it was a big step for me. 😁 It was very cool to be able to say “don’t worry, I’ve got it” to that collaborator.

Chapter 6: Rewriting history and disaster recovery

Another new-to-me command: git reflog

git reflog will list “all changes including history rewrites”, for the local repository. Commits that are deleted on all branches are only kept in the reflog for 90 days.

Git reset can take a list of paths as arguments

An example would be git reset HEAD^ -- path.txt

Actually retired now: Git filter-branch

One technique presented in the book, for rewriting git repository history, is git filter-branch but then I saw a toot by Kirill Müller indicating it’s retired. A replacement is git filter-repo, that I’d have to explore a bit more before being able to use it. The simple example looks convincing enough! According to the book this can be useful for some cases of disaster recovery.

Commit regularly

This is the main message of this chapter, and it’s not an advanced technique at all, just a good habit. 😸

Chapter 7: Personalizing Git

Option for pruning branches automatically

If you want the references to now deleted remote branches to be deleted, you can run git remote prune origin or configure Git so that this will happen every time you pull or fetch.

gert::git_config_global_set("fetch.prune", 1)

Global gitignore

I already knew about the global gitignore file thanks to the handy usethis::git_vaccinate().

An option for autocorrecting misspelled commands

There’s a Git option for autocorrecting misspelled commands, help.autocorrect. It would not help me since my most frequent typo is “gut” instead of “git”, sadly.

Chapter 8: Vendoring dependencies as submodules

If I ever have to work again with a repo that uses submodules 😭, I might go back to this chapter to see which commands to run.

Chapter 12: Creating a clean history

One can build a commit from parts of files

Using git add --patch <path>. This opens an interactive menu letting you choose which hunk(s) to add to a commit. Each hunk can be split into smaller hunks. Then one uses git commit as usual.

The book also mentions that git commit --patch also exists. It seems less intuitive to me to do the hunk selection at that stage.

Chapter 13: merging vs. rebasing

This chapter compares the strategies of two projects:

CMake’s branching and merging strategy.
Homebrew’s rebasing and squashing strategy. “This hides merge commits, evidence of branches, and temporary commits (for example, those that fix previous commits on the same branch) from the main branch.”

To me CMake history looks like a mess or even a nightmare so I guess I’m in favor of the Homebrew’s strategy!

But in each project, it is worth to have a discussion about the chosen strategy, to educate contributors.

I liked reading that in Homebrew, metadata is included into commit messages. For instance I enjoy the fact that GitHub automatically adds co-authors when you squash and merge a PR that add multiple contributors: https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

Chapter 14: Recommended team workflows

Have you heard of GitHub flow, Git flow? These are team workflows.

“The different strategies for deciding how and when to branch, merge or rebase as part of a team are called team workflows.”

I much prefer GitHub flow where the default branch is stable and where there is no long-lived development branch, because in my experience, having such a long-lived development branch has lead to more confusion than gains. For rOpenSci dev guide we went from some sort of Git flow to GitHub flow, where actual relases will be, well, actual releases/tags of the main branch. Changing flows was such a relief!

The author also presented his own two usual flows.

Conclusion

“Git in practice” by Mike McQuaid was a nice train travel companion. I feared it would be too dry or obscure but found it clear and well explained. I learnt a lot, but now, I will have to… practice what I learnt! 😁

Did you read this book, or another Git book? How did you learn Git? Any strong opinion on any of the topics mentioned in those notes?

3 (actually 4) neat R functions

Fri, 20 Oct 2023 00:00:00 +0000

Time for me to throw away my sticky note after sharing what I wrote on it!

`grep(...)` not `which(grepl(...))`

Recently I caught myself using which(grepl(...)),

animals <- c("cat", "bird", "dog", "fish")
which(grepl("i", animals))
#> [1] 2 4

when the simpler alternative is

animals <- c("cat", "bird", "dog", "fish")
grep("i", animals)
#> [1] 2 4

And should I need the values instead of the indices, I know I shouldn’t write

animals <- c("cat", "bird", "dog", "fish")
animals[grepl("i", animals)]
#> [1] "bird" "fish"

but

animals <- c("cat", "bird", "dog", "fish")
grep("i", animals, value = TRUE)
#> [1] "bird" "fish"

How to remember to use grep()? Re-reading oneself, or having code reviewed, probably helps, but why not automate this? When I shared my note to self on Mastodon, Hugo Gruson explained that detecting usage of which(grepl( was part of planned linters to be added to lintr from Google linting suite. This is excellent news!

`strrep()` and other defence tools against poor usages of `paste()`

Yihui Xie wrote a blog post inspired by my own series, where one of the three presented functions was one that was on my sticky note! I’ll still present it: strrep().

strrep() means “string repeat”. Instead of writing

paste(rep("bla", 3), collapse = "")
#> [1] "blablabla"

you can, and should, write

strrep("bla", 3)
#> [1] "blablabla"

I discovered this function because Hugo Gruson telling me about lintr inspired me to skim through lintr reference, where I saw “Raise lints for several common poor usages of paste()”. That linter would also tell you when you use paste(, sep = "") instead of paste0().

`startsWith()` and `endsWith()`

I learned about startsWith() and endsWith() by reading lintr reference but I also got notified about it when running lintr on a package I was working on. Have you ever tried running all linters on your code? Fun experience. Anyhow, one linter is Require usage of startsWith() and endsWith() over grepl()/substr() versions, with an interesting Details section on missing values.

Instead of writing

animals <- c("cat", "cow", "dog", "fish")
grepl("^c", animals)
#> [1]  TRUE  TRUE FALSE FALSE

I should write

animals <- c("cat", "cow", "dog", "fish")
startsWith(animals, "c")
#> [1]  TRUE  TRUE FALSE FALSE

A nice side-effect of the switch, beyond good practice for its own sake and more readability, is that the argument order is more logical in startsWith().

Similarly, instead of writing

animals <- c("cat", "cow", "dog", "fish")
grepl("t$", animals)
#> [1]  TRUE FALSE FALSE FALSE

I should write

animals <- c("cat", "cow", "dog", "fish")
endsWith(animals, "t")
#> [1]  TRUE FALSE FALSE FALSE

Conclusion

In this post I shared about grep() to be used in lieu of which(grepl()), about strrep() (string repetition) to be used in lieu of paste(rep(), collapse ="") and about startsWith() and endsWith() to be used in lieu of some regular expressions with respectively ^ and $.

Reading notes on A Philosophy of Software Design by John Ousterhout

Thu, 19 Oct 2023 00:00:00 +0000

When I see a book recommendation somewhere, be it for work or leisure, I often either order the book or set an alert in my favorite online second-hand bookstore. By the time I am notified the book is available, I sometimes don’t remember why I listed it! That’s what happened for the book A Philosophy of Software Design by John Ousterhout. I decided to trust my past self and buy it. Unfortunately I got the first not the second edition.

For a change I wrote some reading notes, reflecting what resonated more with me at this stage in my software development journey.

First and later impressions

I found the title of the book misleading: the word “philosophy” made it sound unapproachable, but it was approachable, and also rather short (about 169 pages, including examples no one will know you skipped¹).

I got a copy that was annotated. Sadly the notes didn’t provide any novel insight, but the handwriting is good and the notes repeat what’s important, perfect for future skimming.

I appreciated the tone of the book which wasn’t too dogmatic compared to other sources out there.

Complexity in software

The most important concept is the book is complexity, what it is and how to avoid it. The author defined three manifestations of complexity:

Change amplification, meaning it’s hard to change something. It reminds me of this quote I heard from Kara Woo’s posit keynote talk: “If you can’t make changes because you are afraid of breaking something, it’s already broken.".
Cognitive load, at which point I have to tell you to read The Programmer’s Brain by Felienne Hermans. I think about it regularly and see my R friends mentioning it.
Unknown unknowns. 👻

Priority to the interface

A point made at least twice in the book is that you should give priority to simplifying the interface of things not their implementation.

“When developing a module, look for opportunities to take a little bit of extra suffering upon yourself to reduce the suffering of your users”.

This is also present in the author’s idea of “Define errors out of existence”.

This made me think about retries in HTTP requests. When writing a function interfacing an API I could simply return the error, forcing the user to handle the error themselves, or I could add retries, that would potentially make all errors disappear. It’s not a perfect example since retries aren’t hard to add with say httr2 or httr, but that was what I was thinking of.

Another point about the interface is that you only split two things if it simplifies the interface. That’s a better way to think about it than just thinking in number of lines.

Tactical vs strategic programming

The author compares two approaches to programming:

Tactical programming is where you squash bugs and add features while racking up tech debt, not thinking about software design.
Strategic programming is where you take more time to implement things so that the software design remains good or even becomes better over time.

Unsurprisingly the author defends the second approach, explaining how it pays off. The author does acknowledge it might be hard to use the second approach depending on, for instance, deadlines.

Strategic programming, and the author’s insistent advice to “stay strategic” reminded me of a paragraph in the Tidyverse code review guide:

“On the other hand, it is the duty of the reviewer to make sure that each PR is of such a quality that the overall code health of the package is not decreasing as time goes on. This can be tricky, because codebases often degrade through small decreases in code health over time. This isn’t necessarily anyone’s fault. It is a natural process resulting from the accumulation of technical debt over time, which must continually be fought against.”

Comments

“Good comments can make a big difference in the overall quality of a system.”

I appreciated seeing a few chapters about comments. I wrote a post on comments myself earlier this year.

The author write that comments belong in the code, not the commit log, which is true but with the fledge package you can transform commit messages into changelog content. Furthermore, I feel that Git blame can be useful no matter how good your comments are.

The book mentions that if a piece of code is hard to comment maybe you should refactor it. I like how it’s similar to another idea in the book: if a variable is hard to name maybe it indicates some weakness of the code design.

Variable names

Apart from the idea I’ve just mentioned, one piece of advice that is reported in the book resonated with me. It’s a comment by Andrew Gerrand: “The greater the distance between a name’s declaration and its uses, the longer the name should be”.

Consistency

The author underlines the importance of consistency (in names, coding style, design patterns…), that makes your codebase easier to grasp and contribute to. To ensure consistency you should document it which made me think of contributing guides, and/or use automatic tools, which made me think of the lintr package.

Conclusion

I am glad I ordered this book past me had bookmarked! I found it was very readable, with some good advice and ways to express it. I am pretty sure I’ll skim it in the future. My first recommendation of resources about good code would remain The Art of Readable Code by Dustin Boswell and Trevor Foucher, The Programmer’s Brain by Felienne Hermans, Jenny Bryan’s talk “Code Smells and Feels” and Hadley Wickham’s Tidy Design newsletter that goes with a book-in-progress.

If you read the book, what were your take-aways? What is your favorite resource to learn how to program better?

Other reviews of / notes on the book

Unless you blog about it. ↩︎

The real reset button for local mess fom tests: withr::deferred_run()

Mon, 16 Oct 2023 00:00:00 +0000

This post was featured on the R Weekly highlights podcast hosted by Eric Nantz and Mike Thomas.

Following last week’s post on my testing workflow enhancements, Jenny Bryan kindly reminded me of the existence of an actual reset button when you’ve been interactively running tests that include some “local mess”: withr::local_envvar(), withr::local_dir(), usethis::local_project()… The reset button is withr::deferred_run().

It is documented in Jenny’s article about test fixtures:

Since the global environment isn’t perishable, like a test environment is, you have to call deferred_run() explicitly to execute the deferred events. You can also clear them, without running, with deferred_clear().

It is also mentioned in messages from withr.

For some reason it hadn’t clicked for me until now?! But now I know better!

Example: making a `withr::local_` mess and wiping it with `withr::deferred_run()`

Imagine a test file¹

test_that("My function works", {
  temp_dir <- withr::local_tempdir()
  withr::local_options(blop = TRUE)
  usethis::local_project(temp_dir, force = TRUE)
  withr::local_envvar("TEST_SWITCH" = "something")
  expect_something(my_function()) # not actual test code, you get the idea
})

Imagine the test fails and I run this in my R session probably after throwing a browser()² somewhere and running devtools::load_all():

temp_dir <- withr::local_tempdir()
withr::local_options(blop = FALSE)
usethis::local_project(temp_dir, force = TRUE)
#> ✔ Setting active project to '/tmp/RtmpTxY51U/file1dbe2032be39'
withr::local_envvar("TEST_SWITCH" = "something")

Then I do the actual debugging. At the end my session is all messy!

Sys.getenv("TEST_SWITCH")
#> [1] "something"
getOption("blop")
#> [1] FALSE
usethis::proj_sitrep() # cool function!
#> •   working_directory: '/home/maelle/Documents/blog/simplymaelle/content/post/2023-10-16-deferred-run'
#> • active_usethis_proj: '/tmp/RtmpTxY51U/file1dbe2032be39'
#> • active_rstudio_proj: <unset>
#> • Your working directory is not the same as the active usethis project.
#>   Set working directory to the project: `setwd(proj_get())`
#>   Set project to working directory:     `proj_set(getwd())`

Instead of fixing each thing by hand I can run

withr::deferred_run()

And now all is right in my session again, you’ll have to believe me or run the example yourself: indeed, demonstrating all of this in a knitr document makes it a bit more challenging. I swear that outside of a knitr document, this all works even for code that changes the current directory!

Sys.getenv("TEST_SWITCH")
getOption("blop")
usethis::proj_get()

So I can go and debug the next failing test. 😸

What about rlang’s local mess?

rlang itself has rlang::local_interactive() and rlang::local_options(). Luckily they are reset by withr::deferred_run().

I’m not sure exactly where the compatibility comes from, maybe from the standalone defer script. In any case, that’s good news!

Conclusion

So, in summary, if you’re running test code interactively to debug them, and this test code changed some things using withr::local_ or other compatible local_ functions, to get your session back to its initial state without restarting R, run withr::deferred_run()! Thanks Jenny for the reminder!

Test switches are sooo handy. ↩︎
For better debugging advice, refer to Shannon Pileggi’s materials! ↩︎

Two recent enhancements to my testing workflow

Mon, 09 Oct 2023 00:00:00 +0000

I spend a lot of quality time with testthat, that sometimes deigns to praise my code with emojis, sometimes has to encourage me. No one gets it right on their first try apparently?

Anyway, in honor of testthat 3.2.0 release 🎉 👏, I’d like to mention two small things that improved my testing workflow a whole lot!

Running one single test at a time

Under testthat 3.2.0 minor features lies a small gem:

test_file() gains a desc argument which allows you to run a single test from a file. #1776

This is huge! When I am struggling with a single test called “My great function works”, I can now call devtools::test_active_file(desc = "My great function works")! I’m not entirely sure yet it perfectly integrates with snapshot recording, but I’m very happy nonetheless.

Now if I were struggling with more tests at a time (Remember, no one gets it right on their first try 😅), I should use the handy lazytest package.

Getting back to the package directory and project

Update on 2023-10-16: here’s a better, more general solution to fix local mess from a test.

I recently realized that if…

you develop a package using RStudio IDE (which I do!);
your package tests change the current directory / the usethis project (for reasons 😅 ) for instance with withr::local_dir()/usethis::local_project();
you run test code interactively & thus lose your current directory/project…

the wonderful studioapi::getActiveProject() function remembers where the actual project, that is your package, is, very much like clicking on the RStudio hex logo in the Files tab.

Based on that knowledge I added a helper function to the tests of a package I work on, to have it at hand after running devtools::load_all():

back_to_start <- function() {
  current_dev_project <-  rstudioapi::getActiveProject()
  setwd(current_dev_project)
  usethis::proj_set(current_dev_project)
}

So I can make a mess in my console and then run back_to_start() to get back where I started.

Conclusion

In this short post I presented two recent enhancements to my testing workflow: the adoption of the new desc argument of testthat::test_file() to run one single test at a time; and the “discovery” of a handy way to find my way back to the package directory after experimenting in temporary toy directories.

How to translate your package's messages with {potools}

Fri, 06 Oct 2023 00:00:00 +0000

This post was featured on the R Weekly highlights podcast hosted by Eric Nantz and Mike Thomas.

In November I’ll give a talk about multilingualism in R at the Spanish R conference in Barcelona (😍). I can’t wait! Until then, I need to prepare my talk. 😅 I plan to present the rOpenSci “multilingual publishing” project but also other related tools, like potools. In this post, I’ll walk you through a minimal example of using potools to translate messages in an R package!

What is potools?

In R, you can provide messages in different languages by using .po, .pot and .mo files. If you provide messages in English and their translation to Spanish, an user who sets the environment variable LANGUAGE to es will get to see Spanish messages rather than the default English messages.

The potools package by Michael Chirico is the roxygen2 of .po, .pot and .mo files: as you could well write Rd files by hand¹, you could write the translation files by hand… but things are easier with potools.

Create package

I create a package called pockage, with my usual usethis workflow. It initially has a single function, that tries to extract the user’s name via the whoami package, and then says hello using the cli package:

#' Say hello
#'
#' @return Nothing
#' @export
#'
#' @examples
#' speak()
speak <- function() {
  name <- whoami::fullname(fallback = "user")
  cli::cli_alert_info("Hello {name}!")
}

Let’s try it out:

pockage::speak()
#> ℹ Hello Maëlle Salmon!

Install potools

I install potools from GitHub using pak::pak("MichaelChirico/potools").

Register potools style in DESCRIPTION

I shall use potools explicit style, which means it will recognize strings as translatable only if I mark them as so. Therefore I add these lines to DESCRIPTION:

Config/potools/style: explicit

Create `tr_()` function and use it

Following potools’ vignette for developers, I run usethis::use_r("utils-potools") and paste the definition an internal function in that new file:

tr_ <- function(...) {
  enc2utf8(gettext(paste0(...), domain = "R-pockage"))
}

I then modify the speak() function:

#' Say hello
#'
#' @return Nothing
#' @export
#'
#' @examples
#' speak()
speak <- function() {
  name <- whoami::fullname(fallback = "user")
  cli::cli_alert_info(tr_("Hello {name}!"))
}

The difference is that the string “Hello {name}!” is now marked as translatable.

Create the general translation file

I run potools::po_extract() to create the po/R-pockage.pot file.

Create the translation file for French and fill it

Then I run potools::po_create("fr") to create the file holding the translation of the string to French.

I obtain this po/R-fr.po file where I edit two lines (Last-Translator, and msgstr at the bottom):

msgid ""
msgstr ""
"Project-Id-Version: pockage 0.0.0.9000\n"
"POT-Creation-Date: 2023-10-06 10:45+0200\n"
"PO-Revision-Date: 2023-10-06 10:33+0200\n"
"Last-Translator: Malle Salmon\n"
"Language-Team: none\n"
"Language: fr\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=ASCII\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"

#: mensaje.R:9
msgid "user"
msgstr "utilisateur·rice"

#: mensaje.R:10
msgid "Hello {name}!"
msgstr "Salut {name} !"

Compile the translation

I compile with potools::po_compile() which creates the .mo binary for the French language in inst/po.

Try out the translation

I load or install&load the package, then I use the following code, magically using withr to set the LANGUAGE environment variable locally:

withr::with_language("fr", pockage::speak())
#> ℹ Salut Maëlle Salmon !
# as opposed to
pockage::speak()
#> ℹ Salut Maëlle Salmon !

Quite neat, right?

Add languages

I then add two other languages, unstoppable as I am now, by running potools::po_create(c("es", "ca")). I add the Spanish and Catalan translations of the string in respectively po/R-es.po and po/R-ca.po. I compile with potools::po_compile() which creates the .mo binary for the Spanish and Catalan languages in inst/po.

After installing the pockage package again, I get:

withr::with_language("fr", pockage::speak())
#> ℹ Salut Maëlle Salmon !
withr::with_language("es", pockage::speak())
#> ℹ Hola Maëlle Salmon!

# I swear it's pronounced differently
withr::with_language("ca", pockage::speak())
#> ℹ Hola Maëlle Salmon!

Translate more strings

Now imagine I also want to translate the fallback of the whoami function call, for the case whoami can’t identify the user.

#' Say hello
#'
#' @return Nothing
#' @export
#'
#' @examples
#' speak()
speak <- function() {
  name <- whoami::fullname(fallback = tr_("user"))
  cli::cli_alert_info(tr_("Hello {name}!"))
}

I run

potools::po_extract() again;
then potools::po_update();
after which I need to go add a msgstr under msgid "user" in po/R-fr.po, po/R-es.po, po/R-ca.po;
finally I run potools::po_compile().

Conclusion

Find my final pockage package on GitHub.

To translate messages with potools in the explicit style one needs to:

register the potools style in DESCRIPTION;
create and use a tr_() internal function to wrap strings to be translated;
run potools::po_extract() at the beginning of the translation efforts and every time strings wrapped in tr_() are changed, deleted, added;
run potools::po_create() once per non-default language to be supported;
run potools::po_update() after potools::po_extract() every time strings wrapped in tr_() are changed, deleted, added;
run potools::po_compile() every time the translation source files are changed.

potools has one vignette for developers and one for translators that I’d recommend reading, because they provide useful advice beyond the basic workflow that I illustrated here.

which is what I first learnt to do years and years ago. ↩︎

3 R functions that I enjoy

Fri, 29 Sep 2023 00:00:00 +0000

Straight from my sticky note, three functions that I like a lot, despite their not being new at all… But maybe new to some of you?

`sprintf()`, the dependency-free but less neat “glue”

Imagine I want to tell you who I am.

I could write

name <- whoami::fullname()
github_username <- whoami::gh_username()
glue::glue("My name is {name} and you'll find me on GitHub as {github_username}!")
#> My name is Maëlle Salmon and you'll find me on GitHub as maelle!

(Maybe the whoami package by Gábor Csárdi is a sub topic of this section?! So handy!)

The code above is very readable. The nice syntax with curly braces is something one finds again in the cli package.¹

Now, there’s a dependency-free version of the glue code! Albeit a bit uglier 🤐

name <- whoami::fullname()
github_username <- whoami::gh_username()
sprintf(
  "My name is %s and you'll find me on GitHub as %s!",
  name,
  github_username
)
#> [1] "My name is Maëlle Salmon and you'll find me on GitHub as maelle!"

Sure it’s less readable, since the replacements are identified by their position, but I often find myself using it! I clearly remember seeing it in other people’s code and wondering what that was.

It’s a pattern one finds in other languages: the manual page for the function states “Use C-style String Formatting Commands”, and I know the Hugo equivalent.

`append()` and its after argument

To append values to a vector, I mostly use c(), but I recently discovered the base R function append() and its after argument that indicates where to include the new values! By default, the values are appended at the end of the vector.

I most recently used append() to create a test fixture.

x <- c("thing", "stuff", "element")
values <- c("bla", "blop")
append(x, values)
#> [1] "thing"   "stuff"   "element" "bla"     "blop"
append(x, values, after = 2)
#> [1] "thing"   "stuff"   "bla"     "blop"    "element"

It’s not a function I use every day, but it can come in handy depending on what you’re doing!

For my fellow XML fans out there, it reminds me of the .where argument of xml2::xml_add_sibling() and xml2::xml_add_child().

`servr::httw()` to serve a local directory as a website

Do you know about the servr package by Yihui Xie?

Its use case is having a local directory that is the source of a website, and wanting to preview it locally in your browser as if it were served by a real server.

I make use of it when working on the babelquarto package for instance, that builds multilingual Quarto books or websites. In the code of the multilingual books/websites, links to the other language versions are relative so they don’t work if I simply open HTML files in my browser. So, instead, I write servr::httw(<directory-with-the-website-source>).

You can also use servr if you want to preview locally your pkgdown website and be able to use the search function.

There’s servr::httw() that watches changes in the directory, and the more simple servr::httd().

Conclusion

Today I shared about sprintf() for glue-like functionality, append() and its after argument for appending values where you want in a vector, and servr::httw() for serving static files.

Curious about cli? Come to this cool rOpenSci coworking session next week! ↩︎

R functions that shorten/filter stuff: less is more

Thu, 31 Aug 2023 00:00:00 +0000

My sticky note is full! And luckily all functions on it can be squeezed into a similar topic: making things smaller!

Make lists smaller with `purrr::compact()`, `purrr::keep()`, `purrr::discard()`

Once upon a time there was a list (isn’t this the beginning of all R scripts?!)

my_list <- list(
  name = "Maëlle",
  city = "Nancy",
  r_problems_encountered = Inf,
  python_skills = NULL
)

Imagine you want to get rid of NULL elements. That’s what purrr::compact() does!

purrr::compact(my_list)
#> $name
#> [1] "Maëlle"
#> 
#> $city
#> [1] "Nancy"
#> 
#> $r_problems_encountered
#> [1] Inf

Imagine you want to only keep elements that are infinite numbers. For some unknown reason is.infinite() works on strings (but not on NULL) so the following works.

my_list |>
  purrr::compact() |>
  purrr::keep(is.infinite)
#> $r_problems_encountered
#> [1] Inf

Similarly, imagine you want to discard all elements that are character.

purrr::discard(my_list, is.character)
#> $r_problems_encountered
#> [1] Inf
#> 
#> $python_skills
#> NULL

We’ve only used built-in functions as predicate functions (the second argument of all these calls). You can really unleash the power of these three functions when pairing them with your own predicate functions!

Related to these functions, don’t miss Jonathan Carroll’s great post “Four Filters for Functional (Programming) Friends”.

Make string (vectors) smaller with `stringr::str_squish()`, `stringr::str_subset()`

This week when removing stringr from the dependencies of the glitter package I discovered stringr::str_subset(). It will help you keep, in purrr’s parlance, only the elements of a string vector that have a match with a pattern.

string <- c("R", "Python", "Julia", "Ruby")
stringr::str_subset(string, "R")
#> [1] "R"    "Ruby"

It will also help you discard them if you use the negate arguments.

stringr::str_subset(string, "R", negate = TRUE)
#> [1] "Python" "Julia"

In glitter, where I chose to write similar-looking functions for the stringr functions that we used, I replaced stringr::str_subset() with

str_subset <- function(x, pattern, negate = FALSE) {
  grep(pattern, x, value = TRUE, invert = negate)
}

since its docs mentioned this base R’s equivalence.

Another stringr function that makes things smaller is stringr::str_squish() that removes useless whitespace from a string vector: whitespace at the start and end, and repeated whitespace in the middle.

text <- " So much\n\n useless    whitespace    "
stringr::str_squish(text)
#> [1] "So much useless whitespace"

It’s documented on the same page as stringr::str_trim() which is stringr’s own trimws() (a base R function) that only removes whitespace from the start and end. 💇

Make your code shorter with `purrr::partial()`

I am working on a vignette for glitter where I write (well, copy-paste) the same function call over and over again.

blablabla %>%
  spq_perform(
    endpoint = "https://ld.admin.ch/query", 
    request_type = "body-form"
  )

From the developer’s perspective it makes me wonder whether we should add some options as default. From an user’s perspective it reminded me of purrr::partial()! With purrr::partial one easily creates a function that corresponds to a function with some arguments pre-filled.

So, as a glitter user, I could use

my_spq_perform <- purrr::partial(
    spq_perform,
    endpoint = "https://ld.admin.ch/query", 
    request_type = "body-form"
)

blablabla %>%
  my_spq_perform

Quite neat!

Conclusion

In this short (of course) post I presented functions for making lists, string vectors and code shorter by filtering or squishing them. 🔨

Three (four?) R functions I enjoyed this week

Wed, 30 Aug 2023 00:00:00 +0000

There are already three functions of note on a piece of paper on my desk, so it’s time to blog about them!

This post was featured on the R Weekly podcast by Eric Nantz and Mike Thomas.

How does this package depend on this other package? `pak::pkg_deps_explain()`

The pak package by Gábor Csárdi makes installing packages easier. If I need to start working on a package, I clone it, then run pak::pak() to install and update its dependencies. It’s a “convenience function” that is convenient for sure! Bye bye remotes::install_deps().

Anyway, pak is truly a treasure trove, even for challenges that happen less often. Earlier this week I was seeing an error in a GitHub Actions log due to a package that I didn’t know was a dependency of the package I was working on. It was clearly not a direct dependency, it was not listed in DESCRIPTION. It was a dependency of a dependency and I couldn’t guess which one it was… Luckily one doesn’t need to guess! Although I didn’t think of it right away, the pak::pkg_deps_explain() is my friend in such cases¹!

For instance, if you are wondering why usethis depends on httr2,

pak::pkg_deps_explain("usethis", "httr2")
#> 
#> ✔ Updated metadata database: 3.97 MB in 4 files.
#> 
#> ℹ Updating metadata database
#> ✔ Updating metadata database ... done
#> 
#> usethis -> gh -> httr2

usethis depends on gh (client for GitHub API) that in turns depends on httr2.

Next time I encounter a similar problem I can only hope I’ll remember about this function sooner!

Where in the file are there non ASCII characters? `tools::showNonASCIIfile()`

If you get the dreaded R CMD check WARNING “Found the following file with non-ASCII characters” and can’t see at once what character that is in the file, you don’t need to comb through each line of code. You can simply run tools::showNonASCIIfile(<filename>). After that an easy fix can be to replace then with the \uxxxx escape as indicated in the WARNING. I’m not saying it’s always that easy but that’s what happened to me this week! I found the correct escape by using a search engine.

How do these two text files differ? `tools::Rdiff()` or `gert::git_diff_patch()`

I’m working on the babeldown R package that helps translate mostly Markdown files. Support for first-time automatic translation is there, but a next step is to add support for automatic update of a translation based on a git commit. Say I have a file in English and its translation in Spanish and a contributor edits part of the Spanish file. We want to have a function update the English file based on this diff. As part of that work I need to programmatically parse the difference between two text files.

I saw gert::git_diff() but I didn’t find the patch to be easily parsable.

I had better luck with tools::Rdiff() especially after a GitHub search showed me an example using it with Log = TRUE. Before that I was unsuccessfully trying capture.output() to capture what was printed… I should have read the arguments list more carefully.

original_lines <- c(
  "# Title", "",
  "## Subtitle", "",
  "Some info", "",
  "First line of a paragraph",
  "Second line of a paragraph"
)

amended_lines <- c(
  "# BIG Title", "",
  "## Subtitle", "",
  "Some info", "",
  "First line of a paragraph",
  "Second line of a paragraph",
  "More info"
)

file1 <- withr::local_tempfile()
file2 <- withr::local_tempfile()

brio::write_lines(original_lines, file1)
brio::write_lines(amended_lines, file2)

diff <- tools::Rdiff(file1, file2, Log = TRUE)
#> 
#> files differ in number of lines:
diff
#> $status
#> [1] 1
#> 
#> $out
#> [1] "files differ in number of lines" "1c1"                            
#> [3] "< # Title"                       "---"                            
#> [5] "> # BIG Title"                   "8a9"                            
#> [7] "> More info"

What I especially liked in this output, is the lines <line-numbers-in-original-file><letter-indicated-status><line-numbers-in-new-file> where the status can be a (added), d (deleted) or c (changed).

In comparison here’s what gert gives me.

dir <- withr::local_tempdir()
gert::git_init(dir)

brio::write_lines(original_lines, file.path(dir, "file.txt"))
gert::git_add("file.txt", repo = dir)
#>       file status staged
#> 1 file.txt    new   TRUE
gert::git_commit("first commit", repo = dir)
#> [1] "b56a48d10dc033e4c6a3bb81be6e18d27b34032a"

brio::write_lines(amended_lines, file.path(dir, "file.txt"))
gert::git_add("file.txt", repo = dir)
#>       file   status staged
#> 1 file.txt modified   TRUE
commit_of_interest <- gert::git_commit("second commit", repo = dir)

cat(gert::git_diff_patch(commit_of_interest, repo = dir))
#> diff --git a/file.txt b/file.txt
#> index 6d79a01..44a4269 100644
#> --- a/file.txt
#> +++ b/file.txt
#> @@ -1,4 +1,4 @@
#> -# Title
#> +# BIG Title
#>  
#>  ## Subtitle
#>  
#> @@ -6,3 +6,4 @@ Some info
#>  
#>  First line of a paragraph
#>  Second line of a paragraph
#> +More info

I’m not exactly sure yet whether tools::Rdiff() will be enough for my use case but it might!

Conclusion

My sticky note mentioned pak::pkg_deps_explain(), tools::showNonASCIIfile() (what case is this 😅) and tools::Rdiff(), that helped me make progress on R code earlier this week. Time to break out a fresh sticky note and see what ends up on it!

I first wrote it “pak_deps_explain” instead of “pkg_deps_explain”. 😬 ↩︎

Reducing my for loop usage with purrr::reduce()

Wed, 26 Jul 2023 00:00:00 +0000

I (only! but luckily!) recently got introduced to the magic of purrr::reduce(). Thank you, Tobias! I was told about it right as I was unhappily using many for loops in a package¹, for lack of a better idea. In this post I’ll explain how purrr::reduce() helped me reduce my for loop usage. I also hope that if I’m doing something wrong, someone will come forward and tell me!

This post was featured on the R Weekly podcast by Eric Nantz and Mike Thomas.

Before: many for, much sadness

I was starting from a thing, that could be a list, or even a data.frame. Then for a bunch of variables, I tweaked the thing. My initial coding pattern was therefore:

for (var in variables_vector) {
  thing <- do_something(thing, var, other_argument = other_argument)
}

I was iteratively changing the thing, along a variables_vector, or sometimes a variables_list.

Silly example

Ugh, finding an example is hard, it feels very contrived but I promise my real-life adoption of purrr::usage() was life-changing!

# Some basic movie information
movies <- tibble::tribble(
  ~title, ~color, ~elements,
  "Barbie", "pink", "shoes",
  "Oppenheimer", "red", "history"
)

# More information to add to movies
info_list <- list(
  list(title = "Barbie", info = list(element = "sparkles")),
  list(title = "Barbie", info = list(element = "feminism")),
  list(title = "Oppenheimer", info = list(element = "fire"))
)

# Don't tell me this is weirdly formatted data,
# who never obtains weirdly formatted data?!
info_list
#> [[1]]
#> [[1]]$title
#> [1] "Barbie"
#> 
#> [[1]]$info
#> [[1]]$info$element
#> [1] "sparkles"
#> 
#> 
#> 
#> [[2]]
#> [[2]]$title
#> [1] "Barbie"
#> 
#> [[2]]$info
#> [[2]]$info$element
#> [1] "feminism"
#> 
#> 
#> 
#> [[3]]
#> [[3]]$title
#> [1] "Oppenheimer"
#> 
#> [[3]]$info
#> [[3]]$info$element
#> [1] "fire"

add_element <- function(movies, info) {
  movies[movies[["title"]] == info[["title"]],][["elements"]] <-
    toString(c(
      movies[movies[["title"]] == info[["title"]],][["elements"]],
      info[["info"]][[1]]
    ))
  movies
}

Now how do I add each element of the list to the original table? I could type something like:

for (info in info_list) {
  movies <- add_element(movies, info)
}
movies
#> # A tibble: 2 × 3
#>   title       color elements                 
#>   <chr>       <chr> <chr>                    
#> 1 Barbie      pink  shoes, sparkles, feminism
#> 2 Oppenheimer red   history, fire

It’s not too bad, really. But since there’s another way, we can change it.

After

With purrr::reduce()

for (var in variables_vector) {
  thing <- do_something(thing, var)
}

can become

thing <- purrr::reduce(variables_vector, do_something, .init = thing)

And (notice the other argument),

for (var in variables_vector) {
  thing <- do_something(thing, var, other_argument = other_argument)
}

can become

thing <- purrr::reduce(
  variables_vector, 
  \(thing, x) do_something(thing, x, other_argument = other_argument), 
  .init = thing
)

I haven’t completely internalized the pattern above but the documentation of purrr::reduce() states

“We now generally recommend against using … to pass additional (constant) arguments to .f. Instead use a shorthand anonymous function:

Instead of x |> map(f, 1, 2, collapse = “,") do: x |> map((x) f(x, 1, 2, collapse = “,")) This makes it easier to understand which arguments belong to which function and will tend to yield better error messages.”

It might remind you of how things work for dplyr::across() these days.

Back to our silly example!

# Some basic movie information
movies <- tibble::tribble(
  ~title, ~color, ~elements,
  "Barbie", "pink", "shoes",
  "Oppenheimer", "red", "history"
)

# More information to add to movies
info_list <- list(
  list(title = "Barbie", info = list(element = "sparkles")),
  list(title = "Barbie", info = list(element = "feminism")),
  list(title = "Oppenheimer", info = list(element = "fire"))
)

add_element <- function(movies, info) {
  movies[movies[["title"]] == info[["title"]],][["elements"]] <-
    toString(c(
      movies[movies[["title"]] == info[["title"]],][["elements"]],
      info[["info"]][[1]]
    ))
  movies
}

purrr::reduce(info_list, add_element, .init = movies)
#> # A tibble: 2 × 3
#>   title       color elements                 
#>   <chr>       <chr> <chr>                    
#> 1 Barbie      pink  shoes, sparkles, feminism
#> 2 Oppenheimer red   history, fire

If we tweak the add_element() function to add a separator argument to it,

add_element <- function(movies, info, separator) {
  movies[movies[["title"]] == info[["title"]],][["elements"]] <-
    paste(c(
      movies[movies[["title"]] == info[["title"]],][["elements"]],
      info[["info"]][[1]]
    ), collapse = separator)
  movies
}

purrr::reduce(
  info_list, 
  \(movies, x) add_element(movies, x, separator = " - "), 
  .init = movies
)
#> # A tibble: 2 × 3
#>   title       color elements                   
#>   <chr>       <chr> <chr>                      
#> 1 Barbie      pink  shoes - sparkles - feminism
#> 2 Oppenheimer red   history - fire

purrr::reduce(
  info_list, 
  \(movies, x) add_element(movies, x, separator = " PLUS "), 
  .init = movies
)
#> # A tibble: 2 × 3
#>   title       color elements                         
#>   <chr>       <chr> <chr>                            
#> 1 Barbie      pink  shoes PLUS sparkles PLUS feminism
#> 2 Oppenheimer red   history PLUS fire

And voilà!

Conclusion

In this post I presented my approximate understanding of purrr::reduce(), that helped me avoid writing some for loops and instead more elegant code… or at least helped me understand a pattern that in the future I could use elegantly. I can only hope I purrr::accumulate() more experience, as I very much still feel like a newbie.

For more information I’d recommend reading the documentation of purrr::reduce() to be aware of other features, the content on the reduce family in Advanced R by Hadley Wickham… and release-watching the purrr repo to keep up-to-date with latest recommendations. You can also use GitHub Advanced Search to find examples of usage of the function in, say, CRAN packages.

Edit: For another take of / use case of purrr::reduce(), June Choe wrote a nice detailed tutorial “Collapse repetitive piping with reduce()".

The package is glitter, where we store query objects as a list. ↩︎

Three useful (to me) R notions

Mon, 24 Jul 2023 00:00:00 +0000

Following my recent post on three useful (to me) R patterns, I’ve written down three other things on a tiny sticky note. This post will allow me to throw away this beaten down sticky note, and maybe to show you one element you didn’t know?

`nzchar()`: “a fast way to find out if elements of a character vector are non-empty strings”

One of my favorite testing technique is the escape hatch strategy, about which I wrote a post on the R-hub blog: you make part of your code responsive to an environment variable, and you locally set that environment variable in your tests. In your code, to determine whether the environment variable has been set to any non-empty string, you can use the nzchar() function.

nzchar("Hello World")
#> [1] TRUE
nzchar("")
#> [1] FALSE
nzchar(NA)
#> [1] TRUE
nzchar(NA, keepNA = TRUE)
#> [1] NA

# Now with an environment variable
nzchar(Sys.getenv("blopblopblop"))
#> [1] FALSE
withr::with_envvar(
  new = c("blopblopblop" = "bla"),
  nzchar(Sys.getenv("blopblopblop"))
)
#> [1] TRUE

Allow me to mention the obvious: if you’re using nzchar() to examine an environment variable, do not forget Sys.getenv(). I spent too much time looking at the following piece of code last week: nzchar("BABELQUARTO_TESTS_URL") wondering why it didn’t do what I expected it to do… I had to replace it with nzchar(Sys.getenv("BABELQUARTO_TESTS_URL")).

The backports R package, and its README

R is quite stable to say the least, but useful new functions have been added to it over time. For instance, R 3.3.0 saw the birth of trimws(), that trims white space.

trimws(c("useless whitespace ", " all around  "))
#> [1] "useless whitespace" "all around"

To use such functions in your package, you might need to set a dependency on a recent version of R, making it harder for users with restrictive rights on their R install, to use your package. The backports R package can help you avoid to set that dependency: it provides backports for these useful new functions. The README explains how to set up its usage.

I like both the existence of backports and its README because it is a list of these newer functions! I can’t pretend I’m a hipsteR but there are gems I forget or never heard about. Reading the list helps me see what’s there! For similar reasons I find the reference index of the lintr package to be a treasure trove.

Using the first non-missing thing: coalescing

Sometimes you might have an ordered “wish list” in your code: use A, then if A is missing, use B, then if B is missing, use C, and so on and so forth until a default value or error. The pkgdown package for instance has a function called path_first_existing().

I only recently realized this idea is usually called coalescing. dplyr has a coalesce() function.

“Given a set of vectors, coalesce() finds the first non-missing value at each position. It’s inspired by the SQL COALESCE function which does the same thing for SQL NULLs.”

I know close to NULL about SQL, but I did notice the COALESCE function in another query language, SPARQL, for which Lise Vaudor and I are working on a DSL, meaning a helper package, called glitter.

It makes me very happy to learn such an useful new word, or at least it usage in the programming realm.

Conclusion

After writing a bit about nzchar(), the backports package and the idea of coalescence, I can now throw away my sticky note! Let’s see what makes its way onto a new sticky note…

Three useful (to me) R patterns

Tue, 06 Jun 2023 00:00:00 +0000

This post was featured on the R Weekly highlights podcast hosted by Eric Nantz and Mike Thomas.

I’m happy to report that I thought “oh but I know a better way to write that code!” a few times lately when reading old scripts of mine, or scripts by others. It’s a good feeling because it shows progress! I’ve tooted about all three things I’ll present in this post: After reading Julia Evans’ post about blogging, I decided to train the blogging muscle a bit using these low-hanging fruits/toots¹.

Combine a list of default values with a list of custom values

Toot

Imagine 💭

🎚️ you have a default_values list and 👇 want to let the user pass a custom_values list to override some of them.

✨ utils::modifyList(default_values, custom_values)does that!

So say you had code à la

default_values <- list(a = 1, b = 2)
options <- list(b = 56)
temporary_list <- default_values
temporary_list[names(options)] <- options
options <- temporary_list

options
#> $a
#> [1] 1
#> 
#> $b
#> [1] 56

You can write it like so

default_values <- list(a = 1, b = 2)
options <- list(b = 56)
options <- modifyList(default_values, options)
options
#> $a
#> [1] 1
#> 
#> $b
#> [1] 56

I learnt about that function in pkgdown source.

Use a default if the user provided NULL

Toot

Do you know the rlang %||% operator?²

Code like

if (is.null(blop)) {
  blop <- 42
}

can become

blop <- blop %||% 42

Related to this, I’d recommend package developers read the chapter of the Tidyverse design guide on defaults, especially the section on the NULL default.

Extract common values or different / unique values from two vectors

Toot

Say I have a vector a and a vector b, and I need the unique a values that are not in b.

a <- c("thing", "object")
b <- c("thing", "gift")

I tended to write something like

unique(a[!(a %in% b)])
#> [1] "object"

(or without the unique() if a has only distinct values)

that can be

setdiff(a, b)
#> [1] "object"

Similarly, when looking for the unique values of the two vectors combined, instead of

unique(c(a, b))
#> [1] "thing"  "object" "gift"

I can write

union(a, b)
#> [1] "thing"  "object" "gift"

Because I’ve noticed I didn’t know these base R functions well enough, I open the Set Operations manual page more often, by typing ?setdiff for instance.

Salix Dubois helpfully noted the functions can be slower, and that one might not always want to drop duplicates.

Conclusion

In this post I presented three basic (set of) functions (not all base functions) that I’ve found serve me well: utils::modifyList(), rlang::%||% and base Set Operations. I’m glad they’re now part of my R vocabulary.

Note that you might still prefer the longer version of some of these patterns, depending on your needs, your code readers, etc. I won’t judge!

I’m curious to see what three new things I’ll have learnt in a few months (and will try not to beat myself up for not learning about them sooner 😇). If you’re interested about code quality in general, you might enjoy this post by Christophe Dervieux and myself on the R-hub blog.

I hope it’s a myth that your puns need to be good! ↩︎
Please don’t ask me to say this aloud, I have no idea how it’s pronounced. ↩︎

git and GitHub in R for the casual user

Thu, 12 Jan 2023 00:00:00 +0000

If you’ve been taught git and GitHub but practice so rarely that you’re discouraged, what should you do to re-start more easily? Let’s imagine you have to, or really want to, use git and GitHub for your next analysis project. Here’s what I would recommend…

I assume you already own a GitHub account. If not, refer to happygitwithr guidance.

Thanks to the people who shared recommendations on Mastodon, whose names are acknowledged in the rest of the post!

Prepare your tissues…

Or rather your support system. Make sure a collaborator with more git and GitHub practice is on call! If not, remember where you post your R questions. This way if you get stuck somewhere along the line, things will get less desperate.

You might also want to prepare your favorite hot beverage, but you probably won’t be able to read git guidance from tea leaves, hence the need for an actual support plan.

Clear notifications

This step is not completely necessary but could help you for the next re-start. Look at your GitHub notifications! Hopefully you weren’t getting them by email the whole time…

If you see you’ve been getting far too many notifications, tweak the settings!

Notification settings;
Threads (issues, PRs) you are subscribed to;
Repositories you are watching (I find custom watching much more rewarding).

This way next time you open GitHub, your notifications should have a better signal to noise ratio.

Now if you notice you’ve been missing important notifications, try to either commit to log in more often or get those notifications out of GitHub and into your preferred system. For instance, tell collaborators it’s safer to email you. Another example: maybe don’t watch dplyr repository, but do subscribe to the Tidyverse blog.

Inject a bit of motivation

Why are you doing this anyway? 😅 Good reads are the “Big picture” section of happygitwithr (by Jenny Bryan and collaborators) and the Openscapes GitHub illustrated series by Allison Horst and Julie Lowndes.

Decide whether to use GitHub Desktop

After I first shared my post on Mastodon, several people (Robert M Flight, Steven P Sanderson, Jim Gardner, Francisco Rodriguez-Sanchez) recommended using GitHub Desktop. Its nice interface would, as far as I understand, replace usethis. It’d mean you have R open somewhere, do your thing there, and regularly you do the git/GitHub stuff in the GitHub Desktop interface (setting up a GitHub repository, commits, pushes).

Check your R/GitHub situation

Skip if you use GitHub Desktop (and have installed git!). 😉

In this I assume you have R and RStudio IDE installed. One problem at a time. 😸

Tip by Hao Ye: first setup R so that it loads usethis at each session start, so you might skip writing usethis:: in front of each usethis function.

Let’s see if you have git correctly installed and hooked with GitHub, and remediate if not. Run usethis::git_sitrep(). Also run usethis::gh_token_help().

You might also want to run the whole usethis article about git and gitHub credentials. On Linux, you might need Managing GitHub credentials from R, difficulty level linux by Danielle Navarro.

For git installation, refer to happygitwithr guidance.

Do not be surprised if your tokens expired since the last time you used GitHub from R. Also do not be surprised if it all takes a little while. Take notes on what you’re doing. Next time you might be able to re-start after just running the two sitrep functions!

When re-creating tokens, make sure to give them an informative name (related to the computer you use them for, the project you use them for) as this is very useful for token management. I swear that no longer calling my tokens “mytoken1” has improved my life.

Last but not least, run usethis::git_vaccinate(). It will prevent you leaking credentials, or useless files (.DS_Store), on GitHub.

Run a full cycle

Even if your fantastic analysis project is still waiting for you to finish the mise en place, it might be good to run a full cycle to see whether things work.

Create a project with usethis::create_project("../toy").
Initiate git usage with usethis::use_git().
Add a text file containing, for instance “I will excel at git today!”.
git commit this file. Use RStudio git tab, or gert::git_add() then gert::git_commit(), or the command line… whatever you were taught and remember vaguely liking.
Create a GitHub repository with usethis::use_github(), or GitHub Desktop.

Did it work? If not, turn to the docs, a search engine and your support network. If yes, continue!

Initiate your project right

Once you have your project folder set up with create_project() for instance,

Initiate git usage with usethis::use_git().
Make sure .gitignore is correct: is there for instance a gigantic data file you work with locally but that you do not wish to commit to git, lest it is too heavy for git or GitHub? List it in .gitignore. You can use usethis::use_git_ignore() for that.

Upload your project to GitHub

First, where on GitHub do you want to upload your project?

Your personal account or an organization?

If you wish to upload it to an organization, be aware that it might fail if you don’t have the correct access level. In some GitHub organizations any member can create a repository, in others only owners can. Contact an organization owner if needed.

You might want to create a GitHub organization for your project. Even with a single project in it, the big advantage is that you can better give access to your project to collaborators, with more granularity than for a repository hosted in a personal account.

Private or public?

Do you want your repository to be public or private? Sometimes the choice is obvious, for instance your project is top secret or you really don’t want to share it as is right now. Private repositories have a few less features than public repositories (for instance draft pull requests) depending on your GitHub subscription.

`use_github()` (or GitHub Desktop)

Now run usethis::use_github(organisation = "your-account-username-or-an-organization", private = TRUE) with the correct arguments. After this you should see your project in a GitHub repository! 🎉

Make commits as you work

Ideally, make a commit everytime you make a substantial, coherent set of changes. To encourage you, you could fill a jar of marbles, one marble per commit?

Realistically, at least make a commit every time you take a break, especially when leaving at the end of your working session.

Push often, each time you make a commit, with the RStudio git tab, the command line or gert::git_push(), or GitHub Desktop.

If you feel icky submitting ugly commits to your pristine repo, think about the idea of branches. With usethis you

create one with gert::git_branch_create(),
experiment and make the ugly commits in there until the whole set of changes is good to go,
then open a “pull request” on GitHub with usethis helpers,
then you can “squash and merge” it which means the whole thing becomes one single beautiful commit.

With GitHub Desktop you

create a branch,
experiment and make the ugly commits in there until the whole set of changes is good to go,
then open a “pull request” on GitHub,
then you can “squash and merge” it which means the whole thing becomes one single beautiful commit.

Keep track of your TODOS

Since you are working on GitHub, you could use the issue tracker of your repository to write down ideas! You can create labels, milestones (so you could procrastinate a lot here 😅).

On Mastodon, Francisco Rodriguez-Sanchez underlined how handy it can be to use GitHub for project management.

Put the “Closed” sign before you leave

Have you done your last commit, last push and opened your last TODO issue? Have you looked over the history of what you did today and earlier in your GitHub repository?

If you intend to leave GitHub for a while,

make sure collaborators (or users of your package) have a good idea of how to contact you, and what maintenance is expected (for instance create a file CONTRIBUTING.md where you state the maintenance is stop and go).
set a busy status on GitHub.

Procrastinate on your profile

A nice touch would be to make your GitHub profile informative, setting a picture, description, and potentially a GitHub README. If you have GitHub repositories you are proud of, pin them to your profile.

Schedule training sessions?

Your work on checking your git/GitHub situation in RStudio, credentials and such will make your next restart easier. However practice will also help a lot!

Beside trying to get a bit more git into your daily work, you might want to schedule some sessions where you experiment with branches and pull requests in a toy or real repository you own, where you make you first pull request to an external repository. You could run these sessions with other (false) beginners!

Conclusion

First of all, it is awesome you want to use git and GitHub (or GitLab or something else for that matter)! It is not an easy toolset to learn, so no wonder you can get frustated. Hopefully the tips in this post, which I’d summarize as “Remember happygitwithr.com, usethis and/or GitHub Desktop, Openscapes illustrated series; Be patient with yourself”, can help a bit. Good luck!

PS: improve your tooling?

Tip by Robert M Flight: find a diff viewer you like.
Tip by Dave Braze: experiment with other IDEs, use magit.

RSS, fantastic tool for keeping up-to-date

Tue, 26 Apr 2022 00:00:00 +0000

I found an excuse to blog about XML again! Yes, RSS feeds are in practice XML, but for most people, that’s not why they are cool. An RSS (really simple syndication) feed is metadata about all, or the most recent posts published by a website: publication date, content or summary, etc. Much handier in my opinion to use that to get updates among Twitter “noise”.

RSS feed example

Have a look at the RSS feed for this blog over at https://masalmon.eu/post/index.xml

How to consume RSS

Take a list of all your favorite websites whose updates you want to follow. Then you can syndicate all of them to some tool that will regularly look for new posts in the feed and present them to you, potentially with some bells and whistles like letting you mark them as “read” or “to read later”.

That tool could be Feedly. Note that Feedly will find RSS feed addresses directly based on an URL. I’m quite happy with it.

It could also be a local Hugo website you’d build as Hugo has support for XML hence RSS since version 0.90 (see https://github.com/gohugoio/hugo/releases/tag/v0.90.0). I can imagine one would store addresses to RSS feeds in a JSON file, then have a template that fetch all of them and then transform them. Although, to be honest, it’d seem more natural to me to use some sort of R pipeline. Both ideas would be fun exercises. 😉

Beyond the tool used, subscribing to RSS feeds is in my opinion very empowering. Instead of following an organization or person on say Twitter, I much prefer making sure I get notified of their important stuff via a feed (or a newsletter!). Obviously it does not have the social aspect of social media but you can still share and discuss posts “somewhere”.

Last but not least, as mentioned in my previous post about GitHub, you can subscribe to your own GitHub timeline.

How to produce RSS

If you maintain some sort of blog, please make sure it produces an RSS feed! Type the name of your website generator and “RSS” into a search engine. E.g. here are the docs for RSS feeds for Quarto blogs.

Your website generator might let you create distinct RSS feeds for different categories or tags which might useful for your different audiences.

If you created your RSS feed yourself, e.g. by tweaking a Hugo template quite a lot, validate it with this online validator.

How to broadcast RSS

If your website has an RSS feed, aggregators, not only individuals, can subscribe to it!

You could try and syndicate your blog on R bloggers.
At the moment R Weekly no longer uses RSS feeds but it’ll probably soon use them again.
There might be some sort of aggregators for some other communities you’re part of?

Conclusion

In this post I briefly explained how to consume and produce RSS feeds. Note that there are also JSON feeds 😅, that there are newsletter tools for letting people subscribe to your updates by email, and that services such as Feedly have paid features that create feeds for websites that do not produce one, so that you can follow them. RSS feeds are not going away right now, and are regularly the topic of trending discussions on Hacker News (together with some trendy phrases like “slow web”). So it’s not that uncool to be part of the group of old fashion RSSers, right? 😎

Why I like XPath, XML and HTML

Fri, 08 Apr 2022 00:00:00 +0000

One of my favorite tool is XPath, the query language for exploring XML and HTML trees. In this post, I will highlight a few use cases of this “angle-bracket crunching tool” and hope to convince you that it’s an awesome thing to know about and play with.

Many thanks to Christophe Dervieux for useful feedback on this post! Mille mercis !

Brief intro to XPath in R

Say I have some XML,

my_xml <- xml2::read_xml("<wrapper><thing>blop</thing></wrapper>")

I’m using xml2, by Hadley Wickham, Jim Hester and Jeroen Ooms. This package is recommended over the XML package by e.g. the rOpenSci dev guide.

With XPath I can query the “thing” element:

xml2::xml_find_all(my_xml, ".//thing")
#> {xml_nodeset (1)}
#> [1] <thing>blop</thing>

I can extract its content via xml2::xml_text():

xml2::xml_find_all(my_xml, ".//thing") |>
  xml2::xml_text()
#> [1] "blop"

I could also replace the element.

Now, that was an especially simple XPath query. XPath’s strength is to allow you to really take advantage of the structure of the XML or HTML tree. You can extract nodes based on their attributes, on their parents, on their siblings, etc.

Where to learn XPath? Two good websites to get started are

A primary skill to learn is the name of elements, e.g. nodes, attributes, which will help write type better keywords into search engines when trying to figure out a query. 😉

Note that if you are handling HTML, you might enjoy selectr by Simon Potter that creates XPath filters based on CSS selectors.

Knowing XPath, or even knowing it exists, is really empowering. In the rest of this post, I’ll highlight cases where this is useful.

When life gives you XML or HTML

Web scraping

At the beginning of this blog I liked extracting data from websites. I did that with regular expressions. Now I know better and would wrangle HTML as HTML. Goodbye, stringr::str_detect(), hello, xml2::xml_find_all().

A package that’s especially useful for web scraping is rvest by Hadley Wickham. rvest builds upon selectr, and will write XPath for you.

pkgdown

If you use pkgdown to produce the documentation website of for your package, please know that part of its magic comes from various “HTML tweaks” that are powered by XPath, see for instance “tweak-reference.R”.

When life gives you something else…

You can still make it XML to handle it as such, with XPath!

Markdown manipulation with commonmark, tinkr

The commonmark package transforms Markdown to XML. This can be extremely handy to get data on R Markdown or Markdown files.

Now, say you want to modify the Markdown file as XML then get a Markdown file back. It is also possible, with the tinkr package, started by yours truly, now maintained by Zhian Kamvar. The conversion back to Markdown relies on xslt by Jeroen Ooms, a package that can use XSL stylesheets.

Code tree manipulation with xmlparsedata

Imagine you’re writing a domain-specific language where you let users write something like

str_detect(str_to_lower(itemTitle), 'wikidata')

that you want to somehow translate to:

REGEX(LCASE(?itemTitle),"wikidata")

Yes, that’s a real use case, from the glitter package (SPARQL DSL) maintained by Lise Vaudor.

The way we translate the code is to transform it to an XML tree via Gábor Csárdi’s xmlparsedata, then we can apply different tweaks based on XPath.

parse(
  text = "str_detect(str_to_lower(itemTitle), 'wikidata')",
  keep.source = TRUE
) |> 
  xmlparsedata::xml_parse_data(pretty = TRUE) |> 
  xml2::read_xml() |>
  as.character() |>
  cat()
#> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
#> <exprlist>
#>   <expr line1="1" col1="1" line2="1" col2="47" start="49" end="95">
#>     <expr line1="1" col1="1" line2="1" col2="10" start="49" end="58">
#>       <SYMBOL_FUNCTION_CALL line1="1" col1="1" line2="1" col2="10" start="49" end="58">str_detect</SYMBOL_FUNCTION_CALL>
#>     </expr>
#>     <OP-LEFT-PAREN line1="1" col1="11" line2="1" col2="11" start="59" end="59">(</OP-LEFT-PAREN>
#>     <expr line1="1" col1="12" line2="1" col2="34" start="60" end="82">
#>       <expr line1="1" col1="12" line2="1" col2="23" start="60" end="71">
#>         <SYMBOL_FUNCTION_CALL line1="1" col1="12" line2="1" col2="23" start="60" end="71">str_to_lower</SYMBOL_FUNCTION_CALL>
#>       </expr>
#>       <OP-LEFT-PAREN line1="1" col1="24" line2="1" col2="24" start="72" end="72">(</OP-LEFT-PAREN>
#>       <expr line1="1" col1="25" line2="1" col2="33" start="73" end="81">
#>         <SYMBOL line1="1" col1="25" line2="1" col2="33" start="73" end="81">itemTitle</SYMBOL>
#>       </expr>
#>       <OP-RIGHT-PAREN line1="1" col1="34" line2="1" col2="34" start="82" end="82">)</OP-RIGHT-PAREN>
#>     </expr>
#>     <OP-COMMA line1="1" col1="35" line2="1" col2="35" start="83" end="83">,</OP-COMMA>
#>     <expr line1="1" col1="37" line2="1" col2="46" start="85" end="94">
#>       <STR_CONST line1="1" col1="37" line2="1" col2="46" start="85" end="94">'wikidata'</STR_CONST>
#>     </expr>
#>     <OP-RIGHT-PAREN line1="1" col1="47" line2="1" col2="47" start="95" end="95">)</OP-RIGHT-PAREN>
#>   </expr>
#> </exprlist>

To me, having an XML tree at hand makes it easier to think of, and work with, an “abstract syntax tree”.

XPath for all the things

A tool that I haven’t used, but that sounds intriguing, is rpath by Gabriel Becker, an R package implementing xpath-like functionality for querying R objects.

Data documentation with EML

No matter what format your data is, you can create its metadata using the EML package maintained by Carl Boettiger that creates XML metadata following the Ecological Metadata Language. Sure, you might prefer using dataspice maintained by Bryce Mecum (and get JSON).

When you are creating XML or HTML

If the goal of your code or package is to produce XML or HTML, knowing XPath will help you write unit tests (that you might want to complement with snapshot unit tests).

Conclusion

In this post I’ve explained why I find XPath, XML, HTML so useful. Applications are endless, not limited to the examples from this post: web scraping, HTML tweaks, Markdown manipulation, code tree manipulation…

8 (octo!) GitHub Tips

Tue, 05 Apr 2022 00:00:00 +0000

I’m spending quite a lot of my working time on GitHub, so have taken some habits. Maybe some of them can be useful to you!

1: How to get started

I’ve never actually taught git and GitHub, but I like sharing these useful links:

Happy Git and GitHub for the useR by Jenny Bryan, the STAT 545 TAs, Jim Hester. It includes a big picture section “Why Git? Why GitHub?".
Reflections on 4 months of GitHub: my advice to beginners by Suzan Baert. It’s pragmatic and encouraging.
Excuse me, do you have a moment to talk about version control? by Jenny Bryan.

2: Secure your account

Please use a password manager with some sort of cloud backup.

Enable two factor-authentication for your GitHub account. It should not take much time. Make sure you store the recovery keys into your password manager. Also make sure your two factor-authentication app, if that’s an app, has some sort of cloud backup.

If you interact with GitHub programmatically, say with the usethis package, refer to

Managing Git(Hub) Credentials in usethis docs;
Managing GitHub credentials from R, difficulty level linux by Danielle Navarro (if you use Linux, that is!).

3: Refine GitHub interface with… Refined GitHub

Refined GitHub is a browser extension that adds nice features such as the default ordering of issues from most recently udpated to least recently updated. It’s quite neat.

4: Make your profile informative

Try to fill your GitHub profile information (and to keep it up-to-date).

Furthermore,

Pin your most important repositories. They do not have to be your repositories, actually, they only need to be repositories you contributed to.
Publicize the organization memberships you want to show.
Add a profile README (yes, I have still not done this myself).

5: Handle PR states and suggestions with available features

You can make draft pull requests to indicate they are not ready yet. You can revert a pull request to the draft state. I find this most useful.

In the first comment of a pull request, if you add a line Fix #42 (or some other recognized keyword), merging the PR will close issue 42. From the issue 42 itself, one will be able to see the PR is “linked” to it.

When reviewing a pull request, in comments you can make actual change suggestions, “commit suggestions” that the receiver can accept with a click. Using this instead of writing something à la tipo -> typo is quite handy. When you are on the receiving end, to incorporate the feedback you can either accept each suggestion individually or batch-accept them, but only from the files tab of the PR, not the main tab.

6: Custom watch repositories

Watching repositories can be very useful: your own repositories, a favorite package of yours, an important dependency of a project of yours, Hacker News Daily Top 10, etc.

When you watch a repository, you get notified of all issues, PRs, discussions, and also see all commits in your timeline. That last part might be a bit too much. Well, you can customize what to watch by configuring your watch settings for an individual repository. I tend to subscribe to all issues, PRs and discussions, which I then tackle from my notifications inbox, one repository at a time.

7: Add your GitHub timeline to your RSS feed reader

At the very bottom of your GitHub timeline there is a link to your RSS feed, “Subscribe to your news feed” (it has a token in it so do not publicize it!). I have added mine to Feedly. Even if GitHub has recently improved the timeline interface, I like it to be grouped with other tech RSS feeds I subscribe to. That might change, though.

I’ve also added the GitHub blog to Feedly to be aware of new features in a less random way than on Twitter (I’m actively reducing my dependency on Twitter for crucial information).

Conclusion

In this post I have shared eight tips based on my GitHub usage. Thanks to those who taught me some of these! Do you have some advice for GitHub users?

How to become a better R code detective?

Tue, 13 Jul 2021 00:00:00 +0000

Huge thanks to Hannah Frick for her useful feedback on this post! Vielen Dank!

This post was featured on the R Weekly podcast by Eric Nantz.

When trying to fix a bug or add a feature to an R package, how do you go from viewing the code as a big messy ball of wool, to a logical diagram that you can bend to your will? In this post, I will share some resources and tips on getting better at debugging and reading code, written by someone else (or yourself but long enough ago to feel foreign!).

After step 1, there’s not really an order for applying the techniques, but you definitely want to acquire enough knowledge through research before you tinker, otherwise you will be tinkering quite randomly.

Apart from the idea of adding tests, most tips could apply to non-package code. If your motivation is to contribute to R itself please refer to A Guide to Contributing to R Core by Saranjeet Kaur and colleagues.

Step 0: Only deal with well-designed code

This is obviously not an actual tip, but more of a encouragement to try and write better and better code ourselves.

The Tidyverse style guide is a good read, both for learning tips, and for seeing what aspects such a style guide applies to. Of particular interest are e.g. the mention of function comments, and file organization.

Regarding code comments, I was intrigued and impressed by the notion of “explaining variables” mentioned in the tweet below referring a great short post.

Great post on why "explaining variables" and "explaining functions or methods" are better than code comments.

I found that my enthusiasm for writing/maintaining comments peaked many years ago and I'm better off using other techniques to make the non-obvious (more) obvious. https://t.co/yIIOnd8qHz
— Jenny Bryan (@JennyBryan) July 5, 2021

Beyond its being well designed, it’ll also help if the code is well tested: tests can help you understand the goals behind functions, and will help you re-factor without breaking features.

Now, we don’t always choose what code we get, and even if it was well-designed, we might still need detective skills anyway!

Step 1: Clone and build

In sharing this life-changing tip I am merely repeating the talk “Reading other people’s code” by Patricia Aas that I actually listened to as an episode of the All Things Git podcast. The techniques presented by Patricia Aas are not specific to R but many of them are relevant for R codebases.

Now to this tip… Instead of being overwhelmed by the idea of starting to tinker with a codebase, create a local version-controlled project with the codebase in it! E.g. fork a GitHub repo, and use usethis::create_from_github(). Then open it, install the dependencies via remotes::install_deps(dependencies = TRUE), build or load it. Before amending things, create a new branch via e.g. gert::git_branch_create("tinkering"). I suppose that if I were fancy I’d say this step, preparing a local codebase to play with, is your mise en place.

Obviously to reach that stage you’ll need to know what codebase is the one to be working on. However, you’ll probably start from some code in any case, e.g. your currently buggy code.

Make your problem smaller

In case of a bug, you’ll often be advised to make it a minimal reproducible example. While you’ll often hear this when you try to communicate your bug to someone else, it is also great practice to do this for yourself! An important thing to know here is reprex. A reprex is both a concept (reprex for reproducible example) and a package for communicating such examples, respectively promoted and maintained by Jenny Bryan. Why use reprex?

The isolated bug is easier to solve or will be solved by creating it! In Jenny Bryan’s talk “Object of type ‘closure’ is not subsettable” there’s an example of original code and its minified version.
As it is run in an isolated session you can be more sure that it’s reproducible.
You can send your bug in a format ideal for experts! But you might be writing a reprex just for yourself, to accompany some notes in a GitHub issue for instance.

How does reprex work?

You write some code in an editor (including loading libraries, creating toy data etc.).
You copy the code to your clipboard.
You run reprex::reprex() and reprex runs your code in an isolated session!
You get the rendered code on the clipboard (and a preview in RStudio Viewer pane)! Error messages rendered, images uploaded to imgur.
You paste the rendered code somewhere, potentially to show to someone.

To learn more about reprex and adopt it, I’d recommend watching the RStudio webinar about reprex and reading reprex vignettes, in particular “Reprex do’s and don’ts”.

Also in the case of a bug, maybe you don’t need to read this post further if your problem is in the bingo below. Often, you’ll only notice “obvious” mistakes after making the problem smaller (or after taking a break!).

In case of amending the features of a package, it’ll be important to clearly defined the scope of what you’re after. Easier for your work as a code detective, but also for many other reasons, see Sarah Drasner’s post How to Scope Down PRs and Yihui Xie’s post “Bite-sized” Pull Requests.

Pull an end / Follow the trails

As you are not going to read code from cover to cover, you’ll need to find a logical way to explore the code.

I like the phrase follow the trails by Kara Woo in her excellent RStudio conference talk “Box plots - A case study in debugging and perseverance” as well as the phrase pull an end by Patricia Aas in her also excellent talk “Reading Other People’s Code” already mentioned in this post.

Find where to start

Easy case: there’s a message on screen telling you where an error occurred, or you know what function you want to amend.

Alternatively,

You can put the error / warning in a search engine.
If there’s an unclear error you can try to see the traceback i.e. what functions were called leading to that error. In her talk “Object of type ‘closure’ is not subsettable”, Jenny Bryan explains very well what a traceback is. In my .Rprofile I have

options(
error = rlang::entrace, 
rlang_backtrace_on_error = "branch")

thanks to a tweet by Noam Ross reporting a tip by Jenny Bryan. “It gives trimmed tracebacks when using pipes.”

If there’s no error but a warning you could try to convert the warning to an error.

Explore from that starting point

You can use “grepping” as said by Patricia Aas: look for the occurrences of a function or variable name in a local folder, or via GitHub (advanced) search. You can limit the hits to some types of files e.g. R scripts in R/.
In your IDE e.g. RStudio there might be a way to go directly to the definition of function. With the lookup package you can also easily look up the source of functions locally and on GitHub.

How to read code: space… and time

Hopefully the code makes sense on its own. If not, fear not, the next item right here, and the other sections of this post, as well as patience, will help.
Sometimes using git blame or looking at the git history might help understand the context of some aspects of the code, if there’s no code comment referring an issue. Do not actually blame people, though. To make your own git history more informative for such later code archaeology, use branches and squash and merge.

Build your mental model of the code

That’s what Patricia Aas calls “mental machine”. You might want to draw some sort of diagram by hand (or programmatically). Patricia Aas remarks that such diagrams might even be contributed to the codebase as developer documentation.

Browse code by others

The life-hack [now deleted] by Julia Silge for fixing Travis CI builds¹, looking at other people’s configuration files, applies to any code endeavour.

How to find good examples?

The lookup package can help you look up usage of a function on GitHub and in general GitHub Advanced search is really useful;
You might look at the reverse dependencies of a package you are using;
You might try to think of packages doing something similar to yours (e.g. another package munging XML data from an API ; another package wrapping a C library).

Beyond browsing files, `browser()`

Reading code and imagining what it does only goes so long. You can edit the code and see whether, from the outside, it does what you want it to. Sometimes you might also make do with print-debugging i.e. for instance writing print("coucou !") to check whether a part of the code was run, or print(class(x)) to check an assumption about a thing. Sometimes print-debugging is the only technique you might be able to use if non-interactive debugging. It can also be perfect to know where a loop breaks which motivated a tweet by Sharla Gelfand.

But often you will have to go experiment under the hood. For doing that efficiently you will need to learn about browser() and friends! Or only just browser() to start with!

The basic idea is that you just replace the print() command you were about to write with browser(), run the code and voilà! You entered the debugger and can run code line by line, explore options and environment variables, etc. Over time it’ll become a habit of yours, at least that’s what happened to me once I saw the light. 😁

Here are some good resources to learn about debugging tools. These resources also overlap with some of the objectives of this very blog post.

Beyond R

Sometimes the bug or element to tweak will live outside of R. Maybe in some C code you are wrapping, maybe in a CSS file. You will therefore have to learn debugging tools for these things too!

Read tests? Write some for sure

In Patricia Aas’ techniques features the idea of writing and running tests to see what’s the code is supposed to do. They especially mention integration tests, whereas in R packages you’ll mostly find unit tests. Those can also be useful to read, especially when they start breaking after your experiments.

In any case, once you have amended a codebase to fix a bug or add a feature, add tests! In Kara Woo’s talk “Box plots - A case study in debugging and perseverance”, she explained she added tests. In Jenny Bryan’s talk “Object of type ‘closure’ is not subsettable” she uses the word “deter” in the part of the talk where she gives such advice: adding tests and assertions, but also other tips such as running those on continuous integration, “using mind bendy stuff in moderation”, leaving access panels (e.g. verbose modes), writing error messages for humans.

You could even write a failing test at the beginning of your code exploration, even leaving it failing for an easier restart when you come back to the codebase (better than a sticky note for sure!).

Rubberducking to a person

Another technique you will often see mentioned is rubberducking i.e. explaining your problem to a rubber duck. The simple act of phrasing your issue might help you solve it.

However, you might prefer to speak to an actual person, or pretend you are as written in the tweet by Julia Evans below:

yea i don't really like to talk about rubber duck debugging because personally I find it hard to get into the right mindset if I don't have a specific person in mind who I'm talking to, even if I'm just writing them an email / Slack message which I later delete without sending it
— 🔎Julia Evans🔍 (@b0rk) June 28, 2021

I liked seeing that as I sometimes open a Slack conversation with someone as if I were about to ask for their help, and whilst preparing my notes for them, I’ll solve my issue.

Refactoring

Another tip by Patricia Aas is refactoring the code as it might improve your understanding of it. They underline that you should not contribute the results of your refactoring, especially as a first PR, as people might hate you! It’s an exercise for you.

That said, I remember receiving nice PRs from someone who had just read the Clean Code book by Robert C. Martin. That contributor started with a small PR, and were very polite. Since then I’ve seen some bad critic of the book and its author and wouldn’t recommend reading it but these PRs made perfect sense. I can however easily imagine a big refactoring PR would not be happily received!

So, in a nutshell, as said by Patricia Aas, refactor to learn, in your own local copy or your own fork.

Asking for help

As nice as solving a problem on one’s own is, asking for help might be the solution! It is also a skill, or more, a bunch of skills: both how and where to ask for help but also deciding when to ask for help, when it’s no longer worth anyone’s money to have you continuing to work alone on a problem.

How to ask for help?

In the section of this post about making your problem smaller, we mentioned creating a reprex and having a clear scope. These are elements that will be featured in your plea for help.

It can also be good if you have tracked your progress, the various things you have tried.

Where to ask for help?

Where to ask for help depends on your question and the codebase you are working on. If you are working on a pull request in a package where you already have a good working relationship with other developers, or where you were encouraged to open a PR (directly or via the contributing guide), you might get help from other contributors.

I wrote a blog post on where to get help with your R questions.

See also my post on the R-hub blog, How to get help with R package development? R-package-devel and beyond, for general venues where to ask for help about package development. My favorite ones are the rOpenSci forum and the package development category of the RStudio community forum.

Reading other people’s debugging journeys, document yours

Sadly but understandably people will often only take the time to document their debugging journey when the bug is especially tricky or weird. However, few people write actual debugging games.

In the meantime, you might enjoy watching or hearing some debugging journeys. You will notice how these programmers make and invalidate hypotheses.

If you end up documenting your own code detective story, please tell me, I’d like to read it!

Conclusion

In this post I presented various techniques useful for code detectives. Getting better at them will help you debug and amend codebases. Now, I was able to summarize these tips, but I can’t say I never get stuck. 🙂

Where next?

If you have no personal opinion on what to “study” next from the numerous links of this post or elsewhere, my recommendation would be:

Start with Kara Woo’s talk “Box plots - A case study in debugging and perseverance” as it’s both a real and engaging story (with closure i.e. the bug was fixed! happy end!) and covers many useful debugging tools.
Continue with the Debugging advice in the Advanced R book by Hadley Wickham that is very clear and well organized. “Advanced R” might a frightening title but really don’t be afraid.
If you are an RStudio IDE user, you’ll find great use of Amanda Gadrow’s webinar about debugging techniques in RStudio and the corresponding official RStudio documentation.
Then you could watch Jenny Bryan’s talk “Object of type ‘closure’ is not subsettable” as it covers a lot of ground around problem solving in R, with ideas extremely well conveyed (and if you want to binge watch more talks of Jenny Bryan’s, continue with “Code feels, code smells” that’ll help you write better code to start with 🙂).
Lastly, you might be interested in drawing your own lessons from non R specific resources such as the All Things Git podcast episode with Patricia Aas, Julia Evans’ blog posts about debugging and Julia Evans’ comics about debugging.

And then, just wait for the next problem to tackle in your coding practice… One never has to wait very long. 😅

Last words

Last but not least I want to emphasize that there are also human aspects to this process.

we think about debugging as a technical skill (and it absolutely is!!) but a huge amount of it is managing your feelings so you don't get discouraged and being self-aware so you can recognize your incorrect assumptions
— 🔎Julia Evans🔍 (@b0rk) June 11, 2021

I love waking up and jumping back into solving a bug and immediately solving it with a fresh mind. Sleep is my favorite coding tool.
— Kelly Vaughn (@kvlly) April 23, 2021

What are your favourite tips and resources? Are you too eagerly awaiting Julia Evans’ zine about debugging? Please tell me in the comments below!

Travis CI itself is no longer recommended by rOpenSci for instance. ↩︎

Draw me a project

Wed, 30 Jun 2021 00:00:00 +0000

I’ll be giving a remote keynote talk at the Rencontres R (French R conference) on July the 12th, all in French. This blog post is a written version of my presentation, but in English. I decided to not talk about package development for once, but rather about workflows and how to structure & run an analysis.¹

Many thanks to Christophe Dervieux for useful feedback on this post! Merci beaucoup !

This post was featured on the R Weekly podcast by Eric Nantz.

Why discuss the making of R projects? Because any improvement in your workflow might improve the experience of anyone trying to run or audit an analysis later, including yourself. It’s also nice that there are always elements to refine, although that also means one might be procrastinating instead of making actual progress, so beware!

Now why should I discuss the making of R projects? While I am not often in charge of analyses these days, I follow R news quite well so thought I might be able to deliver some useful tips to the audience.

Some “basics”

To start with I’d like to mention some rules or principles that are quite crucial. My main source of information for them is Jenny Bryan.

Like many I identified with Sharla Gelfand’s quote² as reported by Kara Woo on Twitter, “Everything I know is from Jenny Bryan”.

There’s even a logo created by Martin Monkman for all of us agreeing with that statement! It features the letters EIKIFJB for “Everything I know is from Jenny Bryan” as well as a laptop on fire.

Why a laptop on fire?!

The only two things that make @JennyBryan 😤😠🤯. Instead use projects + here::here() #rstats pic.twitter.com/GwxnHePL4n
— Hadley Wickham (@hadleywickham) December 11, 2017

That’s a reference to a talk Jenny Bryan gave years ago, to which she said she’d put your computer on fire if you used setwd("C:\Users\jenny\path\that\only\I\have") or rm(list = ls()) at the beginning of your scripts. This goes against the notion of a “project-oriented workflow”. I do not need to repeat the wisdom of her blog post in mine, so will only use cliff notes & some remarks:

It’s handy to create your project with usethis::create_project(). RStudio IDE and projects work well together cf IDE support for projects and RStudio projects (although you could use the principles of a project-oriented workflow outside of RStudio!).
When reading in data etc. use paths relative to the root of your project, possibly using the here package by Kirill Müller.
Re-start R often, do not e.g. load packages in your .Rprofile, usethis::use_blank_slate().

Really, go and read her post if you haven’t yet!

In my talk I’ll show pictures of the Kei-tora Gardening Contest where participants create beautiful gardens in little trucks, which I find is a good image of a project that should be independent (you can move your truck, you should be able to move your project too).

The Kei-tora Gardening Contest is an annual event put on by the Japan Federation of Landscape Contractors to see who can build the best looking garden on the back of a Kei truck. Here are some of the winners from the past 12 years. https://t.co/IpkRiALn7a pic.twitter.com/vep4K0VZt6
— Doctor Popular (@DocPop) June 10, 2021

As the first part of my talk really is a collection of awesome content by Jenny Bryan, I will also show her rules for naming files. What Shakespeare said about roses isn’t true when programming. 😅

Then I’ll “introduce” version control. I’ll first underline that you want to have a back-up of your stuff somewhere no matter how you do it (so even if you do not use git, think of backing things up with a good system!). I’ll briefly mention some basic git commands, and most importantly, how to run them! Here are my preferences:

Using usethis (usethis::use_git(), usethis::use_github(), etc.) and gert (gert::git_push()) so that I don’t need to leave R.
Using RStudio git pane to not be too far from the R console.
Using the terminal for commands copy-pasted from somewhere on the web (and for the ones I now know by heart!).
Using a git interface like GitKraken (or the editor VSCODE that has a good git integration) for more complicated stuff… which I haven’t really ever fully explored, actually.

I’ll share links to my favorite git resources that I send to anyone who asks (or doesn’t ask):

Excuse Me, Do You Have a Moment to Talk About Version Control?, Jenny Bryan.
Happy Git and GitHub for the useR, Jenny Bryan, the STAT 545 TAs, Jim Hester.

Great observation: if you can write #rstats, you can use Git.
Great advice: when or before things get funky, practice tricky bits and disaster recovery in a bogus repo. https://t.co/XVwHP9K4av
— Jenny Bryan (@JennyBryan) March 4, 2018

Reflections on 4 months of GitHub: my advice to beginners, Suzan Baert.

And in French there’s a great blog post by ThinkR: Travailler avec Git via RStudio et versionner son code.

How to protect your project from external changes

Here’s a maximum credible accident: you write some pretty and handy code munging your data using package::my_favorite_function(). Now you go and update that package and realize my_favorite_function is gone! It was apparently removed for good reasons but now your script is broken!

To prevent that, you need to encapsulate your project. You can track and restore package dependencies of your package by using the renv package by Kevin Ushey. The renv package is the successor of the packrat package by the same author.

Using renv is actually quite easy:

In a new project you run renv::init();
After that you install and remove packages as you normally would (renv is smart and will copy files from your local not-project library to be faster). Regularly run renv::snapshot().³ Metadata about packages (where do they come from) are stored in the renv.lock file that you’d put under version control;
Anyone getting the project runs renv::restore() to have the exact same project library as you.

Also worth of a mention is capsule by Miles McBain that is “an inversion of renv for low effort reproducible R package libraries”.

Now if you want to go further and also freeze the operating system used etc. you could check out Docker.

What structure for your project?

What’s in your project? Probably something like:

Data or the code to get them from a database or a remote resource;
Some code munging and analysing them;
Some output that could be a graph, a report etc.

Now how should you structure your project? It’s important to use a structure that’s consistent between your (team’s) project, and that can be created automatically.

While I have never used the ProjectTemplate package by Kenton White, I really like the blog post Love for ProjectTemplate by Hilary Parker as it underlines advantages that should be requirement for any tool that helps create an analysis.

“Routine is your friend”.
“It’s easier to start somewhere and then customize, rather than start from the ground up."
“Reproducibility should be as easy as possible."
“Finding things should also be as easy as possible."

Now some people find all these advantages by structuring their analyses as R packages. Creating an R package to share code and data you use throughout projects is not subject to debate: it’s great! You can even build Shiny apps as package with golem. Creating your analysis as a package, with dependencies in DESCRIPTION, functions in R/, analysis in e.g. a vignette, is subject to debate.

The advantages are that when doing that you can re-use or refresh your package development skills, and foremost that you can re-use tools made for package development (like devtools and usethis). There’s a paper presenting and promoting the approach, where such packages are called research compendia: Packaging Data Analytical Work Reproducibly Using R (and Friends), Ben Marwick, Carl Boettiger & Lincoln Mullen (2018), The American Statistician, 72:1, 80-88, DOI: <10.1080/00031305.2017.1375986>

There are specific tools for building and using research compendia:

rrtools by Ben Marwick. “The goal of rrtools is to provide instructions, templates, and functions for making a basic compendium suitable for writing a reproducible journal article or report with R."
holepunch by Karthik Ram. “holepunch will read the contents of your R project on GitHub, create a DESCRIPTION file with all dependencies, write a Dockerfile, add a badge to your README, and build a Docker image. Once these 4 steps are complete, any reader can click the badge and within minutes, be dropped into a free, live, RStudio server. Here they can run your scripts and notebooks and see how everything works." (🤫 holepunch works without the compendium structure as well.)
The R-universe project at rOpenSci, built by Jeroen Ooms. As it automatically builds vignettes, it could be used to publish your packaged analysis online.

Now it’s good to know not everyone loves the idea of projects as R packages. Miles McBain wrote a blog post “Project as an R package: An okay idea”.

I found this quote quite interesting:

“My response to advocates of project as a package is: ==You’re wasting precious time making the wrong packages.==”

“Instead of shoehorning your work into the package development domain, with all the loss of fidelity that entails, why aren’t you packaging tools that create the smooth {devtools}/{usethis} style experience for your own domain?"

In my talk my own advice is to use whatever structure you, and your team if you have one, prefers, and to choose a structure that can be created automatically. You could be the one creating the package to create projects, as Miles said – although he also mentioned the risk of “bitrot” for tools maintained and used by few people.

How to run your project?

How do you go from resources and scripts to the analysis output? If your project “only” contains one or a few R Markdown document(s), maybe you can simply use the knit button. Now you might be dealing with some challenges warranting the use of dedicated tools. I’ll briefly present two.

Note that not all tools separate ways to structure and run projects i.e. you could be using a workflow package that’s opinionated about both.

Optimize pipeline with targets

The targets package by Will Landau, reviewed at rOpenSci software peer-review, helps optimizing pipelines by recognizing dependencies between steps (e.g. if you change the raw data you need to re-run everything, but if you change only the model fit you only need to re-run the final plot) and only running those that are needed at the moment. To make a project a targets project you need a script called _targets.R where you load packages, source e.g. functions from R/ and define targets. Now what are targets? Looking at part of a _targets.R from targets manual,

list(
  # Raw data file. Notice the format argument is used.
  tar_target(
    raw_data_file,
    "data/raw_data.csv",
    format = "file"
  ),
  tar_target(
    raw_data,
    read_csv(raw_data_file, col_types = cols())
  ),
  # The dplyr package has been previously loaded
  tar_target(
    data,
    raw_data %>%
      filter(!is.na(Ozone))
  ),
  # The create_plot() function comes from a script that has been previously sourced
  tar_target(hist, create_plot(data)),
  tar_target(fit, biglm(Ozone ~ Wind + Temp, data))
)

the targets are defined in a list. Each of them uses the tar_target() function, has a name, and code that creates it, or, in the case of the raw data file, the path to the file. To build the project you run targets::tar_make() (and to destroy everything if you need to, targets::tar_destroy()). To see the network of dependencies between targets you can run targets::tar_glimpse() and other functions that inspect the pipeline.

There’s a whole ecosystem of packages around targets, the Targetopia e.g. the tarchetypes package defines targets making functions that lets you define targets that need to be re-run after a certain time.

To get started with targets, I’d recommend:

Reading the manual;
Watching the talk Reproducible Computation at Scale in R with {targets} (Will Landau at the RUG of Lille).
Starting with a small project (my current non-expertise level 😅).

To follow evolutions of targets as it keeps getting better, you can:

Release-watch the targets GitHub repo;
Follow Will Landau on Twitter;
Subscribe to rOpenSci newsletter.

Track versions of an analysis with orderly

Imagine you want to keep track of the different versions of an analysis and everything that went into it, and to run analysis comparing versions. The orderly package, maintained by Rich FitzJohn⁴, offers an infrastructure for that kind of workflows.

With orderly you have repos and in repos you have reports/tasks, or only one report/task. Here’s an example with an orderly repo with one report. From an RStudio project I ran orderly::orderly_init("blop") which created the repo in a new folder “blop”, and then orderly::orderly_new("example", "blop") after which I modified files using the orderly introduction vignette.

In the blop/ folder there’s a general orderly configuration that I haven’t needed to touch, orderly_config.yml. There’s also a src/ folder corresponding to the source of my example report.

blop
├── orderly_config.yml
└── src
    └── example
        ├── orderly.yml
        └── script.R

That example report consists of two files so it’s a very small one. You could put anything in there as orderly is not opinionated about that! It is opinionated about being told about packages, resources, scripts, artefacts in the blop/src/example/orderly.yml configuration.

In the example the configuration lists one script and two artefacts:

script: script.R

artefacts:
  - staticgraph:
      description: A graph of things
      filenames: mygraph.png
  - data:
      description: Data that went into the plot
      filenames: mydata.csv

The script creates those two artefacts:

dat <- data.frame(x = 1:10, y = runif(10))
write.csv(dat, "mydata.csv", row.names = FALSE)

png("mygraph.png")
plot(dat)
dev.off()

Now how do you run the project?

You can use the development mode when developing a report, to have things at your current working directory.
You can build a draft version of the report with id <- orderly::orderly_run("example", root = "blop") after which your whole analysis, input and output, appears in blop/draft/example/some-id-that-contains-the-date-and-a-hash.
If you like it you can commit it with orderly::orderly_commit(id, root = "blop") which moves the whole analysis folder from blop/draft/example/some-id-that-contains-the-date-and-a-hash to blop/archive/example/some-id-that-contains-the-date-and-a-hash.

Note that as the draft and archive folders can be gigantic you are expected to back up with some system. You are expected not to use git to track these folders as git does not behave well with very large files. You can still use git to track other files in the orderly repo.

To get started with orderly you can read the orderly website that I found very clear, and you should start with a small project… which again is my current non-expertise level.

To follow orderly news, as it’s actively used and developed, you can:

Release-watch the orderly GitHub repo;
read the blog of the team behind orderly;
follow Rich FitzJohn on Twitter.

Other tools for building analyses

The targets and orderly packages are not the only ones helping run analyses: workflowr, cabinets, etc.. You might even want to build your own!

Conclusion

In this post/talk I have discussed several aspects of drawing a project

Some “basics” that are not all easy (all are habits to take, and some are trickier to learn than others);
Encapsulating your project by tracking its dependencies with e.g. renv;
Structuring your project in a way that suits your team’s wishes, is consistent over time, and can be automated;
Using tools for building outputs that answers your needs (optimizing a pipeline? tracking versions of an analysis projects?).

All in all my tips would be to read everything Jenny Bryan writes 😁, and to not be afraid to change tools over time as new cool tools will appear and as your needs and experience will change. I’d be interested to hear any thoughts in the comments below!

Further resources

These items might be relevant for you:

Course “Reproducible Research Data and Project Management in R” by Anna Krystalli.
Guide “Organisation d’un projet collaboratif de publication PROPRE” de Sébastien Rochette.
Good enough practices in scientific computing Wilson G, Bryan J, Cranston K, Kitzes J, Nederbragt L, et al. (2017) Good enough practices in scientific computing. PLOS Computational Biology 13(6): e1005510. https://doi.org/10.1371/journal.pcbi.1005510
The Turing Way, an open source community-driven guide to reproducible, ethical, inclusive and collaborative data science.

The Little Prince might have asked “Please draw me a sheep”, not a project, but I liked tweaking that quote for a title as one will often end up putting R projects in boxes (folders, maybe packages). ↩︎
Note that Sharla Gelfand themselves is a great source of good ideas! See e.g. their talk Don’t repeat yourself, talk to yourself! Repeated reporting in the R universe from the RStudio::conf 2020. ↩︎
Unless you go for automatic snapshots. ↩︎
Many thanks to Rich for answering some questions of mine! ↩︎

Server-side MathJax rendering with R?

Tue, 15 Jun 2021 00:00:00 +0000

Edit: now there is an R package for server-side MathJax rendering, katex by Jeroen Ooms.

Whilst I most certainly do not write LaTeX formulas on the regular anymore, I got curious about their MathJax rendering on websites.¹ In brief : your website source contains LaTeX code, and the MathJax JS library (self-hosted or hosted on a CDN) transforms it into something humans can understand: some HTML with inline CSS but also some MathML for screen-reader users.² As I quite like the idea of moving things from the client-side to the server-side, I started wondering whether the processing of LaTeX code could happen before an user opens a website. Searching for “server-side MathJax rendering” on the web gave a few hits. A few hits only, sure, which shows how niche the topic is, and meant reading resources was not too verwhelming. 😁 In this post I am reporting on my findings.

Why render MathJax on the server-side

Let me be honest, there is no extremely strong reason for me. Sure loading MathJax from a CDN might be a privacy problem (as the CDN measures traffic) or make your website potentially fragile if the CDN goes down (remember the Fastly outage last week). Now you could still self-host MathJax to mitigate that (and to know exactly who to blame in case of an outage i.e. yourself 😉). Not having MathJax might mean the page takes less time to load… if having the MathJax-rendered HTML weren’t making the page bigger than the same page with LaTeX code instead. But hey I still think it’s an interesting problem.

Something I realized whilst working on this post is that one probably does not want to remove all MathJax JS from the browser, as it includes accessibility options! You need some JS in order to have a menu etc. That’s not something I’ve tackled here, I think that to extract the right MathJax JS to have only the components for the menu but not those for processign input, one would need to understand MathJax much better than I do.

How to render MathJax on the server-side

The recipe would be:

extract LaTeX code. E.g. use XPath with xml2 for HTML; or regular expressions if that’s your jam.
transform the LaTeX string into HTML somehow.
replace the LaTeX code with the HTML.

The first and second steps are things close to e.g. the HTML tweaking pkgdown does. They might well shall entail (un)escaping problems.

Now the second step, how to transform LaTeX code into HTML? The best summary of the state-of-the-art I found is the blog post Math Rendering is Wrong.

Solutions I saw are:

a Node module called mathjax-node-cli. I know there are ways to use JS from R but I got lazy once I saw I had to update my Node installation or whatever the error message was.
a way to run a MathJax API, which I have not tried.
rendering math on a browser and extracting the result.

I will now focus on the last solution as it works well from R these days.

MathJax, but not in the viewer’s browser!

Let’s take a minimal HTML with MathJax loaded from a CDN and an empty paragraph with class “mathp”:

html <- xml2::read_html('<!DOCTYPE html>
<html>
<head>
<title>MathJax</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1">
<script type="text/javascript" src="https://mathjax.rstudio.com/2.7.2/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>

</head>
<body>

<p class="mathp">

</p>

</body>
</html>
')

Now let’s add math to it. Note that a real solution would have to differentiate in-line and block maths whose resulting HTML has a different XPath.

mathp <- xml2::xml_find_first(html, "//p[@class='mathp']")
xml2::xml_text(mathp) <- r"($$x = {-b \pm \sqrt{b^2-4ac} \over 2a}.$$)"
as.character(html)
#> [1] "<!DOCTYPE html>\n<html>\n<head>\n<title>MathJax</title>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n<script type=\"text/javascript\" src=\"https://mathjax.rstudio.com/2.7.2/MathJax.js?config=TeX-AMS-MML_HTMLorMML\"></script>\n</head>\n<body>\n\n<p class=\"mathp\">$$x = {-b \\pm \\sqrt{b^2-4ac} \\over 2a}.$$</p>\n\n</body>\n</html>\n"
file <- withr::local_tempfile(fileext = ".html")
#> Setting deferred event(s) on global environment.
#>   * Execute (and clear) with `withr::deferred_run()`.
#>   * Clear (without executing) with `withr::deferred_clear()`.
xml2::write_html(html, file)

Now we shall load the file in a browser via chromote³ and extract the HTML Pandoc produced. To find the XPath, I examined the HTML with the DevTools of my browser.

library("chromote")
b <- ChromoteSession$new()
b$Page$navigate(sprintf("file://%s", file)) 
#> $frameId
#> [1] "7FC1F01C63F746D0D66CC7E2DC40EF87"
#> 
#> $loaderId
#> [1] "A3031F97EEF09D8BF745DF8C089F448F"
# Make sure we wait long enough
Sys.sleep(2)
doc <- b$DOM$getDocument()
x <- b$DOM$querySelector(doc$root$nodeId, ".MathJax_Display")
(math_html <- b$DOM$getOuterHTML(x$nodeId))
#> $outerHTML
#> [1] "<div class=\"MathJax_Display\" style=\"text-align: center;\"><span class=\"MathJax\" id=\"MathJax-Element-1-Frame\" tabindex=\"0\" data-mathml=\"<math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot; display=&quot;block&quot;><mi>x</mi><mo>=</mo><mrow class=&quot;MJX-TeXAtom-ORD&quot;><mfrac><mrow><mo>&amp;#x2212;</mo><mi>b</mi><mo>&amp;#x00B1;</mo><msqrt><msup><mi>b</mi><mn>2</mn></msup><mo>&amp;#x2212;</mo><mn>4</mn><mi>a</mi><mi>c</mi></msqrt></mrow><mrow><mn>2</mn><mi>a</mi></mrow></mfrac></mrow><mo>.</mo></math>\" role=\"presentation\" style=\"text-align: center; position: relative;\"><nobr aria-hidden=\"true\"><span class=\"math\" id=\"MathJax-Span-1\" style=\"width: 10.144em; display: inline-block;\"><span style=\"display: inline-block; position: relative; width: 9.552em; height: 0px; font-size: 106%;\"><span style=\"position: absolute; clip: rect(0.307em, 1009.47em, 3.054em, -1000em); top: -2.182em; left: 0em;\"><span class=\"mrow\" id=\"MathJax-Span-2\"><span class=\"mi\" id=\"MathJax-Span-3\" style=\"font-family: MathJax_Math-italic;\">x</span><span class=\"mo\" id=\"MathJax-Span-4\" style=\"font-family: MathJax_Main; padding-left: 0.278em;\">=</span><span class=\"texatom\" id=\"MathJax-Span-5\" style=\"padding-left: 0.278em;\"><span class=\"mrow\" id=\"MathJax-Span-6\"><span class=\"mfrac\" id=\"MathJax-Span-7\"><span style=\"display: inline-block; position: relative; width: 7.116em; height: 0px; margin-right: 0.12em; margin-left: 0.12em;\"><span style=\"position: absolute; clip: rect(2.879em, 1007em, 4.443em, -1000em); top: -4.753em; left: 50%; margin-left: -3.498em;\"><span class=\"mrow\" id=\"MathJax-Span-8\"><span class=\"mo\" id=\"MathJax-Span-9\" style=\"font-family: MathJax_Main;\">−</span><span class=\"mi\" id=\"MathJax-Span-10\" style=\"font-family: MathJax_Math-italic;\">b</span><span class=\"mo\" id=\"MathJax-Span-11\" style=\"font-family: MathJax_Main; padding-left: 0.222em;\">±</span><span class=\"msqrt\" id=\"MathJax-Span-12\" style=\"padding-left: 0.222em;\"><span style=\"display: inline-block; position: relative; width: 4.567em; height: 0px;\"><span style=\"position: absolute; clip: rect(3.073em, 1003.54em, 4.268em, -1000em); top: -4.009em; left: 1em;\"><span class=\"mrow\" id=\"MathJax-Span-13\"><span class=\"msubsup\" id=\"MathJax-Span-14\"><span style=\"display: inline-block; position: relative; width: 0.858em; height: 0px;\"><span style=\"position: absolute; clip: rect(3.139em, 1000.42em, 4.197em, -1000em); top: -4.009em; left: 0em;\"><span class=\"mi\" id=\"MathJax-Span-15\" style=\"font-family: MathJax_Math-italic;\">b</span><span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"position: absolute; top: -4.298em; left: 0.429em;\"><span class=\"mn\" id=\"MathJax-Span-16\" style=\"font-size: 70.7%; font-family: MathJax_Main;\">2</span><span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span></span></span><span class=\"mo\" id=\"MathJax-Span-17\" style=\"font-family: MathJax_Main; padding-left: 0.222em;\">−</span><span class=\"mn\" id=\"MathJax-Span-18\" style=\"font-family: MathJax_Main; padding-left: 0.222em;\">4</span><span class=\"mi\" id=\"MathJax-Span-19\" style=\"font-family: MathJax_Math-italic;\">a</span><span class=\"mi\" id=\"MathJax-Span-20\" style=\"font-family: MathJax_Math-italic;\">c</span></span><span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"position: absolute; clip: rect(3.56em, 1003.57em, 3.958em, -1000em); top: -4.69em; left: 1em;\"><span style=\"display: inline-block; position: relative; width: 3.567em; height: 0px;\"><span style=\"position: absolute; font-family: MathJax_Main; top: -4.009em; left: -0.084em;\">−<span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"position: absolute; font-family: MathJax_Main; top: -4.009em; left: 2.873em;\">−<span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"font-family: MathJax_Main; position: absolute; top: -4.009em; left: 0.388em;\">−<span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"font-family: MathJax_Main; position: absolute; top: -4.009em; left: 0.885em;\">−<span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"font-family: MathJax_Main; position: absolute; top: -4.009em; left: 1.382em;\">−<span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"font-family: MathJax_Main; position: absolute; top: -4.009em; left: 1.879em;\">−<span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"font-family: MathJax_Main; position: absolute; top: -4.009em; left: 2.376em;\">−<span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span></span><span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"position: absolute; clip: rect(2.983em, 1001.02em, 4.536em, -1000em); top: -4.103em; left: 0em;\"><span style=\"font-family: MathJax_Size1;\">√</span><span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span></span></span></span><span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"position: absolute; clip: rect(3.167em, 1001.01em, 4.196em, -1000em); top: -3.323em; left: 50%; margin-left: -0.514em;\"><span class=\"mrow\" id=\"MathJax-Span-21\"><span class=\"mn\" id=\"MathJax-Span-22\" style=\"font-family: MathJax_Main;\">2</span><span class=\"mi\" id=\"MathJax-Span-23\" style=\"font-family: MathJax_Math-italic;\">a</span></span><span style=\"display: inline-block; width: 0px; height: 4.009em;\"></span></span><span style=\"position: absolute; clip: rect(0.811em, 1007.12em, 1.238em, -1000em); top: -1.281em; left: 0em;\"><span style=\"display: inline-block; overflow: hidden; vertical-align: 0em; border-top: 1.3px solid; width: 7.116em; height: 0px;\"></span><span style=\"display: inline-block; width: 0px; height: 1.061em;\"></span></span></span></span></span></span><span class=\"mo\" id=\"MathJax-Span-24\" style=\"font-family: MathJax_Main;\">.</span></span><span style=\"display: inline-block; width: 0px; height: 2.182em;\"></span></span></span><span style=\"display: inline-block; overflow: hidden; vertical-align: -0.8em; border-left: 0px solid; width: 0px; height: 2.662em;\"></span></span></nobr><span class=\"MJX_Assistive_MathML MJX_Assistive_MathML_Block\" role=\"presentation\"><math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\"><mi>x</mi><mo>=</mo><mrow class=\"MJX-TeXAtom-ORD\"><mfrac><mrow><mo>−</mo><mi>b</mi><mo>±</mo><msqrt><msup><mi>b</mi><mn>2</mn></msup><mo>−</mo><mn>4</mn><mi>a</mi><mi>c</mi></msqrt></mrow><mrow><mn>2</mn><mi>a</mi></mrow></mfrac></mrow><mo>.</mo></math></span></span></div>"

The math_html above is the MathJax-rendered math HTML we were after!

Now we create a minimal HTML without MathJax and add the math HTML to it.

html <- xml2::read_html('<!DOCTYPE html>
<html>
<head>
<title>MathJax</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>

<p id="mathp" class="mathp">

</p>

</body>
</html>
')
mathp <- xml2::xml_find_first(html, "//p[@id='mathp']")
xml2::xml_text(mathp) <- math_html$outerHTML

Sadly the step above means the math HTML will have been escaped. One could find an actual fix, but my fix will be to unescape the HTML à la pkgdown i.e. literally using an internal function of pkgdown’s.

# from https://github.com/r-lib/pkgdown/blob/23eb05ceceda1c44573b254dd8b96e92cd91f825/R/html-build.R#L49
unescape_html <- function(x) {
  x <- gsub("&lt;", "<", x)
  x <- gsub("&gt;", ">", x)
  x <- gsub("&amp;", "&", x)
  x
}
html <- xml2::read_html(unescape_html(as.character(html)))
xml2::write_html(html, "example.html")

Now we can look at the example and see that it has indeed some math! It’s probably lacking fonts, the absence of any MathJax JS means that we see the math twice, one of them being a div for assistive technology.

Conclusion

In this post I explored server-side MathJax rendering. I have not created a workable solution : An actually acceptable solution would necessitate more knowledge of MathJax, be far more detail-oriented and would check the accessibility of the produced document! Nevertheless, it was interesting to me to use chromote to extract HTML produced by MathJax and to learn more about MathJax. I don’t think I’ll feel like writing more LaTeX than now but I am sure curious to also check out MathJax “competitors” like KaTeX.⁴

MathJax is not the only way to render math for the web but it is the most popular one. ↩︎
I found the video Accessible Math on the Web: A Server/Client Solution interesting, albeit a few years old already. Yes, I watched a video about SAS documentation. ↩︎
One could also choose to use crrri but this is left as an exercise to the reader. ↩︎
If you use Katex you might be interested in a related PR to rmarkdown. ↩︎

Stingy Beanie baby webscraping

Fri, 15 Jan 2021 00:00:00 +0000

I’ve just finished teaching blogging with R Markdown at R-Ladies Bangalore. This has two consequences: I need to calm down a bit after the stress of live demoing & co, and I am inspired to, well, blog with R Markdown! As I’ve just read a fascinating book about the beanie baby bubble and as I’ve seen rvest is getting an update, I’ve decided to harvest Beaniepedia. Both of these things show I spend too much time on Twitter, as the book has been tweeted about by Vicki Boykis, and the package changes have been tweeted about by Hadley Wickham. I call that staying up-to-date, of course.

So, as a little challenge for today, what are the most common animals among Beanie babies? Do I even need much webscraping to find this out?

How does one webscrape these days

I’ve always liked webscraping, as I think I enjoy getting and transforming data more than I enjoy analyzing it. Compared to my former self,

I know a bit more about XPath (starting with knowing it exists!) so I don’t use regular expression to parse HTML.
I use polite for polite webscraping!
I know it’s best to spend more time pondering about strategies before hammering requests at a website.

As to rvest recent changes, I had a quick look at the changelog but since I hadn’t used it in so long, it’s not as if I had any habit to change!

How to harvest Beaniepedia

I’ve noticed Beaniepedia has a sitemap for all beanies so from that I can extract the URLs to all Beanie pages. That’s a necessary step.

Now from there I could either

Scrape each of these pages, respectfully slowly, and extract the table that includes the beanie’s information;
Use a more frugal strategy by parsing URLs. E.g. from the path of https://beaniepedia.com/beanies/beanie-babies/january-the-birthday-bear-series-2/ I can extract the category of the Beanie (a beanie baby as opposed to, say, an attic treasure) and the animal by splitting january-the-birthday-bear-series-2 into pieces and see whether one is an animal. How would I recognize animals? By extracting the word coming after “the”.

I’ll choose the second strategy and leave the first one as an exercise to the reader. 😉

From XML to animal frequencies

Let’s get to work! A sine qua non condition is obviously the website being ok with our scraping stuff. The polite package would tell us whether the robots.txt file were against our doing this, and I also took time looking whether the website had any warning. I didn’t find any so I think we’re good to go.

session <- polite::bow(
  url = "https://beaniepedia.com/beanies/sitemap.xml",
  user_agent = "Maëlle Salmon https://masalmon.eu"
  )
sitemap <- polite::scrape(session)
sitemap

#> {xml_document}
#> <urlset schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
#>  [1] <url>\n  <loc>https://beaniepedia.com/beanies/</loc>\n  <lastmod>2021-01 ...
#>  [2] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/jerry-the-mi ...
#>  [3] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/snowball-the ...
#>  [4] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/jemima-puddl ...
#>  [5] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/jeff-gordon- ...
#>  [6] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/jeff-burton- ...
#>  [7] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/jeepers-the- ...
#>  [8] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/jeanette-the ...
#>  [9] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/jaz-the-cat/ ...
#> [10] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/japan-the-be ...
#> [11] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/laughter-the ...
#> [12] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/righty-2000- ...
#> [13] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/lefty-2004-t ...
#> [14] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/lefty-the-do ...
#> [15] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/lefty-the-do ...
#> [16] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/january-the- ...
#> [17] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/january-the- ...
#> [18] <url>\n  <loc>https://beaniepedia.com/beanies/beanie-babies/janglemouse- ...
#> [19] <url>\n  <loc>https://beaniepedia.com/beanies/attic-treasures/burrows-th ...
#> [20] <url>\n  <loc>https://beaniepedia.com/beanies/attic-treasures/klause-the ...
#> ...

The sitemap object is an XML document. I will extract URLs with the xml2 package.

sitemap <- xml2::xml_ns_strip(sitemap)
urls <- xml2::xml_text(xml2::xml_find_all(sitemap, ".//loc"))
head(urls)

#> [1] "https://beaniepedia.com/beanies/"                                          
#> [2] "https://beaniepedia.com/beanies/beanie-babies/jerry-the-minion-2/"         
#> [3] "https://beaniepedia.com/beanies/beanie-babies/snowball-the-snowman/"       
#> [4] "https://beaniepedia.com/beanies/beanie-babies/jemima-puddle-duck-the-duck/"
#> [5] "https://beaniepedia.com/beanies/beanie-babies/jeff-gordon-24-the-bear/"    
#> [6] "https://beaniepedia.com/beanies/beanie-babies/jeff-burton-no-31-the-bear/"

Now I need to parse the URLs. In an URL path like beanies/beanie-babies/jerry-the-minion-2/ the second part is the category, the third part is the Beanie Baby name. I, as if I were a good collector 🙃, am not interested in Attic treasures, only in Beanie Babies.

urls_df <- urltools::url_parse(urls)
urls_df <- dplyr::filter(urls_df, stringr::str_detect(path, "beanie-babies"))

This gives me 632 Beanie babies. Let’s parse the last part of their path. An earlier attempt ignored that some Beanie babies don’t have any “the” in their names, e.g. the Hello Kitty ones. This is a limitation of my stingy approach. The error messages by dplyr were most helpful! “The error occurred in row 185.” is so handy!


get_animal <- function(parsed_path) {
  
  if (all(unlist(parsed_path) != "the")) {
    return(NA)
  }
  
  animals <- unlist(parsed_path)[which(unlist(parsed_path) == "the") + 1]
  animals[length(animals)] # thanks, "The End the bear"
}

library("magrittr")
animals_df <- urls_df %>%
  dplyr::rowwise() %>%
  dplyr::mutate(parsed_path = stringr::str_split(path, "/", simplify = TRUE)[1,3]) %>%
  dplyr::mutate(parsed_path = stringr::str_split(parsed_path, "-")) %>%
  dplyr::mutate(animal = get_animal(parsed_path))

Now we’re getting somewhere!

dplyr::count(
  animals_df,
  animal,
  sort = TRUE
)

#> # A tibble: 150 x 2
#> # Rowwise: 
#>    animal      n
#>    <chr>   <int>
#>  1 bear      222
#>  2 cat        38
#>  3 dog        32
#>  4 rabbit     23
#>  5 NA         15
#>  6 pig        12
#>  7 unicorn     9
#>  8 polar       7
#>  9 giraffe     6
#> 10 penguin     6
#> # … with 140 more rows

Is this result surprising? Probably not! Now, let’s have a look at the ones we did not identify.

animals_df %>%
  dplyr::filter(is.na(animal)) %>%
  dplyr::pull(path)

#>  [1] "beanies/beanie-babies/hello-kitty-rainbow-with-cupcake/"    
#>  [2] "beanies/beanie-babies/boston-red-sox-key-clip/"             
#>  [3] "beanies/beanie-babies/i-love-you-bears/"                    
#>  [4] "beanies/beanie-babies/zodiac-horse/"                        
#>  [5] "beanies/beanie-babies/hong-kong-toy-fair-2017-brown/"       
#>  [6] "beanies/beanie-babies/hello-kitty-bunny-costume/"           
#>  [7] "beanies/beanie-babies/hello-kitty-pink-tartan/"             
#>  [8] "beanies/beanie-babies/hello-kitty-gold-angel/"              
#>  [9] "beanies/beanie-babies/rock-hello-kitty/"                    
#> [10] "beanies/beanie-babies/hello-kitty-i-love-japan-usa-version/"
#> [11] "beanies/beanie-babies/hello-kitty-i-love-japan-uk-version/" 
#> [12] "beanies/beanie-babies/zodiac-ox/"                           
#> [13] "beanies/beanie-babies/zodiac-tiger/"                        
#> [14] "beanies/beanie-babies/happy-birthday-sock-monkey/"          
#> [15] "beanies/beanie-babies/zodiac-goat/"

Fair enough, and nothing endangering our conclusion that bears win.

Conclusion

In this post I set out to find out what animals are the most common among Beanie babies. I thought I’d freshen my rvest-ing skill but thanks to the sitemap, that’s my rusty dplyr knowledge I was able to update a bit. In the end, I learnt that 35% of Beanie babies, at least the ones registered on Beaniepedia, are bears. Thanks to Beaniepedia maintainer for allowing this fun!

Storytime preparedness with av

Thu, 10 Dec 2020 00:00:00 +0000

My kids got a cool electronic storyteller as a gift. It is basically a pretty cube that you shake to make it play tracks. La conteuse merveilleuse comes with pre-loaded songs and stories, but you can also add your own. Very handy, as we had e.g. CDs that came with magazines. According to the anti-manual¹, to add a mp3 or wav file you own, you first need to convert it to the storyteller’s expected format by using the company’s online converter, la moulinette (the mill). In this post I shall explain how I, a cool anti-mom, used R, in particular the av package, instead.

Official guidance on how to convert audio files on the Conteuse merveilleuse

The storyteller can be plugged into a computer like an USB stick, using its specific red cable. The interface of the storyteller is the file system, which is rather intuitive to me. There is a folder for each face of the cube. When you shake the cube three times with e.g. the lion face facing upwards, it plays a random track from the LION folder.

As a side note, something that’s fascinating to me is that the settings of the cube, like the maximal volume, are to be tweaked via a text file called settings.txt! Not even reglages.txt! Furthermore, your storyteller’s ID (needed to register the device on the company’s website) is stored as a text filename.

You can buy new tracks from the company’s website, record your own tracks in their studio, and upload your local stories to the website for then getting them as compatible mp3. You are limited to 5 conversions a day, of files up to 100Mb. I had some connection issues and couldn’t even get a few conversions, and besides, some of the tracks (from our local kid theater, unsurprisingly currently closed) I wanted to use were a bit bigger than 100Mb.

Using R instead

When reading the FAQ on the website, sadly not linkable as it’s in the website section where you’re logged in, I saw a sentence describing the mp3 formats for “geeks”: mono, 64 kbps, 16 khz, constant bitrate. Maybe I felt targeted by the word geek, in any case seeing this made me realize I could probably use other tools to convert tracks. I remembered about Jeroen Ooms’ av package, and decided to read its docs, in particular of av::av_audio_convert().

Now, I still had no clear idea what all the numbers in the format description meant, but I realized av has a function very similar to magick::image_info(): av::av_media_info()!

I used it on a track pre-loaded on the cube.

# I did not add the space character
av::av_media_info("/media/maelle/JOYEUSE-AD/FR/NUAGE/Je respire.mp3")

#> $duration
#> [1] 426.564
#> 
#> $video
#> NULL
#> 
#> $audio
#>   channels sample_rate    codec frames bitrate layout
#> 1        1       16000 mp3float     NA   64000   mono

This helped me see the parameter values of av::av_audio_convert() I needed to use: channels = 1 and sample_rate = 16000. I did not tweak the bitrate, I didn’t see how, and anyway my tracks had a lower bit rate (so lower quality) than the company’s ones.

After the conversion I’ll describe later, here’s the info I got about new tracks.

av::av_media_info("/media/maelle/JOYEUSE-AD/FR/LION/choum.mp3")

#> $duration
#> [1] 265.356
#> 
#> $video
#> NULL
#> 
#> $audio
#>   channels sample_rate    codec frames bitrate layout
#> 1        1       16000 mp3float     NA   24000   mono

av::av_media_info("/media/maelle/JOYEUSE-AD/FR/ROUGE/leila.mp3")

#> $duration
#> [1] 762.984
#> 
#> $video
#> NULL
#> 
#> $audio
#>   channels sample_rate    codec frames bitrate layout
#> 1        1       16000 mp3float     NA   24000   mono

No need to tell av what format to use as the output filename extension gives it away.

I wrote two conversions functions

One for tracks coming with a magazine, to crop out the intro song. This was crucial for user-friendliness of the cube as the title of the story comes after the 41 seconds (admittedly very cute) intro song which is a long time without knowing whether you want to listen or shake again to get another random track.

convert_belles <- function(audio_path) {
  av::av_audio_convert(
    audio = audio_path,
    output = file.path("output", basename(audio_path)),
    channels = 1,
    start_time = 41,
    sample_rate = 16000
  )
}

One for other tracks needing no cropping, but including some “wav” files.

convert_autres <- function(audio_path) {
  av::av_audio_convert(
    audio = audio_path,
    output = file.path("output", gsub("wav", "mp3", basename(audio_path))),
    channels = 1,
    sample_rate = 16000
  )
}

I could have written a single function instead with one more parameter. 🤷

To apply the functions

I put the tracks in two folders (by hand).
For each folder I used fs and purrr.

to_convert <- fs::dir_ls("input/belles-h")

convert_belles <- function(audio_path) {
  av::av_audio_convert(
    audio = audio_path,
    output = file.path("output", basename(audio_path)),
    channels = 1,
    start_time = 41,
    sample_rate = 16000
  )
}

purrr::walk(
  to_convert,
  convert_belles
)

After that I manually copy-pasted output tracks to the face folders of the cube. Now the tracks corresponding to theater stories are all in the ROUGE face for instance.

In reality I started by converting one track, then put it in a face folder of the cube and tried playing it.

Conclusion

In this post I presented my small script converting mp3 and wav tracks to mp3 compatible with my offspring’s storyteller. To me these were a few empowering lines of utility code. I think it exemplifies the usefulness of av for audio manipulation. Now, maybe I’ll go record some tale about R to put on my kids’ cube…

Apparently when you’re a cool new startup you don’t call your docs a manual! ↩︎

Single-source publishing for R users

Fri, 06 Nov 2020 00:00:00 +0000

A big part of my work includes putting content about R online, in blog posts and online books. I’m therefore very interested in the technical infrastructure that allows us R users to produce beautiful products out of R Markdown or Markdown source files. In this post I shall summarize my recent experiments around making HTML and PDF versions of books. Thanks to Julie Blanc’s inspiring post in French, I have learnt this is called single-source publishing. So let’s dive into the incredible machines allowing single-source publishing for R users. 😉

What I want for writing books

~~I want technical tools that allow me to procrastinate whilst learning a ton.~~ Just kidding, or not. 😇 My official criteria are:

For a book(let), I want a beautiful, usable website as well as a beautiful, usable PDF since some folks might prefer that for printing or offline usage.
I want to be able to write content in R Markdown or Markdown files, with some helpers for elements that Markdown doesn’t allow, for instance I might use a custom block to show an alert.
I want continuous deployment for my books, using git and a continuous integration service, so that every change in the source is reflected in the online versions of my book without my needing to think about it.
I would prefer to knit R Markdown files only once.
I don’t want to have to learn two ways to style things. ¹

Now, let’s dive into three workflows I’ve used or experimented with.

The standard way: bookdown gitbook and pdf_book

We R users are very lucky to be able to use bookdown for making books. You write content in Rmd files in a single folder, you order their filenames in a config file and choose an output format when knitting. There are many books out there built using bookdown, and often they’re knit twice so that there’s an HTML and PDF version. See for instance rOpenSci dev guide, that is deployed with GitHub Actions.

Advantages of bookdown include its widespread usage and stability, as well as its docs, both by its authors and users. For instance:

There is a bookdown book by bookdown maintainer Yihui Xie.
Emil Hvitfeldt wrote a post about deploying your bookdown project to Netlify with Github Actions.

A downside of bookdown ~~for the procrastinators out there~~ is, in my opinion, that if you want a PDF and HTML versions of a book, you have to knit it twice. Furthermore, if you introduce some fancy things such as a custom block, you will have to define it both with CSS and with LaTeX environments. Now, I have written a whole book with LaTeX so I used to partially know how to use it, but these days I’d prefer learning about CSS only as there’s already a lot to digest in there. 😅

So let’s dive into two DIY ways of publishing a book in HTML and PDF.

Hugo and pagedjs-cli

In this proof-of-concept, content is stored in Markdown files, that could have been generated with hugodown. Actors are Hugo, a powerful static website generator that you get in the form of a binary, and pagedjs-cli, a Node package that prints HTML to PDF.

You can find a repo corresponding to this experiment on GitHub, and the resulting book on Netlify.

This proof-of-concept is based on the great hugo-book theme. My “work” was just to add a bit of setup magic (Netlify config, config/ dir).

The idea is to run two Hugo builds with different configurations. In the words of Hadley Wickham, “hugo provides a bewildering array of possible config locations”. In this proof-of-concept/incredible machine, this array of possibilities is key to success. The two configurations use different layout directories, layout directories being the templates telling Hugo where to put your words, e.g. every blog post on its own page, and a page showing summaries of all of them.

One Hugo build, hugo -d public --environment 'website', creates the HTML website with a link to “book.pdf” somewhere, in the public/ folder.
The second one, hugo -d public2 --environment 'pdf', creates a website with a single HTML page containing all the book content.
Then that single page is printed to PDF using pagedjs-cli and copied as “book.pdf” to public/. pagedjs-cli public2/all/index.html -o public/book.pdf"
The live website uses the content of the public/ folder so it has the website and the PDF.

Netlify can run all these commands

[build]
publish = "/public"
command = "hugo -d public --environment 'website' && hugo -d public2 --environment 'pdf' && pagedjs-cli public2/all/index.html -o public/book.pdf"

The dependency on pagedjs-cli is indicated with package.json (the DESCRIPTION of Node projects).

Now if the content lived in R Markdown files, we would need to use a service like GitHub Actions to knit things.

Advantages

The Markdown files that are the source of both versions of the book would be generated only once if you started from R Markdown.
Styling both versions rely on CSS. I haven’t added any print styles but if you used this workflow for real, then it’d be better to, of course.
Compared to creating a gitbook with bookdown, one could customize the HTML website more easily, and use any Hugo feature.

Further work

If I decided to take this further, that is! All the work would be aimed at making the PDF version better

Improve the Hugo layout to add various tables of contents (for the whole book, for chapters).
Actually work on the CSS print sheet for the PDF version.
Make Katex, Mermaid etc. work for the PDF version.

The work presented by Julie Blanc is a similar pipeline, where content lives in Markdown files rendered to two HTML versions by Jekyll, another static website generator. One HTML version is a website, the other one is a printable version thanks to Paged.js (the PDF is not pre-generated but you can get it from any modern browser). The big differences with my experiment are:

Julie Blanc’s work is not an experiment, it’s deployed to production!
Both versions of the book are absolutely stunning, there was actual CSS work involved.

bookdown::bs4_book(), xml2 and pagedjs-cli

In another experiment, I used bookdown’s brand-new HTML book format, bs4_book() by Hadley Wickham (only available in bookdown’s dev version). I used this one because I find it looks better than gitbook, and because it uses Bootswatch themes, you get to use divs such as a alerts. See bookdown’s docs about custom blocks.

The source is open. See the HTML version and PDF version. This time I took time to learn a bit about print CSS. 😸

Here the steps are knit using the bs4_book() template for HTML and then for getting a PDF

tweaks of the HTML e.g. moving the chapter table of contents from the sidebar to the main content, and merging of all chapters using xml2 and XPath (so yeah you’re ditching LaTeX in favor of a workflow involving XPath, so modern 🙃) in build.R. I could have made my life easier by using selectr for translating ” CSS3 Selectors and translates them to XPath 1.0 expressions. “.
some print CSS, see stylesheet. It turns out that following Paged.js docs about print CSS is not that hard. See my stylesheet, I changed the display of some things in print, and I used properties to add the page number and chapter title to the footer.
pagedjs-cli.

Locally I sourced build.R to see whether it works, but then I also wrote a GitHub Actions workflow that installs dependencies, runs the script, and deploys the docs/ folder to a gh-pages branch.

Note that Bootstrap provides classes you can add to elements for changing their display in print. So to not show the burger menu in print, you can either do as I did with a CSS property

@media print {
  .btn.btn-outline-primary.d-lg-none.ml-2.mt-1 {
    display: none;
}

}

or you’d add the .d-print-none to that element in your HTML template. If you can control said HTML template, that is.

Further work needed in this experiment is listed in GitHub issues. For instance footnotes aren’t tackled yet.

Conclusion

So, as an R user writing content in R Markdown, you have different possibilities for respecting the principles of single-source publishing. I would still recommend using bookdown or other official rmarkdown formats, knitting twice² if you need two versions, as it is the only stable and widely used solution, but I found it fun to explore other workflows. Both the DIY ways I presented use pagedjs-cli to generate a PDF out of HTML. I strongly recommend following the work done by the lovely Paged.js folks in the world of paged media, and to learn more about CSS for print. If you want to produce only paged content out of your R Markdown file, check out the pagedown package and its fantastic output formats.

If you too want to experiment with homemade pipelines, have fun customizing things. 😉 Now, of course, the possibility of customization might actually be a curse when you are trying to write something. But this was not a blog post about productivity. 😁

At this point, let me thank Romain Lesur (one of pagedown authors) for his work promoting Paged.js in the R community. 🙏

Yes, I should probably re-style this very website. ↩︎
Potentially, knitr cache might make knitting a thing a second time very fast? 🤔 ↩︎

How I Taught Scientific Blogging with R Markdown, Online

Wed, 08 Jul 2020 00:00:00 +0000

Last week I had the pleasure to lead an online course about “Scientific Blogging with R Markdown”, invited by Najko Jahn and Anne Hobert from SUB Göttingen. To follow the example set by the incredible Alison Hill, I’ll write a summary of what I’ve learnt and would like to do better next time.

The topic

The topic of the course was “Scientific Blogging with R Markdown”. For months I would sometimes write down some ideas, from “present distill” to “show web developer console”, that I had whilst reading things online. I was already interested in topics related to R Markdown blogging and website development, but having the course on my agenda made me pay even more attention. I did my note taking in a physical notebook and then I set up a GitHub repo to use its issue tracker.

A bit later I had to start drafting an actual schedule, and decided I would try and not recommend a single tool, but instead show three ways of blogging with R Markdown: using the distill package whose goal is scientific and technical writing; using Hugo since that’s what I use at work; using WordPress since this comes up on Twitter every so often and seems like a good alternative workflow for when one does not use Git yet. I thought it might help each attendee see a tool that suit them better, and that it might mean everyone would learn something. I suppose it also helped reduce my responsability if folks ended up not liking their tool, since I did not recommend a single one. 😜

The course summary I wrote was

Are you an R user who works in science? Would you like sharing more online? How about starting a new blog, with R Markdown (Rmd)?

In this 2-hour course with live coding, we’ll go on three short adventures:

setting up a scientific Rmd blog with the Distill framework and the distill package
setting up a scientific Rmd blog with the Hugo website generator and the hugodown package
adding Rmd posts to a Wordpress blog.

We’ll prepare for the adventures by defining what we expect of an Rmd blog. We’ll end the course by reflecting on each adventure as well as mentioning important future paths such as how to promote your blog. You should leave the course ready to start a scientific R Markdown blog with your tool of choice, and knowing where to find more resources and help.

The practical plan

When I first said yes to giving the course a while ago, it was supposed to happen in person in Göttingen in Germany, but unsurprisingly that plan changed a few months ago. Online it would be! With Zoom. Anne Hobert and Najko Jahn, who organized the workshop and taught basic R Markdown on the first day, were technical helpers on the second day.

My course was to take place on the second day, in two hours. In the afternoon attendees would have time to work together in breakout rooms or on their own, and I stayed online for questions, as did Anne and Najko.

Anne had the idea to open my course to more attendees than the workshop first day i.e. open it to 10 attendees outside of the funding stream, bringing the maximum to 20 attendees. We decided to advertise it via R-Ladies communication channels (telling R-Ladies could share the info with their friends and colleagues from under-represented groups), and Anne gave spots on a first come first serve basis.

There was an online pad associated with the course, where attendees could write who they were (so cool to see that) and their questions. We had a short call at the beginning of the week to ensure we could hear each other. The three of us read The Carpentries guidance for teaching online and e.g. used the idea to ask attendees to write “hand” and “hand helper” in the Zoom chat if they wanted to ask a question to the instructor (me) or a helper.

A course website

Since it was a course about blogging, setting up a website seemed natural. It was also an excellent way to productively procrastinate if I’m being honest, but in the end I really liked filling and working with the course website so I’d say it was time well spent?

I used two Hugo themes,

Hugo theme learn (that has a nice table of contents on the left, search, copy-paste button for code…)
Hugo theme reveal-hugo (for slides)

Slides for each section are listed in the menu and opened in a new tab (thanks to a custom menu layout, compared to the original Hugo learn theme).

Some Markdown content was generated with R Markdown, using hugodown.

The website is deployed by Netlify.

Slides could be printed to PDF using Decktape which I had done in a concept but I did not pursue it further.

Why use Hugo for both the website and slidedecks, and not, say Hugo+hugodown for pages and xaringan for slides? This way the source of slides is html produced by Hugo from Markdown content. It allowed me to use:

downlit syntax highlighting for slides created from R Markdown with hugodown output format;
Chroma syntax highlighting for other languages;
emojis!
Shortcodes in slides, which I ended up doing for diagrams.

Also, because slides are in the content, they are indexed by the Hugo learn theme so searchable!

I learnt a few Hugo tricks whilst setting up the website which was fun per se, and I really liked the end product, as mentioned above. Two highlights for me were

to have a table of contents on the left that I could use on the day of the course
the page with snippets for demos that solved my problem “How am I going to organize the small pieces of code I need for demos?”.

The website also had questions for topics not mentioned during the course, where I e.g. stuffed the content from the notes I had taken.

The newest tools 😬

In two parts of the course I actually demo-ed packages in development which is not optimal, but I had excuses each time.

For using R Markdown and Hugo, I never fully used blogdown myself. When one day in my GitHub timeline I saw Hadley Wickham create hugodown, I got very excited because it fit more with the way I had used Hugo and R Markdown. I started using it so that was the tool I wanted to share about. I think one can trust hugodown’s maintainer. 😁
For using R Markdown and WordPress, the existing function relies on unmaintained packages (XML and RCurl), whereas there is a REST API associated with WordPress websites nowadays. I wrote a minimal package called goodpress with which it is possible to write directly from R Markdown to WordPress. It does not work for all WordPress packages yet, but I think that with Bob Rudis’ efforts in pressur, with some work (and contributors 😉) things could work for all WordPress websites. I would only recommend goodpress on existing websites at the moment, where it can only be a better idea than trying to use RCurl IMHO.

I hope my different warning signs will have helped the learners see what’s stable vs what’s not.

What we did

During the course two hours, I shared my whole desktop with the attendees¹. I had several slidedecks integrated in my website.

The slidedecks related to the three tools had a slide telling it was time for a demo, and then a few slides ending with a break countdown (I had to remove the last break). For the demos, I had printed my notes after knitting them to PDF. The demos were my doing website things in RStudio and Firefox. Compared to the demos on the website, for hugodown I only showed create_site_academic(), not how to make another theme hugodown-compatible.

In the end I took time to present my decks about reproducibility and about interactions with readers (including blog promotion and analytics).

It was a bit intense, which I need to think about if I teach this again.

In the few hours after my course, some attendees remained in breakout rooms (I got one question I think, about a Pandoc installation issue) and two of them showed their experiments in the final sharing session (where there were more than the two still online, although some people understandably had to leave).

Questions I got

And my tentative answers.

(about distill) Is there a difference between knitting and using the build tab?

I think at that point I realized I’d need to use distill more regularly to differentiate the different parts and the magic happening from RStudio.

Apparently from RStudio the website gets built when one edits the site configuration, and when one knits a post. You need to knit posts; and if you edit about.Rmd you need to either knit it or re-build the website. My impression is that you need to knit with intent, but most often the build part happens magically.

What is the difference (pros/cons) between hugo and distill?

I mentioned it a bit, distill is perfect for scientific/technical blogging but not very flexible. Hugo is very flexible which might be a curse. 😉 With Hugo you can really build any website: for this course website I mixed two Hugo themes, with a few custom layouts for instance (so even the slides used hugodown). But it requires time for learning. You could try both before committing, and you could still switch tools one day even if that’d take time.

Is it possible to have one citation style file for all posts/folders with Hugo or does it need to be in the same folder as the post?

For distill not yet https://github.com/rstudio/distill/issues/82
For hugodown not yet either https://github.com/r-lib/hugodown/issues/28

So for both distill and hugodown, it’s not possible yet but at least you can store .bib at the root and refer to it as ../../../refs.bib or so in your post YAML.

How to limit the readership of your blog - if i want my website to only be viewable to the people in my company?

For WordPress websites having private posts seem to be built-in.

For Hugo and distill it depends on how you deploy. I.e. from https://hugodown.r-lib.org/articles/deploy.html if you choose say Amazon S3 you could look for “Amazon S3 private website”. Such a feature might not be in the free tier of the service.

Or you could deploy to a server that your team has?

You could distribute it with your Hugo website (in the post folder or from static – I haven’t tried), otherwise sharing it in a GitHub repo works; but better for bigger datasets would be to use Figshare/Zenodo/etc.

Using utterance.es, you have some sort of notification if someone comments on your post?

Yes, same as other GitHub notifications (so email or web depending on your settings).

Is there a way to collaborate on the same blog?

Yes definitely. At rOpenSci we use GitHub for that (we even have a guide, blogguide.ropensci.org), you could use GitLab too. Adding posts by pull requests is nice because of the pull requests review infrastructure. Without a git platform I think it might be less natural to edit collaboratively but you could still find an alternative workflow.

Is it a good idea to use the distill format for the lab where you work instead of being a personal blog?

I’ve seen distill for groups, I think it works well for that too, if your colleagues are ok editing from R / letting you edit things.

Do you know cheap hosting services?

That question was a good reminder of my privilege: for me a custom domain costing a few dollars a month is not expensive. For others, especially in some countries, it can be a luxury. In that case, using GitHub pages, free Netlify domains, etc. is a good workaround and better than having no website.

I shortly mentioned one should add a privacy page mentioning what data is collected, and Anne Hobert reminded attendees that in Germany all websites need to have a page called Impressum/imprint.

Possible improvements

I might get even more ideas from participants² but here are a few ones:

I think next time it’d make sense to set up a second device to see the Zoom screen at all times. Or a second screen if Zoom lets me share one screen only.
I would not be that ambitious regarding what one can show with hugodown.
I’d like to know more about distill and WordPress to feel more at ease showing them, or to let someone else teach these parts.
I am not sure how, but I’d like to interact more with attendees.

A big limitation of the format, and the way I taught (giving a few recipes but not ensuring attendees live with a website), is that I can’t know attendees will actually go and build a website. Or maybe I’ll know, in a few weeks, if/when I receive URLs. I think that with websites, there’s a need for technical knowledge for sure, but maybe also some sort of accountability/support group.

Thank you

I’d like to thank Najko Jahn and Anne Hobert for trusting me to give this course and for being such helpful and nice collaborators. Thanks to the participants, I’m looking forward to seeing new blogs but no pressure. 😉 My course website has a credits page where I’m sure I forgot names. 😬

Next time?

I’d love to teach the topic again, re-using and building upon my materials. If you’re the organizer of a meetup for under-represented R users, get in touch to see if my skills and availabilities can be a fit for your group. If you’re organizing something else, let’s talk business i.e. in that case I wouldn’t teach for free. 😉 In all cases, I’m looking forward to keep learning new things myself.

If you taught or attended a workshop about blogging and R Markdown, feel free to share your own insights and good ideas. 🙏

Sharing my whole screen makes me quite nervous. I had written a list of things to do before hand such as making sure Slack and Mattermost were closed and wouldn’t send any notification. ↩︎
Participants were sent a pin.up link to add feedback on sticky notes à la Carpentries. ↩︎

How to showcase CSS+JS+HTML snippets with Hugo?

Tue, 21 Apr 2020 00:00:00 +0000

I’ve recently found myself having to write a bit of CSS or JS for websites made with Hugo. Note for usual readers: it is a topic not directly related to R, but you might have played with either or both CSS and JS for your R blog or Shiny app. On a scale from Peter Griffin programming CSS window blinds to making art with CSS, I’m sadly much closer to the former; my JS knowledge is not better. I often scour forums for answers to my numerous and poorly formulated questions, and often end up at code playgrounds like Codepen where folks showcase snippets of HTML on its own or with either or both CSS and JS, together with the resulting HTML document. You can even edit the code and run it again. Quite neat!

Now, as I was listening to an episode from the Ladybug podcast about blogging, one of the hosts, Ali Spittel, mentioned integrating Codepens into blog posts¹, which sounds useful indeed, and I started wondering: how could one showcase CSS+JS+HTML snippets on a Hugo website? In this post I shall go over three solutions, with and without Codepen, in all cases based on custom shortcodes.

The easiest way: Embed a Codepen

As reported in a 2018 blog post by Ryan Campbell and a 2019 blog post by Jeremy Kinson, Jorin Vogel created and shared a Hugo shortcode for Codepen.

Save it under layouts/pen.html and voilà, you can call it! Below I’ll embed a cool CSS art snippet by Sarah L. Fossheim.

Gives

For a blog post showcasing code of yours, it might get a bit tiring to create and keep track of Codepens. Moreover, you might want more ownership of your code.

The DIY way: load HTML, CSS, and JS code into an iFrame

I was completely stuck trying to find out how to create and embed my own iframe and then luckily found a perfect post by Josh Pullen “How to Load HTML, CSS, and JS Code into an iFrame”, with a perfect definition of the problem “If you’ve ever used JSFiddle, Codepen, or others, this problem will be familiar to you: The goal is to take some HTML, CSS, and JS (stored as strings) and create an iframe with the code loaded inside.". 🙌 Good stuff!

Based on the code in the post, I created a shortcode called “snippet.html”. It is a paired shortcode: there is input between two markers, and one option passed via the first marker. It expects input like

{{< snippet an-id-unique-for-the-post >}}
```css
// css code
```

```html
// HTML code
```

```js
// js code
```
{{< /snippet >}}

The shortcode itself is shown below.

<h4>My snippet {{ .Get 0 }}</h4>
{{ $content := .Inner }}
{{ $content := replaceRE "```html" "\n **HTML code:** \n```html" $content }}
{{ $content := replaceRE "```css" "\n **CSS code:** \n```css" $content }}
{{ $content := replaceRE "```js" "\n **JS code:** \n```js" $content }}

{{ $content | markdownify }}

{{ $css := replaceRE "```html(.|\n)*?```" "$1" .Inner }}
{{ $css := replaceRE "```js(.|\n)*?```" "$1" $css }}
{{ $css := replaceRE "```css" "$1" $css }}
{{ $css := replaceRE "```" "$1" $css }}

{{ $js := replaceRE "```html(.|\n)*?```" "$1" .Inner }}
{{ $js := replaceRE "```css(.|\n)*?```" "$1" $js }}
{{ $js := replaceRE "```js" "$1" $js }}
{{ $js := replaceRE "```" "$1" $js }}

{{ $html := replaceRE "```css(.|\n)*?```" "$1" .Inner }}
{{ $html := replaceRE "```js(.|\n)*?```" "$1" $html }}
{{ $html := replaceRE "```html" "$1" $html }}
{{ $html := replaceRE "```" "$1" $html }}

<b>Result:</b><br>

<iframe id="{{ .Get 0 }}" allowfullscreen
      style="width:100%;height:100%;"></iframe>

<script src="/js/blob.js" type="text/javascript"></script> 
<script type="text/javascript">
document.addEventListener('DOMContentLoaded', function() {
   mySnippet('{{ $html }}', '{{ $js }}', '{{ $css }}', '{{ .Get 0 }}');
}, false);
</script>

Lines 2 to 7 create and markdownify the three highlighted blocks, with a note on the language before each. Using markdownify means the code will be highlighted using Chroma without my having to make any further efforts.

Then there is some non elegant string manipulation going on to extract the CSS, JS and HTML, until line 58.

After that I use the code from Josh Pullen’s post to create blob URLs and an iframe.

I create an iframe with the ID given as argument of the shortcode.
Once the page is loaded, I call a function defined below and saved under /js/blob.js, again recycling code from Josh Pullen’s post, with the HTML, CSS and JS code, as well as the frame ID, as argument. The code creates blob URLs with the JS and CSS, then a blob URL with the HTML calling the JS and CSS, and finally assign that last blob URL to the iframe. ✨

function mySnippet(html, js, css, id) {
  const getGeneratedPageURL = ({ html, css, js }) => {
  const getBlobURL = (code, type) => {
    const blob = new Blob([code], { type });
    const url = URL.createObjectURL(blob);
    return url;
  };

  const cssURL = getBlobURL(css, 'text/css')
  const jsURL = getBlobURL(js, 'text/javascript')

  const source = `
    <html>
      <head>
      <meta charset="utf-8"/>
        ${css && `<link rel="stylesheet" type="text/css" href="${cssURL}" />`}
        ${js && `<script src="${jsURL}" type="text/javascript"></script>`}
      </head>
      <body>
        ${html || ''}
      </body>
    </html>
  `;
  return getBlobURL(source, 'text/html');
};

const url = getGeneratedPageURL({
  html: html,
  css: css,
  js: js
});

const getid = "#" + id;

const iframe = document.querySelector(getid);
iframe.src = url;
};

Demos

In this demo below I use w3schools.com tutorial about buttons.

{{< snippet number42 >}}
```css
p {
  color: red;
}
button {
  background-color: pink;
}
```

```html
<p> Some text </p>
<button onclick="getTime()">What time is it??</button>
<p id="demo"></p>
```

```js
function getTime() {
  document.getElementById('demo').innerHTML = Date();
}
```
{{< /snippet >}}

Gives

My snippet number42

CSS code:

p {
  color: red;
}
button {
  background-color: pink;
}

HTML code:

<p> Some text </p>
<button onclick="getTime()">What time is it??</button>
<p id="demo"></p>

JS code:

function getTime() {
  document.getElementById('demo').innerHTML = Date();
}

Result:

Below is another, simpler, one.

{{< snippet number66 >}}
```css
p {
  color: blue;
}
```

```html
<p> There is no JS in this snippet. </p>
```

{{< /snippet >}}

Gives

My snippet number66

CSS code:

p {
  color: blue;
}

HTML code:

<p> There is no JS in this snippet. </p>

Result:

To-dos

Clearly, my custom shortcode could do with… styling, which is sort of ironic, but this is left as an exercise to the reader. 😉

The mix: own your code, present it through Codepen

A page of Codepen docs caught my attention: “Prefill Embeds “ CodePen Prefill Embeds allow you to enhance code that you are already displaying on your own website and transform it into an interactive environment.

Using them make you rely on Codepen, of course, but you can therefore use all of Codepen fixings (even preprocessing!).

I created another shortcode as a proof-of-concept, not encompassing all features. In this case I was able to use nested shortcodes. In the previous solution I didn’t find how I could do that given I needed to use the content of each block on its own and together in the iframe.

The shortcode expects input like

{{< prefillembed "A title for the pen" >}}
  {{< pcode css >}}
  // CSS code
  {{< /pcode >}}
  
  {{< pcode html >}}
  // HTML code
  {{< /pcode >}}
  
  {{< pcode js >}}
  // JS code
  {{< /pcode >}}
  
{{< /prefillembed >}}

The main shortcode code is quite simple:

<div 
  class="codepen" 
  data-prefill='{
    "title": "{{ .Get 0 }}"}'
  data-height="400" 
  data-theme-id="1"
  data-default-tab="html,result" 
>
  {{ .Inner }}
</div>
<script async src="https://static.codepen.io/assets/embed/ei.js"></script>

The sub-shortcodes are not much more complicated. An important aspect is the escaping of HTML, that Codepen docs warn about. I felt quite proud knowing about htmlEscape but it was not enough, I had to pipe the output into safeHTML so I was no longer so full of myself after that. 😸

<pre data-lang="{{ .Get 0 }}">
  {{ if eq (.Get 0) "html" }}
    {{ .Inner| htmlEscape | safeHTML }}
  {{ else }}
    {{ .Inner }}
  {{ end }}
</pre>

Demo

{{< prefillembed "My Pen" >}}

  {{< pcode css >}}
  p {
    color: red;
  }
  button {
    background-color: pink;
  }
  {{< /pcode >}}
  
  {{< pcode html >}}
  <p> Some text </p>
  <button onclick="getTime()">What time is it??</button>
  <p id="demo"></p>
  {{< /pcode >}}
  
  {{< pcode js >}}
  function getTime() {
    document.getElementById('demo').innerHTML = Date();
  }
  {{< /pcode >}}
  
{{< /prefillembed >}}

gives

  
    
  p {
    color: red;
  }
  button {
    background-color: pink;
  }

  
    
  <p> Some text </p>
  <button onclick="getTime()">What time is it??</button>
  <p id="demo"></p>

  
    
  function getTime() {
    document.getElementById('demo').innerHTML = Date();
  }

To-dos

This shortcode could do with more parameterization to allow using all features of Codepen’s prefill embeds.

Conclusion

In this post I went over three ways to showcase CSS+JS+HTML snippets with Hugo: adding a custom shortcode for embedding Codepen; creating a custom shortcode thanks to which the code is displayed in highlighted code blocks but also loaded into an iframe; creating a custom shortcode that uses Codepen prefill embeds. Each approach has its pros and cons depending on whether or not you want to rely on Codepen. Please don’t hesitate to share your alternative approaches or your extensions of my shortcodes!

Taking a step back, such shortcodes, if much improved, could maybe be shared in a Hugo theme as a developer toolbelt²? Even if copy-pasting shortcodes from someone else’s repo, with attribution, works well too. 🙂 It could contain shortcodes for developer websites that use OEmbed (so not Stack Overflow, not GitHub), and unfurling workarounds for others. Quite a lot to explore!

I looked up the episode transcript to find out which of the hosts said that because I can’t recognize their voices (yet?). 😁 ↩︎
I am using this term because of Steph Locke’s Hugo utility belt. ↩︎

What to know before you adopt Hugo/blogdown

Sat, 29 Feb 2020 00:00:00 +0000

Fancy (re-)creating your website using Hugo, with or without blogdown? Feeling a bit anxious? This post is aimed at being the Hugo equivalent of “What to know before you adopt a pet”. We shall go through things that can/will break in the future, and what you can do to prevent future pain.

I’m writing this post with R users in mind, which means I shall use R analogies and mentioning blogdown, but I hope some aspects are generalizable to other potential Hugo adopters. Some aspects are specific to deploying your website using git and an online git platform such as GitHub.

Why Hugo/blogdown?

If you’re reading this you’ve probably heard of Hugo somewhere. It’s getting quite popular in the R community thanks to the blogdown package, whose associated book features an excellent intro to why Hugo (and blogdown).

I myself have used Hugo for this website (with a bit of blogdown) and dived into more details whilst working on tweaks to the rOpenSci website. I really enjoy working with this website generator, partly because it’s fast and open-source¹.

What can break or evolve?

When you have a website created with Hugo, there are two to four main actors:

Hugo itself which you could think of as R;
A Hugo theme which defines the layout and style of your website. You could think of the theme as an R package. Your theme is a folder inside your website folder. You need one, no matter how minimal.
the other things your website theme might depend on, such as Font Awesome.
potential fourth actor: your tweaks to the theme which might be your editing an R package source; or maintaining a script/a second package on to of that R package to add some syntactic sugar or so to the existing functions of the package.

So what’s going to break and evolve?

Just like R packages need to keep up with changes to R sometimes, Hugo themes need to keep up with Hugo changes. Hugo is a trustworthy framework, that keeps improving, which however means sometimes removing support for some things or ways of doing them. Example: this commit to the theme I’m using made a small change for making the theme compatible with Hugo>0.57.
Just like R packages need to keep up with changes to their external dependencies (web APIs, data format standard), Hugo themes need to keep with changes to external dependencies (font providers, JS scripts, etc). Example: a commit to the theme I’m using that updates the templates to use the new syntax for Font Awesome icons.
Just like R packages often improve with time, Hugo themes evolve. And just as you might like updating packages to get the best new tools, you might like getting the fancy features offered by a new version of your theme. For instance look at this release notes for some version of Hugo academic theme: there are bug fixes but also cool improvements!
Just like your taste or needs of R packages in your script might evolve, e.g. you might want to update old scripts to use data.table instead of dplyr, you might even want to switch themes!

How to reduce the likelihood of breakages?

Choose your theme wisely and keep in touch!

When choosing a theme i.e. a collection of layouts for your website, you’ll have aesthetics and practicalities in mind. E.g. if you’re a prolific blogger you’ll want the posts to be quite prominent. As the blogdown book mentions, also look at the theme’s popularity and activity before adopting it. This way you can have more trust in the theme’s responding to Hugo changes and to bug reports.

Now, you’ll have to know when the theme gets updated. How?

You could go hardcore git and make the theme a git submodule of your website repo.
You could watch i.e. subscribe to the GitHub releases of that theme, or, if the maintainer(s) use a less formal workflow, watch all activity from that repo.
Any other idea? Maybe involving GitHub Actions, Dependabot, something else? Maybe just a reminder to have a coffee date with your theme repo once in a while?

What if your theme gets orphaned?

What if you chose your theme wisely but it lost its maintainer(s)? In that case you’ll need to look into adopting it or rolling out your own version, or changing themes.

Make well-defined tweaks to the theme

Although you’ve adopted a theme, you’ll probably want to personalize it a bit. If you do so, do it with a good file structure and documentation hygiene! As very well explained in the blogdown book,

some tweaks are directly supported by the theme via the website config file (think of it as an R function parameters) e.g. adding your name to the homepage rather than Jane Doe’s;
some tweaks require your writing layout files (think of it as writing a wrapper for an R package/re-writing your own version of some functions) e.g. adding some fun sentence to the footer which the original theme doesn’t support. In this case you should store your custom layouts, like the fancy footer partial template, in a folder called layouts/ at the root of your website folder; not in the theme folder. Hugo will give priority to layouts/ stuff when defined, to use them on top of theme stuff; and you’ll easily see what you changed. Your future self will probably find much joy in your present self’s documenting the why and how of your custom layouts in some sort of developer notes. The more custom layouts you write, the greater your future responsability.

If you define CSS and JS files, they’d live in static/ and be referred to in the website config file.

IMHO Have your content as Markdown, not html files

With blogdown you can use .Rmd, .RMarkdown or .md as your website source, refer to this exhaustive and clear comparison. I am strongly in favour of never using .Rmd in a blogdown site because its output is an html file, not a .markdown/.md file. html files seem less portable to me. I like the idea of being able to take my Markdown content and rather easily move them to a new theme (or even a new framework, like when I migrated this website from Jekyll).

Now, as explained in the table mentioned above, using .RMarkdown/.md has its limitations. For html widgets I can recommend looking at the setup Steph Locke created for the RECON learn website.

Update your theme

So if you’ve tweaked cleanly, you can update your theme when needed! If you work with version control, and you probably should, make a branch and work on the update in that branch. If you don’t, backup your website’s current state first.

To “update your theme”, you need to replace the theme folder of your website folder with the new theme files.

You could do update the theme manually, or use blogdown::install_theme() with force = TRUE.

Build your website, look at what needs to be changed:

maybe the config parameter names changed (like an R function which would have renamed one of the parameters);
your custom layout might need some work too.

To figure out what needs to be changed, you’ll probably want to read the changelog (or commit history) of your theme, and maybe even Hugo changelog.

Often, changes in your theme, and work needed on your website, won’t be dramatic: the theme folder update, maybe one config parameter, a few lines diff in your custom layouts.

Follow Hugo news?

If you wrote no custom layouts and use a very well maintained theme, you might never need to keep up with Hugo changes yourself. However, if you’ve written Hugo custom layouts, or strive to become a contributor to your theme, you might want to read Hugo changelogs, follow Hugo’s source repository, or Hugo’s Twitter account, etc.

Edit in May 2020. After asking on the Hugo forum how to follow Hugo more automatically for websites I (co-)maintain on GitHub, I ended up creating a GitHub Actions workflow that will look for Hugo updates once a month, and if needed open a PR amending Hugo version in the Netlify config file.

Don’t live on the edge

If you have a workflow on a continuous integration system updating your website every day from an external data source, like Noam Ross and Jim Hester do (yes that’s a very cool and very fancy setup!), use a specific Hugo version there, don’t let the workflow install Hugo’s latest version because it could break your website without your noticing. Update Hugo with intent, after reading a bit and/or testing it doesn’t break your website.

What if I just never update Hugo or my theme?

Edit in August 2024: I think pinning Hugo and the theme at a version that works for you is a fantastic idea! You can use hugodown to pin the Hugo version (and probably blogdown too).

No, it’s not a good solution in my opinion. Never updating Hugo (neither locally nor on say Netlify) nor your theme probably means your website will still build as it used to. However, updates to Hugo/themes contain both improvements and bug fixes so it’s better to know you’ll probably update the tools at least once in a while.

Quick fixes to bad news

Imagine you made yourself a pretty website to showcase your cool posts and informative slidedecks. In the meantime, you changed laptops and today Saturday you want to post the link to your talk from last week, before you head out for a barbecue with friends. Your laptop doesn’t have Hugo installed so you install its latest version using blogdown::install_hugo(), add your talk page, build the website, look at the preview and… notice it all looks wonky! Your website is broken! If you remember the beginning of this post, what happened is probably an update in Hugo breaking your theme version. You’ll eventually need to update your theme.

But you don’t have time right now before your barbecue to do that, let alone to learn how to do that if it’s the first time, so what can you do apart from not posting your talk content?

If your site is deployed by gh-pages

I.e. your build your website locally and then push the rendered content to a gh-pages branch of a GitHub repo, you need to retrograde Hugo before doing that. Note that blogdown::install_hugo() has a version argument, refer to Hugo changelogs to see what version you had last used.

If your site is deployed by Netlify

If you’re lucky, you can just push your content, and since the Hugo version of your Netlify’s config file hasn’t changed, your website will build smoothly. You could make a PR to your own website to get a preview there before merging.

Later

Eventually, one day soon, get to updating Hugo again, looking at your theme’s changes, if needed extracting your tweaks from the theme folder if you made them there (I’ve done that, and now I sure stick to using the layouts/ folder as mentioned earlier). Use some of the tips of the beginning of the post to reduce the likelihood of the scenario happening again.

Conclusion

In this blog post I presented what the maintenance of a Hugo website entails in my experience. I haven’t mentioned other possible sources of breakage that are not specific to Hugo:

URLs in the content getting obsolete;
Your changing deploy workflows (so use your own domain instead of a Netlify’s one?);
Your website source disappearing (so use backups);
etc.

Coming back to Hugo, if you encounter problems I’d unsurprisingly recommend Hugo docs and Hugo forum.

Now, if this all sounds overwhelming, I don’t think these tech skills are harder than R skills but time is a limited resource so maybe you could outsource some of your website’s creating and maintenance. Maybe you can hire someone, or do a skill swap?

Don’t hesitate to share your own experience and advice on maintaining Hugo websites!

Also because I love the font of its docs website and looking at pages full of curly braces. ↩︎

Goodbye, Disqus! Hello, Utterances!

Wed, 02 Oct 2019 00:00:00 +0000

Removing Disqus from my blogdown blog had been on my mind for a while, ever since I saw Bob Rudis’ tweet enjoining Noam Ross to not use it for his brand-new website. The same Twitter thread introduced me to Utterances, a “lightweight comments widget built on GitHub issues”, which I have at last installed to my blog in lieu of Disqus. How did I manage to not lose anything of value? How easy was it to switch tools? Read on to learn more!

Was saying goodbye to Disqus hard?

Removing Disqus was neither emotionally nor technically hard.

Comments I (kinda) let go of

To deal with my fear of loss, I exported all the comments to an XML. It means I have a backup, that in theory I could explore within R! I’ll show some XML wrangling a bit later. The export does contain Disqus usernames and names which means I’m now responsible for person identifying information that Disqus collected.

Thanks a lot to commenters in the past system for taking the time to leave me a note: it helped me keep blogging!

The Disqus XML contains information about

posts that are in fact comments, not blog posts, along with their threading information i.e. both whose blog post the comment was on, and if relevant, which comment the comment is an answer to;
threads, one thread per comment but not necessarily one comment per thread, and potentially more than one thread per blog post. Each top-level comment is a thread, but a website page with no comment also gets a thread, obviously empty.

In the code below, I shall rectangle the information about threads (id, title) and about comments (message, author name, date, thread id, was it spam, was it deleted) using the xml2 package.

export <- xml2::read_xml("data/disqus-export.xml")
export

## {xml_document}
## <disqus schemaLocation="http://disqus.com/api/schemas/1.0/disqus.xsd http://disqus.com/api/schemas/1.0/disqus-internals.xsd" xmlns="http://disqus.com" xmlns:dsq="http://disqus.com/disqus-internals" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
##  [1] <category dsq:id="5516420">\n  <forum>masalmon</forum>\n  <title>Ge ...
##  [2] <thread dsq:id="5169974011">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [3] <thread dsq:id="5177145143">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [4] <thread dsq:id="5192091523">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [5] <thread dsq:id="5490361085">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [6] <thread dsq:id="5490455492">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [7] <thread dsq:id="5496622766">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [8] <thread dsq:id="5496655533">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [9] <thread dsq:id="5499530252">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [10] <thread dsq:id="5499566886">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [11] <thread dsq:id="5503605733">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [12] <thread dsq:id="5503629818">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [13] <thread dsq:id="5519522381">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [14] <thread dsq:id="5521172753">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [15] <thread dsq:id="5523856167">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [16] <thread dsq:id="5523862926">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [17] <thread dsq:id="5544585525">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [18] <thread dsq:id="5544919490">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [19] <thread dsq:id="5544919711">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [20] <thread dsq:id="5544920033">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## ...

Explaining how to parse XML is beyond the scope of this blog post, but let me just mention I used a search engine to answer questions such as “XPath extract nodes by name”, “xml2 namespace”. I reckon my code could be more elegant if I knew more XPath.

Let me start with threads.

# extract nodes corresponding to comments
threads_nodes <- xml2::xml_find_all(export, "d1:thread")
threads_nodes

## {xml_nodeset (283)}
##  [1] <thread dsq:id="5169974011">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [2] <thread dsq:id="5177145143">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [3] <thread dsq:id="5192091523">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [4] <thread dsq:id="5490361085">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [5] <thread dsq:id="5490455492">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [6] <thread dsq:id="5496622766">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [7] <thread dsq:id="5496655533">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [8] <thread dsq:id="5499530252">\n  <id/>\n  <forum>masalmon</forum>\n  ...
##  [9] <thread dsq:id="5499566886">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [10] <thread dsq:id="5503605733">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [11] <thread dsq:id="5503629818">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [12] <thread dsq:id="5519522381">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [13] <thread dsq:id="5521172753">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [14] <thread dsq:id="5523856167">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [15] <thread dsq:id="5523862926">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [16] <thread dsq:id="5544585525">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [17] <thread dsq:id="5544919490">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [18] <thread dsq:id="5544919711">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [19] <thread dsq:id="5544920033">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## [20] <thread dsq:id="5544931944">\n  <id/>\n  <forum>masalmon</forum>\n  ...
## ...

# extract interesting information from each node,
# information that is potentially nested
threads <- tibble::tibble(
  thread_id = xml2::xml_attr( threads_nodes, 
                              "id"),
  title = xml2::xml_text(
    xml2::xml_find_all(threads_nodes, "d1:title")
    ))

Now on to comments…

# extract nodes corresponding to comments
comments_nodes <- xml2::xml_find_all(export, "d1:post") 
comments_nodes

## {xml_nodeset (206)}
##  [1] <post dsq:id="2916181700">\n  <id/>\n  <message><![CDATA[<p>La poss ...
##  [2] <post dsq:id="3118121105">\n  <id/>\n  <message><![CDATA[<p>Great b ...
##  [3] <post dsq:id="3118236082">\n  <id/>\n  <message><![CDATA[<p>Nice Ma ...
##  [4] <post dsq:id="3118757895">\n  <id/>\n  <message><![CDATA[<p>Thanks  ...
##  [5] <post dsq:id="3118758593">\n  <id/>\n  <message><![CDATA[<p>Thanks  ...
##  [6] <post dsq:id="3118783826">\n  <id/>\n  <message><![CDATA[<p>I love  ...
##  [7] <post dsq:id="3118785805">\n  <id/>\n  <message><![CDATA[<p>Thanks! ...
##  [8] <post dsq:id="3121124991">\n  <id/>\n  <message><![CDATA[<p>Chouett ...
##  [9] <post dsq:id="3121137475">\n  <id/>\n  <message><![CDATA[<p>Oops qu ...
## [10] <post dsq:id="3121945521">\n  <id/>\n  <message><![CDATA[<p>I poste ...
## [11] <post dsq:id="3122653372">\n  <id/>\n  <message><![CDATA[<p>Another ...
## [12] <post dsq:id="3122759373">\n  <id/>\n  <message><![CDATA[<p>Thank y ...
## [13] <post dsq:id="3123851616">\n  <id/>\n  <message><![CDATA[<p>Great s ...
## [14] <post dsq:id="3126340152">\n  <id/>\n  <message><![CDATA[<p>Once ag ...
## [15] <post dsq:id="3126343942">\n  <id/>\n  <message><![CDATA[<p>Thank y ...
## [16] <post dsq:id="3140125477">\n  <id/>\n  <message><![CDATA[<p>The ide ...
## [17] <post dsq:id="3140136118">\n  <id/>\n  <message><![CDATA[<p>Thank y ...
## [18] <post dsq:id="3140161542">\n  <id/>\n  <message><![CDATA[<p>Not rel ...
## [19] <post dsq:id="3140329480">\n  <id/>\n  <message><![CDATA[<p>Yes def ...
## [20] <post dsq:id="3150961786">\n  <id/>\n  <message><![CDATA[<p>That's  ...
## ...

# extract interesting information from each node,
# information that is potentially nested
comments <- tibble::tibble(
  thread_id = xml2::xml_attr(
    xml2::xml_find_all(comments_nodes, "d1:thread"), 
    "id"),
  message = xml2::xml_text(
    xml2::xml_find_all(comments_nodes, "d1:message")
    ),
  date =  xml2::xml_text(
    xml2::xml_find_all(comments_nodes, "d1:createdAt")
    ),
  deleted = xml2::xml_text(
    xml2::xml_find_all(comments_nodes, "d1:isDeleted")
    ),
  spam = xml2::xml_text(
    xml2::xml_find_all(comments_nodes, "d1:isSpam")
    ),
  author = xml2::xml_text(
    xml2::xml_find_all(
    xml2::xml_find_all(comments_nodes, "d1:author"), 
    "d1:name")))
  
comments <- dplyr::mutate(comments,
                          date = anytime::anydate(date),
                          deleted = deleted == "true",
                          spam = spam == "true")

I was then able to join the two tables.

comments <- dplyr::left_join(comments, threads,
                             by = "thread_id")
readr::write_csv(comments, "comments.csv")

comments[2:12, c("message", "title")]

## # A tibble: 11 x 2
##    message                                        title                    
##    <chr>                                          <chr>                    
##  1 "<p>Great blog post, thank you! I really appr… French villages and a so…
##  2 <p>Nice Maëlle!</p><p>Good, interesting quest… French villages and a so…
##  3 <p>Thanks Nick!</p>                            French villages and a so…
##  4 <p>Thanks Lisa! Toponomy is quite fascinating… French villages and a so…
##  5 "<p>I love this! I wonder, would it be possib… French villages and a so…
##  6 <p>Thanks! I got a similar question on Twitte… French villages and a so…
##  7 "<p>Chouette! Mais la mer c'est \"sea\", \"se… French villages and a so…
##  8 <p>Oops quelle faute de frappe idiote, je la … French villages and a so…
##  9 "<p>I posted an update! <a href=\"http://www.… French villages and a so…
## 10 <p>Another good one, Maëlle! It's a great exa… More water, a bit more a…
## 11 <p>Thank you! Lol on ninja grep skills, I wis… More water, a bit more a…

I got a pretty nice rectangle in the end, that could be used for some text analysis, but in my case, I mostly view it as memorabilia. I hope any comment warranting action had been tackled. I wrote 93 (sum(comments$author == "Maëlle Salmon")) out of the 206 comments (Disqus classified a few comments as spam, 15 to be exact, of which a few were informative comments. ), because I tend to answer comments, if only with a simple thank you! Now that I have GitHub issues as comments, I could also answer with emojis.

It was fun because I didn’t even remember about some notes I had gotten. I hope you don’t feel slighted, dear reader, now that I keep these old comments to myself! Be happy you’re no longer tracked.

Getting rid of Disqus

I removed Disqus from my website, and my website from Disqus!

To remove Disqus from a Hugo website, one can, depending on the website’s theme:

use Hugo’s Disqus GDPR setting if the website theme uses Hugo’s built-in Disqus template (i.e. if there’s a mention of a internal Disqus partial, not of a Disqus partial stored somewhere in the theme files);
or remove one’s Disqus username from the config file, or remove the Disqus partial mention in the post template. To do that, the best process is not to edit the e.g. post/single.html file in the theme folder, but instead to save it and edit it under layouts/post/single.html. Refer to the blogdown book for more information.

After that, there should be no mention of Disqus scripts in your website source.

Then, to delete my website from my Disqus account, following Disqus guidance, I logged into my account and, after exporting my website’s comments, confirmed I wanted to do that and voilà!

Adding Utterances

To add Utterances, a “lightweight comments widget built on GitHub issues”, I also followed official guidance:

I installed the app to my website GitHub repo.
In my custom single.html layout for posts (again, refer to the blogdown book for clear information on the topic), I added the script tag corresponding to Utterances. The docs linked above help you configure the appearance of the widget.

Here’s the Utterances script tag for my blog. The label needs to have been created for your website repo, since Utterances can only use existing issue labels.

<script src="https://utteranc.es/client.js"
        repo="maelle/simplymaelle"
        issue-term="title"
        label="comments 💬"
        theme="github-light"
        crossorigin="anonymous"
        async>
</script>

I then wrote a comment under one of my posts to make sure things worked. I also took a few seconds to add my site to the list of sites using Utterances. Overall, I found the process easy, as I had the first time I installed Utterances on a blogdown website.

I am aware that a downside of Utterances is that it is tied to GitHub. However, I am not afraid of switching to yet another tool in the future, and hope not too many potential commenters will be turned off by the necessity to use a GitHub account to comment.

Conclusion

In this post I explained how I removed Disqus from my blog and vice versa, and how I installed Utterances instead. What I did not do was analyzing the Disqus data. I moreover did not try to transfer old comments to the new system, partly because that’d mean removing commenters’ ownership on what they had written.

In his tweet that motivated me to say goodbye to Disqus, Bob wrote “Yet another service where you and your site visitors are the product. Plus it acts like malicious javascript.”. I therefore feel a bit better about this website! Now, I didn’t have a ton of comments, and I don’t have a real need for moderating comments, so maybe you’ll make a different decision for your website. I’d be glad to hear about it… in the comments section below! Also feel free to tell me if you explore your own Disqus data in a more thorough way than I!

How did Axios rectangle Trump's PDF schedule? A try with R

Mon, 11 Feb 2019 00:00:00 +0000

Last week, Axios published a very interesting piece reporting on Trump’s private schedule thanks to an insider’s leak. The headlines all were about Trump’s spending more than 60% of his time in “executive time” which admittedly was indeed the most important aspect of the story. I, however, also got curious about Axios’ work to go from the PDF schedules to the spreadsheet they made public. In this post, I’ll have a got at using rOpenSci’s Jeroen Ooms’ pdftools package and some data-wrangling stubborness of mine to try and rectangle Trump’s PDF schedules.

Available data, available software

Axios published both the spreadsheet they made from the data leaked, and PDF corresponding to the schedules retyped in the format White House staff received them. I tried finding out how they went from the latter to the former, but this was apparently not reported. Quite a niche topic, I agree. Here’s the untidy PDF below.

pdf_file <- "data/Axios-President-Donald-Trump-Private-Schedules.pdf"
library("magrittr")
pdftools::pdf_render_page(pdf_file, page = 1) %>%
  magick::image_read()

Seeing the PDF format above reminded me of the cool new pdftools feature Jeroen Ooms announced back in December: low level text extraction.

# High level text extraction
pdftools::pdf_text(pdf_file)[1]

## [1] "                       SCHEDULE OF THE PRESIDENT\n                          Wednesday, November 7, 2018\n8:00 AM        EXECUTIVE TIME\n(3 hr)         Location:           Oval Office\n11:00 AM       MEETING WITH THE CHIEF OF STAFF\n(30 min)       Location:           Oval Office\n11:30 AM       EXECUTIVE TIME\n(1 hr)         Location:           Oval Office\n12:30 PM       LUNCH\n(1 hr)         Location:           Private Dining Room\n1:30 PM        EXECUTIVE TIME\n(3 hr, 30 min) Location:           Oval Office\nRON:           The White House\n"

# Low level text extraction
head(pdftools::pdf_data(pdf_file)[[1]])

##   width height   x  y space       text
## 1    60     12 220 72  TRUE   SCHEDULE
## 2    15     12 283 72  TRUE         OF
## 3    23     12 301 72  TRUE        THE
## 4    62     12 327 72 FALSE  PRESIDENT
## 5    54     12 236 99  TRUE Wednesday,
## 6    46     12 293 99  TRUE   November

While the pdftools::pdf_text() function returns “a character vector of length equal to the number of pages in the file”, pdftools::pdf_data returns a list of as many data.frame’s as there are pages, each one indicating the coordinates and width of each text piece. So powerful! Trump’s schedule appeared to be a perfect use case for this functionality!

Special tale for Ubuntu users

Edit: Benjamin Louis posted a link to a helpful thread about installing Poppler’s latest version and pdftools on Ubuntu 18.04 in the comments. Thank you!

Sadly, as noted in Jeroen’s tech note, the new Poppler version that is necessary for pdftools to support low level text extraction… is not available on Linux yet, and I’ve just switched to Ubuntu from Windows (an accidental New Year’s resolution after Windows disappeared from my laptop). Jeroen wrote “Linux users probably have to wait for the latest version of poppler to become available in their system package manager (or compile from source).”. I thought, OK, let me just compile Poppler from source then. I’m not sure I’d recommend trying this at home so take my experience report with a bit of caution. I downloaded Poppler’s latest version, unzipped it, created a build folder inside the poppler folder and ran something like this (inspired by this gist):

cmake ..
make
sudo make install

It seemed to work but then re-installing pdftools failed. Jeroen thankfully advised me to remove my old Poppler installation with a drastic command that affected all softwares depending on Poppler (that’s why you might not want to reproduce this at home) like GIMP or Libre Office Writer: sudo apt-get remove libpoppler*. But, after I did that, I was able to install pdftools using the newest Poppler version.

General strategy

I started work by carefully looking at the PDF. There were one-page daily schedules like on page 1,

pdftools::pdf_render_page(pdf_file, page = 1) %>%
  magick::image_read()

but also schedules continued on several pages,

pdftools::pdf_render_page(pdf_file, page = 3) %>%
  magick::image_read()

Because of that, I decided to extract the dates of pages that had one, then assign to each undated page the latest available date, before splitting the PDF low level text data into dates and working on a pipeline by date.

Splitting the low level text data by date

In a first step, I wrote a pipeline adding the page number to each data.frame from pdftools::pdf_data().

pdf_data <- pdftools::pdf_data(pdf_file)

read_page <- function(page, pdf_data){
  df <- pdf_data[[page]]
  if(nrow(df) > 0) {
    df$page <- page
    df
  } else {
    NULL
  }
}

pdf_text <- purrr::map(1:pdftools::pdf_info(pdf_file)$pages,
                       read_page,
                       pdf_data = pdf_data)

# remove empty pages
pdf_text <- purrr::discard(pdf_text, is.null)

head(pdf_text[[1]])

##   width height   x  y space       text page
## 1    60     12 220 72  TRUE   SCHEDULE    1
## 2    15     12 283 72  TRUE         OF    1
## 3    23     12 301 72  TRUE        THE    1
## 4    62     12 327 72 FALSE  PRESIDENT    1
## 5    54     12 236 99  TRUE Wednesday,    1
## 6    46     12 293 99  TRUE   November    1

Then for each of them I tried extracting the date.

get_date <- function(page_data){
  if(page_data$text[[1]] == "SCHEDULE"){
    date <- as.character(
      glue::glue_collapse(page_data$text[6:8],
                          sep = " "))
    date <- lubridate::parse_date_time(date, 
                                       orders = "BdY", 
                                       locale = "en_US.utf8", tz = "EST")
    date <- as.Date(date)
  }else{
    date <- NA
  }
  tibble::tibble(date = date,
                 page = page_data$page[1])
}

dates <- purrr::map_df(pdf_text, get_date)

Finally, I created a gigantic data.frame of all the data.frame’s from pdftools::pdf_text(), sorted it by page just to be sure it was sorted by page, assigned to each row the latest non missing page, and then split the data by date which was a better unit to work from. In order to not mess up coordinates, within each date I added 700, 1400, etc. to y-coordinates from the second, third, etc. page.

split_days <- function(pdf_text, dates){
  pdf_text %>% 
  dplyr::bind_rows() %>%
  dplyr::left_join(dates, by = "page") %>%
  dplyr::arrange(page) %>%
  dplyr::mutate(date = zoo::na.locf(date)) %>%
  dplyr::group_by(date) %>%
  dplyr::mutate(y = y + (page - min(page)) * 700) %>%
  dplyr::ungroup() %>%
  split(.$date)
}
  
pdf_text <- split_days(pdf_text, dates)

length(pdf_text)

## [1] 50

pdf_text[[2]]

## # A tibble: 243 x 8
##    width height     x     y space text       page date      
##    <int>  <int> <int> <dbl> <lgl> <chr>     <int> <date>    
##  1    60     12   220    72 TRUE  SCHEDULE      2 2018-11-08
##  2    15     12   283    72 TRUE  OF            2 2018-11-08
##  3    23     12   301    72 TRUE  THE           2 2018-11-08
##  4    62     12   327    72 FALSE PRESIDENT     2 2018-11-08
##  5    44     12   240    99 TRUE  Thursday,     2 2018-11-08
##  6    46     12   287    99 TRUE  November      2 2018-11-08
##  7     8     12   336    99 TRUE  8,            2 2018-11-08
##  8    21     12   347    99 FALSE 2018          2 2018-11-08
##  9    19     12    72   140 TRUE  8:00          2 2018-11-08
## 10    17     12    94   140 FALSE AM            2 2018-11-08
## # … with 233 more rows

tail(pdf_text[[2]])

## # A tibble: 6 x 8
##   width height     x     y space text         page date      
##   <int>  <int> <int> <dbl> <lgl> <chr>       <int> <date>    
## 1    28     12   229  1069 FALSE House           3 2018-11-08
## 2    51     12   288   799 TRUE  Christopher     3 2018-11-08
## 3    31     12   342   799 FALSE Liddell         3 2018-11-08
## 4    30     12   288   813 FALSE Closed          3 2018-11-08
## 5    21     12   288   826 TRUE  Oval            3 2018-11-08
## 6    28     12   312   826 FALSE Office          3 2018-11-08

At that point I took a minute to compare the dates I had with the ones in Axios’ clean data.

clean_data <- readr::read_csv("data/axios_data.csv")
unique(clean_data$date[!clean_data$date %in% dates$date])

##  [1] "2018-11-12" "2018-11-16" "2018-11-20" "2018-11-21" "2018-11-22"
##  [6] "2018-12-06" "2018-12-24" "2018-12-25" "2018-12-26" "2018-12-27"
## [11] "2018-12-28" "2018-12-31" "2019-01-01"

Wat?! I had a look at the PDF, and actually, these dates are not even in the PDF I’m fighting, so no need for me to try and revise my R script.

Digesting each date’s data

This part was the most crucial, and difficult one. I wanted to obtain a table with one line by event. My strategy was to first extract times by filtering text data whose x-coordinate was in an interval I had decided on by looking at some pages, clean these times to obtain time, duration and y-coordinate of the top of the time info, and then assign words from the right of the page to each event thanks to fuzzy joins: words correspond to an event if their y-coordinate is between the y-coordinate of the time of that event and the y-coordinate of next event.

On top of that I had to decide what to do with pages with times in several timezones, like below

pdftools::pdf_render_page(pdf_file, page = 13) %>%
  magick::image_read()

I simply removed all times that where in parentheses, assumed a time with no timezone information had the same timezone as the last event with a timezone, and that the first event of the day, if lacking a timezone, was in EST. In real life if I were really working on such data I’d probably still make such a general assumption but then review the result page by page, preferably with several pairs of eyes.

get_times <- function(page_data){
  times <- dplyr::filter(page_data,
                         dplyr::between(x, 72, 117))
  rest <- dplyr::filter(page_data,
                        !dplyr::between(x, 72, 117))
  
  times <- times %>%
    dplyr::group_by(y) %>%
    dplyr::summarize(text = glue::glue_collapse(text, sep = " "))
  
  times <- dplyr::filter(times,
                         !(grepl("\\(", text)&grepl("M", text)))
  clean_times <- dplyr::mutate(times,
                               time = dplyr::if_else(
                                 grepl("M", text)|grepl("RON", text),
                                 text,
                                 ""),
                               duration = dplyr::if_else(
                                 grepl("\\(", dplyr::lead(text)),
                                       dplyr::lead(text),
                                       "")
                               )
  clean_times <- dplyr::filter(clean_times, time != "")
  clean_times <- dplyr::select(clean_times, y, time, duration)
  clean_times <- dplyr::mutate(clean_times,
                               timezone = gsub(".*\\:.*M ", "", time), 
                               timezone = ifelse(timezone == time,
                                                 NA, timezone))
  if(is.na(clean_times$timezone[1])){
    clean_times$timezone[1] <- "EST"
  }
  clean_times <- dplyr::mutate(clean_times,
                               timezone = zoo::na.locf(timezone))
  
  clean_times <- dplyr::mutate(clean_times,
                               timezone = trimws(timezone),
                               timezone = gsub(" ", "",timezone),
                               time = gsub("M.*", "M", time))
  
  clean_times <- dplyr::mutate(clean_times,
                               start = y, 
                               end = ifelse(dplyr::lead(y) != start,
                                            dplyr::lead(y) - 1, start), 
                               end = ifelse(is.na(end),
                                            99999, end))
  if(nrow(clean_times) == 0){
    print(page_data$date[1])
    return(NULL)
  }
  
  rest <- dplyr::mutate(rest,
                        start = y, end = y)
  
  df <- fuzzyjoin::interval_left_join(clean_times,
                                      rest,
                                      by = c("start", "end"))
  df <- dplyr::select(df, time, duration, timezone, x, y.y, text)
  df <- dplyr::rename(df, y = y.y)
  
  df <- df %>%
    dplyr::group_by(time, duration, timezone) %>%
    dplyr::mutate(event = as.character(
      glue::glue_collapse(text[y == min(y)], sep = " "))) %>%
    dplyr::group_by(time, duration, timezone, y, event) %>%
    dplyr::summarize(text = as.character(
      glue::glue_collapse(text, sep = " ")
    )) %>%
    dplyr::group_by(time, duration, timezone, event) %>%
    dplyr::mutate(text = ifelse(toupper(event) != event,
                                as.character(
                                  glue::glue_collapse(text, sep = " ")
                                ), text)) %>%
    dplyr::ungroup()
  
  if(nrow(df) == 0){
    df <- tibble::tibble(event = as.character(
      glue::glue_collapse(page_data$text, sep = " ")
    ))
  }
  
  df$date <- page_data$date[1]
  df
}

all_events <- purrr::map_df(pdf_text,
                            get_times)

head(all_events)

## # A tibble: 6 x 7
##   time    duration    timezone     y event         text          date      
##   <chr>   <chr>       <chr>    <dbl> <chr>         <chr>         <date>    
## 1 1:30 PM (3 hr, 30 … EST        302 EXECUTIVE TI… EXECUTIVE TI… 2018-11-07
## 2 1:30 PM (3 hr, 30 … EST        315 EXECUTIVE TI… Location: Ov… 2018-11-07
## 3 11:00 … (30 min)    EST        180 MEETING WITH… MEETING WITH… 2018-11-07
## 4 11:00 … (30 min)    EST        194 MEETING WITH… Location: Ov… 2018-11-07
## 5 11:30 … (1 hr)      EST        221 EXECUTIVE TI… EXECUTIVE TI… 2018-11-07
## 6 11:30 … (1 hr)      EST        234 EXECUTIVE TI… Location: Ov… 2018-11-07

Cleaning each event

At that point I was fairly happy but I knew that a code review would have been great in real life, and that in any case I needed more work. I wanted to have a duration in minutes for each event as well as a start and end as times, and I wanted each event’s info to be in a single row.

I first made each event a row.

gather_events <- function(all_events){
    all_events %>%
    dplyr::group_by(event, text) %>%
    dplyr::mutate(clean_text = trimws(gsub(event[1], "", text))) %>%
    dplyr::ungroup() %>%
    dplyr::select(- text) %>%
    dplyr::rename(text = clean_text)
  
  # events with no text
  no_text_events <- all_events %>%
    dplyr::group_by(time, duration, 
                    timezone, date) %>%
    dplyr::filter(all(!nzchar(text))) %>%
    dplyr::summarise(event = unique(event),
                     text = NA,
                     y = min(y))
  
  
  # events with text
  text_events <- all_events  %>%
    dplyr::group_by(time, duration, 
                    timezone, date) %>%
    dplyr::filter(any(nzchar(text))) %>%
    dplyr::filter(nzchar(text)) %>%
    dplyr::summarize(y = min(y),#y gives us the *certain* order of events in a day
                     event = ifelse(length(text[!grepl("\\:", text)]) > 0,
                                    as.character(
                                 glue::glue_collapse(paste(unique(event, text[!grepl("\\:", text)&text!=event]))), sep = " "),
                       event),
                     text = ifelse(length(text[grepl("\\:", text)]) > 0,
                                   toString(text[grepl("\\:", text)]),
                                   ""))
  all_events <- dplyr::bind_rows(no_text_events, text_events)
  all_events <- dplyr::ungroup(all_events)
  all_events <- dplyr::arrange(all_events, date, y)
  all_events
}
all_events <- gather_events(all_events)

Before binding the events with text and no text I had checked by had that no events got lost (using dplyr::n_groups()).

head(all_events)

## # A tibble: 6 x 7
##   time    duration   timezone date       event          text              y
##   <chr>   <chr>      <chr>    <date>     <chr>          <chr>         <dbl>
## 1 8:00 AM (3 hr)     EST      2018-11-07 EXECUTIVE TIME Location: Ov…   140
## 2 11:00 … (30 min)   EST      2018-11-07 MEETING WITH … Location: Ov…   180
## 3 11:30 … (1 hr)     EST      2018-11-07 EXECUTIVE TIME Location: Ov…   221
## 4 12:30 … (1 hr)     EST      2018-11-07 LUNCH          Location: Pr…   261
## 5 1:30 PM (3 hr, 30… EST      2018-11-07 EXECUTIVE TIME Location: Ov…   302
## 6 RON:    ""         EST      2018-11-07 The White Hou… ""              342

head(all_events$text)

## [1] "Location: Oval Office"         "Location: Oval Office"        
## [3] "Location: Oval Office"         "Location: Private Dining Room"
## [5] "Location: Oval Office"         ""

Further work aimed at extracting “Location”, “Note”, etc. into specific columns. This code is especially ugly.

extract_info <- function(all_events){
  all_events %>%
  dplyr::group_by(time, duration, timezone, date, event, y) %>%
  dplyr::mutate(location = ifelse(any(grepl("Location", 
                                                        unlist(strsplit(text, ",")))),
                                  gsub("Location\\: ", "", 
                                       unlist(strsplit(text, ","))[grepl("Location", 
                                                        unlist(strsplit(text, ",")))]),
                                  NA),
                project_officer = ifelse(any(grepl("Project Officer", 
                                                        unlist(strsplit(text, ",")))),
                                  gsub("Project Officer\\: ", "", 
                                       unlist(strsplit(text, ","))[grepl("Project Officer", 
                                                        unlist(strsplit(text, ",")))]),
                                  NA),
                press = ifelse(any(grepl("Press", 
                                                        unlist(strsplit(text, ",")))),
                                  gsub("Press\\: ", "", 
                                       unlist(strsplit(text, ","))[grepl("Press", 
                                                        unlist(strsplit(text, ",")))]),
                                  NA)) %>%
  dplyr::ungroup() %>%
  dplyr::mutate(location = trimws(location),
                project_officer = trimws(project_officer),
                press = trimws(press))
}
  
all_events <- extract_info(all_events)

At this point, again, in a real work environment I’d spend time checking data carefuly, but instead, I’ll just have a rough look at the clean data vs my own cleaned data.

dplyr::count(clean_data, listed_project_officer, sort = TRUE)

## # A tibble: 39 x 2
##    listed_project_officer     n
##    <chr>                  <int>
##  1 <NA>                     369
##  2 Ambassador John Bolton    33
##  3 Sarah Sanders             25
##  4 Lindsay Reynolds          18
##  5 Shahira Knight            18
##  6 William McGinley          18
##  7 Christopher Liddell       13
##  8 John DeStefano            10
##  9 Mercedes Schlapp           9
## 10 Tim Pataki                 7
## # … with 29 more rows

dplyr::count(all_events, trimws(project_officer), sort = TRUE)

## # A tibble: 32 x 2
##    `trimws(project_officer)`     n
##    <chr>                     <int>
##  1 <NA>                        467
##  2 Ambassador John Bolton       32
##  3 Sarah Sanders                23
##  4 Shahira Knight               18
##  5 William McGinley             16
##  6 Lindsay Reynolds             15
##  7 Christopher Liddell          13
##  8 John DeStefano               10
##  9 Mercedes Schlapp              9
## 10 Tim Pataki                    7
## # … with 22 more rows

dplyr::count(clean_data, listed_location, sort = TRUE)

## # A tibble: 36 x 2
##    listed_location             n
##    <chr>                   <int>
##  1 Oval office               310
##  2 <NA>                       75
##  3 Oval Office                48
##  4 Private dining room        34
##  5 Buenos Aires, Argentina    21
##  6 oval office                 9
##  7 Grand foyer                 7
##  8 Roosevelt room              6
##  9 Cabinet Room                5
## 10 East Room                   5
## # … with 26 more rows

dplyr::count(all_events, trimws(location), sort = TRUE)

## # A tibble: 34 x 2
##    `trimws(location)`            n
##    <chr>                     <int>
##  1 Oval Office                 377
##  2 <NA>                        151
##  3 Private Dining Room          35
##  4 Buenos Aires                 18
##  5 Grand Foyer                   8
##  6 Cabinet Room                  7
##  7 Roosevelt Room                7
##  8 Biloxi                        4
##  9 Diplomatic Reception Room     4
## 10 McAllen                       4
## # … with 24 more rows

I surprisingly have more events in my data than in Axios’ clean data, which would require further exploration. Part of that might be due to my data listing “RON” as events, where RON means “Remainder of the Night” which is not an event, but a way to indicate where the President spent the rest of his time on that day. On one day, the whole schedule is actually “RON”.

dplyr::filter(all_events, date == lubridate::ymd("2018-11-23"))

## # A tibble: 1 x 10
##   time  duration timezone date       event text      y location
##   <chr> <chr>    <chr>    <date>     <chr> <chr> <dbl> <chr>   
## 1 RON:  ""       EST      2018-11-23 Mar-… ""      180 <NA>    
## # … with 2 more variables: project_officer <chr>, press <chr>

For some reasons, in its clean data Axios classified that day as executive time, although the PDF does not indicate anything about it.

dplyr::filter(clean_data, date == lubridate::ymd("2018-11-23"))

## # A tibble: 1 x 11
##    week date       time_start time_end duration listed_title top_category
##   <dbl> <date>     <time>     <time>      <dbl> <chr>        <chr>       
## 1     3 2018-11-23 08:00      17:00           9 Executive t… executive_t…
## # … with 4 more variables: listed_location <chr>,
## #   listed_project_officer <chr>, detail_category <chr>, notes <chr>

I then had a go at cleaning times and durations.

clean_time_info <- function(all_events){
  all_events %>%
  dplyr::mutate(start = lubridate::parse_date_time(paste(date, time),
                                                   orders = "Ymd HM p", 
                                                   locale = "en_US.utf8", tz = "EST")) %>%
  dplyr::mutate(duration = gsub("\\(", "", duration),
                            duration = gsub("\\)", "", duration)) %>%
  dplyr::mutate(hours = ifelse(grepl("hr", duration),
                               as.numeric(trimws(gsub("hr.*$", "", duration))),
                               0),
                minutes = ifelse(grepl("min", duration),
                               as.numeric(trimws(stringr::str_remove(
                                 gsub("^.*\\,", "", duration),
                                 "min"))),
                               0)) %>%
  dplyr::mutate(duration = 60*hours + minutes) %>%
  dplyr::select(- hours, - minutes) %>%
  dplyr::group_by(date) %>%
  dplyr::mutate(end = ifelse(is.na(duration),
                             as.POSIXct(dplyr::lead(start),
                                        origin = "1970-01-01"),
                             as.POSIXct(start + lubridate::minutes(duration),
                                        origin = "1970-01-01")))
}
  
all_events <- clean_time_info(all_events)
head(all_events)

## # A tibble: 6 x 12
## # Groups:   date [1]
##   time  duration timezone date       event text      y location
##   <chr>    <dbl> <chr>    <date>     <chr> <chr> <dbl> <chr>   
## 1 8:00…      180 EST      2018-11-07 EXEC… Loca…   140 Oval Of…
## 2 11:0…       30 EST      2018-11-07 MEET… Loca…   180 Oval Of…
## 3 11:3…       60 EST      2018-11-07 EXEC… Loca…   221 Oval Of…
## 4 12:3…       60 EST      2018-11-07 LUNCH Loca…   261 Private…
## 5 1:30…      210 EST      2018-11-07 EXEC… Loca…   302 Oval Of…
## 6 RON:         0 EST      2018-11-07 The … ""      342 <NA>    
## # … with 4 more variables: project_officer <chr>, press <chr>,
## #   start <dttm>, end <dbl>

I stopped there, not willing to further work on the data. More work on timezones, and on filling missing durations based on context, would be useful. I’d be delighted to see someone cleaning the dataset more thoroughly than I did here!

Conclusion

In this post I took advantage of the nifty pdftools::pdf_data() function to extract text from Trump’s PDF schedule along with coordinates, in the hope to be able to rectangle the data. Part of that went well in particular thanks to fuzzyjoin and Tidyverse tooling, part of that went less well, in particular due to the inexact data formatting and less than optimal indication of timezones… and my not desiring to spend my week cleaning the data. I have no idea how Axios prepared the dataset, but if I were to clean such data for actual work, after scripting as much as possible I’d request at least one colleague to review the code, and this person and I would also sample a few if not all events and compare their information in the PDF and the resulting data.

The advantage of scripting at least part of the cleaning is reproducibility, for being able to explain what happened to the data, but also to apply the code to new data. This morning I read that more schedules had been leaked so I tried my code on it.

pdf_file2 <- "data/newschedules.pdf"
pdf_data2 <- pdftools::pdf_data(pdf_file2)

pdf_text2 <- purrr::map(1:pdftools::pdf_info(pdf_file2)$pages,
                       read_page,
                       pdf_data = pdf_data2) %>%
  purrr::discard(is.null)

dates2 <- purrr::map_df(pdf_text2, get_date)

all_events2  <- pdf_text2 %>%
  split_days(dates2) %>%
  purrr::map_df(get_times) %>%
  gather_events() %>%
  extract_info() %>%
  clean_time_info()

head(all_events2)

## # A tibble: 6 x 12
## # Groups:   date [1]
##   time  duration timezone date       event text      y location
##   <chr>    <dbl> <chr>    <date>     <chr> <chr> <dbl> <chr>   
## 1 8:00…      180 EST      2019-02-04 EXEC… Loca…   141 Oval Of…
## 2 11:0…       30 EST      2019-02-04 MEET… Loca…   180 Oval Of…
## 3 11:3…       15 EST      2019-02-04 EXEC… Loca…   219 Oval Of…
## 4 11:4…       30 EST      2019-02-04 INTE… Proj…   258 Oval Of…
## 5 12:1…       30 EST      2019-02-04 EXEC… Loca…   310 Oval Of…
## 6 12:4…       60 EST      2019-02-04 LUNC… Proj…   349 Private…
## # … with 4 more variables: project_officer <chr>, press <chr>,
## #   start <dttm>, end <dbl>

I think that the fact that my (imperfect!) pipeline seems to work on the new schedules shows the power of scripting one’s data cleaning… as well as the hard work Axios’ journalists did! If you work on the PDF data, I’d be glad to hear how you handled it, so share links to gists and posts in the comments below! If you prefer playing with the data, check out this cool Shiny app by Garrick Aden-Buie.

Keeping up to date with R news

Fri, 25 Jan 2019 00:00:00 +0000

I’ve now given my talk about “How to be a resilient R user” three times, at R-Ladies Strasbourg and R-Ladies Paris in person, and at R-Ladies San José via Google Hangouts. It was fun! I covered part of the content of that talk in a blog post about where to get R help. Today, it’s time for a post full of my personal opinions! I’ll cover the rest of the talk: why and how stay up to date with R news?

Why it’s important to stay informed

Following the news is a big part of being a resilient R user, because it helps you solve the problems you didn’t know you had (maybe you’ll see a package that can replace a cumbersome script of yours!) and the problems you will have (one day, you’ll need to make a map, and be thankful you already vaguely know sf exists). Obviously, staying aware helps because you learn about tools and their applications. But it also helps because it makes you learn about people and organizations! It is highly valuable to get a sense of who’s working on what and of where the developments on a topic happen, because when you consciously look for something later, it’ll help you find your way more easily.

Where to get your R news fix

Here is a probably non exhaustive list of where to get R news. Pick the ones that fit your interests and workflow better, and do not hesitate to tell me of other information sources in the comments!

Infinite information flows: Twitter, GitHub and dev.to

Both Twitter and GitHub offer some sort of timeline, as does dev.to.

On Twitter, there’s the timeline made of content (re-)tweeted (or liked, thanks to that new random aspect of Twitter…) by accounts you follow. You choose your own adventure by selecting which accounts to follow! There’s also the #rstats feed, either recent tweets by popularity or all tweets by publication time. I like browsing the latter to see interesting questions, news or applications of packages, R-Ladies boasting, etc. Keys to a good experience are, in my opinion
- Muting the spam and accounts you do not like. In particular, keep the habit to mute the big data blabla companies that start polluting the feed, you’ll see gems more easily.
- Not being a completist. I’ve been told it’s stressful to see the flow never stops. Well, make peace with that, you won’t read everything and you’ll be fine.
- Practice! I know I’ve gotten more deliberate, and much faster at browsing the content. Interestingly it probably means that content producers need to be wise in their choice of emojis, illustrations etc. to catch people’s attention, but that’s not today’s topic.
On GitHub, I only use the timeline fed by accounts I follow and repos I watch: repos created or forked, new releases, people following each other (kind of cute when you see long time R friends following each other!), repos starred. It can help discovering new code, and new people. By the way, adding an avatar and minimal information to your GitHub account is crucial in general, and useful for the few seconds of screen time you get on someone’s timeline after being followed by someone they follow: if your account is empty, nothing is read, otherwise, people will see “Jane Doe started following Ada Lovelace, computer scientist” which is more informative than “Jane Doe started following alvlc”.

Note that there’s no ad on GitHub timelines… yet.

When reading my GitHub timeline and seeing what I think are people experimenting with #rstats usethis/devtools, I wonder whether the default description could be sold as ad space 😉 pic.twitter.com/eUVkUqdZyX
— Maëlle Salmon 🐟 (@ma_salmon) 9 de març de 2018

Thanks to rOpenSci’s excellent newsletter maintained by Scott Chamberlain, I’ve learnt about dev.to. Scott wrote “A shout out to a probably lesser know place to share and gain R knowledge: If you haven’t heard of dev.to, it’s a sort of developer forum for sharing articles and discussing. They have an R tag https://dev.to/t/r and an rstats tag https://dev.to/t/rstats - the former has more traffic." I have not started reading it myself, but it’s good to know!

News aggregators

This is a category I made up, where I’m putting these information sources that are less overwhelming than Twitter because already more organized and curated.

R mailing lists could be your thing, be it one of the field specific ones, or the more general R-announce.
R Weekly is a great newsletter organized in categories making it easier to browse it and select what picks your interest. The very first section, highlights, is made of a few links on which R Weekly team voted. You can subscribe by RSS feed, email, or follow the Twitter account. R Weekly content stems from RSS feeds R Weekly subscribes to, and pull requests. Note that if you want to know how to get your content on folks’ radar, you should check out this post of mine about blog promotion.
rOpenSci biweekly newsletter is as written earlier excellent. It includes news about rOpenSci packages, their application in scientific articles and in blog posts, reminders of recent content on rOpenSci blog, and since this week related news (that’s how I read about dev.to), as well as call for maintainers.
R-Bloggers aggregates content from R blogs, re-rendering the whole content on their website. It can be a way to discover interesting blogs.
You could create your own news aggregators made up of RSS (or JSON?) feeds of blogs and newsletters you enjoy. I don’t have a routine for that, but have heard of Feedly and co.

R conferences and meetups

R conferences (satRdays, useR!, RStudio::conf, LatinR, ConectaR, etc.) and meetups (R-Ladies meetups, R User groups) will allow you to stay aware of new developments as well as to make R friends. How to know which ones are taking place?

R Weekly has a section about upcoming events.
There’s a weekly post about events on RStudio community forum.
Jumping Rivers maintains a list of R conferences and local user groups.

Now, attending events is obviously fantastic, but you can also make the most of what’s online!

Conference programs can give you a good insight into what’s new, trendy, and interesting to catch up on. Especially topics you see in several programs.
Slidedecks are often made available and links aggregated on the conference website or by volunteers (here’s a links collection for RStudio::conf 2019 maintained by Karl Broman).
If you’re lucky you’ll even find a recording for the talk you’re interested in! See Rstudio::conf 2019 videos, SatRday Cape Town 2018 recordings, useR! 2018 videos. A cheap way to get information!

At this point, I’d like to add a short side note about R-Ladies abstract peer-review system organized by Jennifer Thompson! The idea is to encourage R-Ladies to speak at R conferences by giving them feedback on their abstracts. R-Ladies, submit your abstract. R-Ladies and allies, volunteer as abstract reviewer.

Other sources of information

Maybe you’re into podcasts? If so, check out e.g. Not So Standard Deviations, Credibly Curious and The R Podcast. The latest R Podcast episode is a crossover between the three since R Podcast’s host Eric Nantz is joined by Credibly Curious co-host Nick Tierny and Not So Standard Deviations co-host Hilary Parker! Jumping Rivers maintains a list of data science podcasts.

I like this monthly post by RStudio Joseph Rickert, selecting 40 packages among all new CRAN releases, organized in sections which makes it easy to digest.

What to do with the information

Now that you know how to get a ton of info, what should you do with it?

Assess it!

Assess the information you get much like you would assess any piece of news, and maybe ask your R friends what they think. If hesitating to try out a new package, always perform a mini assessment to estimate whether it’s worth your time and trust. Assessing a package is not the topic of this post, but have a look at e.g. criteria used in this rOpenSci unconf project, Jeff Leek’s post about how he decides to trust an R package, rOpenSci packaging guidelines.

Store it somehow

Maybe you don’t need any system because you have a good memory, maybe you do need one. I’m not an organization coach so will only throw a few links/ideas. Figuring out what works best for you is very important, it’ll make this time investment reading news worth it.

Joseph Stachelek wrote about how to make a dashboard out of his Twitter favorites.
You can search your GitHub starred repos so starring a repo is a bit like bookmarking it.
Suzan Baert maintains a public treasure trove of interesting links, for herself and anyone interested.
Lise Vaudor wrote a blog post in French about se souvenir de tout, including a pic of the cheatsheets near her desk.

Maybe needless to say, if you like something you read or use, go forth and post it in your networks, to help feed the information flows. :-)

Conclusion

In this post inspired by my recent R-Ladies talks, I summarized my views on why, where and how to stay aware of news about R. I had included the topic in my talks because I’ve been asked “But how do you know about this?!” and answered “Oh, I simply read about it”. After a few similar conversations, I realized that maybe it’d be useful to share my personal experience. I hope you find your own optimal routine, discover awesome stuff… and remember about it when needed!

Your and my 2019 R goals

Tue, 01 Jan 2019 00:00:00 +0000

Here we go again, using a Twitter trend as blog fodder! Colin Fay launched an inspiring movement by sharing his R goals of 2019.

My #RStats goals for 2019:

1️⃣ Becoming entirely fluent with {data.table}

2️⃣ Getting at ease with {Rcpp}

What are yours?#rdatatable #rcpp
— Colin 🤘🌱🏃‍♀️ (@_ColinFay) December 29, 2018

It’s been quite interesting reading the objectives of other tweeps: what they want to learn, make, how they want to get involved in the community, etc. As Mike Kearney, rtweet’s maintainer, underlined, it is excellent reading material!

… but also blogging material! Let me fetch and tokenize these tweets to summarize them!

Disclaimer: I later saw that Jason Baik got the same idea and was faster than I, find the analysis here.

Collect Twitter data

If you’re using rtweet for the first time, check out its website for information about use and setup and also refer to Twitter API docs themselves for knowing more about rate limitation and e.g. learning that the search endpoint won’t let you get tweets older than 6-9 days old.

tweets <- rtweet::search_tweets("Rstats goals 2019",
                                include_rts = FALSE)

I obtained 87 tweets from 85 unique users. Definitely not big data, but not bad!

Tokenize tweets

I then set out to tokenize the tweets into words using the specific tokenizers::tokenize_tweets() tokenizer via the tidytext package. If you’re new to tidytext I’d recommend reading the book written by its authors. A token in natural language processing can be a word, line, etc. which is a totally different concept from a token for rtweet functions (your API credentials).

The tweet tokenization is a “tokenization by word that preserves usernames, hashtags, and URLS”. So awesome, and today is the first time I find an occasion to use it! I also removed stopwords.

library("magrittr")

stopwords <- rcorpora::corpora("words/stopwords/en")$stopWords

tokens <- tweets %>%
  dplyr::select(text) %>%
  tidytext::unnest_tokens(token, text,
                        token = "tweets",
                        drop = FALSE) %>%
  dplyr::filter(!token %in% stopwords)

Analyze tweets

Most mentioned topics

I first was able to draw a figure similar to Jason Baik’s one, with the most common tokens. I too removed digits.

library("ggalt")
tokens %>%
  dplyr::mutate(token = stringr::str_remove_all(token, "[^\x01-\x7F]")) %>%
  dplyr::mutate(token = stringr::str_remove_all(token, "[[:digit:]]")) %>%
  dplyr::filter(! token %in% c("", "#rstats", "goals")) %>%
  dplyr::count(token, sort = TRUE) %>%
  dplyr::mutate(token = reorder(token, n)) %>%
  head(n = 18) %>%
  ggplot()  +
  geom_lollipop(aes(token, n),
                size = 1.5, col = "salmon") +
  hrbrthemes::theme_ipsum(base_size = 12,
                          axis_title_size = 12) +
  coord_flip()

What actions?

In this figure I identify verbs like learn, finish, write, build and contribute. Let me look at a sample of lines for each of them. This is a sample of lines for a small sample of verbs.

lines <- tweets %>%
  dplyr::select(text) %>%
  tidytext::unnest_tokens(line, text,
                        token = "lines")

sample_verb <- function(verb, lines){
  set.seed(42)
  dplyr::filter(lines, stringr::str_detect(line, paste0(verb, " "))) %>%
    dplyr::sample_n(3)
}

samples <- purrr::map_df(c("learn", "finish", "write", "build", "contribute"), sample_verb, lines)

knitr::kable(samples)

line
3️⃣ learn how to make r packages and write my code so it could be made into an r package more easily
1. learn how to do spatial analysis in r
2️⃣ learn better way to automate feature engineering (neural nets) for text
1️⃣ finally finish all the courses and certifications i started last year on #coursera and #datacamp
2️⃣ finish my track and field r package
3). finish that text mining project i started in october
4️⃣ write an advanced shiny book with bookdown 🎈
3️⃣ learn how to make r packages and write my code so it could be made into an r package more easily
1️⃣ write the htmlwidgets book
- build my first #rstats package (aiming for 2 but 1 would be great :d)
2⃣ build a shiny web app to explore tx staar data
- use f(x) regularly & build own package. cease patching.
5⃣ contribute to foss https://t.co/oh7mwcq50r
2 contribute more to #rstats community through #scicomm, #stackoverflow, etc
3) contribute to #swdchallenge (with r, duh)

These actions are quite varied, e.g. writing is applied to software as well as reading material. My goal was to summarize tweets, but I keep thinking reading all of them is interesting!

Packages?

I wondered how many of the tokens correspond to a package name. I limited myself to CRAN packages, by using the available.packages() function, but one could have a look at the source code of the available package to get an idea of how to find names of packages from Bioconductor and GitHub.

cran_pkgs <- as.character(
  available.packages(contrib.url('https://cran.r-project.org', 'source'))[,"Package"])
pkg_tokens <- dplyr::mutate(tokens,
                            token = gsub("#", "", token)) %>%
  dplyr::filter(token %in% cran_pkgs)

Using the data I’ll look at tweets with the most packages, and most frequent packages.

pkg_tokens %>%
  dplyr::group_by(text) %>%
  dplyr::mutate(pkg_text = paste(toString(token), text)) %>%
  dplyr::count(pkg_text, sort = TRUE) %>%
  head(n = 3) %>%
  dplyr::pull(pkg_text)

## [1] "portfolio, blogdown, rmarkdown, knitr, shiny, maps I really like seeing all these #rstats 2019 goals. My own, in order of urgency:\n1) Finish my personal website and online portfolio using blogdown\n2) Get rolling with project workflows, rmarkdown, and knitr \n3) Create shiny apps for custom interactive maps"                                           
## [2] "inference, projects, import, rvest, httr, xml2 My #rstats 2019 goals:\n1. Improve my statistical modeling and inference skills\n2. Develop business literacy and apply it in data analysis projects\n3. Continue to post on my blog (1 post every 2 months)\n4. Learn to import data using DBI, rvest, httr, and xml2"                                           
## [3] "shiny, templates, shiny, shiny, bookdown, shiny My #RStats goals for 2019: \n\n1<U+FE0F><U+20E3> Improve shinydashboardPlus, bs4Dash and argonDash ..<U+0001F973> \n\n2<U+FE0F><U+20E3> Release new shiny templates              \n3<U+FE0F><U+20E3> Open a consulting service for https://t.co/k3PAbxyVMa about shiny \n4<U+FE0F><U+20E3> Write an advanced shiny book with bookdown <U+0001F388>\n#rstats #shiny #consulting https://t.co/Fyc7MhaeW8"

There are false positives, e.g. projects was here meant as a word, not a package name. What about the most popular packages among the tweets?

dplyr::count(pkg_tokens, token, sort = TRUE)

## # A tibble: 66 x 2
##    token         n
##    <chr>     <int>
##  1 shiny        16
##  2 blogdown      9
##  3 projects      9
##  4 rmarkdown     9
##  5 tidyverse     6
##  6 bookdown      5
##  7 purrr         4
##  8 track         4
##  9 caret         3
## 10 markdown      3
## # ... with 56 more rows

In this table, we get a glimpse at current popular packages, apart from “projects”, “track” and “markdown”. If I’m reading the list correctly they’re all developed at RStudio!

Conclusion

In this post I followed an approach similar to Jason Baik’s to summarize tweets about 2019 R goals announced on Twitter: I collected tweets with rtweet and then used tidytext and the tidyverse to summarize them. Goals often included learning about stuff, building packages (find my list of resources and don’t miss this offer by Steph de Silva), and mentions of RStudio packages.

What about my own R goals, that I haven’t tweeted? I have not made any list, but have exciting projects at work, and hope to keep semi-consistently posting on this blog. In January I’ll also get to start 2019 by giving two R talks, one at R-Ladies Paris and a remote one at ConectaR 2019! Happy 2019, I hope you can meet your own R goals!

My #Best9of2018 tweets

Fri, 28 Dec 2018 00:00:00 +0000

As 2018 nears its end, it’s time for me to look back on my R/Twitter year with the same simple method as last year: let me identify and webshoot my 9 best tweets of 2018!

Downloading and opening my Twitter data

Like in 2017 I tweeted too much and therefore was unable to rely on rtweet::get_timeline() (or rtweet::get_my_timeline()) to download my tweets so I exported data via the Tweets tab of https://analytics.twitter.com/. Last year, I downloaded one file per quarter but somehow had to download one per month this time. It was monotonous but not horrible.

library("magrittr")

dir("data", full.names = TRUE) %>%
  purrr::map_df(readr::read_csv) %>%
  unique() -> my_tweets

4196 tweets!

Getting the top 9 tweets

Similarly to my 2017 post, I chose the number of likes as criterion to define my best tweets.

my_tweets <- dplyr::arrange(my_tweets, - likes)
my_tweets <- janitor::clean_names(my_tweets)

knitr::kable(my_tweets[1:9, "likes"])

likes
162
151
128
112
95
94
91
86
75

best9 <- my_tweets$tweet_permalink[1:9]

The table above shows that my best tweets were not extremely popular.

Webshooting and combining the tweets

My 2017 blog post inspired Bob Rudis to contribute a tweet_shot() function to rtweet, relying on the webshot package, so that’s what I used.

shot <- function(permalink){
  rtweet::tweet_shot(permalink) %>%
    magick::image_crop(magick::geometry_area(width = 517,
                                             height = 517,
                       y_off = 88)) %>%
    magick::image_border("salmon", "20x20")
}

images <- purrr::map(as.character(best9),
                     shot)

I improved the collage code with two tweaks:

I created rows then columns instead of the other way round, because that’s what Instagram does. I did not pay attention to this last year but Andrew Caines told me this in a comment.
I did not save the intermediary images to disk. I’ve upped my magick game!

make_row <- function(i, images){
  images[((i-1)*3+1):((i-1)*3+3)] %>%
    magick::image_join() %>%
    magick::image_append()
}

purrr::map(1:3, make_row,
           images = images) %>%
  magick::image_join() %>%
  magick::image_append(stack = TRUE) %>%
  magick::image_border("salmon", "20x20")

Worth noting from my R year are, according to these tweets,

my re-designing my website;
the slides of my R-Ladies Strasbourg talk about how to be a resilient R user;
my starting work at rOpenSci and Locke Data (yay!);
my blog post about rainbowing a set of pictures;
another blog post featuring magick, about combining hexes but I’d recommend reading this post by Mitchell O’Hara Wild instead;
my blog post about where to get R help that might be the personal blog post I’m the proudest of.

What about your R year, on Twitter and elsewhere? Have a good last days of 2018, and a happy 2019!

Advent of Code: Most Popular Languages

Sat, 15 Dec 2018 00:00:00 +0000

You might have heard of the Advent of Code, a 25-day challenge involving a programming puzzle a day, to be solved with the language of your choice. I’ve noted the popularity of this activity in my Twitter timeline but also in my GitHub timeline where I’ve seen the creation of a few advent-of-code or so repositories.

AoC is largely an exercise in figuring how to write your favourite language as if were C or C++ 😁, which can be fun ... in moderation
— Jenny Bryan (@JennyBryan) December 12, 2018

If I were to participate one year, I’d probably use R. Jenny Bryan’s tweet above inspired me to try and gauge the popularity of languages used in the Advent of Code. To do that, in this post, I shall use the search endpoint of GitHub V3 API to identify Advent of Code 2018 repos.

Searching repositories on GitHub

Study design ;-)

GitHub’s V3 API offers a search endpoint, that however gives you less results than doing the same search via the web interface, even when using pagination right (or at least, when believing I use pagination right!). I’m however willing to use that sub-sample as basis for my study of language popularity. It’s actually a sub-sub-sample, since I’m only looking at Advent of code projects published on GitHub.

In order to circumvent the sub-sub-sampling a bit, I’ll do the search in two steps:

Searching for Advent of code 2018 in general among repos, and extracting the language of the repos.
Searching for Advent of code 2018 for each of these languages separately and extracting the total count of hits.

Note that I am not filtering the repos by activity, so some of them could very well have been created for a few days only. If they are empty though, they do not get assigned a language.

Regarding the language of repos, GitHub assigns a language to each repository. This information can be wrong, which is e.g. mentioned in rOpenSci’s development guide. Furthermore, my using this piece of information means I’m disregarding the fact that some people actually use a mix of technologies to solve the puzzles.

Actual queries

I first defined a function to search the API whilst respecting the rate limiting. I even erred on the side of caution and queried very slowly.

.search <- function(page){
  gh::gh("GET /search/repositories",
         q = "adventofcode 2018",
         page = page,
         fork = FALSE)
}

search <- ratelimitr::limit_rate(.search,
                                ratelimitr::rate(10, 60))

I then wrote two other functions to help me rectangle the API output for each repository.

empty_null <- function(x){
  if(is.null(x)){
    ""
  }else{
    x
  }
}

rectangle <- function(item){
  tibble::tibble(full_name = item$full_name,
                 language = empty_null(item$language))
}

I created a function putting these two pieces together.

get_page <- function(page){
  results <- try(search(page), silent = TRUE)

  # an early return
  if(inherits(results, "try-error")){
    return(NULL)
  }

  purrr::map_df(results$items,
                rectangle)
}

And I then ran the following pipeline.

total_count <- search(1)$total_count
pages <- 1:(ceiling(total_count/100))

results <- purrr::map_df(pages, get_page)
results <- unique(results)

languages <- unique(results$language)
languages <- languages[languages != ""]

This got me 814 repos, with 46 non empty languages. Repo names are quite varied: rdmueller/aoc-2018, petertseng/adventofcode-rb-2018, NiXXeD/adventofcode, Arxcis/adventofcode2018, Stupremee/adventofcode-2018, phaazon/advent-of-code-2k18.

With that information obtained, I was able to run a query by language.

.get_one_language_count <- function(language){
  gh::gh("GET /search/repositories",
         q = glue::glue("adventofcode 2018&language:{language}"),
         fork = FALSE)$total_count -> count
  tibble::tibble(language = language,
                 count = count)
}

get_one_language_count <- ratelimitr::limit_rate(.get_one_language_count,
                                 ratelimitr::rate(10, 60))

counts <- purrr::map_df(languages,
                        get_one_language_count)

In total, the counts table contains information about 2080 repositories, a bit less than half the number of Advent of code 2018 repositories I’d find via the web interface.

Advent of Code’s languages popularity

I’ll concentrate on the 15 most popular languages in the sample, which automatically excludes R with… 8 repositories only.

library("ggplot2")
library("ggalt")
library("hrbrthemes")
library("magrittr")

counts %>%
  dplyr::arrange(- count) %>%
  head(n = 15) %>%
  dplyr::mutate(language = reorder(language, count))   %>%
  ggplot() +
  geom_lollipop(aes(language, count),
                size = 2, col = "salmon") +
  hrbrthemes::theme_ipsum(base_size = 16,
                          axis_title_size = 16) +
  coord_flip() +
  ggtitle("Advent of Code Languages",
          subtitle = "Among a sample of GitHub repositories, with language information from GitHub linguist")

The results are not surprising after reading e.g. the insights from Stack Overflow’s 2018 survey, although the Python domination is crazy! I find interesting to reflect on the fact that Jenny Bryan says that the challenge is best for C or C++, that are not the most popular languages in these samples… but still more popular than R, ok.

Conclusion

In this post I used GitHub V3 API to get a glimpse at the popularity of languages used to solve the Advent of Code. Further work could include looking at the completion of the challenge by language, potentially using the GitHub activity of each repo as an (imperfect) proxy.

I do not take part in the challenge myself, my principal Advent’s specific activity being instead the lazy and delightful watching of the Swedish TV channel SVT Adventskalender for kids. Incidentally, this year’s storyline includes a Christmas competition, which however features competitive eating of saffron buns and gingerhouse building rather than programming puzzles… Do you participate to Advent of Code this year? If so, with which language and why?

Make a trailer for your slidedeck with av

Sun, 07 Oct 2018 00:00:00 +0000

rOpenSci post-doc hacker Jeroen Ooms has just released a cool new package, av, that he wrote “will become the video counterpart of the magick package which [rOpenSci uses] for working with images.". av provides bindings to the FFmepg libraries for editing videos. It’s already become a renderer for gganimate by Thomas Lin Pedersen, but av allows more than making videos of graphics. In this post, I’ll show how to use av and webshot to make a trailer/sneak preview of a slidedeck, i.e. a short video featuring the first few slides on music!

Why make a trailer of your slidedeck?

No, making a trailer of a slidedeck isn’t only a pretext for trying out av! In case you hadn’t noticed, I’m all about promoting your work once you’ve put in the work. I’ve even written a whole post about blog promotion. Promoting a slidedeck after a talk, or a talk before the event, is quite similar to promoting a blog post both in terms of benefits and technically. In particular, a tweet about it will be more powerful if featuring an illustration, and why not making it animated?

For more tips about advertising for a thing via a talk, check out the excellent “Carpe Talk” article on tidyverse.org by Jenny Bryan and Mara Averick.

Capture the first few slides

In this post, I’ll use the slidedeck that Jeroen shared after giving a talk about R infrastructure after the uRos conference. I’ll capture the first few slides minus the ones where he introduced himself, because I wanted the focus of the trailer to be on the talk topic. I didnd’t know webshot::webshot() was vectorized, thanks to Jeroen for telling me that which shortened the code below.

fs::dir_create("slides")

urls <- sprintf("https://jeroen.github.io/uros2018/#%d",
                c(1:2, 5:10))
webshot::webshot(urls, 'slides/webshot.png')

At that point, I had a folder with 8 PNGs. If your talk isn’t html, but say Power Point, you could export to PDF, and use the rOpenSci pdftools package to render slides to images. Note that the PNGs I got didn’t have the right font, which is something I could solve by using decktape instead of webshot, to render the html slides to PDF and then to PNG with pdftools.

Create the video

The video creation was extremely simple and fast once I had selected a soundtrack. Note that having a soundtrack is optional!

Jeroen’s talk was about building stuff so I looked for construction sounds and selected this sample published by Audio Hero on Zapsplat

slides <- fs::dir_ls("slides")
slides <- c(slides[1], slides, slides[length(slides)])

av::av_encode_video(slides,
                    # it could be a fraction!
                    framerate = 1,
                    # if too short there'll be silence 
                    # at the end
                    # if too long it'll be truncated
                    audio = audio,
                    output = "trailer.mp4")

Trailer of Jeroen's slidedeck. Your browser does not support the video tag.

Tada! Thanks to Audio Hero for the sound sample. Now, if you want to read more of that cool slidedeck, you’ll find it here.

Conclusion

In this post, I was able to write R code to take screenshots of the first few slides of a slidedeck, and to combine them in a video with sounds related to the slidedeck topic. That new av package is quite promising!

Regarding accessibility of the produced trailer, on Twitter you can’t add an alternative description to a video, so it might be nice to either replace the music with a voice over, or to tweet once with music/sound and once with voice over? I guess the tweet itself can also help counterbalance the information lost by being informative and featuring a link to the actual slidedeck. If you’re as much into listening to your own voice as I am into listening to mine (i.e. not at all!), you might want to try rOpenSci’s googleLanguageR package that helps use Google text to speech API.

Finally, if you use a cool Xaringan theme like Emi Tanaka’s ninja theme, you’d a bit disappointed to create a preview that loses the nice animated transitions. Luckily, in his tech note Jeroen promised “In future versions we also want to add things like screen capturing and reading raw video frames and audio samples for analysis in R.” so one could depend on screen capturing rather than screen shooting. I’m looking forward to future versions of av, and encourage you to give it a try!

Spookify: Halloween Name Generation in R

Fri, 05 Oct 2018 00:00:00 +0000

It’s October, time for spooky Twitter names! If you’re on this social media platform, you might have noticed some of your friends switching their names to something spooky and punny. Last year I was “Maelstrom Salmon”, which I find scary but is arguably not that funny. Anyhow, what if you want to switch your name but have no inspiration? In this post, we shall explore R’s abilities to help us with that with the help of webscraping, phonetic spelling and string distance algorithms, and the magic of randomness!

My strategy for spooky name generation will be to replace each part of a name (e.g. first and last names) with the phonetically closest Halloween-related words. Therefore, the technical challenges here were to find a list of Halloween-related words, and to measure phonetical string distance.

Getting spooky words

The otherwise very useful package rcorpora doesn’t have a “spooky” category, so I decided to scrape this webpage of Halloween words for kids. It was one of the first search results, and being for kids it couldn’t be offensive. Like the last two times I webscraped, I used the very cool polite package!

There were two less useful sections on the page, with French and Spanish words, that I very inelegantly removed after looking up their div ID in the page source.

# introduce myself
session <- polite::bow("https://holidappy.com/holidays/halloween-words-and-halloween-vocabulary-lists",
                       user_agent = "Maëlle Salmon https://masalmon.eu/")

# scrape
page <- polite::scrape(session)

## No encoding supplied: defaulting to UTF-8.

# remove the two sections I don't want
French <- xml2::xml_find_all(page,
                            xpath = "//div[@id='mod_37589248']")

xml2::xml_remove(French)

Spanish <- xml2::xml_find_all(page,
                              xpath = "//div[@id='mod_37589143']")

xml2::xml_remove(Spanish)

# extract the words
words <- xml2::xml_find_all(page,
                            xpath = "//td[contains(@class, 'tableCell')]")

words <- xml2::xml_text(words)
head(words)

## [1] "All Hallows' Eve\n\t" "lantern\n\t"          "prank\n\t"           
## [4] "costume\n\t"          "make-believe\n\t"     "sweets\n\t"

I cleaned the words a bit. Some of them were not words so I split them. I hesitated between this solution and keeping only one-word phrases… I’m still not sure it was the best decision!

words <- stringr::str_remove_all(words, "\\\n\\\t")
words <- trimws(words)
words <- tokenizers::tokenize_words(words)
words <- unlist(words[lengths(words) > 0])
words <- words[!stringr::str_detect(words, "[0-9]")]
head(words)

## [1] "all"     "hallows" "eve"     "lantern" "prank"   "costume"

I obtained 258 words.

Matching names and spooky words

Once I had this vector of Halloween-related words, all I needed was a way to compute a phonetical distance between each of them and a name. It turned out more complicated than I thought! The soundex method of the stringdist package returned 1 or 0 only, so I set out to search another package supporting phonetical comparison. I found the phonics package that has a paper in JOSS, that implements more algorithms translating strings to phonetic “codes”. I ended up using the phonics::nysiis() function, corresponding to the New York State Identification and Intelligence System phonetic algorithm. That package also has an algorithm suitable for German, via phonics::cologne(), and one for French, via phonics::statcan(), but I haven’t explored that further.

phonics::nysiis("pain")

## [1] "PAN"

phonics::nysiis("pane")

## [1] "PAN"

phonics::nysiis("train")

## [1] "TRAN"

I was at a loss as to how best compare output from phonics::nysiis() and very hackily used… stringdist::stringdist() with a default method! Below is the function finding the closest match for any name part.

spookify_word <- function(word, seed = 42, words){

  phon <- phonics::nysiis(words)
  dist <- stringdist::stringdist(phonics::nysiis(word),
                                 phon)
  set.seed(seed)
  sample(words[dist == min(dist)], 1)
}
spookify_word("Jane", words = words)

## [1] "bones"

So spookify_word() samples a word among the closest matches… that aren’t necessarily very close to the original name. But that’s fine!

Mixing it all in a chauldron function

I finally created a function taking a whole name, say “Maëlle Salmon” or “Ada Colau Ballano”, tokenizes it into words thanks to the tokenizers package, gets the closest match for each part and then collapses everything together, capitalizing first letters thanks to snakecase… before announcing the result with the cool cowsay package! I made sure to only use animals that are associated with Halloween.

spookify <- function(name, seed,
                     words){
  name_parts <- unlist(tokenizers::tokenize_words(name))
  spook_names <- purrr::map_chr(name_parts,
                                spookify_word,
                                seed, words)
  spook_name <- paste(spook_names, collapse = " ")
  spook_name <- snakecase::to_upper_camel_case(spook_name,
                                               sep_out =  " ")
  
  set.seed(seed)
  animal <- sample(c("owl", "spider",
                     "pumpkin", "ghost",
                     "bat"), 1)

  cowsay::say(what = paste0("Byebye ", name, "\n",
                           "You are now ", spook_name),
              by = animal)

}

And because I didn’t want to make fun of anyone else than me, I created 7 fake names using the charlatan package and ran spookify on them!

set.seed(42)
names <- charlatan::ch_name(n = 7)
purrr::walk2(names, 1:length(names),
             spookify, words = words)

## Colors cannot be applied in this environment :( Try using a terminal or RStudio.

## 
## 
##  ----- 
## Byebye Tyrik Graham PhD
## You are now Trick Grinning Giant 
##  ------ 
##     \   
##      \
##               |
##               |
##               |
##              __
##           | /  \ |
##          \_\\  //_/
##           .'/()\'.
##            \\  //  [nosig]
## 

## Colors cannot be applied in this environment :( Try using a terminal or RStudio.

## 
##  ----- 
## Byebye Jeremy Rippin
## You are now Horrify Rotten 
##  ------ 
##     \   
##      \  
##       \
##        /\___/\
##        {o}{o}|
##        \ v  /|
##        |    \ \
##         \___/_/       [ab] 
##           | |

## Colors cannot be applied in this environment :( Try using a terminal or RStudio.

## 
##  ----- 
## Byebye Margarett Purdy
## You are now Boogers Party 
##  ------ 
##     \   
##      \  
##       \
##        /\___/\
##        {o}{o}|
##        \ v  /|
##        |    \ \
##         \___/_/       [ab] 
##           | |

## Colors cannot be applied in this environment :( Try using a terminal or RStudio.

## 
## 
##  ----- 
## Byebye Miss Tilla Funk DDS
## You are now Ooze Vil Fangs Dead 
##  ------ 
##     \   
##      \
##                   ___
##                ___)__|_
##           .-*'          '*-,
##          /      /|   |\     \
##         ;      /_|   |_\     ;
##         ;   |\           /|  ;
##         ;   | ''--...--'' |  ;
##          \  ''---.....--''  /
##           ''*-.,_______,.-*'  [nosig]
## 

## Colors cannot be applied in this environment :( Try using a terminal or RStudio.

## 
## 
##  ----- 
## Byebye Wilton Carroll IV
## You are now Cartoon Cruella Eve 
##  ------ 
##     \   
##      \
##               |
##               |
##               |
##              __
##           | /  \ |
##          \_\\  //_/
##           .'/()\'.
##            \\  //  [nosig]
## 

## Colors cannot be applied in this environment :( Try using a terminal or RStudio.

## 
## 
##  ----- 
## Byebye Tevin Conn
## You are now Coffin Bones 
##  ------ 
##     \   
##      \
##      .-.
##     (o o)
##     | O \
##      \   \
##       `~~~' [nosig]
## 

## Colors cannot be applied in this environment :( Try using a terminal or RStudio.

## 
## 
##  ------------- 
## Byebye Dr. Ballard Langworth III
## You are now Dish Blood Cemetery Dish 
##  -------------- 
##               \   
##                \  
##                 \
##         __.--'\     \.__./     /'--.__
##     _.-'       '.__.'    '.__.'       '-._
##   .'                                      '.
##  /                                          \
## |                                            |
## |                                            |
##  \         .---.              .---.         /
##   '._    .'     '.''.    .''.'     '.    _.'
##      '-./            \  /           \.-'
##                       ''mrf

Note that I reported the message I got from cowsay. It’s not too bad, cowsay is not designed for use in R Markdown I think.

I found the results not fantastic, but not too bad either! I especially liked “Boogers Party”!

Mucus, anyone?

In conclusion, in this post I showed how to build a corpus of Halloween-related words by responsibly webscraping, and how to more or less hackily compare strings based on their pronunciation. I obviously tried the function on my own name (without the accent, useless in this case) and wasn’t too happy…

spookify("Maelle Salmon", 42, words)

## Colors cannot be applied in this environment :( Try using a terminal or RStudio.

## 
## 
##  ------------- 
## Byebye Maelle Salmon
## You are now Mucus Bulging 
##  -------------- 
##               \   
##                \  
##                 \
##         __.--'\     \.__./     /'--.__
##     _.-'       '.__.'    '.__.'       '-._
##   .'                                      '.
##  /                                          \
## |                                            |
## |                                            |
##  \         .---.              .---.         /
##   '._    .'     '.''.    .''.'     '.    _.'
##      '-./            \  /           \.-'
##                       ''mrf

But then, at least, it is scary.

O'Reilly animals in trouble? Conservation status of book covers

Sat, 25 Aug 2018 00:00:00 +0000

What can a kaka, a kakapo, an European rabbit and a grey heron have in common? Well, they might co-habit in the bookshelf of an R user, since they’re all animals on the covers of popular R books: “R Packages”, “R for Data Science”, “Text mining with R” and “Efficient R programming”, respectively. Their publisher, O’Reilly, has now based its brand on covers featuring beautiful gravures of animals.

Recently, while wondering what the name of R for Data Science bird was again (I thought it was a kea!), I was thrilled to find the whole O’Reilly menagerie, i.e. a list of books and corresponding animals! The website also features a link to “A short history of the O’Reilly animals” that was an amazing read. In it was noted that “The animals are in trouble.”, with a few examples of endangered species. It inspired me to actually try and assess the conservation status of O’Reilly animals using responsible webscraping, taxonomic name resolving and IUCN Redlist API querying…

Scraping the menagerie: an utter delight!

I had a great time webscraping the menagerie, not only thanks to my now reasonable experience doing such things, but also thanks to

my using the wonderful polite package for webscraping, that makes me feel so good about myself. Read more about this package in my previous post.
the webpage having really good structured html with specific classes.

The menagerie is divided into pages of 20 books, so I mapped over all possible offsets up to the number of animals indicated on the website, 1227.

library("magrittr")
home_url <- "https://www.oreilly.com/animals.csp"

session <- polite::bow(home_url,
                       user_agent = "Maëlle Salmon https://masalmon.eu/")

get_twenty <- function(offset, session){
  # offset parameter to get all books 20 by 20
  params <- glue::glue("?x-o={offset}")
  
  # scraping with content parameter
  # cf https://github.com/dmi3kno/polite/issues/6
  # https://www.oreilly.com/animals.csp?x-o=720 was problematic
  # (German characters)
  page <- polite::scrape(session, params = params, 
                         content = "text/html;charset=iso-8859-1")
  # get all animal rows
  rows <- rvest::xml_nodes(page,
                           xpath = "//div[@class='animal-row']")
  # extract book titles
  rows %>%
    rvest::xml_nodes(xpath = "a[@class='book']") %>%
    rvest::xml_nodes(xpath = "h1[@class='book-title']") %>%
    rvest::html_text() -> book_titles
  
  rows %>%
    rvest::xml_nodes(xpath = "h2[@class='animal-name']") %>%
    rvest::html_text() -> animal_names
  
  tibble::tibble(book = book_titles,
                 animal = animal_names)
  
}

no_animals <- 1227 # by hand!

offsets <- (0:floor(no_animals/20))*20

purrr::map_df(offsets, get_twenty, session = session) %>%
  readr::write_csv("oreilly_animals.csv")

I got 1134 rows, each corresponding to a book, with animals potentially repeated.

animals

## # A tibble: 1,134 x 2
##    book                            animal                                 
##    <chr>                           <chr>                                  
##  1 Mobile Design and Development   12-Wired Bird of Paradise              
##  2 Windows PowerShell for Develop~ 3-Banded Armadillo                     
##  3 Jakarta Commons Cookbook        Aardvark                               
##  4 Clojure Cookbook                Aardwolf                               
##  5 Ubuntu: Up and Running          Addax, aka Screwhorn Antelope          
##  6 Social eCommerce                Adjutant (Storks)                      
##  7 BioBuilder                      Aegina Citrea, narcomedusae, jellyfish 
##  8 JRuby Cookbook                  African Civet                          
##  9 C# 5.0 Pocket Reference         African Crowned Crane aka Grey Crowned~
## 10 Programming C# 5.0              African Crowned Crane aka Grey Crowned~
## # ... with 1,124 more rows

In the short history of animals, Edie Freedman mentions having discovered “that there were intriguing correspondences between specific technologies and specific animals”. This made me curious about my last name, Salmon!

animals %>%
  dplyr::filter(stringr::str_detect(animal, "[Ss]almon")) %>%
  knitr::kable()

book	animal
Values, Units, and Colors	Salmon
CSS Text	Salmon
CSS Fonts	Salmon
Selectors, Specificity, and the Cascade	Salmon
Transitions and Animations in CSS	Salmon2

I have no idea what trait of salmons make them good at design, other than my not sharing that trait with them. When my friend Adrien and I wrote a (non O’Reilly) book years ago, we selected a frog for the cover based on its being pretty, which is much less cool than O’Reilly branding!

From animals common names to scientific names?

Now, you’ll have noticed the names of animals are written in English. My ultimate goal being the querying of IUCN Red List API, and this API only accepting scientific names (contrary to the website of the same organization), I needed to resolve the common names to scientific names. This is a hard problem! My strategy here was:

Cleaning the names a bit to remove the parts after “aka” for instance.

clean <- function(animal){
  semiclean <- animal %>%
    stringr::str_remove_all("aka.*") %>%
    stringr::str_remove_all("\\,.*") %>%
    stringr::str_remove_all("\\(.*") 
  
  if(semiclean == "12-Wired Bird of Paradise"){
    semiclean <- "Twelve-Wired Bird of Paradise"
  }
  
  if(semiclean == "3-Banded Armadillo"){
    semiclean <- "Three-Banded Armadillo"
  }
  stringr::str_remove_all(semiclean, "[0-9]")
}


animals <- dplyr::mutate(animals, animal_clean = purrr::map_chr(animal, clean))

Using the rOpenSci taxize package that has a handy comm2sci function. This function works for anyone, but it’s better to request a key for the database used, EOL by default (see e.g. taxize::use_eol() for more info).
Not being too optimistic since the databases taxize queries cannot do wonders, no matter how good they are.

Note that for each species, the first scientific name returned is selected, because there’s no other criterion to go by. That’s how I’ll end up with a Salmon catfish for Salmon, too bad.

animal_names <- unique(animals$animal_clean)

# scientific names
good_comm2sci <- memoise::memoise(taxize::comm2sci)

get_name <- function(common_name){
  sci_names <- good_comm2sci(common_name)
  
  # don't get the name of who defined the species
  sci_name <- stringr::word(sci_names[[1]][1], start=1, end = 2)
  
  tibble::tibble(common_name = common_name,
                 sci_name = sci_name)
}
scientific_names <- purrr::map_df(animal_names, get_name)

animals <- dplyr::left_join(animals,
                            scientific_names,
                            by = c("animal_clean" = "common_name"))

I got names for 694 books, out of 1134, getting 555 animals. It’s not bad, but this number also needs to be treated with caution. See for instance:

animals %>%
  dplyr::filter(stringr::str_detect(animal, "Galapagos")) %>%
  knitr::kable()

book	animal	animal_clean	sci_name
PHP Cookbook	Galapagos Land Iguana	Galapagos Land Iguana	Conolophus marthae
Upgrading to PHP 5	Galapagos Tortoise	Galapagos Tortoise	Chelonoidis nigra

I noticed the iguana while perusing my results, and a quick internet search taught me that there are three species of terrestrial iguanas in the Galapagos, the most common one, and the one probably present on the book cover, being Conolophus subcristatus, not Conolophus marthae! I’ve noticed a few other mistakes, so I’ll need to handle the results with care. I now wish the menagerie had a bit more Latin in it!

Querying the IUCN Red List

Indeed, scientific names of species are the key to a wealth of data! Traits data, taxonomic information… and conservation status thanks to the IUCN Red List, an impressive assessment of species at the global scale. One can programmatically query it using the rOpenSci rredlist package! That’s what I did, adding a waiting time of 2 seconds between API calls, as recommended by the IUCN folks. Note that I have an API key because I asked for it, see more info by typing rredlist::rl_use_iucn() after installing rredlist, and be patient since it can last a few days before one gets one.

slow_rl_search <- ratelimitr::limit_rate(rredlist::rl_search,
                                         rate = ratelimitr::rate(1, 2))

get_status <- function(sci_name){
  message(sci_name)
  results <- slow_rl_search(sci_name)$result
  
  if(!is.null(results)){
    results$sci_name <- sci_name
  }
  
  results
}

animals <- dplyr::filter(animals, !is.na(sci_name))
purrr::map_df(unique(animals$sci_name), get_status) %>%
  readr::write_csv("oreilly_animals_status.csv")

status <- readr::read_csv("oreilly_animals_status.csv")

animals <- readr::read_csv("oreilly_animals_scientific.csv")

status <- dplyr::filter(status, !is.na(category))
animals <- dplyr::left_join(animals, status, by = "sci_name")

str(animals)

## Classes 'tbl_df', 'tbl' and 'data.frame':    1134 obs. of  32 variables:
##  $ book              : chr  "Mobile Design and Development" "Windows PowerShell for Developers" "Jakarta Commons Cookbook" "Clojure Cookbook" ...
##  $ animal            : chr  "12-Wired Bird of Paradise" "3-Banded Armadillo" "Aardvark" "Aardwolf" ...
##  $ animal_clean      : chr  "Twelve-Wired Bird of Paradise" "Three-Banded Armadillo" "Aardvark" "Aardwolf" ...
##  $ sci_name          : chr  "Seleucidis melanoleuca" "Tolypeutes tricinctus" "Cucumis humifructus" "Proteles cristata" ...
##  $ taxonid           : int  NA 21975 NA 18372 NA 22697721 NA 41589 22692046 22692046 ...
##  $ scientific_name   : chr  NA "Tolypeutes tricinctus" NA "Proteles cristata" ...
##  $ kingdom           : chr  NA "ANIMALIA" NA "ANIMALIA" ...
##  $ phylum            : chr  NA "CHORDATA" NA "CHORDATA" ...
##  $ class             : chr  NA "MAMMALIA" NA "MAMMALIA" ...
##  $ order             : chr  NA "CINGULATA" NA "CARNIVORA" ...
##  $ family            : chr  NA "CHLAMYPHORIDAE" NA "HYAENIDAE" ...
##  $ genus             : chr  NA "Tolypeutes" NA "Proteles" ...
##  $ main_common_name  : chr  NA "Brazilian Three-banded Armadillo" NA "Aardwolf" ...
##  $ authority         : chr  NA "(Linnaeus, 1758)" NA "(Sparrman, 1783)" ...
##  $ published_year    : int  NA 2014 NA 2015 NA 2016 NA 2015 2016 2016 ...
##  $ category          : chr  NA "VU" NA "LC" ...
##  $ criteria          : chr  NA "A2cd" NA NA ...
##  $ marine_system     : logi  NA FALSE NA FALSE NA FALSE ...
##  $ freshwater_system : logi  NA FALSE NA FALSE NA TRUE ...
##  $ terrestrial_system: logi  NA TRUE NA TRUE NA TRUE ...
##  $ assessor          : chr  NA "Miranda, F., Moraes-Barros, N., Superina, M. & Abba, A.M." NA "Green, D.S." ...
##  $ reviewer          : chr  NA "Loughry, J." NA "Dloniak, S.M.D. & Holekamp, E." ...
##  $ aoo_km2           : chr  NA NA NA NA ...
##  $ eoo_km2           : chr  NA "937000" NA NA ...
##  $ elevation_upper   : int  NA NA NA 2000 NA 550 NA 2500 NA NA ...
##  $ elevation_lower   : int  NA NA NA 0 NA 0 NA 0 0 0 ...
##  $ depth_upper       : num  NA NA NA NA NA NA NA NA NA NA ...
##  $ depth_lower       : int  NA NA NA NA NA NA NA NA NA NA ...
##  $ errata_flag       : logi  NA NA NA NA NA NA ...
##  $ errata_reason     : chr  NA NA NA NA ...
##  $ amended_flag      : logi  NA NA NA NA NA NA ...
##  $ amended_reason    : chr  NA NA NA NA ...

There are 1134 books, 499 with a conservation status from the IUCN Red List, although this includes “DD” meaning “Data Deficient”. I am hesitant to actually show the proportion of species in each category for those for which I got data for, because the resolution of common names to scientific names isn’t certain… Take the following table with a pinch of salt!

dplyr::count(animals, category) %>%
  knitr::kable()

category	n
CR	14
DD	7
EN	42
EW	1
EX	6
LC	348
LR/cd	1
LR/lc	5
LR/nt	4
NT	23
VU	48
NA	635

See the following page for more precise information about categories. LC is least concern. Let’s have a look at the extinct species.

animals %>%
  dplyr::filter(category == "EX") %>%
  dplyr::select(book, animal, sci_name) %>%
  knitr::kable()

book	animal	sci_name
Java Data Objects	Bilby, Rabbit-eared Bandicoot (Macrotis lagotis)	Macrotis leucura
Building and Testing with Gradle	Bush Wren	Xenicus longipes
Designing Mobile Payment Experiences	Crested Pigeon	Microgoura meeki
SSH, The Secure Shell: The Definitive Guide	Land Snail	Amastra crassilabrum
Java NIO	Pigfooted Bandicoot	Chaeropus ecaudatus
Java I/O	White Rabbit	Macrotis leucura

I searched for the covers and names and could assess that in that table, there are 4 false positives due to the ambiguity of common names! Only the Bush wren and the Pigfooted Bandicoot got scientific names corresponding to what they look like, and are extinct, which is quite sad.

Now, to reverse-engineer what Edie Freedman wrote in the short history of O’Reilly animals, “Many of the animals that appear on our covers are critically endangered—the tarsier from Learning the vi & Vim Editors, the lorises from sed & awk, the Hawksbill turtle from Getting Started with CouchDB, the tiger from Running Mac OS X Tiger, and the African elephant on Hadoop: The Definitive Guide, just to name a few.”, let’s look at what we got for them.

animals %>%
  dplyr::filter(book %in%
                  c("Hadoop: The Definitive Guide",
                    "Learning the vi and Vim Editors",
                    "sed & awk",
                    "Getting Started with CouchDB",
                    "Running Mac OS X Tiger")) %>%
  dplyr::select(book, animal, sci_name, category) %>%
  knitr::kable()

book	animal	sci_name	category
Hadoop: The Definitive Guide	African Elephant, young	Elephantulus rozeti	LC
Getting Started with CouchDB	Hawksbill Turtle	Eretmochelys imbricata	CR
sed & awk	Slender Loris “Awk”	NA	NA
Running Mac OS X Tiger	Sumatran Tiger	Parantica tityoides	LR/nt
Learning the vi and Vim Editors	Tarsier, full-body, standing on hind feet, b/w engraving	Tarsius pelengensis	EN

Again, our name resolution wasn’t very good!

The elephant should be Loxondota africana, vulnerable species
The turtle is right.
For the loris we should have gotten this species that’s endangered.
The Sumatran tiger, Panthera tigris ssp. sumatrae , is critically endangered
There are several Tarsier species, I’m not sure which one is the right one.

So all in all, we got some truth but also some wrong names and hence wrong conservation statuses!

Conclusion: hoping for a menagerie of scientific names

In this post, I exemplified responsible webscraping with the use of the polite package to get a table of all animals on O’Reilly book covers from the dedicated menagerie. I tried resolving the common names to scientific names using taxize::comm2sci, which was only partly successful. I got conservation status for the scientific names using the rredlist package, programmatic interface to the IUCN Red List. The results would be better if O’Reilly published scientific names of animals, but nonetheless this workflow helped me identify two extinct species, the Bush wren of Building and Testing with Gradle and the Pigfooted Bandicoot of Java NIO. I can’t but hope the list of such now sad book covers won’t grow any longer…

ALLSTATisticians in decline? A polite look at ALLSTAT email Archives

Tue, 31 Jul 2018 00:00:00 +0000

Webscraping ALLSTAT

Life on the edge of easy responsible webscraping

As underlined in The Ethical Scraper’s principles by James Densmore (that I found in the slides of Hanjo Odendaal’s useR! 2018 webscraping tutorial), a first step is to research whether you can get the data without webscraping (I found no API). Note that an ideal case would be to have received emails since the list’s creation in 1998, and having stored them, so that one could use this local copy. Or one could email the list maintainers of course!

Then, there is some good practice to keep in mind whilst webscraping, also underlined in the principles list, and most importantly all encompassed in the awesome new package polite by Dmytro Perepolkin! I knew the packages wrapped by Dmytro, but tended to only use robotstxt e.g. here when webscraping, (but using others in API packages). Now with very little effort, with polite you will

Seek permissions via robotstxt,
Introduce yourself with an user agent, by default the package’s name and URL,
Take it slowly thanks to ratelimitr,
Never ask twice thanks to memoise.

The polite package is brand-new, still in development, which in general means you might want to stay away from it for a while, but I was eager to try it and pleased by its working very well! Being an early adopter also means I saw my issues promptly closed with some solution/new code by Dmytro!

I started work by bowing which in polite’s workflow means both creating a session (with user-agent, delay between calls) and checking that the path is allowed.

home_url <- "https://www.jiscmail.ac.uk/cgi-bin/webadmin"

session <- polite::bow(home_url,
                       user_agent = "Maëlle Salmon https://masalmon.eu/")

And then it was time to scrape and parse…

Actual webscraping a.k.a solving XPath puzzles

When confronted with a page I’d like to extract info from, I try to identify how I can write XPath to get the elements I want, which means looking at the page source. I used to transform whole pages to text before using regexp on them, which was clearly suboptimal, thanks to Eric Persson for making me switch to XPath.

My strategy here was to get the subject, date, sender and size of each email from archive pages. Here is such an archive page. Nowadays there is one archive page by month, but there used to be one by year, so I got the list of archive pages from this general page.

library("magrittr")

polite::scrape(session, params = "?A0=ALLSTAT") %>%
  rvest::xml_nodes("li") %>%
  rvest::xml_nodes("a") %>%
  rvest::html_attr("href") %>%
  purrr::keep(function(x) stringr::str_detect(x, "\\/cgi-bin\\/webadmin\\?A1\\=")) %>%
  stringr::str_remove("\\/cgi\\-bin\\/webadmin\\?A1\\=ind") %>%
  stringr::str_remove("\\&L\\=ALLSTAT") -> date_strings

This is not very elegant but this got me only the “1807” and such I needed for the rest of the scraping. polite::scrape is a wrapper to both httr::GET and httr::content and does the rate limiting (by default 1 call every 5 seconds, delay parameter of polite::bow) and memoising. I actually ended up only scraping emails metadata from 2007 because it took ages to parse the 2006 page. It made me, according to a search via the website, miss these data scientists job openings: one in 2000 and one in 2004.

date_strings <- date_strings[stringr::str_length(date_strings) != 2]
# or
date_strings <- purrr::discard(date_strings, function(x) stringr::str_length(x) == 2)

I created a function getting the metadata out of each archive page. The trickiest points here were:

That the rows of the archive table could have two classes, which is the way alternate coloring was obtained. I therefore used | in XPath '//tr[@class="normalgroup"]|//tr[@class="emphasizedgroup"]'.
That there was no different class/formatting for subject, date, sender, so I got all of them at once, and then used the modulo operator, %%, to assign them to the right vector.

get_emails_meta_by_date <- function(date_string, session){
  message(date_string)
  params <- glue::glue("?A1=ind{date_string}&L=ALLSTAT&F=&S=&O=T&H=0&D=0&T=0")
  
  everything <- try(polite::scrape(session, params = params),
                    silent = TRUE)
  
  # at the time of writing one couldn't pass encoding to scrape
  # but now one can https://github.com/dmi3kno/polite/issues/6#issuecomment-409268730
  if(is(everything, "try-error")){
    everything <- httr::GET(paste0(home_url,
                                   params)) %>%
      httr::content(encoding = "latin1")
  }
  
  everything <- everything %>%
    # there are two classes that correspond
    # to the table having two colours of rows!
    rvest::xml_nodes(XPath = '//tr[@class="normalgroup"]|//tr[@class="emphasizedgroup"]') %>%
    rvest::xml_nodes("span")
  
  everything %>%
    rvest::xml_nodes(XPath = "//td") %>%
    rvest::xml_nodes("span") %>%
    rvest::xml_nodes("a") %>%
    rvest::html_text() -> subjects
  
  everything %>%
    rvest::xml_nodes(XPath = "//td[@nowrap]") %>%
    rvest::xml_nodes(XPath = "p[@class='archive']") %>%
    rvest::html_text() -> big_mess
  
  senders <- big_mess[seq_along(big_mess) %% 3 == 1]
  senders <- stringr::str_remove(senders, " \\<\\[log in to unmask\\]\\>")
  
  dates <- big_mess[seq_along(big_mess) %% 3 == 2]
  dates <- lubridate::dmy_hms(dates, tz = "UTC")
  
  sizes <- big_mess[seq_along(big_mess) %% 3 == 0]
  sizes <- stringr::str_remove(sizes, " lines")
  sizes <- as.numeric(sizes)
  
  tibble::tibble(subject = subjects,
                 sender = senders,
                 date = dates,
                 size = sizes) %>%
    readr::write_csv(glue::glue("data/emails_meta{date_string}.csv"))
  
}

I chose to save the metadata of each archive page in its own csv in order to make my workflow less breakable. I could have used purrr::map_df but then it’d be harder to re-start, and it was hard on memory apparently.

fs::dir_create("data")
purrr::walk(date_strings,
              get_emails_meta_by_date,
              session = session)

Analyzing ALLSTAT jobs

Filtering jobs

ALLSTAT encourages you to use keywords in emails’ subjects, so many job openings contain some variant of “job”, and that’s the sample on which I shall work.

library("magrittr")
library("magrittr")

fs::dir_ls("../../static/data/allstat") %>%
  purrr::map_df(readr::read_csv) -> emails

jobs <- dplyr::filter(emails, 
                      stringr::str_detect(subject,
                                          "[Jj][Oo][Bb]"))

Out of 32761 emails I got 9245 job openings.

I created two dummy variables to indicate the presence of data scientist or statistician in the description. With the definition below, the “statistician” category might contain “biostatitisticians” which is fine by me.

jobs <- dplyr::mutate(jobs,
                      data_scientist = stringr::str_detect(subject, 
                                                           "[Dd]ata [Ss]cientist"),
                      statistician = stringr::str_detect(subject, 
                                                          "[Ss]tatistician"))

176 subjects contain the word “data scientist”, 2546 the word “statistician.”, 20 both.

dplyr::filter(jobs, data_scientist, statistician) %>%
  dplyr::select(subject, sender, date) %>%
  knitr::kable()

subject	sender	date
JOB: Statisticians/Data Scientists with Unilever	Murray, Peter	2008-02-12 08:47:56
Jobs: Senior Data Scientists/Statisticians	Cox, Trevor	2008-03-12 16:24:48
20 new job ads for data scientists, statisticians	Vincent Granville	2013-06-16 00:18:27
23 new jobs for statisticians, data scientists	Vincent Granville	2013-07-27 20:05:48
19 new job ads for statisticians and data scientists	Vincent Granville	2013-10-12 23:21:23
Job Data scientist / Statistician	Andrea Schirru	2014-07-22 16:38:59
JOB: Computational Statistician / Data Scientist	David Hastie	2014-07-16 11:36:34
Job Openings: Statisticians and Data Scientists at Open Analytics (Belgium)	Tobias Verbeke	2015-06-30 09:53:21
JOBS x 2: Data Scientist/Medical Statistician at University of Manchester	Matthew Sperrin	2016-07-28 10:42:52
JOB: Data scientist - Statistician (KTP associate) @ University of Essex	Aris Perperoglou	2016-09-12 10:33:32
JOB: Principal Statistician/ Data Scientist- Pharma - Perm - Centralised Monitoring- Data Safety - Global Search-UK	Sabrina Andresen	2016-10-17 16:05:12
JOB: Research Associate - Statistician / Data Scientist, University of Manchester (Centre for Musculoskeletal Research)	Jamie Sergeant	2016-11-01 10:21:07
JOB - Statistician / Data Scientist at Cefas, UK	David Maxwell	2017-08-04 16:15:25
Job: Data Scientist/Statistician at Essex University	Aris Perperoglou	2017-08-21 13:43:43
Job: Statistician/Data Scientist	Roisin McCarthy	2017-11-06 16:49:18
Job Openings: Statisticians and Data Scientists at Open Analytics (Belgium)	Tobias Verbeke	2017-12-05 20:48:39
JOB \| StatsJobs - Data Scientist/Statistician, Zurich Insurance	James Phillips	2018-01-30 08:59:49
JOB \| Junior Medical statistician / Real-world data scientist - Centre of Excellence for Retrospective Studies, IQVIA (London)	Venerus, Alessandra	2018-03-05 10:33:09
JOB \| Senior Medical statistician / Real-world data scientist - Centre of Excellence for Retrospective Studies, IQVIA (London)	Venerus, Alessandra	2018-03-05 10:32:51
Job: Catalyst Project Research Data Scientist / Statistician	Aris Perperoglou	2018-06-25 14:17:18

Are the job titles synonymous for the organizations using slashes? I am especially puzzled by “Senior Medical statistician / Real-world data scientist”! I filtered them out and created a category variable.

jobs <- dplyr::filter(jobs,
                      !(data_scientist&statistician))
jobs <- dplyr::mutate(jobs,
                      category = dplyr::case_when(data_scientist ~ "data scientist",
                                                  statistician ~ "statistician",
                                                  TRUE ~ "other"),
                      category = factor(category,
                                        levels = c("statistician",
                                                   "data scientist",
                                                   "other"),
                                        ordered = TRUE))

jobs <- dplyr::mutate(jobs,
                      year = lubridate::year(date),
                      year = as.factor(year))

Here are some examples of positions for each:

head(jobs$subject[jobs$category=="statistician"])

## [1] "FW: JOB: Senior/Lead Statistician - GlaxoSmithKline"                   
## [2] "JOB - Clinical Trial Statisticians and SAS Programmers"                
## [3] "JOB - Medical statistician in Salvador, Brazil (2-3 years)"            
## [4] "JOB - Medical Statistician, Oxford"                                    
## [5] "JOB OPPORTUNITY - 39408 - Entry-Level Statisticians - GSK"             
## [6] "JOB: Biostatistician - Contract/Consultancy Opportunity - Switzer land"

head(jobs$subject[jobs$category=="data scientist"])

## [1] "JOB: Data Scientist, Unilever"                                 
## [2] "JOB: Data Scientist, Unilever"                                 
## [3] "JOB: Data Scientist / Machine Learning (BELFAST)"              
## [4] "JOB: Data Scientists in Belfast"                               
## [5] "JOBS: Data Scientists"                                         
## [6] "[Job] Hacker or Botnet Developer, Data Scientist (Telecommute)"

head(jobs$subject[jobs$category=="other"])

## [1] "Job - Data manager Dept of Haematology, Imperial College, London"                  
## [2] "Re: JOB - Health Economic Modelling and Value Demonstrations"                      
## [3] "JOB - Lectureship / readership"                                                    
## [4] "JOB - Mathematical Modeller at the Health Protection Agency, Centre for Infections"
## [5] "JOB - PhD position"                                                                
## [6] "JOB - SAS Senior Statistical Programmer, Oxford"

Are data scientists on the rise?

library("ggplot2")

ggplot(jobs) +
  geom_bar(aes(year, fill = category)) +
  viridis::scale_fill_viridis(discrete = TRUE) +
  theme(legend.position = "bottom") +
  hrbrthemes::theme_ipsum(base_size = 14) +
  xlab("Year (2018 not complete yet)") +
  ylab("Number of job openings") +
  theme(axis.text.x = element_text(angle = 90, hjust = 1)) +
  ggtitle("ALLSTAT mailing list 2007-2018")

I like this bar plot that shows how the total number of job openings fluctuates, but it’s hard to see differences in proportions.

ggplot(jobs) +
  geom_bar(aes(year, fill = category),
           position = "fill") +
  viridis::scale_fill_viridis(discrete = TRUE) +
  theme(legend.position = "bottom") +
  hrbrthemes::theme_ipsum(base_size = 14) +
  xlab("Year (2018 not complete yet)") +
  ylab("Number of job openings")  +
  theme(axis.text.x = element_text(angle = 90, hjust = 1))+
  ggtitle("ALLSTAT mailing list 2007-2018")

According to this plot, although there seems to be more and more data scientists’ jobs advertised on ALLSTAT… Statisticians don’t need to get worried just yet.

Who offers data scientists’ jobs?

dplyr::count(jobs, category,
                sender) %>%
  dplyr::group_by(category) %>%
  dplyr::arrange(category, - n) %>%
  dplyr::filter(sender %in% sender[1:5])

## # A tibble: 15 x 3
## # Groups:   category [3]
##    category       sender                     n
##    <ord>          <chr>                  <int>
##  1 statistician   James Phillips           106
##  2 statistician   Sabrina Andresen          82
##  3 statistician   James Miller              45
##  4 statistician   Angela Smythe             40
##  5 statistician   Helena Newman-Mitchell    37
##  6 data scientist James Phillips            86
##  7 data scientist Sportradar HR              5
##  8 data scientist Jason Howlin               4
##  9 data scientist Deborah Gee                3
## 10 data scientist Christos Mitas             2
## 11 other          James Phillips           671
## 12 other          Angela Smythe            223
## 13 other          Helena Newman-Mitchell   103
## 14 other          Jason Howlin              91
## 15 other          James Miller              85

Seeing James Phillips’ name so often made me have a look at their emails: this person sends emails on the behalf of a website called StatsJobs.com! We can also assume that other super-senders actually work for job aggregators of some sort.

What are the openings about?

To make a more thorough description of the different categories, one would need to get the email bodies, which I decided against for this post. I simply used the subjects, and compared word usage between the “data scientist” and “statistician” categories as in this chapter of the Tidy text mining book by Julia Silge and David Robinson.

library("tidytext")
data("stop_words")
words <- dplyr::filter(jobs, category != "other") %>%
  unnest_tokens(word, subject, token = "words") %>%
  dplyr::filter(!word %in% stop_words$word,
                !word %in% c("job", "statistician", 
                             "jobs", "statisticians",
                             "data", "scientist",
                             "scientists",
                             "datascientistjobs"))

word_ratios <- words %>%
  dplyr::count(word, category) %>%
  dplyr::group_by(word) %>%
  dplyr::filter(sum(n) >= 10) %>%
  dplyr::ungroup() %>%
  tidyr::spread(category, n, fill = 0) %>%
  dplyr::mutate_if(is.numeric, dplyr::funs((. + 1) / sum(. + 1))) %>%
  dplyr::mutate(logratio = log(`data scientist` / statistician)) %>%
  dplyr::arrange(desc(logratio))

word_ratios %>%
  dplyr::group_by(logratio < 0) %>%
  dplyr::top_n(15, abs(logratio)) %>%
  dplyr::ungroup() %>%
  dplyr::mutate(word = reorder(word, logratio)) %>%
  ggplot(aes(word, logratio, fill = logratio < 0)) +
  geom_col(show.legend = FALSE) +
  coord_flip() +
  ylab("log odds ratio (data scientist / statistician)") +
  scale_fill_manual(name = "", 
                    values = c("#21908CFF", "#440154FF"),
                    labels = c("data scientist", "statistician")) +
  hrbrthemes::theme_ipsum(base_size = 11)+
  ggtitle("ALLSTAT mailing list 2007-2018")

What does this figure show? On the left are words more frequent in statistician job openings, on the right words more frequent in data scientist job openings. Well, there might be geographical clusters for each of the category, which I don’t believe though: is the data scientist vs. statistician debate a Cambridgeshire vs. Oxford battle? I was surprised that no word related to academia made an appearance because I thought “university” or an equivalent word would be more representative of statisticians. I am less surprised, though, by words such as “trials”, “medical”, “clinical” being more prevalent in the “statistician” category. The word “fixed” as in “fixed-term” contract is more prevalent for data scientist job openings, which doesn’t sound too cool?

Conclusion

In this post, I had the pleasure to use the brand-new polite package that made responsible webscraping very smooth! Armed with that tool, I extracted metadata of job openings posted on the ALLSTAT mailing lists. I made no statistical analyses, but the main take-away is that statisticians aren’t in danger of extinction just yet.

Where to get help with your R question?

Sun, 22 Jul 2018 00:00:00 +0000

Last time I blogged, I offered my ~~obnoxious~~ helpful advice for blog content and promotion. Today, let me again be the agony aunt you didn’t even write to! Imagine you have an R question, i.e. a question related to how you can do something with R, and your search engine efforts haven’t been too successful: where should you ask it to increase your chance of its getting answered? You could see this post as my future answer to stray suboptimal Twitter R questions, or as a more general version of Scott Chamberlain’s excellent talk about how to get help related to rOpenSci software in the 2017-03-07 rOpenSci comm call.

I think that the general journey to getting answers to your R questions is first trying your best to get answers locally in the documentation of R, then to use a search engine, and then to post a well-formulated question somewhere. My post is aimed at helping you find that somewhere. Note that it’s considered best practice to ask in one somewhere at once, and to then move on to another somewhere if you haven’t got any answer… or if someone kindly redirects you to a better venue!

One thing this post isn’t about is how to ask for help to humans, which is a topic that’s e.g. covered very well in Jenny Bryan’s talk about reprex in the same 2017-03-07 rOpenSci comm call, but I’ll link out to useful resources I’ve found. This post is also not about how to ask for help to e.g. Google, and I don’t know of a good search engine guide yet although e.g. “It can be particularly helpful to paste an error message into a search engine to find out whether others have solved a problem that you encountered.” in https://www.r-project.org/help.html is true.

Public question platforms vs. safe spaces?

I’ll start this post with a general comment for newbies who aren’t at ease enough yet to post their question, no matter what type of questions (see later), to anywhere public: find your safe and friendly space!

Lean on your current friend(s)

Do you have an R friend? Maybe this colleague who actually convinced you to start using R? Well, ask this person for help! Remind them it’s their fault you’re struggling now.

And if you’re not a newbie but are reading this now, be this R friend! Mentor newbies on their way to be better R users and askers.

Find new friends

Join a community, and ask your questions there. More specifically, you can join the R4DS community; your local/remote R-Ladies community (as a reminder, this doesn’t mean women only but instead “minority genders (including but not limited to cis/trans women, trans men, non-binary, genderqueer, agender)"); this French-speaking community. Such communities most often have Slack workspaces or equivalent, with an r-help channel, as well as a code of conduct and a general friendly attitude. Build your confidence there, and soon, you’ll be out asking (and answering!) R questions in the wilderness as well.

Note, if you know of any list of such friendly communities, please tell me. I only know of this list (of lists) of R User Groups and Conferences by David Smith.

Spotlight on my typology of R questions

One thing I especially like about Scott Chamberlain’s slidedeck is his typology of methods to get help, “methods and what they’re good for”. I’ll build this post based on a typology of questions, that also intersects with a typology of your newbiness-nervousness.

I tend to think of R questions as pertaining to one of these categories:

is this a bug, why am I stuck with this code? how do I install rJava on my laptop? (problem!) Problems encompass actual bugs, and problems where the problem is your code rather than the package (I’ll admit that I find joy in my code’s sometimes not being the problem, although this doesn’t solve the problem.).
is there a tool in R for doing foo? how do I learn about bar? (question!)
what’s the best plotting library in R? dependencies, is less more? (debate!)

I agree that the difference between debate and question might be thin, in my mind question questions have more of a trivia aspect, with answers being easier.

Where to ask your problem question

Scenario: you wrote R code, and it doesn’t do what you expected it to, which includes the glorious cases of not running at all or of using packages you haven’t even been able to install. What do you do then?

If your problem is linked to a package in particular, i.e. you know the function that’s causing you sorrow comes from the foo package, then your question is a foo question or bug report. So what you need to find now is the platform to report bugs and ask questions about foo.
- It might be written in foo docs/metadata. Have a look in the docs, and observe the website or development repository of the package. E.g. if you use blogdown, its BugReports URL is https://github.com/rstudio/blogdown/issues and if you were to try and open an issue there with your bug or other problem, you’d get to see the template that tells you to use Stack Overflow first for general questions. GitHub repos can also contain question guidance as part of their CONTRIBUTING.md. If you use INLA and have questions, you’ll find http://www.r-inla.org/help useful. I remember having many questions about INLA during my PhD, and sometimes having to ask them in the mailing list when the docs couldn’t help me any further.
- You’ll rarely have to email the maintainer of a package, but you might have to, e.g. if you wonder whether the package will be further maintained or as Scott said here, if your problem is specific to some sensitive data and an rOpenSci package.
- Sometimes foo provider can also be a clue. If foo is an rOpenSci package, you can ask for help on rOpenSci’s forum or on Stack Overflow using the “ropensci” tag. If foo is an RStudio package, you can ask for help on RStudio’s forum… You get the gist.
When there’s no other indication or you’re not sure foo or bar is the actual culprit, use a more general site like Stack Overflow, tagging your question with “r” and other relevant labels. I was pleased to see that the bottom of https://www.r-project.org/help.html includes a list of question venues including Stack Overflow, but also all R mailing lists.

Where to ask your question or debate question

How to find out whether there’s a tool to do image manipulation in R? How to know what’s best practice for your special package development challenge? How to get a book recommendation?

Maybe Twitter, to which I’ll dedicate a whole section a bit later. But a good guess is also trying to locate where the experts, or people interested in the answer, normally interact. Your finding your happy place will probably be a bit of a trial-and-error process, so while asking your first question might be more difficult, things should be easier as you learn to navigate the different communities and their activity. Here is some inspiration:

The R SIG (special interest group) mailing lists. Now I don’t use them at all because I’m not keen on receiving and sending many emails, but I can see there’s a great diversity of groups, maybe one that’ll suit your interests!
If your question is related to open science, reproducibility, documenting data, extracting data out of something (a PDF! a PNG!), package development, rOpenSci’s discuss forum might be the right venue.
The RStudio community forum has many categories including the tidyverse and the RStudio IDE of course but also about package development; R in production, at scale, and in complex environments, etc.
For a question related to outbreak analytics and R programming, check out the R Epidemics Consortium mailing list or Slack.

What is Twitter good for?

Anyone can use Twitter for what they want, depending on the accounts they follow and on their way to consume their timelines and hashtags, so the following is probably even more personal to me than the rest. I think Twitter questions that have higher chances to be answered (because I like answering them!) are the trivia ones, i.e. “is there something to do foo”, and the more controversial ones if only for having fun. That said, if you don’t have a huge following yet, or post at the “wrong” time, even when using the right hashtags or so your question might be ignored, keep that in mind.

What Twitter is not good for is, in my opinion:

Asking a question about a package by tagging the maintainer especially when the package docs state there’s another venue for that. But then, as a maintainer, one should probably handle such mistakes gracefully (like Yihui did here).
Too much code I think, even if tools like Sean Kross’ codefinch can help. Your tweet with a question can circumvent the difficulty by advertising a gist like codefinch makes you do (i.e. a tweet with an “attachment”), or by being a link to your actual question on Stack Overflow (i.e. a tweet to try and draw attention to your question).
Building an archive of questions and answers. It won’t be easy at all to search whether a question has already been answered. You can counteract this loss of knowledge by asking questions somewhere else, or building something out of the answers you’ve been given, e.g. a blog post listing resources for geospatial analysis in R or resources about transport and GIS in R like Steph Locke and Steph de Silva, respectively, did after getting answers on Twitter.
Surprise questions. At least I am scared of questions phrased like this: “Who has time to help me with ?” (e.g. XML, web scraping). They sound like a trap to me! I’d rather know more precisely what the person wants.

Conclusion

I think it’s as important to ask your questions in the right place as it is to ask them in the right way. I hope this post provided some clues as to know where to ask R questions. I’d like to make an important remark from Mara Averick here: “You should not take being redirected to a different forum as a criticism! It’s 100% a means of helping you get the help you need!”. Truth!

I’ll round up this post with a few links of resources to learn how to ask your question:

The guidelines and code of conduct of the venues where you’ve chosen to post your question, if there are some!
the 2017-03-07 rOpenSci comm call “How to Ask Questions so they get Answered! Possibly by Yourself!” was so, so good. I’ve already mentioned the talk by Jenny Bryan about reprex and Scott Chamberlain’s talk about getting help about rOpenSci’s software. Furthermore, JD Long reviewed the historical challenges of getting R help.
I’ve seen very useful posts on RStudio Community forum when browsing the FAQ tag. In particular, I like “Tips for Introducing Non-Programming-Problem Discussions” and this more general list of FAQs.

Good luck with your questions! The Disqus thingy below is good for suggestions and questions about this post!

Get on your soapbox! R blog content and promotion

Mon, 16 Jul 2018 00:00:00 +0000

This year I had the chance to speak at two R-Ladies meetups (I might have invited myself to those meetups to make the most of being in town, create your own happiness!), one in Cape Town in March, one in Seattle in May. It was a blast both times! I gave the same talk twice, and decided it was about time to write it up.

My talk in March was aimed at pairing, well tripleting, with Marie Dussault’s talk about setting up your blogdown website, and Stephanie Kovalchik’s talk about sports blogging so it is not about these topics. What it is is my non data-driven, quite personal view on blog content and promotion, which hopefully features some useful tips for any wannabe R blogger!

Why R blogging and why blogging about R blogging

The reason why I suggested the topic of the Cape Town R-Ladies meetup to be blogging is that it helped me get plugged in into the R community, so I liked the idea of sharing my experience, especially with R-Ladies. I don’t have Google analytics for my blog so I don’t actually know how successful my blog is, but I’ve got enough positive feedback to consider I’m doing something right. That said, given my lack of data to support anything I’ll say, please take everything with a pinch of salt!

R Blog content

What to blog about?

There are plenty of R things you could blog about!

Your latest R obsession? Me in https://www.masalmon.eu/2018/02/22/hexcombine/
Something you’ve just implemented? Omayma Said http://omaymas.github.io/prophet_explore/
Something you like! Kasia Kulma https://kkulma.github.io/2017-12-16-star-wars-vs-star-trek-word-battle/
Fun packages? Find inspiration in Mine Çetinkaya-Rundel’s list http://www2.stat.duke.edu/courses/Fall17/sta112.01/assignment/08-mini-hw.html

I especially like the point Edwin Thoen made in this post: you can blog about things you’ve just learnt, it’s actually a good thing!

Then, an important aspect of R blogging is that you don’t need to blog about R code! Examples of relevant R content I’ve seen include:

A list of things that helped you learn Lorna Maria Aine in https://medium.com/@lornamariak/r-libraries-to-aid-you-learn-data-science-in-2018-30eaaf7cc050
Your experience at an event e.g. Jasmine Dumas in https://jasdumas.github.io/2017-05-28-runconf17-experience/
Your experience at work e.g. Julia Silge in https://juliasilge.com/blog/one-year-at-stack/

Now, if you still have trouble finding ideas, you can find inspiration by… reading other blogs! I’d particularly recommend:

R Weekly newsletter https://rweekly.org/
#rstats on Twitter https://twitter.com/search?q=%23rstats&src=savs

How to get stuff done?

I’m not a self-help master so I’ll be very brief here. It’s obviously important to proofread and do your best but don’t let stress and perfectionism spoil the fun. Most often you won’t get paid for blogging, so you should enjoy it! Do not get too ambitious in each post, and at some point, just ship it. Each post needs to be good enough, just good enough. I particularly liked this article of Kasia Kulma’s about getting stuff done when blogging and recommend it for better phrased thoughts than mine: https://kkulma.github.io/2017-12-29-end-of-year-thoughts/

Another aspect of getting blogging done is I think not losing your precious ideas. I used to have an empty GitHub repo with issues, just for the pleasure of then closing them, now I tend to write ideas in a notebook, maybe for the pleasure of striking through them. Other strategies I’ve heard of include having a Trello board or taping sticky notes on a cupboard.

Besides, when you have an idea, explore it by googling it. If you find similar work, link it in your post. Last but not least, use your own authentic voice when blogging. I think it makes blogging easier, and more enjoyable for readers, at least I like how informal blogging can be and laugh as much as I learn when reading e.g. Brooke Watson. As regards the language, if you’re not an English native speaker, you can choose to blog in English or your own language, or both like Daniela Vázquez https://d4tagirl.com/ !

Blog promotion

Promoting your posts is crucial, because no matter how good your blog is, you need to help it find its audience!

How to tweet about your post

Twitter is an important medium for publicizing your work. If you don’t have an account, consider getting one. Here is my recipe for tweeting a blog post.

Title of your post, maybe a short intro, but do not add a self-deprecatory comment here. Please, don’t.
Link to your post.
#rstats and other hashtags if needed, but do not use too many hashtags, in particular because it creates visual clutter (and in my opinion can give the (wrong) impression of being too thirsty).
A figure or a gif, to make your tweet stand out (in the timelines of Twitter users, in the #rstats timeline)
Emojis! If you like them of course.
You can tag people who inspired you, or whose great package is featured in your post, but proceed with caution. Tagging someone whose package isn’t too popular (e.g. me, ahah) will result in making this person’s happy and often in your tweet getting re-tweeted by them. Also keep in mind that having too many Twitter handles in your tweet is like having too many hashtags, it doesn’t always look good.

Add your blog to lists and feeds aggregator!

There are lists of blogdown blogs where you can list your blog:

https://github.com/sellorm/awesome-blogdown
https://support.rbind.io/about/ if you use their domain name (I don’t know much about rbind.io)

You should also create an RSS feed for your blog if it doesn’t have one yet. Since this post of mine isn’t too technical, I’d recommend googling the name of your website tool and “RSS feed”. Once you have an RSS feed featuring the entire content of your posts, your readers will be able to subscribe, and you can submit your RSS feed to aggregators!

My favorite one, maybe because I’m an R-Weekly team member https://rweekly.org/submit (you can also submit individual links to R Weekly, see https://github.com/rweekly/rweekly.org#r-weekly)
The most famous one, and oldest one, where my own blog is featured, maintained by Tal Galili https://www.r-bloggers.com/add-your-blog/ It might take a while for your feed to be added, don’t take it personally.

Take advantages of networks and other voice-amplifying opportunities

First of all, depending on the topic of your post or of your blog in general, share links to your posts with your potential (niche) audience!

Then, if you’re an R-Lady (as a reminder, this doesn’t mean women only but instead “minority genders (including but not limited to cis/trans women, trans men, non-binary, genderqueer, agender)"), share your posts with your local chapter or R-Ladies remote and don’t hesitate to ping R-Ladies Global (quoting your tweet, not adding the handle in your tweet, in order to keep it clean).

If you use one of rOpenSci’s packages, please post the link to rOpenSci’s discussion forum in the Use Cases category and tag Scott Chamberlain (@sckott) for inclusion in the rOpenSci biweekly newsletter.

Last but not least, I’m getting (even more) preachy and recommend being the blog reader you want to see

Leave comments: compliments, questions, suggestions, corrections.
Share links with your network!
Link it back on your blog.

That’s all from me, now I’ll follow my own advice and ship this post! Happy R blogging!

Edited to add: I didn’t mention Search Engine Optimization because I don’t know much about that, but wanted to add a link to the blogdown book relevant section: https://bookdown.org/yihui/blogdown/useful-resources.html#helping-people-find-your-site

Really not a fish? Scraping my mathematical family tree

Mon, 18 Jun 2018 00:00:00 +0000

It’s nearly been two years since I defended my PhD thesis! On top of allowing me to call myself doctor, having a PhD in statistics gives me the honour to feature in the data of the Mathematics Genealogy Project. Today, I decided to webscrape my mathematical ancestors.

My blogging ancestors?

Before blogging, following my own advice I googled my idea and found these similar efforts:

Nathalie Vialaneix scraped her own mathematical tree using the R XML package and drew it using the igraph package and GraphViz. My post is basically a remix of her post.
Heike Hofmann started writing a package to scrape the Mathematical Genealogy Project, and apparently abandoned it.
Thibault Vatter published a GitHub repo mixing R, Python and a tool called Scrapy to scrape the website.

I wrote my own version of the webscraping because I wanted to use rvest like I usually do these days.

Scraping the Mathematical Genealogy Project

Now that I have added tags to all my old blog posts, I can look back at my previous webscraping efforts and use my recent scripts… as well as see how much progress I made since my reckless webscraping days, when I didn’t check I was allowed to webscrape, and when I used on string manipulation rather than XPath and friends.

Nowadays before webscraping I use the rOpenSci robotstxt package to read what the website owners will let me do.

> robotstxt::robotstxt("https://www.genealogy.math.ndsu.nodak.edu")$text
User-agent: msnbot
Crawl-delay: 30

User-agent: Browsershots
Disallow: 

User-agent: *
Disallow: /submit-data.php
> robotstxt::paths_allowed("https://www.genealogy.math.ndsu.nodak.edu/id.php")

 www.genealogy.math.ndsu.nodak.edu
[1] TRUE

From the above I deduced that

there is no recommended delay for scrapers that are not msnbot, so I used 5 seconds between calls, which seemed lagom.
I was allowed to scrape mathematicians’ pages. Hooray!

Then, I looked at the code of my most recent webscraping blog post that was building on code by Bob Rudis: after looking at the source of one mathematician’s page, I used html_attrs and html_text to extract the elements I needed from each page.

.get_advisors <- function(id_string, sleep_time){
  # small break to be nice
  Sys.sleep(sleep_time)
  
  # try to get the page
  page <- glue::glue("https://www.genealogy.math.ndsu.nodak.edu/{id_string}") %>%
  httr::GET() 
  
  # try until it works but not more than 5 times
  try <- 1
  while(httr::status_code(page) != 200 & try <= 5){
    Sys.sleep(sleep_time)
    page <- glue::glue("https://www.genealogy.math.ndsu.nodak.edu/{id_string}") %>%
      httr::GET() 
    try = try + 1
  }
  
  # Now get student's name
  student_name <- httr::content(page) %>%
    rvest::xml_nodes(xpath = '//h2[@style="text-align: center; margin-bottom: 0.5ex; margin-top: 1ex"]') %>%
    rvest::html_text() %>%
    stringr::str_remove("\\\n")
  
  # Get all nodes corresponding to advisors
  # Thanks to their... formatting but it works
  all_advisors <- httr::content(page) %>%
  rvest::xml_nodes(xpath = "//p[@style='text-align: center; line-height: 2.75ex']") %>%
    rvest::html_nodes("a")
  
  # Export results
  tibble::tibble(student_name = student_name,
                 id_string_student = id_string,
                 name = purrr::map_chr(all_advisors, rvest::html_text),
                 id_string = purrr::map_chr(all_advisors, rvest::html_attr,
                                            "href"))
}

Now, since sometimes advisors will be encountered more than once in the data, I used memoise to create a handy function wrapper that will cache results.

# Cache results in case a mathematician comes up several times
get_advisors <- memoise::memoise(.get_advisors)

Finally, to get all my ancestors, I had to iteratively get the ancestors of each of my ancestors… until when? Heike Hofmann wrote a function working a given number of steps, Nathalie Vialaneix stopped when there was no advisor on an advisor’s page, I stopped when the data.frame stopped growing because that’s what I found the easiest to implement.

me <- "id.php?id=207686"
# initial data.frame
df <- get_advisors(me, 5)
new_df <- df
keep_growing <- TRUE
while(keep_growing){
  # get size to compare to size after a bit more scraping
  nrow1 <- nrow(df)
  # get advisors for all new lines 
  # from the previous iterations
  new_df <- purrr::map_df(new_df$id_string, get_advisors, sleep_time = 30)
  df <- unique(rbind(df, new_df))
  # if the data.frame didn't grow, stop
  if(nrow(df) == nrow1){
    keep_growing <- FALSE
  }
}

# save results
readr::write_csv(df, "math_ancestry.csv")

I tested the above on very ancient mathematicans who didn’t have too many ancestors to check it was working, and stopping.

At the end of my data gathering, I had a nice table of 261 mathematicians including yours truly! By the way, the Mathematical Genealogy Project maintainers state “Throughout this project when we use the word “mathematics” or “mathematician” we mean that word in a very inclusive sense. Thus, all relevant data from mathematics education, statistics, computer science, or operations research is welcome.” which is the reason why I feel fine calling myself a mathematician in this post.

Showing (off) my mathematical family tree

The approach I used below, defining nodes and edges for integration by DiagrammeR before exporting to igraph and then to GraphViz (not an R package, but interfaced by both DiagrammeR and igraph), might seem a bit complicated since the DiagrammeR package itself exports to GraphViz format… but not with the classic default look I liked on Nathalie’s Vialaneix blog.


library("magrittr")
library("DiagrammeR")

# create nodes
labels <-  unique(c(df$student_name, df$name))
nodes_df <- create_node_df(n = length(labels))
nodes_df$label <- labels

# create edges
edges_df <- df[, c("name", "student_name")]
edges_df <- dplyr::left_join(edges_df, nodes_df,
                             by = c("name" = "label"))
edges_df <- dplyr::rename(edges_df, from = id)
edges_df <- dplyr::left_join(edges_df, nodes_df,
                             by = c("student_name" = "label"))
edges_df <- dplyr::rename(edges_df, to = id)

# special character that'd make GraphViz throw an error
nodes_df <- dplyr::mutate(nodes_df,
                   label = stringr::str_replace_all(label, "'", " "))

# create a DiagrammeR dgr_graph object
dgr <- create_graph(nodes_df = nodes_df,
                    edges_df = edges_df[, c("to", "from")],
                    directed = TRUE)

# export the object to igraph format
# and then write it to a GraphViz DOT file
DiagrammeR::to_igraph(dgr) %>%
  igraph::write.graph(file = "graph.dot",
                      format = "dot")

I vaguely got the ambition to use some sort of htmlwidget to have a zoomable and pretty network but didn’t want to spent too much time doing it, so PNGs it is! Here is how I hacked my way to a PNG.

DiagrammeR::grViz("graph.dot") %>%
  htmlwidgets::saveWidget("lala.html")

webshot::webshot("lala.html",
                 selector="#htmlwidget_container",
                 file = "tree.png",
                 zoom = 10)
file.remove("lala.html")

I then cropped two zooms by hand!

So here is my whole tree…

It is unreadable unless you open it in its own tab and zoom. Like Nathalie Vialaneix, I think I only have male mathematical ancestors.

The most recent part of the tree obviously features my PhD advisor Michael Höhle, who by the way has a very smart statistics blog! I also see I’m related to CRAN’s Kurt Hornik, which doesn’t mean I get my packages on CRAN whilst bypassing gatekeeping though.

I was then very pleased to recognize some famous mathematicians’ names, not the most ancient ones I’m afraid, but look, Poisson! I was very excited about this because my PhD thesis featured count data, and because Poisson means fish in French… So maybe I’m a fish after all. Besides, seeing Carl Friedrich Gauß also made me happy… I had no idea the picture below was actually a family portrait!

Growing this post?

In conclusion, I scraped and drew my mathematical family tree using data from the Mathematical Genealogy Project. Extensions of my post could include making a package like Heike Hofmann’s one to make it easier for anyone to get their data; preparing a nicer visualization (note that you can buy posters to support the project as explained on this page), and getting more data from each mathematician’s page to draw once’s ancestry on a map, summarize thesis topics with tidy text analysis… Now, I’ll probably remain the black sheep of my mathematical family by blogging about kitsch plots!

Bad Stock Photos of My Job? Data Science on Pexels

Sun, 06 May 2018 00:00:00 +0000

I couldn’t miss the fun Twitter hashtag #BadStockPhotosOfMyJob thanks to a tweet by Julia Silge and another one by Colin Fay. The latter inspired me to actually go and look for what makes a data science photo… What characterizes “data science” stock photos?

My (not bad) stock photo source

Pexels metadata for the win

Where to find information related to stock photos? In two previous blog posts of mine I used Pexels, a website providing CC0 pictures which is quite nice. My goal was to obtain the titles and the tags of stock photos of “data science”: for instance if you look at this picture, its tags are “business”, “contemporary”, “computer”, etc. Pexels tags are very useful metadata, saving me the effort to use machine learning methods to analyse images.

Responsible webscraping

When researching this post I discovered that Pexels has an API, documented here but this API does not get you the title nor the tags associated to a picture so only webscraping could get me what I needed.

Webscraping is a powerful tool allowing one to rectangle webpages but with great power comes great responsability. Being able to scrape a webpage does not mean you are allowed to. You could get sued or your IP could get blocked. I am far from being an expert but I often read Bob Rudis’ blog where I learnt about rOpenSci’s robotstxt package that does “robots.txt file parsing and checking for R” which in plain language means it checks for you what a webpage legally allows you to do. See below,

# how I'll find pictures
robotstxt::paths_allowed("https://www.pexels.com/search")

## [1] TRUE

# where tags live
robotstxt::paths_allowed("https://www.pexels.com/photo")

## [1] TRUE

robots.txt files often also tell you how often you can hit a page by defining a “crawling delay”. Sadly Pexels robots.txt doesn’t:

robotstxt::get_robotstxt("https://www.pexels.com")

## Sitemap: https://s3.amazonaws.com/pexels/sitemaps/sitemap.xml.gz

But Bob Rudis, who was patient and nice enough to answer my questions, told me that I should probably respect the rate limit defined in Pexels API docs. “Do not abuse the API. The API is rate-limited to 200 requests per hour and 20,000 requests per month.” As I recently explained in a post on Locke Data’s blog, these days to limit rate of a function I use the very handy ratelimitr package by Tarak Shah.

limited_get <- ratelimitr::limit_rate(httr::GET,
                                      ratelimitr::rate(200, 60*60),# not more than 200 times an hour
                                      ratelimitr::rate(1, 5))#not more than 1 time every 5 seconds

Elegant webscraping

At the time of the two aforelinked blog posts I had used RSelenium to scroll down and get the download link of many pictures, but Bob Rudis wrote an elegant and cool alternative using query parameters, on which I’ll build in this post.

I first re-wrote the function to get all 15 pictures of each page of results.

get_page <- function(num = 1, seed = 1) {
  message(num)
  limited_get(
    url = "https://www.pexels.com/search/data science/",
    query = list(
      page=num,
      seed=seed
    )
  ) -> res
  
  httr::stop_for_status(res)
  
  pg <- httr::content(res)
  
  tibble::tibble(
    url = rvest::html_attr(rvest::html_nodes(pg, xpath = "//a[@class='js-photo-link']"), "href"),
    title = rvest::html_attr(rvest::html_nodes(pg, xpath = "//a[@class='js-photo-link']"), "title"),
    tags = purrr::map(url, get_tags)
  )
  
}

I re-wrote it because I needed the “href” and because it seems that the structure of each page changed a bit since the day on which the gist was written. To find out I had to write “a[@class='js-photo-link’]” I inspected the source of a page.

Then I wrote a function getting tags for each picture.

get_tags <- function(url){
  message(url)
  url <- paste0("https://www.pexels.com", url)
  res <- limited_get(url)
  httr::stop_for_status(res)
  pg <- httr::content(res)
  nodes <- rvest::html_nodes(pg, xpath = '//a[@data-track-label="tag" ]')
  rvest::html_text(nodes)
}

And finally I got results for 20 pages. I chose 20 without thinking too much. It seemed enough for my needs, and each of these pages had pictures.

ds_stock <- purrr::map_df(1:20, get_page)
ds_stock <- unique(ds_stock)
ds_stock <- tidyr::unnest(ds_stock, tags)

I got 300 unique pictures.

What’s in a data science stock photo?

Now that I have all this information at hand, I can describe data science stock photos!

Data science tags

library("ggplot2")
library("ggalt")
library("hrbrthemes")
tag_counts <- dplyr::count(ds_stock, tags, sort = TRUE)[1:10,]

dplyr::mutate(tag_counts,
              tags = reorder(tags, n)) %>% 
ggplot() +
  geom_lollipop(aes(tags, n),
                size = 2, col = "salmon") +
  hrbrthemes::theme_ipsum(base_size = 16,
                          axis_title_size = 16) +
  xlab("Tag") +
  ggtitle("300 data science stock photos",
          subtitle = "Most frequent tags. Source: https://www.pexels.com") +
  coord_flip()

So the most common tags are data, technology, business and computer. Not too surprising!

Data science scenes

Now, let’s have a look at titles that are in general more descriptive of what’s happening/present on the photo (i.e. is the computer near a cup of coffee or is someone working on it). I tried using a technique described in Julia Silge’s and David Robinson’s Tidy text mining book: “Counting and correlating pairs of words with the widyr package” described in this section of the book but it wasn’t too interesting because most correlation values were too low. One issue was probably my having too few titles: only half of pictures have titles! So I’ll resort to plotting most common bigrams, which I learnt in the Tidy text mining book as well.

stopwords <- rcorpora::corpora("words/stopwords/en")$stopWords

ds_stock %>%
  dplyr::filter(!is.na(title)) %>%
  dplyr::select(title) %>%
  unique() %>%
  tidytext::unnest_tokens(bigram, title,
                          token = "ngrams", n = 2)  %>%
  tidyr::separate(bigram, c("word1", "word2"), sep = " ",
                  remove = FALSE)  %>%
  dplyr::filter(!word1 %in% stopwords) %>%
  dplyr::filter(!word2 %in% stopwords)%>%
  dplyr::count(bigram, sort = TRUE) %>%
  dplyr::mutate(bigram = reorder(bigram, n)) %>%
  head(n = 10)%>%
ggplot() +
  geom_lollipop(aes(bigram, n),
                size = 2, col = "salmon") +
  hrbrthemes::theme_ipsum(base_size = 16,
                          axis_title_size = 16) +
  ggtitle("300 data science stock photos",
          subtitle = "Most frequent bigrams in titles. Source: https://www.pexels.com")+
  coord_flip()

So there’s a lot of holding computer happening, and these laptops are either black or white… And well Macbook Pro probably looks more professional?

Hold my laptop and watch…

my trying to find a good post conclusion! In this post, I tried to responsibly and elegantly scrape rich photo metadata from Pexels to characterize stock photos of data science. Using tags, and most common bigrams in titles, I found that data science stock photos are associated with data, business and computers; and that they often show people holding computers. Now, you’ll excuse me while I try and comfort my poor pink laptop that feels a bit too un-data-sciency.

Lintr Bot, lintr's Hester egg

Fri, 30 Mar 2018 00:00:00 +0000

Remember my blog post about automatic tools for improving R packages? One of these tools is Jim Hester’s lintr, a package that performs static code analysis. In my experience it mostly helps identifying too long code lines and missing space, although it’s a bit more involved than that. In any case, lintr helps you maintain good code style, and as mentioned in that now old post of mine, you can add a lintr unit test to your package which will ensure you don’t get lazy over time.

Now say your package has a lintr unit test and lives on GitHub. What happens if someone makes a pull request and writes looong code lines? Continuous integration builds will fail but not only that… The contributor will get to know Lintr Bot, lintr’s Hester (Easter) egg!

Lintr Bot, a lazy little thing?

Lintr Bot has a GitHub profile. I am actually responsible for its avatar, because when I discovered it a while back I offered Jim to design it based on this free image. Now, Lintr Bot has a face but if you look at its GitHub activity, you might wonder, what does it even do? It’s perfectly fine for a human to have nearly no GitHub activity, but remember that Lintr Bot lives on GitHub only, so what does it do all day?

Well Lintr Bot goes around and comments on Pull Requests in R packages, see for instance this comment. As mentioned earlier, a contributor who forgets even a slight code style mistake will be reminded of it by Lintr Bot if the package has a lintr unit test. This is quite useful!

We might also wonder how often Lintr Bot actually works? After all, since GitHub activity timeline doesn’t show comments on Pull Requests, we’re at a loss when it comes to estimating how active this bot is…

Lintr Bot’s year in review

That’s where my ghrecipes package, presented in my blog post about Bob Rudis’ R packages, will come into play. I’ve been working on a function called spy which can return either Pull Requests or Issues an user commented on during a given period of time. As creepy as it might sound, it could help you trace back all the desperate comments you wrote when debugging something last week for instance, without your having had to track them in the first place. ghrecipe::spy is a work in progress, but let’s see what it can return me for Lintr Bot!

Note that there’s no way to filter for comments between a date range, well as far as I understand the “search” functionalities of GitHub V4 API, so I can only filter Pull Requests updated in a given range.

lintrbot_act <- ghrecipes::spy(user = "lintr-bot", type = "PullRequest",
                               updated_after = "2017-03-30",
                               updated_before = "2018-03-30")
lintrbot_act <- dplyr::filter(lintrbot_act,
                              as.Date(created_at) >= anytime::anydate("2017-03-30"))

We get 228 pull requests in one year! Lintr Bot is definitely not a lazy little thing!

Here are 5 Pull Requests that got a visit from Lintr Bot.

knitr::kable(lintrbot_act[1:5,])

owner	repo	title	created_at	state	author	url	no_comments	id
PredictiveEcology	reproducible	mergeCache	2018-03-29 18:29:33	MERGED	eliotmcintire	https://github.com/PredictiveEcology/reproducible/pull/18	2	18
PredictiveEcology	SpaDES.core	change objectSize to objSize – name conflict with R.oo::objectSize	2018-03-26 23:48:22	MERGED	eliotmcintire	https://github.com/PredictiveEcology/SpaDES.core/pull/57	1	57
PredictiveEcology	reproducible	memoise – loadFromRepo	2018-03-26 17:41:25	MERGED	eliotmcintire	https://github.com/PredictiveEcology/reproducible/pull/17	3	17
PredictiveEcology	quickPlot	thin fixes – bugfixes, SpatialPolygonsDataFrames	2018-03-26 05:32:12	MERGED	eliotmcintire	https://github.com/PredictiveEcology/quickPlot/pull/13	2	13
ptl93	AEDA	added getEps	2018-03-25 17:12:44	MERGED	MiGraber	https://github.com/ptl93/AEDA/pull/39	5	39

Let’s have a look at the time series of comments.

library("ggplot2")
library("hrbrthemes")
ggplot(lintrbot_act) +
  geom_segment(aes(x = created_at, xend = created_at),
               y = 0, yend = 1) +
  theme_ipsum(base_size = 16,
              strip_text_size = 16) +
  ggtitle("Lintr Bot's work events",
          subtitle = "Each vertical line indicates Lintr Bot's commenting on a pull request")

So it seems that Lintr Bot works quite regularly, with some intense periods! I won’t look at the number of repos over time to differentiate activity due to more activity in a few repos or due to several repos, but I’m curious to see who was helped by Lintr Bot.

library("magrittr")
dplyr::group_by(lintrbot_act, owner) %>%
  dplyr::summarize(repos = toString(unique(repo)), 
                   n = n()) %>%
  dplyr::arrange(- n) %>%
  dplyr::filter(n >= 5) %>%
  knitr::kable()

owner	repos	n
RTMC	tmc-rstudio, tmc-r-tester, tmc-r	64
mlr-org	mlr	38
PredictiveEcology	reproducible, SpaDES.core, quickPlot, SpaDES.tools, SpaDES.shiny	26
Azure	doAzureParallel	16
HealthCatalyst	healthcareai-r	16
neuropsychology	psycho.R	11
ropensci	drake	11
hbc	bcbioRNASeq, bcbioSingleCell	5

I must say that besides the recently onboarded rOpenSci package drake by Will Landau, I don’t know any of these repos!

Hire Lintr Bot?

What about you? Would you let that cute little bot help you?

Note, a good package to improve code style automatically is styler by Kirill Müller and Lorenz Walthert, so if Lintr Bot has a lot to tell you, give styler::style_pkg() a try!

Storrrify #satRdayCapeTown 2018

Sat, 24 Mar 2018 00:00:00 +0000

One week ago I was in Cape Town for the local satRday conference, where I had the honor to be one of the two keynote speakers, the other one being sports analytics extraordinaire Stephanie Kovalchik (You can read Stephanie Kovalchik’s account of the conference in this blog post). It was a fantastic experience! The event was very well organized, and 100% corresponds to its description as a “one day conference packed with R goodness”. You can watch all talks on Youtube. In my talk, I presented rOpenSci onboarding system of packages and… wore a hard hat!

It’d be a bit hard for me to really write a good recap of satRday that’d do it justice! Instead, I’ll use rtweet and a bit of html hacking to storrrify it (like Storify, but in R) using my live tweets!

Getting my satRday tweets

To get my own satRday Cape Town tweets, I’ll first get my timeline and then filter tweets containing the official conference hashtag, "#SatRdayCapeTown”. I shall only keep my tweets.

my_tweets <- rtweet::get_timeline(user = "ma_salmon", n = 5000)
my_satrday_tweets <- dplyr::filter(my_tweets,
                                   stringr::str_detect(text, "[sS]at[rR]day[cC]ape[tT]own"),
                                   !is_retweet)
my_satrday_tweets <- dplyr::arrange(my_satrday_tweets, created_at)

I got 54 tweets.

Embedding tweets

I wanted to embed tweets, not to rtweet::tweet_shot them, because I wanted to display clickable links. I used the Twitter API embed endpoint after reading this webpage. This endpoint isn’t currently included in rtweet.

library("magrittr")
embed_tweet <- function(tweet_id){
  url <-glue::glue("https://twitter.com/ma_salmon/status/{tweet_id}") 
  
  httr::GET("https://publish.twitter.com/oembed",
            query = list(url = url,
                         hide_thread = "true")) %>%
    httr::content() %>%
    .$html
}

I’ll now write all html chunks in an html file.

fs::file_create("satrdaycapetown.html")
embed_and_save <- function(tweet_id){
  embed_tweet(tweet_id) 
}

purrr::map_chr(my_satrday_tweets$status_id, embed_tweet) %>%
  paste() %>%
  writeLines("satrdaycapetown.html", useBytes = TRUE)

And voilà!

I’ll include all tweets below (via a Jekyll include, with the html living in _includes, this is probably very hacky…), happy reading! Thanks again to the organizers, participants and sponsors of SatRday Cape Town for making it such an enjoyable experience!

"GitHub is a treasure trove of public activity just waiting to be analyzed" YES! More on my own 💙 for GH data at #satRdayCapeTown in a few days! 🛫 https://t.co/O7n4EIqpbj
— Maëlle Salmon 🐟 (@ma_salmon) March 13, 2018

💯 Things are getting real down here! #satRdayCapeTown https://t.co/V74HhcVZ9c
— Maëlle Salmon 🐟 (@ma_salmon) March 14, 2018

A good way to get familiar with my accent, dear #satRdayCapeTown attendees 😉 poke @DataWookie https://t.co/HOGoWIqF6I
— Maëlle Salmon 🐟 (@ma_salmon) March 15, 2018

.@theoni_p wrote a legend for the sticky notes at my #satRdayCapeTown package dev workshop! (@thecarpentries style!) 👩 🎨

The nearby @thecarpentries workshop uses pink for "confused" so a helper was scared when she came in & saw all the pink notes in our room 😂 pic.twitter.com/H0ubBpZYNF
— Maëlle Salmon 🐟 (@ma_salmon) March 16, 2018

Now I exactly know what to improve next time I teach #rstats package development in case someone wants me to do that again! 😀

Thanks #SatRdayCapeTown participants for a great day and insightful feedback! 🚦 pic.twitter.com/l6h6GCYvyx
— Maëlle Salmon 🐟 (@ma_salmon) March 16, 2018

And thanks a lot to the helpers, @jonmcalder &co who made everything run smoothly! 🌟🌟🌟 #SatRdayCapeTown https://t.co/mxVGpK8oqn
— Maëlle Salmon 🐟 (@ma_salmon) March 16, 2018

Material for my #satRdayCapeTown #rstats 📦 development workshop https://t.co/lGnaBXaHYy

Slides https://t.co/lZmUHnWzVv pic.twitter.com/YuYse37zo6
— Maëlle Salmon 🐟 (@ma_salmon) March 16, 2018

Roberto Benetto opening #SatRdayCapeTown with an overview of #rstats sf! 🌐🗺 And a pun that made everyone laugh 👏 (wait for the recording to hear it 😜) pic.twitter.com/lxRoQkDKMR
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

He's a fan of reading GitHub issues of #rstats packages to see what problems and ideas people have. So am I! 🕵️ ♂️🕵️ ♀️ #SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

⚡talks time at #SatRdayCapeTown! 🏃 ♂️🏃 ♀️
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Christopher Waspe about the flowcar package. Nothing to do with 🚗! It's about ecological networks! (No URL found/given sorry 😢) ⚡ #SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

#SatRdayCapeTown speakers please mention your Twitter handle if you have one & don't have it on your title slide 😉
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Vishalin Pillay about automatic report generation! RMarkdown 💙💚💛 #SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Maphale Matlala about testing the IUCN Red List 🚦 for Ecosystems methodology on 🇿🇦 datasets #SatRdayCapeTown using the redlistr #rstats package https://t.co/e8QhgwEcKd pic.twitter.com/yI18oNhUyg
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Slides for my #SatRdayCapeTown talk about the @rOpenSci onboarding process of #rstats packages 📦https://t.co/OOPepRT2G3

Poke co-editors @_inundata @noamross @sckottie

And here are the faces of onboarded packages! pic.twitter.com/TAXqxr21A1
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@painblogR is showing us how to make data analysis purrr with 🥕🥕🥕 data as an example #SatRdayCapeTown pic.twitter.com/sCeRMFcfCd
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@painblogR 's code https://t.co/23C6Puqxm8

And 🥕🥕🥕 dataset https://t.co/CiwKJks3im

A really neat intro to purrr, including pros and cons (nesting hell anecdote 😂) 👌#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Sean Soutar showing us how to gather data from dynamic web forms with Docker & the @rOpenSci RSelenium #rstats package https://t.co/WdTgaSzJZe ⭐ #SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Futurology! (extreme risk prediction) with Alice Elizabeth Coyne 📈🔮 #SatRdayCapeTown pic.twitter.com/2G3HPOxT1g
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Super #SatRdayCapeTown organizer-in-chief @DataWookie speaking about productivity hacks, we'd better all take notes! 🗒✏
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@DataWookie productivy hacks
✅ configure ahead
✅ ditch RStudio 😱 & use the shell/R command line instead (now I want to debate functions&pkgs vs parameterized scripts with him 😁🤺)
✅ use the ☁️#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Now listening to @realseem about his journey from Python🐍 to R, with game making as motivator 🎮

Very entertaining and enlightening lightning talk! #SatRdayCapeTown pic.twitter.com/pl7QaNSNzx
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Schalk Heunis about using R for racing! #SatRdayCapeTown
Using R to train the ConvNet of a self-driving RC 🚗

Find his project at https://t.co/eyPS2gLKnL

He mentioned
⭐ https://t.co/yzhQX9qvSe
⭐ https://t.co/M6xzQOX4rC
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Neil Rankin says satRday 2017 changed his life, he discovered useful 🛠! He e.g. mentions @JennyBryan 's https://t.co/G8vxGml00t

He's now talking about identifying bias in 🇿🇦 graduate recruitment

#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Such a fascinating talk about bias. 😮

Neil Rankin used tidy text analysis to analyze personal biographies by self assigned race (thanks to @juliasilge 's presence last year).#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Jed Stephens walked us through some scary factors data 😉 #SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Deveshnie Mudaly, SQL wiz &engaging speaker, uses a cool example in her talk about using SQL from #rstats ☄

PSA! She's looking for collaborators on a Harry Potter R package! Find her at https://t.co/oUJZ1unLH0 #SatRdayCapeTown pic.twitter.com/1pAjcC3zRC
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Unsurprisingly @rugbystatsguy is presenting about 🏈! Either put your Twitter handle on your title slide or... have one people can guess 😉

He mentioned the @rOpenSci plotly package https://t.co/zKD6YuxRv5 #SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Time for @StatsOnTheT 's keynote at #SatRdayCapeTown ! 🎾 pic.twitter.com/qPDQywiOPr
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@StatsOnTheT works for GIG that will revolutionize 🎾 through data science#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

If you are into 🎾 & #rstats check out @StatsOnTheT 's impressive package produced by her collecting tons of data from the web!https://t.co/mueGjGXyVc #SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

This shows

✅ numbers calculated by @StatsOnTheT's group (win probability computed before and during events!)

✅ one of the reasons she 💙s her job!
#SatRdayCapeTown pic.twitter.com/iDwrnGBGfF
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Now we'll get to see how @StatsOnTheT 's group gets there 👌

📽🔩#rstats ☁️🗃#SatRdayCapeTown pic.twitter.com/quQaUmES2E
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Live demo by @StatsOnTheT 😎 Here is my own metrics XML 😉

<talk>
<content>fascinating</content>
<speaking>extremely engaging</speaking>
</talk>#satRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@StatsOnTheT tips for #rstats software for real time

🎾 Think modularly

🎾 Profile bottlenecks using lineprof https://t.co/L8zfOoN7aX

🎾Embrace the #tidyverse

🎾Further optimize with e.g. datatable, parallel/foreach/other parallel processing tools, Rcpp#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@StatsOnTheT : GIG reports are generated automatically & in #rstats RMarkdown

She shows us an example of a parameterized report cf https://t.co/U6HoKDoIwJ #SatRdayCapeTown pic.twitter.com/Py5S0o7RSZ
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@StatsOnTheT's tips for real time #rstats RMarkdown

⚾️ Doc as function

⚾️ Make your summaries and plots generalizable

⚾️ Separate content from style with a CSS style sheet#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Find the material for @StatsOnTheT awesome talk here https://t.co/VvAk7wSODo 💁 ♀️

Pro tip, nagivate to the repo root https://t.co/qviYxENses for even more #rstats goodness from her workshop 👩 🏫 and her @RLadiesCapeTown talk about sports blogging 💜💁 ♀️#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

It was good promotion of sf, with nice examples! The joke wasn't about sf though.

Roberto Benetto's code lives here https://t.co/IRY46TN4RP #SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@UbuntR314 said @juliasilge's last year inspired him to use sentiment analysis!

He developed an internal package for analysing news survey data and will soon blog about it at https://t.co/XUYYSrHQbM #SatRdayCapeTown pic.twitter.com/EWcVP3yCxT
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@naasvanheerden explains his workflow at work using R packages and Shiny apps... and says the packages will be improved thanks to my workshop yesterday 😎😊#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

In his nice talk @naasvanheerden mentioned #rstats coming to the rescue... ICYMI @alice_data made a real logo for that! 😉https://t.co/zCre4ZWyGG #SatRdayCapeTown pic.twitter.com/CSYdqERUA1
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

The very energetic @SaintlyVi 🏃 ♀️ is now introducing CKAN & #OpenData portals

She mentioned the @rOpenSci ckanr #rstats package https://t.co/fWLm4YNrIj #SatRdayCapeTown pic.twitter.com/GhS4Qm4zyb
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Why did I say "mentioned"?! @SaintlyVi has now walked us through an example of #rstats @rOpenSci ckanr use & is now presenting us a typical ckanr workflow! #SatRdayCapeTown pic.twitter.com/dWyzxrs5jh
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Here is @SaintlyVi intro code to #rstats @rOpenSci ckanr https://t.co/QNonaMT6aJ 🏃 ♀️

I also liked her promotion of @RLadiesCapeTown! 💜#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@OpenSourceQuant 📈📉 on blotter https://t.co/BMjUgWvDQW

Hey it doesn't have a README nor a repo description yet 😱😜#SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@pssGuy about creating interactive plots in 5 minutes ⏱ starring #rstats 🛠

✅ the #tidyverse
✅ @rOpenSci plotly https://t.co/zKD6YuxRv5
✅ @MilesMcBain datapasta that he tried to pronounce à la 🇦🇺 😉 https://t.co/lyjnUOcKTH https://t.co/tO3ME03c9R #SatRdayCapeTown pic.twitter.com/VfrNcX8RON
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Last talk, by David Lubinsky, is about profiling using #rstats profvis! (poke @_ColinFay)https://t.co/Ol2KfeBwcQ #SatRdayCapeTown
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Why profile #rstats code according to David Lubinsky

▶️ not wasting resources in production
▶️ often easy to get improvements
▶️ R is more than a prototyping language
▶️ especially useful with Shiny#SatRdayCapeTown pic.twitter.com/l34yKrqML1
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Such an interesting, clear and enjoyable live demo of @winston_chang 's profvis package by David Lubinsky 👌https://t.co/Ol2KfeBwcQ #SatRdayCapeTown pic.twitter.com/RSnNdihuuq
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

.@realseem won a book signed by @hadleywickham thanks to his great ⚡talk this morning! Well deserved!#SatRdayCapeTown pic.twitter.com/4pVymr8wDe
— Maëlle Salmon 🐟 (@ma_salmon) March 17, 2018

Flying home today&tomorrow from #SatRdayCapeTown after a great conference, thx a ton to the organizers, participants &sponsors! 🌴✈⛄ pic.twitter.com/OQWa650oF3
— Maëlle Salmon 🐟 (@ma_salmon) March 18, 2018

Not sure if relevant but it seems that sports analytics extraordinaire @StatsOnTheT
presented such models at her #satRdayCapeTown workshop https://t.co/YyQyUKFaxP
— Maëlle Salmon 🐟 (@ma_salmon) March 19, 2018

New #rstats post! "Storrrify #satRdayCapeTown 2018"https://t.co/1HimAko2yf

Thanks again to the organizers, participants and sponsors of this great conference! pic.twitter.com/HHJjhLUUR9
— Maëlle Salmon 🐟 (@ma_salmon) March 24, 2018

hrbrpkgs: list Bob Rudis' packages

Sun, 04 Mar 2018 00:00:00 +0000

Recently I needed to count lines of code for a project at work work (this is an expression of the person honored in this post), and happened to discover that Bob Rudis had started an R package wrapping the Perl CLOC script. Of course! He has packages for a lot of things! And he’s always ready to help: after I asked him a question about the package, and made a pull request to renew its wrapped CLOC script, he made it all pretty and ready-to-go!

He himself defined his Stack Overflow Driven-Development (SODD) workflow in a blog post: someone will ask him a question on Stack Overflow, and he’ll write a long answer eventually becoming a package, that will or will not make it to CRAN… Which is the motivation of this blog post. How can I output a list of all packages Bob has on GitHub?

The magic of GitHub API V4, `ghql` and jq/`jqr`

I’ve recently started using GitHub API V4 instead of V3, that you might have seen on this blog in my post about random seeds and my post about initial commit messages. I’ll probably introduce my workflow more formally and extensively elsewhere but here are a few appetizers:

Find out why GitHub is using GraphQL. I was particularly receptive to the idea of getting just the data I needed from the API.
rOpenSci has a very good general purpose GraphQL R client, ghql by Scott Chamberlain, that includes R6 classes of course (being a package of Scott’s) and whose doc examples are for… GitHub API V4!
One can build queries in GitHub V4 explorer that has auto-completion and the docs on the right so it made, and still makes, things easier for me as a GraphQL n00b.
One can wrangle JSON using jq via the rOpenSci jqr package which is a real gem. Carl Boettiger wrote about it here, I trusted him and gave it a try and my mind was blown. Now I do not find the syntax intuitive but guess what, there’s an explorer for that too! So I could writeClipboard JSON output from the API and use the explorer to write code.

The results of my recent discoveries, and my using the API for work work stuff, can be found in my very nascent/work-in-progress ghrecipes package. By the way if you’re after something similar i.e. high-level, but for GitHub API V3, check ghapi out.

Using my own GitHub package

I first retrieved all GitHub repositories of Bob’s. My ghrecipes package iterates for you thanks to a piece of code by Scott Chamberlain.

You’ll note some things could be made easier if the output of my own functions was a bit different so you can imagine that yes, I’ll break my own package from time to time, but at least this gives a taste of what can be done.

You can read the code of my ghrecipes::get_repos function on GitHub.

library("magrittr")
repos <- ghrecipes::get_repos("hrbrmstr")
repos <- tidyr::separate(repos, name, into = c("owner", "repo"),
                         remove = FALSE, sep = "/")
# remove forks
repos <- dplyr::filter(repos, !is_fork)

I found 274 repos.

head(repos) %>%
  knitr::kable()

name	owner	repo	created_at	updated_at	description	is_fork
hrbrmstr/dbClone	hrbrmstr	dbClone	2011-04-17 03:04:54	2017-11-12 15:07:47	OS X native-port of Moloch’s dbClone utility (http://marcoramilli.blogspot.com/2011/04/dropbox-cloner.html)	FALSE
hrbrmstr/slopegraph	hrbrmstr	slopegraph	2012-05-29 01:15:20	2017-10-19 08:00:32	A ‘slopegraph’ (‘table-chart’) generator in Python using Cairo/RaphaÃ«l. Currently handles a two column chart with many output options. Look at the ‘/examples’ directory for sample configurations, data files and output formats.	FALSE
hrbrmstr/sandy	hrbrmstr	sandy	2012-10-28 13:09:36	2017-07-04 11:42:37	An example of how to plot path & cone of Hurricane SANDY in Râ€¦	FALSE
vz-risk/veris	vz-risk	veris	2013-01-04 17:19:19	2018-03-02 11:41:31	Vocabulary for Event Recording and Incident Sharing (VERIS)	FALSE
hrbrmstr/netintel	hrbrmstr	netintel	2013-02-08 17:21:15	2018-02-12 18:08:53	A collection of “network intelligence” utilities for R. ASN info, IP reputation, etc.	FALSE
vz-risk/public-data	vz-risk	public-data	2013-03-07 13:46:19	2018-02-07 20:40:36	data from publicly disclosed incidents	FALSE

Now as you see I’m nearly done since my own function returns the GitHub description of the repo (I had forgotten about that when starting to write this post!) and since Bob seems to often write a description for his repos which we should all do; but I only want packages. Fear not, I already have a recipe for this as well, is_package_repo! It looks whether a repo contains man, R, DESCRIPTION and NAMESPACE. Now I’m pretty sure the code below could be hugely simplified, but I’m in a hurry.

is_pkg <- function(df){
  message(df$name)
  tibble::tibble(owner = df$owner,
                 repo = df$repo,
                 is_pkg = ghrecipes::is_package_repo(owner, repo))
}

are_pkgs <- split(repos, repos$name) %>%
  purrr::map_df(is_pkg)

are_pkgs <- dplyr::left_join(repos, are_pkgs, by = c("owner", "repo"))
hrbrpkgs <- dplyr::filter(are_pkgs, is_pkg)

I found 170 packages whose whole table I’ll put at the very end of this post. I can’t wait to browse it!

The end?

What can you do with this list? Explore it to find the package(s) you didn’t know you needed, or the package(s) you’d like to contribute to, because I can recommend contributing to Bob’s packages, and he encourages people to!

I’ve got a GH full of diverse #rstats 📦 repos that all need tweaks/enhancements ranging from docs to ƒ()s. I’ll gladly help anyone — esp new PRers — contribute.
— hrbr'Challa (@hrbrmstr) 13 de gener de 2018

And as promised here is the table!

library("magrittr")
hrbrpkgs %>%
  # thanks https://stackoverflow.com/questions/30901027/convert-a-column-of-text-urls-into-active-hyperlinks-in-shiny
  dplyr::mutate(link = glue::glue("https://github.com/hrbrmstr/{repo}"),
                link = glue::glue("<a href='{link}'>{link}</a>")) %>%
  dplyr::mutate(created_at = lubridate::as_date(created_at),
                updated_at = lubridate::as_date(updated_at)) %>%
  dplyr::rename(package = repo) %>%
  dplyr::select(package, description, created_at, updated_at, link) %>%
  dplyr::arrange(package) %>%
  knitr::kable()

package	description	created_at	updated_at	link
adobecolor	Use Adobe swatch files as R color palettes	2015-03-20	2016-05-22	https://github.com/hrbrmstr/adobecolor
albersusa	Tools, shapefiles & data to work with an “AlbersUSA” composite projection in R	2016-03-28	2018-03-03	https://github.com/hrbrmstr/albersusa
algorithmia	R interface to the Algorithmia API	2016-07-22	2017-05-15	https://github.com/hrbrmstr/algorithmia
asam	⚓ R package providing tools to access, download, update, process and visualize Anti-shipping Activity Messages (ASAM) Database Files	2015-09-19	2017-12-29	https://github.com/hrbrmstr/asam
berate	Provide Insightful Motivation During Interactive R Sessions	2015-10-29	2017-09-10	https://github.com/hrbrmstr/berate
bismer	Generate Shakespearean Insults with R	2017-03-26	2017-07-19	https://github.com/hrbrmstr/bismer
blocklist	💂‍♂️ Tools to Work with the <blocklist.de> ‘API’ in R	2016-04-04	2016-04-04	https://github.com/hrbrmstr/blocklist
bloomsky	Tools to Work with ‘BloomSky’ Weather Station Data in R	2017-04-19	2017-12-08	https://github.com/hrbrmstr/bloomsky
bom	Tools to Identify and Work with Byte Order Marks in R	2016-10-01	2017-11-30	https://github.com/hrbrmstr/bom
burrp	🔖 Tools to Import and Process ‘PortSwigger’ ‘Burp’ Proxy Data	2017-02-18	2017-07-08	https://github.com/hrbrmstr/burrp
cdcfluview	😷 R package to Retrieve U.S. Flu Season Data from the CDC FluView Portal (WHO & ILINet)	2015-01-11	2018-01-27	https://github.com/hrbrmstr/cdcfluview
censys	R interface to the Censys “cyber”/scans search engine â€¢ https://www.censys.io/tutorial	2016-01-11	2017-12-21	https://github.com/hrbrmstr/censys
cfhttr	👷‍♂️ Workaround Cloudflare Anti-DDoS Protection	2017-12-10	2018-01-08	https://github.com/hrbrmstr/cfhttr
cloc	R package to the perl cloc script (which counts blank lines, comment lines, and physical lines of source code in source files/trees/archives)	2015-07-02	2018-02-17	https://github.com/hrbrmstr/cloc
cloudcidrs	Tools to Obtain and Work with Cloud Provider CIDR Blocks in R	2016-10-04	2016-11-06	https://github.com/hrbrmstr/cloudcidrs
crafter	🔬 An R package to work with PCAPs	2015-08-13	2018-02-04	https://github.com/hrbrmstr/crafter
curlconverter	➰ ➡️ ➖ Translate cURL command lines into parameters for use with httr or actual httr calls (R)	2016-01-28	2018-01-15	https://github.com/hrbrmstr/curlconverter
cymruservices	🐉 package that provides interfaces to various Team Cymru Services	2015-07-22	2017-05-15	https://github.com/hrbrmstr/cymruservices
czdaptools	R tools for downloading zone data from ICANN’s CZDS application	2015-10-02	2018-01-13	https://github.com/hrbrmstr/czdaptools
darksky	☁️ R interface to the Dark Sky API	2016-09-21	2017-12-29	https://github.com/hrbrmstr/darksky
databox	Tools to Work with the databox API in R	2016-08-01	2017-05-15	https://github.com/hrbrmstr/databox
ddsecblog	knitr format for ddsec blog posts	2015-10-06	2015-10-06	https://github.com/hrbrmstr/ddsecblog
decapitated	Headless ‘Chrome’ Orchestration in R	2017-05-02	2018-02-13	https://github.com/hrbrmstr/decapitated
decapitated	Chrome headless but rly websockets	2017-05-25	2018-01-08	https://github.com/hrbrmstr/decapitated
devd	Install, Start and Stop ‘devd’ Instances from R	2017-04-29	2017-05-09	https://github.com/hrbrmstr/devd
dmarc	Tools to Machinate ‘DMARC’ in R (WIP libopendmarc wrapper)	2018-02-07	2018-02-07	https://github.com/hrbrmstr/dmarc
docxtractr	✂️ Extract Tables from Microsoft Word Documents with R	2015-08-24	2018-02-17	https://github.com/hrbrmstr/docxtractr
domaintools	R API interface to the DomainTools API	2015-08-09	2016-01-01	https://github.com/hrbrmstr/domaintools
dtupdate	The dtupdate package has functions that try to make it easier to keep up with the non-CRAN universe	2014-08-21	2018-01-24	https://github.com/hrbrmstr/dtupdate
epidata	📉 Tools to Retrieve Economic Policy Institute Data Library Extracts in R	2017-01-04	2017-11-28	https://github.com/hrbrmstr/epidata
exiv	📷 Read and Write ‘Exif’ Image/Media Tags with R	2017-11-14	2018-02-17	https://github.com/hrbrmstr/exiv
fishpals	A package to generate fish color palettes	2017-09-21	2017-11-12	https://github.com/hrbrmstr/fishpals
flowdockr	R package to work with the Flowdock API	2016-02-12	2017-11-12	https://github.com/hrbrmstr/flowdockr
forceaccounted	The Force, Accounted (in R)	2016-01-08	2018-01-08	https://github.com/hrbrmstr/forceaccounted
freepst	An ‘rJava’ wrapper around ‘java-libpst’ https://github.com/rjohnsondev/java-libpst to read PST/OST files	2017-08-23	2018-02-21	https://github.com/hrbrmstr/freepst
gdns	Tools to work with the Google DNS over HTTPS API in R	2016-04-09	2017-07-10	https://github.com/hrbrmstr/gdns
getdns	ðŸ•µðŸ½â€â™€ï¸ Query Domain Name System Using the ‘getdns’ Application Programming Interface	2018-01-25	2018-01-25	https://github.com/hrbrmstr/getdns
ggalt	🌎 Extra Coordinate Systems, Geoms, Statistical Transformations & Scales for ‘ggplot2’	2015-09-08	2018-03-01	https://github.com/hrbrmstr/ggalt
ggcounty	🌐 Generate ggplot2 geom_map county maps	2014-04-17	2017-12-15	https://github.com/hrbrmstr/ggcounty
gghazard	Improved Base and Grid Plots for Survival Hazard Cox Regression	2015-08-31	2017-12-03	https://github.com/hrbrmstr/gghazard
greynoise	Query ‘GreyNoise Intelligence ‘API’ in R	2017-11-13	2018-01-29	https://github.com/hrbrmstr/greynoise
gzmem	⏫⏬ Partial resurrection of the Rcompression package since memCompress/memDecompress are brain dead	2016-09-26	2017-11-30	https://github.com/hrbrmstr/gzmem
harbor	An R package for controlling docker containers on local and remote hosts	2014-10-22	2018-02-21	https://github.com/hrbrmstr/harbor
hgr	🏃 Tools to Work with the ‘Postlight’ ‘Mercury’ ‘API’ â€” https://mercury.postlight.com/web-parser/ â€” in R	2017-04-19	2017-11-12	https://github.com/hrbrmstr/hgr
hilbert	super basic non-vectorized hilbert Rcpp example for an SO question	2017-01-15	2017-01-15	https://github.com/hrbrmstr/hilbert
hilbertv4	Create and Annotate ‘Hilbert Curve’ ‘IPv4’ Heatmaps in R	2018-02-05	2018-03-02	https://github.com/hrbrmstr/hilbertv4
hrbraddins	Additional Addins for RStudio	2017-03-12	2018-01-26	https://github.com/hrbrmstr/hrbraddins
hrbrmisc	personal R pkg	2015-08-05	2017-12-16	https://github.com/hrbrmstr/hrbrmisc
hrbrmiscpp	C/C++-backed R functions that have no real home yet	2016-06-30	2017-03-22	https://github.com/hrbrmstr/hrbrmiscpp
hrbrmrkdn	Personal R Markdown templates	2016-01-20	2016-12-06	https://github.com/hrbrmstr/hrbrmrkdn
hrbrthemes	🔏 Opinionated, typographic-centric ggplot2 themes and theme components	2017-02-11	2018-03-01	https://github.com/hrbrmstr/hrbrthemes
hrbrthemeslite	hrbrthemes w/o spell check	2017-02-27	2017-02-27	https://github.com/hrbrmstr/hrbrthemeslite
htmltidy	🚯 Tidy Up and Test XPath Queries on HTML and XML Content in R	2016-05-06	2018-02-21	https://github.com/hrbrmstr/htmltidy
hubway	R interface to the Hubway “API”	2016-07-14	2016-07-14	https://github.com/hrbrmstr/hubway
humanparser	👱‍♂️ R package to decompose full (human) names into component parts	2015-08-20	2015-11-21	https://github.com/hrbrmstr/humanparser
hyphenatr	R interface to Hunspell hyphenation	2016-03-14	2017-04-03	https://github.com/hrbrmstr/hyphenatr
infermedica	(migrated from gitlab to keep a copy in the event intermedica decides to be stupid and violate AGPL)	2018-01-26	2018-01-26	https://github.com/hrbrmstr/infermedica
ipapi	An R package to geolocate IPv4/6 addresses and/or domain names using ip-api.com’s API	2015-03-09	2017-11-06	https://github.com/hrbrmstr/ipapi
iptools	🍴 A toolkit for manipulating, validating and testing IP addresses and ranges, along with datasets relating to IP addresses. While it primarily has support for the IPv4 address space, more extensive IPv6 support is intended.	2014-08-29	2018-02-23	https://github.com/hrbrmstr/iptools
irced	☎️ Putting the “R” into IRC	2016-09-24	2017-10-25	https://github.com/hrbrmstr/irced
jasmine	simple json parser (just a parser)	2016-04-05	2016-09-12	https://github.com/hrbrmstr/jasmine
jericho	📔 Extract plain or structured text from HTML content in R	2017-09-04	2018-02-14	https://github.com/hrbrmstr/jericho
jerichojars	Java Archive Wrapper Supporting the ‘jericho’ R Package	2017-09-05	2017-11-12	https://github.com/hrbrmstr/jerichojars
Johnson	NA	2017-11-22	2017-11-22	https://github.com/hrbrmstr/Johnson
jsonview	JSON pretty printer & viewer in R	2016-01-22	2018-03-02	https://github.com/hrbrmstr/jsonview
jwatjars	Java ‘.jar’ Files for ‘jwatr’	2017-08-18	2017-11-12	https://github.com/hrbrmstr/jwatjars
jwatr	📇 Tools to Query and Create Web Archive Files Using the Java Web Archive Toolkit in R	2017-08-18	2017-11-24	https://github.com/hrbrmstr/jwatr
keybase	🏯 Tools to Work with the ‘Keybase’ ‘API’	2017-03-04	2018-02-11	https://github.com/hrbrmstr/keybase
knitrengines	An R package to collect and seamlessly add new language engines to knitr	2015-09-07	2017-11-03	https://github.com/hrbrmstr/knitrengines
kumo	wordclouds (more info soon)	2018-01-03	2018-01-03	https://github.com/hrbrmstr/kumo
kumojars	wordclouds (more info soon)	2018-01-03	2018-01-03	https://github.com/hrbrmstr/kumojars
lineworkmaps	Use Project Linework maps easily in R	2015-07-21	2017-02-21	https://github.com/hrbrmstr/lineworkmaps
localgeo	small R package for geocoding of US city/state to lon/lat	2015-01-09	2017-11-26	https://github.com/hrbrmstr/localgeo
lodes	Retrieve Data from LEHD Origin-Destination Employment Statistics Server in R	2017-01-20	2018-02-23	https://github.com/hrbrmstr/lodes
longurl	ℹ️ Small R package for the URL Expander API	2015-06-23	2017-11-12	https://github.com/hrbrmstr/longurl
maRalago	Is ðŸŠ at Mar-a-Lago	2017-04-14	2017-10-23	https://github.com/hrbrmstr/maRalago
markdowntemplates	✅🔻 A collection of alternate R markdown templates	2016-02-04	2018-03-03	https://github.com/hrbrmstr/markdowntemplates
melting5jars	MELTING 5 jars	2018-01-06	2018-01-10	https://github.com/hrbrmstr/melting5jars
metis	Helpers for Accessing and Querying Amazon Athena using R, Including a lightweight RJDBC shim	2017-05-22	2018-02-26	https://github.com/hrbrmstr/metis
metricsgraphics	📈 htmlwidget interface to the MetricsGraphics.js D3 chart library	2015-01-05	2017-12-22	https://github.com/hrbrmstr/metricsgraphics
mgrs	🌐 An R Package to Convert ‘MGRS’ (Military Grid Reference System) References To and From Other Coordiante Systems	2017-04-09	2017-10-30	https://github.com/hrbrmstr/mgrs
mhn	🍯 Analyze and Visualize Data from Modern Honey Network Servers with R	2015-08-22	2017-05-12	https://github.com/hrbrmstr/mhn
moviemeter	Tools to work with the MovieMeter API in R	2016-10-08	2016-10-09	https://github.com/hrbrmstr/moviemeter
mqtt	📞 Interoperate with ‘MQTT’ Message Brokers with R	2017-12-14	2017-12-20	https://github.com/hrbrmstr/mqtt
mrt	Tools to Retrieve and Process ‘BGP’ Files in R	2016-07-18	2017-10-19	https://github.com/hrbrmstr/mrt
msgxtractr	📇 Extract contents from Outlook ‘.msg’ files in R	2017-08-23	2018-02-24	https://github.com/hrbrmstr/msgxtractr
mtblr	R package to work with mtbls	2016-04-23	2016-04-24	https://github.com/hrbrmstr/mtblr
myip	Tools to Determine Your Public ‘IP’ Address in R	2016-07-04	2017-05-16	https://github.com/hrbrmstr/myip
mywx	Tools to Query the ‘MetMalaysia’ Web Service ‘API’	2017-08-13	2017-11-19	https://github.com/hrbrmstr/mywx
ndjson	♨️ Wicked-Fast Streaming ‘JSON’ (‘ndjson’) Reader in R	2016-09-07	2018-01-03	https://github.com/hrbrmstr/ndjson
netintel	A collection of “network intelligence” utilities for R. ASN info, IP reputation, etc.	2013-02-08	2018-02-12	https://github.com/hrbrmstr/netintel
newsflash	Tools to Work with the Internet Archive and GDELT Television Explorer in R	2017-01-26	2018-03-01	https://github.com/hrbrmstr/newsflash
newsfreq	R package to search for keyword frequencies in news articles via newsfreq.com	2015-01-31	2016-11-12	https://github.com/hrbrmstr/newsfreq
nifffty	Small R package to post events to IFTTT Maker channel/recipes	2015-06-19	2018-01-02	https://github.com/hrbrmstr/nifffty
nominatim	🌏 Tools for Working with the ‘Nominatim’ API in R	2015-07-28	2018-02-06	https://github.com/hrbrmstr/nominatim
notary	🔏📦 Signing & verification of R packages	2017-05-25	2017-12-24	https://github.com/hrbrmstr/notary
ohby	R interface to the ‘ohby’ URL & content shortener	2016-06-13	2016-06-13	https://github.com/hrbrmstr/ohby
omdbapi	R package to access the OMDB API (http://www.omdbapi.com/)	2015-06-16	2018-01-14	https://github.com/hrbrmstr/omdbapi
opengraph	Tools to Mine ‘Open Graph’-like Tags From ‘HTML’ Content	2017-12-25	2017-12-26	https://github.com/hrbrmstr/opengraph
overpass	ℹ️ Tools to Work With the OpenStreetMap (OSM) Overpass API in R	2015-08-10	2018-01-21	https://github.com/hrbrmstr/overpass
pantone	R package for Pantone colors of the year	2013-12-06	2017-11-12	https://github.com/hrbrmstr/pantone
passivetotal	Useful tools for working with the PassiveTotal API in R	2015-06-11	2017-10-04	https://github.com/hrbrmstr/passivetotal
passwordrandom	🔒 Access the PasswordRandom.com API in R	2015-08-02	2017-10-23	https://github.com/hrbrmstr/passwordrandom
pastebin	📋 Tools to work with the pastebin API in R	2017-02-05	2017-05-15	https://github.com/hrbrmstr/pastebin
pdfbox	Create, Maniuplate and Extract Data from PDF Files (R Apache PDFBox wrapper)	2017-10-22	2018-01-02	https://github.com/hrbrmstr/pdfbox
pdfboxjars	Java ‘.jar’ Files for ‘pdfbox’	2017-10-22	2017-11-14	https://github.com/hrbrmstr/pdfboxjars
pigeon	🐴 Parse Portable Game Notation (‘PGN’) Files in R	2017-09-25	2017-12-03	https://github.com/hrbrmstr/pigeon
pluralize	An R package to “Pluralize and Singularize Any Word”	2016-01-06	2017-12-25	https://github.com/hrbrmstr/pluralize
porc	🐗 Tools to Work with ‘Snort’ Rules, Logs and Data	2017-12-08	2017-12-30	https://github.com/hrbrmstr/porc
pressur	ä·® Query and Orchestrate the ‘WordPress’ ‘API’ with R	2017-12-27	2018-01-04	https://github.com/hrbrmstr/pressur
punycode	R code to work with punycode domains	2015-06-03	2017-10-23	https://github.com/hrbrmstr/punycode
qrencoder	🔳 Make QR codes in R via libqrencode	2015-08-02	2018-01-24	https://github.com/hrbrmstr/qrencoder
r7snr	R tools to work with Rapid7 Sonar scans	2016-07-15	2017-05-15	https://github.com/hrbrmstr/r7snr
radb	🔬 Tools to Query the ‘Merit’ ‘RADb’ Network Route Server	2017-04-02	2017-04-07	https://github.com/hrbrmstr/radb
radviz	R htmlwidget for radviz!	2016-03-03	2017-12-08	https://github.com/hrbrmstr/radviz
rappalyzer	🔬 :: WIP :: R port of Wappalyzer	2017-09-22	2017-10-02	https://github.com/hrbrmstr/rappalyzer
RBerkeley	🐱 Oracle ‘Berkeley DB’ Interface for R	2015-07-27	2017-08-14	https://github.com/hrbrmstr/RBerkeley
rdatainfo	helper pkg for the macOS QuickLook plugin	2016-08-06	2016-08-06	https://github.com/hrbrmstr/rdatainfo
resolv	ldns DNS resolver wrapper libary for R	2014-04-26	2017-11-16	https://github.com/hrbrmstr/resolv
Rforecastio	☁️ Simple R interface to forecast.io weather data	2013-09-08	2017-12-08	https://github.com/hrbrmstr/Rforecastio
rgeocodio	Tools to Work with the https://geocod.io/ API	2017-03-05	2017-12-25	https://github.com/hrbrmstr/rgeocodio
rosette	Tools to Work with the Rosette API in R	2016-10-10	2017-05-16	https://github.com/hrbrmstr/rosette
rpwnd	🙅‍♀️ The Most Benignly Malicious R Package on the Internet	2017-03-31	2018-01-31	https://github.com/hrbrmstr/rpwnd
safebrowsing	R interface to the XHR Google Safe Browsing “API”	2016-03-05	2016-03-05	https://github.com/hrbrmstr/safebrowsing
sanders	WIP!!! â€” Web-scraping and Web-crawling Content Parsing, Validation and Sanitization Helpers	2017-09-21	2017-11-12	https://github.com/hrbrmstr/sanders
satcat	🛰️ Tools to Work with the ‘CelesTrak’ Satellite Catalog API in R	2017-01-14	2017-12-26	https://github.com/hrbrmstr/satcat
saucy	Searching for Automorphisms in Underlying CNF, yes?	2017-12-17	2017-12-27	https://github.com/hrbrmstr/saucy
scamtracker	R pacakge interface to the BBB ScamTracker : https://www.bbb.org/scamtracker/us	2016-01-11	2017-04-08	https://github.com/hrbrmstr/scamtracker
scimple	➖➖➖🔶➖➖ Tidy Simultaneous Confidence Intervals for Multinomial Proportions	2017-03-03	2018-02-08	https://github.com/hrbrmstr/scimple
securitytxt	ðŸ” Identify and Parse Web Security Policies Files in R	2017-10-09	2017-11-12	https://github.com/hrbrmstr/securitytxt
sentimental	Tools to Work with the ‘text-processing.com’ API in R	2016-07-24	2017-05-20	https://github.com/hrbrmstr/sentimental
sergeant	💂‍♂️ Tools to Transform and Query Data with ‘Apache’ ‘Drill’	2016-06-03	2018-03-01	https://github.com/hrbrmstr/sergeant
shodan	🌑 R package to work with the Shodan API	2015-08-07	2018-02-16	https://github.com/hrbrmstr/shodan
slackr	#️⃣ A package to send webhook API messages to Slack.com channels/users from R	2014-09-04	2018-03-02	https://github.com/hrbrmstr/slackr
SnakeCharmR	SnakeCharmR - R and Python Integration	2016-05-18	2017-04-06	https://github.com/hrbrmstr/SnakeCharmR
speedtest	📐 Measure upload/download speed/bandwidth for your network with R	2017-11-10	2018-01-02	https://github.com/hrbrmstr/speedtest
spiderbar	Lightweight R wrapper around rep-cpp for robot.txt (Robots Exclusion Protocol) parsing and path testing in R	2017-08-14	2018-01-02	https://github.com/hrbrmstr/spiderbar
splashr	💦 Tools to Work with the ‘Splash’ JavaScript Rendering Service in R	2017-02-03	2018-01-03	https://github.com/hrbrmstr/splashr
ssllabs	Tools to Work with the SSL Labs API in R	2016-06-05	2016-07-25	https://github.com/hrbrmstr/ssllabs
sslsaran	Tools to Work with Certificate Transparency (‘CT’) Logs and Various ‘CT’ ‘APIs’	2018-02-18	2018-02-20	https://github.com/hrbrmstr/sslsaran
statebins	Alternative to choropleths of US States ala http://bit.ly/statebins	2014-08-26	2018-03-01	https://github.com/hrbrmstr/statebins
streamgraph	〰️ htmlwidget for creating streamgraph visualizations in R	2015-02-12	2018-01-12	https://github.com/hrbrmstr/streamgraph
stringore	Tidy Regular Expression Operations with Extensive Character Encoding Support	2018-01-01	2018-01-03	https://github.com/hrbrmstr/stringore
swatches	ðŸŽ¨ Read, Inspect, and Manipulate Color Swatch Files	2015-03-21	2018-02-02	https://github.com/hrbrmstr/swatches
taucharts	📊 An R htmlwidget interface to the TauCharts javascript library	2015-08-03	2017-09-08	https://github.com/hrbrmstr/taucharts
terminator	Rcpp implementation of day/night terminator generator	2018-01-25	2018-01-27	https://github.com/hrbrmstr/terminator
threatcrowd	R tools to work with the ThreatCrowd API	2016-01-20	2017-08-18	https://github.com/hrbrmstr/threatcrowd
tidyweb	Easily Install and Load Modern Web-Scraping Packages	2017-07-04	2018-02-21	https://github.com/hrbrmstr/tidyweb
tinyjs	tinyjs R wrapper pkg	2016-04-05	2016-10-29	https://github.com/hrbrmstr/tinyjs
tlsobs	🔭 Tools to Work with the ‘Mozilla’ ‘TLS’ Observatory ‘API’ in R	2017-05-14	2017-12-11	https://github.com/hrbrmstr/tlsobs
triage	Tools to Aid in Debugging Issues Across R Sessions	2017-11-23	2017-11-23	https://github.com/hrbrmstr/triage
uaparserjs	Parse Browser ‘User-Agent’ Strings into R Data Frames	2016-08-04	2018-02-08	https://github.com/hrbrmstr/uaparserjs
urltools	Elegant URL handling in R	2014-12-07	2018-03-03	https://github.com/hrbrmstr/urltools
vegalite	R ggplot2 “bindings” for Vega-Lite	2016-02-23	2018-03-01	https://github.com/hrbrmstr/vegalite
verisr	R package for working with VERIS data	2013-06-03	2017-07-14	https://github.com/hrbrmstr/verisr
verisr	R package for working with data stored within VERIS framework	2013-09-20	2018-02-16	https://github.com/hrbrmstr/verisr
voteogram	U.S. House and Senate Voting Cartogram Generators in R	2017-05-06	2017-12-25	https://github.com/hrbrmstr/voteogram
waffle	🍁 Make waffle (square pie) charts in R	2015-03-18	2018-03-01	https://github.com/hrbrmstr/waffle
wand	🎩 R interface to libmagic - returns file mime type	2016-08-12	2018-01-24	https://github.com/hrbrmstr/wand
warc	📇 Tools to Work with the Web Archive Ecosystem in R	2016-09-07	2018-02-18	https://github.com/hrbrmstr/warc
wayback	⏪ Tools to Work with the Various Internet Archive Wayback Machine APIs	2017-02-26	2018-01-28	https://github.com/hrbrmstr/wayback
webhose	🔨 Tools to Work with the ‘webhose.io’ ‘API’ in R	2017-09-27	2017-12-29	https://github.com/hrbrmstr/webhose
whoisxmlapi	❔ R package to interface with the WhoisXMLAPI.com service	2015-08-09	2015-08-09	https://github.com/hrbrmstr/whoisxmlapi
wondr	Tools to Work with there CDC WONDER API in R	2016-10-04	2017-12-29	https://github.com/hrbrmstr/wondr
xlsxtractr	Extract Things From Excel (xlsx) Files in R	2016-09-16	2018-02-10	https://github.com/hrbrmstr/xlsxtractr
xmlview	📃 Format, Query and Pretty Print ‘HTML’/‘XML’ Content in R (RStudio viewer or browser)	2016-01-12	2017-11-12	https://github.com/hrbrmstr/xmlview
xslt	lightweight XSLT processing package for R based on xmlwrapp	2015-07-08	2017-11-12	https://github.com/hrbrmstr/xslt
zellingenach	A visual exploration of the spatial patterns in the endings of German town and village names in R	2016-01-03	2017-08-09	https://github.com/hrbrmstr/zellingenach
zkcmd	Tools to Administer ‘Zookeeper’ Instances with Four-letter Commands in R	2017-01-01	2017-01-03	https://github.com/hrbrmstr/zkcmd
zoneparser	A V8-powered R package to parse Domain Name System (DNS) zone files	2015-10-01	2015-10-01	https://github.com/hrbrmstr/zoneparser

Combine your hex stickers with magic(k)

Thu, 22 Feb 2018 00:00:00 +0000

Hex stickers remind me of Pogs, except they’re cooler because you can combine them together! Some people do that very smartly.

Now when I forget how to do an #rstats analysis I can just check the back of my laptop #runconf16 pic.twitter.com/DY7UgeBzYV
— David Robinson (@drob) 1 d’abril de 2016

I’ve got a pretty random hex stickers combination on my laptop, but after all it could be worse…

My boyfriend decided he is post-hex / post-#tidyverse pic.twitter.com/ADtyORMxHL
— Hilary Parker (@hspter) 13 de setembre de 2017

Now since I’m a magick/collage fan, you can bet I’ve wondered how to use R in order to combine stickers automatically! Say I have a bunch of sticker PNGs, how could I produce a map to design my laptop style? Read to find out more…

Getting some hex stickers to play with

I do agree that in real life you’ll most often not have PNGs of the stickers but hey I can dream… especially since I’m sure that one will soon be able to identify stickers on a conference haul pic in the blink of an eye with the Rvision package. I got stickers from hex.bin whose existence I learnt from this tweet:

A plea to my favorite #rstats 📦 developers: please submit your BIG hex sticker image files (with vector versions 🙏) to https://t.co/7KWg3d2fZA! @old_man_chester @nj_tierney @samfirke they will be right at home 👇 pic.twitter.com/AIqgnBIx1X
— Alison Hill (@apreshill) 21 de febrer de 2018

And well if more R packages are added to it my Rvision dream could come true.

Note: When I say I got them from hex.bin, I mean I lazily downloaded the “hexagons/” folder of this GitHub repository by using a link as explained in this SO answer.

Combining a few of them at random

I’ll sample 42 stickers from my nice little internet haul.

library("magrittr")
set.seed(42)
ultimate_sample <- fs::dir_ls("hexagons") %>%
  sample(size = 42)

After that I’ll create rows which is pretty straightforward.

no_row <- 6
no_col <- 7

row_paths <- split(ultimate_sample, rep(1:no_col, each = no_row))

# let's be crazy

read_append <- . %>%
  magick::image_read() %>%
  magick::image_append()

rows <- purrr::map(row_paths, read_append)

Now rows have to be combined together, not just stacked, with a small shift. This will depend on the size of pics. They seem to be the same size which is great. Note that the thing that gets displayed below is what magick::image_info gives you about any magick object, so I’ll use that.

magick::image_read(ultimate_sample[1:10])
info <- magick::image_read(ultimate_sample[1]) %>%
  magick::image_info()
height <- info$height
width <- info$width

The col parameter below is my laptop background hex code. I needed it before pasting the hex stickers.

my_laptop <- magick::image_blank(width = width * no_col,
                    height = height * no_row * 0.9,
                    col = "#FF6987")

for(i in 1:no_row){
  if(i/2 == floor(i/2)){
    offset1 <- 0
  }else{
    offset1 <- (width/2) 
  }
  
  offset2 <- (i-1)*(height*0.75)
  
  my_laptop <- magick::image_composite(my_laptop, rows[[i]],
                                       offset = paste0("+", offset1,
                                                       "+", offset2))
}

magick::image_write(my_laptop, "data/my_laptop.png")

This looks fine, the only reason why it’s not perfect is probably the stickers not having exactly the same dimensions. Moreover, one of them doesn’t have transparent borders! That’s a mean design. Jeroen Ooms told me I could correct it by smartly using image_fill with color = “transparent” and some fuzz and image_trim before that to remove margins but I was lazy.

Now, what could one do? One could order hex stickers by color using the code in my blog post about rainbowing a set of pictures. Or one could just combine all hex stickers together, then make them gray and use the resulting image as a place to collect stickers. #goals

magick::image_convert(my_laptop, colorspace = "Gray") %>%
  magick::image_write("data/my_laptop_goals.png")

Galentine's day cards

Fri, 26 Jan 2018 00:00:00 +0000

Remember the nascent series of blog posts about Parks and recreation? Well, we’re still at one post, but don’t worry, here is a new one, and I’m sure the series will eventually be a real one. I’m looking at you, my R-Ladies friends. That said, today is not a day for passive agressive hints, because I’ve decided it’s Galentine’s day and I’ll show you how to craft cards for your R-Ladies friends from your R prompt!

Galentine’s day is a celebration Leslie Knope throws every year for her female friends, showering them with love and gifts. Since she’s very talented, she can even write them poems, but in the R version of Galentine’s day, we’ll make do with simple cards featuring Lesliesque compliments. Indeed, Leslie Knope is also known for her creative complimenting of her best friend Ann. I’ve decided to adopt her tone by using three praising adjectives followed by an animal name, and to dedicate them to imaginary friends called Ann, Donna, April and Shauna. Had I not decided to steal names from the TV show, I could have used the charlatan::ch_name() from the rOpenSci charlatan package, possibly extracting first names thanks to the humaniformat package.

Create the compliments

Here I need animal names and praising adjectives for a pretty frivolous use case. Now sometimes you could want to randomly draw words to make identifiers for, say, participants in your epidemiological study because that’d be easier to deal with than numbers. The ids package actually provides human readable identifiers in the style <adjective>_<animal> (following gfycat.com) but sadly that adjective wouldn’t necessary be positive, so I’ll use the praise package to get adjectives and the rcorpora package to get animal names. I’ll set the random seed to ensure reproducibility, not forgetting to be careful about where to generate it.

set.seed(123)

compliment <- function(name, animal){
  praise::praise(paste0(name, ", you ${adjective}, ${adjective}, ${adjective} ", animal))
}

compliments <- tibble::tibble(name = c("Ann", "April", "Donna", "Shauna"),
                              animal = sample(rcorpora::corpora(which = "animals/common")$animals, size = 4),
                              compliment = purrr::map2_chr(name, animal, compliment))

Create the cards

I first thought I could use cute animal pics for the cards, my husband even suggested I use the animal from the compliment which I could have done using Pexels as in my bubblegum puppies post or my rainbowing post, using the gist posted by Bob Rudis to avoid using RSelenium, but I decided to go a simpler route (which I’ll call using Kasia’s strategy to get stuff done), generating random colours, and putting text on a single-colour background. Let’s assume Leslie Knope would have written a poem on the other side, which you could do using Katie Jolly’s approach. Anyway, how did I generate random colours? Well, using charlatan! Really, a quite useful package by Scott Chamberlain, peer-reviewed by Brooke Anderson and TJ Mahr at rOpenSci onboarding repo.

set.seed(123)
compliments <- dplyr::mutate(compliments, 
                             background = charlatan::ch_hex_color(n = 4), 
                             text_colour = charlatan::ch_hex_color(n = 4))

And then I could use magick! I’ve convinced myself that although I use magick for non serious stuff, I could use these skills on scientific image data one day. Note that I could have packed the whole code of the post in a single function, but I wanted to progressively build the card (compliments, then colours, then the image).

library("magrittr")

create_card <- function(who, compliments){
  compliments <- dplyr::filter(compliments, name == who)
  magick::image_blank(600, 400,
                      color = compliments$background) %>%
    magick::image_annotate(text = compliments$compliment,
                           color = compliments$text_colour,
                           location = "+50+100",
                           boxcolor = "white",
                           size = 20) %>%
    magick::image_annotate(text = "Love, Leslie",
                           color = compliments$text_colour,
                           location = "+200+300",
                           boxcolor = "white",
                           size = 20)%>%
    magick::image_annotate(text = "HAPPY GALENTINE'S DAY!",
                           boxcolor = compliments$text_colour,
                           location = "+100+200",
                           color = "white",
                           size = 20) %>%
    magick::image_write(paste0("data/galentines", who, ".png"))
}

purrr::walk(compliments$name, create_card, compliments)

Yes, I signed the cards “Leslie”. I know the text could be centered more but I don’t really care.

Display the cards

In real life, you’d print and send these cards to your friends, but since my imaginary friends Ann, April, Donna and Shauna probably read my blog, I’ll let the cards here for them to see!

Not too shabby for random text and colours!

Rainbowing a set of pictures

Sun, 07 Jan 2018 00:00:00 +0000

I’ve now done a few collages from R using magick: the faces of #rstats Twitter, We R-Ladies with Lucy D’Agostino McGowan, and a holiday card for R-Ladies. The faces of #rstats Twitter and holiday card collages were arranged at random, while the We R-Ladies one was a mosaic forming the R-Ladies logo. I got the idea to up my collage skills by trying to learn how to arrange pics by their main colour, like a rainbow. The verb rainbow doesn’t exist, and “rainbowing” doesn’t mean ordering by colour, but I didn’t let this stop me.

It was the occasion to grab some useful knowledge about colours, not useless for someone who did not even know about Pantone’s Colors of the Year a few weeks ago…

This post has nothing to do with Kesha’s new album. However, you can listen to it while reading since it’s so good, but maybe switch to something older from her when I use “$”.

Getting some pics to play with

The first pictures I tried to arrange were all the pictures ever posted by R-Ladies local chapters on their Twitter account. While it was fun to grab them all, it was not very interesting to play with them as so many of them were pictures of screens. I therefore grabbed “nature” pictures from Pexels using the same method [as when creating the Bubblegum Puppies] following this Stack Overflow thread. I chose “nature” as a keyword because 1) it lead to many hits 2) it offered a good variety of colours.

library("rvest")
library("RSelenium")
library("magrittr")

rD <- rsDriver()
remDr <- rD[["client"]]


# open the webpage
remDr$navigate("https://www.pexels.com/search/nature/")

# scroll down
for(i in 1:130){      
  remDr$executeScript(paste("scroll(0,",i*10000,");"),
                      args = list("dummy"))
  # be nice and wait
  Sys.sleep(1)    
}

# https://www.pexels.com/faq/

page_content <- remDr$getPageSource() 
remDr$close()

get_link_from_src <- function(node){
  xml2::xml_attrs(node)["src"] %>%
    as.character() %>%
    stringr::str_replace("\\?h.*", "")
  
}

xtract_pic_links <- function(source) {
  css <- '.photo-item__img'
  read_html(source[[1]]) %>%
    html_nodes(css) %>%
    purrr::map_chr(get_link_from_src)    
}

links <- xtract_pic_links(page_content)
links <- links[1:1400]

# save
dir.create("nature")
save_pic <- function(url){
  Sys.sleep(1)
  name <- stringr::str_replace(url, ".*\\/", "")
  
  try(magick::image_read(url) %>%
        magick::image_write(paste0("nature/", name)),
      silent = TRUE)
}

purrr::walk(links, save_pic)

Extracting the main colour and making pics size-compatible

In the following code, I extracted the main colour from each pic using Russel Dinnage’s method as presented in this blog post from David Smith. Before that I had to install two packages from Github, rblocks and rPlotter.

This code also serves another role: since I wanted to paste pics together at some point, I decided to make them all of the same dimensions by adding a border with magick. I had learnt how to do that when preparing R-Ladies Global holiday card, but this time instead of using the same colour every time (R-Ladies’ official purple), I used the main colour I’d just extracted. The most important points to make a picture a square are to know magick::image_info gives you the height and width of an image… and to somehow understand geometry which was embarrassingly a hurdle when I did that.

The code to extract colours didn’t work in a few cases which I did not investigate a lot: I had downloaded more pics than what I needed because I had experienced the issue when working with R-Ladies meetups pics, and had seen it was because of seemingly bicolor pics.

dir.create("formatted_pics")

format_image <- function(path){
  image <- magick::image_read(path)
  info <- magick::image_info(image)
  
  # find in which direction I need to add pixels
  # to make this a square
  direction <- ifelse(info$height > info$width,
                      "height", "width")
  scale_number <- as.numeric(info[direction]/500)
  image <- magick::image_scale(image, paste0(info["width"]/scale_number,
                                             "x", 
                                             info["height"]/scale_number))
  newinfo <- magick::image_info(image)
  
  # colours
  colours <- try(rPlotter::extract_colours(path, num_col = 1), silent = TRUE)
  
  # one pic at least was problematic 
  if(!is(colours, "try-error")){
    colour <- colours[1]
    
    image <- magick::image_border(image, colour, paste0((500-newinfo$width)/2, "x",
                                                        (500-newinfo$height)/2))
    info <- magick::image_info(image)
    # odd numbers out!
    if(info$height/2 != floor(info$height/2)){
      image <- magick::image_crop(image, "0x500+0")
    }
    if(info$width/2 != floor(info$width/2)){
      image <- magick::image_crop(image, "500x0+0")
    }
    magick::image_write(image,
                        stringr::str_replace(path, "nature", "formatted_pics"))
    tibble::tibble(path = path,
                   colour = colour)
  }else{
    NULL
  }
  
  
  
}

pics_main_colours <- purrr::map_df(dir("nature", full.names = TRUE), format_image)
readr::write_csv(pics_main_colours, path = "pics_main_colours.csv")

And because I’m apparently a bad planner, I had to reduce pictures afterwards.

# we need smaller images
reduce_image <- function(path){
  magick::image_read(path) %>%
    magick::image_scale("50x50!") %>%
    magick::image_write(path)
}

purrr::walk(dir("formatted_pics", full.names = TRUE),
            reduce_image)

Preparing a function to order and paste pictures

This function has a collage part which you might recognize from my the faces of #rstats Twitter blog post, and a ordering pictures according to a variable part that’s new and uses a bit of tidy eval… Maybe I’ll really learn tidy eval this year! pics_info needs to be a data.frame with the path to pictures and well the variable one wants to use to order them.


library("rlang")

make_column <- function(i, files, no_rows){
  magick::image_read(files[(i*no_rows+1):((i+1)*no_rows)]) %>%
    magick::image_append(stack = TRUE) %>%
    magick::image_write(paste0("cols/", i, ".jpg"))
}


make_collage <- function(pics_info, no_rows, no_cols, ordering_col){
  pics_info <- dplyr::arrange(pics_info, !!!syms(ordering_col))
  pics_info <- pics_info[1:(no_rows*no_cols),]
  pics_info <- dplyr::mutate(pics_info, column = rep(1:no_cols, each = no_rows))
  pics_info <- dplyr::group_by(pics_info, column) %>%
    dplyr::arrange(!!!syms(ordering_col)) %>%
    dplyr::mutate(row = 1:no_rows) %>%
    dplyr::ungroup()
  
  pics_info <- dplyr::arrange(pics_info, column, row)
  
  dir.create("cols")
  purrr::walk(0:(no_cols-1), make_column, files = pics_info$path,
              no_rows = no_rows)
  
  banner <- magick::image_read(dir("cols/", full.names = TRUE)) %>%
    magick::image_append(stack = FALSE) 
  
  unlink("cols", recursive = TRUE)
  
  return(banner)
  
}

The function returns a magick object that one can then write to disk as a PNG for instance.

I first tested it using a random approach added to the data.frame created in the next section, and show the result here to give an idea of the variety of pictures. For many of them, however, the main colour that you can see in their border is greyish.

set.seed(42)
pics_info <- dplyr::mutate(pics_info, random = sample(1:nrow(pics_info), nrow(pics_info)))
make_collage(pics_info, 19, 59, "random") %>% 
  magick::image_write("data/2018-01-07-rainbowing-banner_random.png")

Testing a first (bad) approach: using hue

Once I had the main colour as an hex code, I had no idea how to order the colours and thought a good idea would be to use hue, which is the main wave length in a colour. Most observed colours are a mix of wave lengths unless you’re using a laser for instance. To get hue from a colour identified by its hex code, one needs two functions: colorspace::hex2rgb and grDevices::rgb2hsv. The latter one outputs hue, saturation and value. Hue is the main wavelength, saturation the amount of that wavelength in the colour and value the amount of light in the colour. The smaller the saturation, the less representative the hue is of the main colour. Add to that the fact that the main colour can also be only a little representative of your original picture… Ordering by hue isn’t too perfect, but I tried that anyway.

# now work on getting the hue and value for all pics
# create a data.frame with path, hue, value 
get_values <- function(path, pics_main_colours){
  print(path)
  # get main color
  colour <- pics_main_colours$colour[pics_main_colours$path == stringr::str_replace(path,
                                                                      "formatted_pics",
                                                                      "nature")]
  
  # translate it
  rgb <- colorspace::hex2RGB(colour)
  values <- grDevices::rgb2hsv(t(rgb@coords))
  
  tibble::tibble(path = path,
                 hue = values[1,1],
                 saturation = values [2,1],
                 value = values[3,1])
}

# all values
pics_col <- purrr::map_df(dir("formatted_pics", full.names = TRUE),
                          get_values, pics_main_colours)
                          



make_collage(pics_info, 19, 59, "hue") %>% 
  magick::image_write("banner_hue.png")

So this is not too pretty. Blue and green pictures seem to cluster together but there are very dark pictures which we’d intuitively put aside.

So I thought a bit more and decided to first assign main colours to a reference colour and then order pictures based on this…

Choosing a better approach: RGB and distances

The first challenge was to choose reference colours which’d be a rainbow slices. I could have looked up RGB codes corresponding to ROYGBIV (red, orange, yellow, green, blue, indigo and violet.) but I had read about xkcd colors survey in this interesting post and therefore decided to use XKCD colors, available in R via the xkcdcolors package. I chose to use the 18 most common ones, and add black to that lot. It was no longer really a rainbow, I agree. The colors present in the pictures were ordered by hand by my husband, and I like his choices.

Then to assign each pic to a reference colour via its main colour, I calculated the Euclidian distance between that colour and all reference colours to find the closes reference colours, using the RGB values.

library("xkcdcolors")
library("magrittr")
# version of colorspace::hex2RGB returning a df
hex2rgb <- function(hex){
  result <- colorspace::hex2RGB(hex)@coords
}

# https://stackoverflow.com/questions/45328221/unnest-one-column-list-to-many-columns-in-tidyr
colors <- tibble::tibble(name = c(xcolors()[1:18], "black"),
                         hex = name2color(name),
                         rgb = purrr::map(hex, hex2rgb)) %>% 
  dplyr::mutate(rgb = purrr::map(rgb, tibble::as_tibble)) %>%
  tidyr::unnest()

# for each colour I want the closest one.
find_closest_colour <- function(hex, colors){
  test <- tibble::tibble(hex = hex) %>% 
    dplyr::mutate(rgb = purrr::map(hex, hex2rgb),
                  rgb = purrr::map(rgb, tibble::as_tibble)) %>%
    tidyr::unnest() 
  
  distance <- stats::dist(rbind(test[, c("R", "G", "B")],
                                colors[, c("R", "G", "B")]))
  
  colors$name[which.min(as.matrix(distance)[,1][2:(nrow(colors) + 1)])]
}

imgs_col <- dplyr::mutate(pics_main_colours,
                          xkcd_col = purrr::map_chr(colour, find_closest_colour,
                                                    colors = colors))


readr::write_csv(imgs_col, path = "imgs_xkcd_col.csv")

Once I had this information about each pic, I could order the pictures, after having defined the order of the reference colours.

pics_info <- readr::read_csv("imgs_xkcd_col.csv")
pics_info <- dplyr::mutate(pics_info, 
                           path = stringr::str_replace(path, "nature", "formatted_pics"))

pics_info <- dplyr::mutate(pics_info,
                           xkcd_col = factor(xkcd_col, ordered = TRUE,
                                             levels = c("black","brown","red","magenta","pink",
                                                        "lime green","green","dark green","teal",
                                                        "light blue","sky blue","blue","purple","grey")))

This looks much better, but of course the initial set (and maybe the used extraction method as well) don’t provide for enough colours to make this extremely pretty. I’m not sure how useful this end product is, but hey I got to look at pretty landscapes full of colours from my grey rainy city, and learnt a lot along the way… Besides, maybe you will find a cool use case of some of the colour methods featured here and will tell me about it in the comments?

Cheer up, Black Metal Cats! Bubblegum Puppies

Thu, 04 Jan 2018 00:00:00 +0000

Do you know the Black Metal Cats Twitter account? As explained in this great introduction, it “combines kitties with heavy metal lyrics”. I know the account because I follow Scott Chamberlain who retweets them a lot, which I enjoy as far as one can enjoy such a dark mood. Speaking of which, I decided to try and transform Black Metal Cat tweets into something more positive… The Bubblegum Puppies were born!

Getting Black Metal Cats tweets

It won’t come as a surprise for the loyal readers of this blog that I just had to use rtweet. I kept only original standalone tweets and removed the picture link from the tweet.

black_tweets <- rtweet::get_timeline("evilbmcats")
black_tweets <- dplyr::filter(black_tweets, is.na(reply_to_user_id), !is_retweet, !is_quote)
black_tweets <- dplyr::select(black_tweets, text, created_at, status_id)
black_tweets <- dplyr::mutate(black_tweets, text = stringr::str_replace(text, "https.*", ""))
readr::write_csv(black_tweets, path = "data/2018-01-03-bubblegumpuppies_cats.csv")

Now that the dark material is ready, let’s sweeten it…

Modifying the tweet text

Black Metal Cats tweet heavy metal lyrics so as you can imagine, they’re sad. How to make them happy while keeping the text similar enough to the original one? And this without too much effort? My simplistic strategy was to identify negative words via sentiment analysis and to replace them with positive words.

Note, had I not wanted to stay close to the original tweet, I could have just chosen lyrics picked from this dataset for instance, and filtered them by sentiment via the sentimentr package.

Finding negative words

I computed sentiment using tidytext copy-pasting code from this post of mine.

library("tidytext")
library("magrittr")
bing <- get_sentiments("bing")

sentiment <- dplyr::select(black_tweets, text) %>%
  dplyr::mutate(saved_text = text) %>%
  unnest_tokens(word, text) %>%
  dplyr::inner_join(bing, by = "word") %>%
  dplyr::filter(sentiment == "negative")

It was a bit disappointing since out of the 97 only 63 were represented in that data.frame. But well, this shall do! I just looked rapidly at some non included tweets.

dplyr::filter(black_tweets, !text %in% sentiment$saved_text) %>%
  head() %>%
  knitr::kable()

text	created_at	status_id
Black covfefe cats.	2017-12-29 21:01:06	946848575446663168
Black coffee cats.	2017-12-28 00:15:48	946172798203916288
Say goodbye to the light.	2017-12-26 15:50:00	945683121336344576
The storm will never end.	2017-12-23 15:49:00	944595705204678656
Satan, your time has come. This world must end.	2017-12-18 22:36:00	942886191107575808
Timely.	2017-12-13 00:43:30	940743951287504896

Ok so some of them are probably negative tweets that the sentimentr package would help detect but that do not contain negative words.

Replacing words

I seriously considered using the wordnet package because of this Stack Overflow question “Getting antonyms using the wordnet package” but I was not brave enough, my strength failed me in front of the Java needs of that package.

I decided to use praise to get positive words, and cleanNLP (as in this post) to try and identify correctly negative words as adjective or verbs for instance in order to be able to replace them. The right annotation for that is a token.

library("cleanNLP")
init_spaCy()

get_token_with_text <- function(x){
  obj <- run_annotators(x, as_strings = TRUE)
  entities <- get_token(obj)
  entities$text <- x
  entities
}

possibly_get_tokens <- purrr::possibly(get_token_with_text,
                                         otherwise = NULL)

tokens <- purrr::map_df(sentiment$saved_text, possibly_get_tokens)
head(tokens) %>%
  knitr::kable()

id	sid	tid	word	lemma	upos	pos	cid	text
1	1	1	We	-PRON-	PRON	PRP	0	We are the creatures you fear.
1	1	2	are	be	VERB	VBP	3	We are the creatures you fear.
1	1	3	the	the	DET	DT	7	We are the creatures you fear.
1	1	4	creatures	creature	NOUN	NNS	11	We are the creatures you fear.
1	1	5	you	-PRON-	PRON	PRP	21	We are the creatures you fear.
1	1	6	fear	fear	VERB	VBP	25	We are the creatures you fear.

Once here, I joined sentiment and tokens.

tokens <- dplyr::mutate(tokens, word = enc2native(word))
tokens <- dplyr::mutate(tokens, word = tolower(word))
tokens <- dplyr::left_join(sentiment, tokens, by = c("saved_text" = "text", "word"))
head(tokens) %>% knitr::kable()

saved_text	word	sentiment	id	sid	tid	lemma	upos	pos	cid
We are the creatures you fear.	fear	negative	1	1	6	fear	VERB	VBP	25
In darkness we walk.	darkness	negative	1	1	2	darkness	NOUN	NN	3
Standing tall, side by side, we shall build thy throne upon the burning ashes.	burning	negative	1	1	15	burn	VERB	VBG	64
This is the dawn of the infernal reign.	infernal	negative	1	1	7	infernal	ADJ	JJ	24
A breath of the past so distant and so unreal. A soul condemned to haunt a frozen burial ground.	condemned	negative	1	2	3	condemn	VERB	VBD	54
A breath of the past so distant and so unreal. A soul condemned to haunt a frozen burial ground.	condemned	negative	1	2	3	condemn	VERB	VBD	54

The praise package provides adjectives that I’ll use to replace adjectives and to add an exclamation at the beginning of each text. Nouns, present and preterit verbs will be replaced with love/loves/loved because hey, this is pop music inspiration. I’ll lose capital letters and won’t bother too much, puppies probably do not care either. What I prepare below is what Hilary Parker called a dictionary table in this tweet. VBZ is for instance a verb in the present form like “haunts”.

modifiers <- tibble::tibble(pos = c( "NN",  "VBG", "JJ", 
                                     "VBD", "VB",  "NNS", 
                                     "VBP", "VBN", "NNP", "VBZ"),
                            modifier = c("love", "${adjective}", "${adjective}",
                                         "loved", "love", "lovers",
                                         "love", "${adjective}",
                                         "love", "loves"))
knitr::kable(modifiers)

pos	modifier
NN	love
VBG	${adjective}
JJ	${adjective}
VBD	loved
VB	love
NNS	lovers
VBP	love
VBN	${adjective}
NNP	love
VBZ	loves

Now let’s get to work on the sweetening of the texts at last! Some occurrences of “hate” and “evil” remained, and I removed them by hand.

# praise has some randomness, let's make it reproducible
set.seed(42)

modifiable_tweets <- dplyr::left_join(tokens, black_tweets, 
                                      by = c("saved_text" = "text"))

modifiable_tweets <- dplyr::left_join(modifiable_tweets, modifiers, 
                                      by = "pos")

# for easier use with map
replace_all <- function(pattern, replacement, x){
  stringr::str_replace_all(x, pattern = pattern,
                           replacement = replacement)
}

modified_tweets <- dplyr::group_by(modifiable_tweets, saved_text) %>%
  dplyr::summarize(praise_template = purrr::map2_chr(word, modifier, replace_all,
                                                  x = tolower(saved_text[1]))[1],
                   praise_template = paste("${exclamation}!", praise_template),
                   praise_template = stringr::str_replace_all(praise_template,
                                                              "hate", "love"),
                   praise_template = stringr::str_replace_all(praise_template,
                                                              "evil", "love"),
                   new_text = praise::praise(praise_template)) 

modified_tweets <- dplyr::select(modified_tweets, saved_text, new_text)

puppies_and_cats <- dplyr::left_join(modified_tweets, black_tweets, by = c("saved_text" = "text"))
head(puppies_and_cats) %>% knitr::kable()

saved_text	new_text	created_at	status_id
A breath of the past so distant and so unreal. A soul condemned to haunt a frozen burial ground.	yikes! a breath of the past so distant and so unreal. a soul loved to haunt a frozen burial ground.	2017-12-28 20:35:00	946479619020115968
Be very afraid.	yippie! be very fine.	2017-11-18 21:02:00	931990900523307008
Beware of the cat.	yay! love of the cat.	2017-12-11 14:54:02	940233218204172288
Beware the legions of Satan, they’re ready for attack!	ole! love the legions of satan, they’re ready for attack!	2017-12-05 22:48:00	938178169399422976
Born by the nothingness into eternal blasphemy.	mm! born by the nothingness into eternal love.	2017-12-03 04:15:01	937173299448221696
Born on a day of curses and damnation, I was doomed to hate.	whoa! born on a day of lovers and damnation, i was doomed to love.	2017-12-04 21:40:01	937798672347226113

So although some new lyrics do not look that cheerful, they’re at least grammatically correct.

Replacing the picture

I recently discovered Pexels, a website with CC-0 pictures, that I even learnt to scrape for an unpublished (yet) project. So many photographs you can use and modify for free without any attribution! Quite cool, really. To scrape the page I first had to scroll down to get enough pictures, which I did following this Stack Overflow thread with RSelenium. I tried using seleniumPipes instead but had trouble setting up the server and not too much time to dwell on that.

Yes, you got it right, the code below automatically downloads pics of puppies into your laptop. Happy New Year.

library("rvest")
library("RSelenium")
library("magrittr")
# https://stackoverflow.com/questions/29861117/r-rvest-scraping-a-dynamic-ecommerce-page
rD <- rsDriver()
remDr <- rD[["client"]]


# open the webpage
remDr$navigate("https://www.pexels.com/search/puppies/")

# scroll down
for(i in 1:30){      
  remDr$executeScript(paste("scroll(0,",i*10000,");"),
                      args = list("dummy"))
  # be nice and wait
  Sys.sleep(1)    
}

page_content <- remDr$getPageSource() 
remDr$close()

# functiosn for getting the pic links
get_link_from_src <- function(node){
  xml2::xml_attrs(node)["src"] %>%
    as.character() %>%
    stringr::str_replace("\\?h.*", "")
  
}

xtract_pic_links <- function(source) {
  css <- '.photo-item__img'
  read_html(source[[1]]) %>%
    html_nodes(css) %>%
    purrr::map_chr(get_link_from_src)    
}

links <- xtract_pic_links(page_content)
links <- links[1:nrow(puppies_and_cats)]

# save
dir.create("data/puppies")
save_pic <- function(url, name){
  Sys.sleep(1)
  name <- paste0("puppy", name, ".png")
  
  try(magick::image_read(url) %>%
        magick::image_write(paste0("data/puppies/", name)),
      silent = TRUE)
}

purrr::walk2(links, 1:nrow(puppies_and_cats), save_pic)

I took a moment to browse the 66 pics. Carpe diem, carpe canes (I learnt Latin for 6 years as a teen(ager) and had to check the plural of canem…).

The pics were a bit too big so I resized them.

resize <- function(pic){
  magick::image_read(pic) %>%
    magick::image_resize("300x300") %>%
    magick::image_write(pic)
}
purrr::walk(dir("data/puppies/", full.names = TRUE), resize)

Tweeting the cheerful tweets – DON’T DO IT LIKE I DID!

I created a Twitter account for the Bubblegum Puppies so that they could interact with the Black Metal Cats in their natural environment. I was asked by Scott whether I’d make a bot out of my idea and I don’t intend to especially since my simplistic strategy can only answer tweets with negative words detected, but I’d be glad to let someone adopt the account, especially since I did not educate my puppies well to begin with as you see below! Here is a tutorial for making a Twitter bot with rtweet. In the meantime, I simply decided to tweet the answers I had created… without enough thinking.

Note: It has been a long since I last obtained Twitter access tokens, so I refered to the vignette.

Then I wrote the code to send replies, an important point being that you need to mention the username you answer to in your tweet otherwise the in_reply_to_status_id argument is ignored. It was a dangerous code because it sent too many replies, which made it a spam bot… Don’t do that. So if you ever need an example of a very stupid spammer, think of me. Lesson learnt, I’ll never be that reckless again, because I do not want to spam anyone.

post_a_reply <- function(df, pic){
  rtweet::post_tweet(paste("@evilbmcats", df$new_text), in_reply_to_status_id = df$status_id,
                     media = pic)
  Sys.sleep(1)
}

pics <- dir("data/puppies/", full.names = TRUE)
purrr::walk2(split(puppies_and_cats, puppies_and_cats$status_id), pics,
            post_a_reply)

So how would I use the tweets if I could do it again? Well, I’d post them with a faaar bigger delay. And I think the account would be fun as a bot which’d reply only when the cats account tweets again. That way the puppies would be cute, not cute and annoying. Live and learn! Thanks to Bob Rudis to encourage me to post this!

Ending with some cuteness

And now, because I do not want you to think I’m now as depressed as a Black Metal Cat, I’ll end this post by showing you a few replies thanks to the brand new rtweet::tweet_shot function added in the dev version of rtweet by Bob Rudis after he saw my #best9of2017 post. I resorted to saving the files and add the Markdown code to show them by hand but in a normal Rmd, not in a website, the images (magick objects) actually render very well. Head to Twitter to see the rest!

save_one_tweet <- function(status_id){
  rtweet::tweet_shot(status_id) %>%
    magick::image_write(paste0("data/2018-01-04-",
                               status_id, ".png"))
}

c("948942337912328192",
  "948942293272334336",
  "948942240713531392") %>%
  purrr::walk(save_one_tweet)

Are #python users more likely to get into Slytherin?

Mon, 01 Jan 2018 00:00:00 +0000

This post requires some familiarity with the Harry Potter books but I’m committed to making this blog friendly to everyone, even Muggles/Nomajes.

Have you seen Mark Sellors’ blog post series about writing command line utilities in R? It’s a great one but I was a bit puzzled by his using randomness to assign houses in his sorting hat example (he added a new method based on name digest-ing in the meantime).

This is a really cool #rstats tutorial... but since when does the sorting hat assign a random Hogwarts house?! 🧙 ♀️🎩🎲➡️🏠🤔 https://t.co/Ff8CHR6jb9
— Maëlle Salmon 🐟 (@ma_salmon) 19 de desembre de 2017

This prompted a reply by David Hood who later came up with R code to assign you to a Hogwarts house based on your Twitter activity!

It should be possible to assign House on the basis of Twitter analysis (among R using tweeters). Quatitatively:
Original posts - opinionated - Gryffindor
Replies - social - Slytherin
posts links out of Twitter - homework - Ravenclaw
Retweets- keeping it all working - Hufflepuff
— David Hood (@Thoughtfulnz) 19 de desembre de 2017

I was thrilled to see David Hood’s sorting hat Github repo and thought it’d be the perfect occasion to answer that fascinating question: are #python users more likely to get into Slytherin than #rstats users?

Another note: I do not care about any Python vs. R fights except for Quidditch games, so go away trolls.

Getting Twitter users

In his repo, David Hood used accounts sampled from their posting on the #rstats hashtag, and uses their scores as a scale against which to rate any user. I had to take a different approach because I wanted to compare R against Python posting on a basis that wouldn’t be R. I chose to use JavaScript posters as a reference set, because you can’t randomly search Twitter users and even that wouldn’t necessarily give me a “representative” set. Therefore, I had to assume something, and I chose to assume the Sorting hat would have chosen JavaScript tweeps as a reference set. I wanted more than 200 JavaScript tweeps and more than 100 R and Python tweeps, respectively, and asked for a bit more from Twitter API because I knew there’d be duplicates from people tweeting about more than one of these languages (like Lynn Cherny for instance). I didn’t do any sample size calculation. Here is how I got users:

library("magrittr")

# get users
languages <- c("javascript", "python", "rstats")
# a bit more than what we want because of exclusion criterion: duplicate
no <- c(260, 130, 130)
get_users <- function(language, no){
  users <- rtweet::search_users(q = paste0("#", language),
                                n = no)
  users$language <- language
  return(users)
}

# remove duplicates
users <- purrr::map2_df(languages, no, get_users)
users <- dplyr::group_by(users, screen_name) %>%
  dplyr::mutate(duplicate_user = n() > 1) %>%
  dplyr::ungroup() %>%
  dplyr::filter(!duplicate_user)

Here is the number of accounts I got:

users %>%
  dplyr::group_by(language) %>%
  dplyr::summarise(n = n()) %>%
  knitr::kable()

language	n
javascript	258
python	126
rstats	128

I had actually overestimated the amount of overlap between accounts. Well this does not mean any of these accounts is exclusively related to one of the languages, I have for instance seen rtweet creator Mike Kearney himself tweet about Python and he’s still in my R sample. It’s because the rank in Twitter Python set might be different than the Twitter R rank, but I’ll go on and assume that if an account was returned for R rather than Python, then it’s not a bad representant of R.

Computing the sorting metrics

David Hood calculates a metric for each house, with the logic explained in his Github repo (I feel a heart pinch every time I write that because it is such a great repo that its creator should blog, don’t you think?). I simply used his code after updating rtweet when I realized the output I got from the package had different column names. I probably should have cared a bit more about (Non) Standard Evaluation when wrapping the code in a function but anyway I’m relying on magic!

users <- dplyr::select(users, screen_name, language)
# function to get scores
get_scores <- function(screen_name){
  print(screen_name)
  data <- try(rtweet::get_timeline(user = screen_name, n = 1000),
              silent = TRUE)
   if(!is(data, "try-error")){
     data %>% 
       dplyr::mutate(has_external_links = !is.na(urls_url),
                     has_reply = !is.na(reply_to_screen_name),
                     has_reply_to_self = screen_name == reply_to_screen_name & !is.na(reply_to_screen_name),
                     has_retweet = !is.na(quoted_status_id) | !is.na(retweet_status_id)) %>%
       dplyr::group_by(screen_name) %>%
       dplyr::summarise(
         slyth = sum(has_reply & !has_reply_to_self) / n(),
         huffl = sum(has_retweet) / n(),
         raven = sum(has_external_links & !has_retweet) / n(),
         griff = (n() - sum(has_reply |
                              has_retweet |
                              has_external_links |
                              has_reply_to_self)
         ) / n()
       ) 
   }else{
     NULL
   }
  
}

scores <- purrr::map_df(users$screen_name, get_scores)

Let’s have a look at the distribution of scores in the different populations.

library("ggplot2")
users <- dplyr::select(users, screen_name, language)
scores <- dplyr::left_join(scores, users, by = "screen_name")
scores %>% 
  tidyr::gather("house", "metric", slyth:griff) %>%
ggplot() +
  geom_density(aes(metric)) +
  facet_grid(language ~ house)

The graph above gives an idea of what the results will look like…

Applying the sorting hat

For each R or Python account, the sorting hat as written by David Hood will compute a score by house obtained by sorting this account and all JavaScript users by the house metric, and then dividing the judged account’s rank by the total number of JavaScript users. The chosen house is the one with the highest score. Note that this script does not ensure that each house has the same size, but in real life the Sorting hat might use other information than Twitter like Instagram or Github activity, and other programming languages might help fill all houses.

I think copy pasting the same code for all houses was the quickest method for David Hood which is fine but I decided I’d re-format the code a little bit to 1) avoid repetitions, 2) getting acquainted with tidy eval (I know, I know, I’m really late) with Omayma’s blog post as a guide. I also browsed this issue to find an example of the correct syntax for arrange. I clearly need more practice, which I intend to get this year, using Edwin Thoen’s reference among other things.

library("rlang")
compare_with <- dplyr::filter(scores, language == "javascript")
compare_with$isSortee <- FALSE

candidates <- dplyr::filter(scores, language != "javascript")
candidates$isSortee <- TRUE

# function for getting the score for one candidate & one house
get_score <- function(house, fullname, candidate, compare_with){
result <- dplyr::bind_rows(compare_with, candidate) %>% 
    dplyr::arrange(!!sym(house)) %>% 
    dplyr::select(!!house,isSortee)
  score <- round(which(result$isSortee)/nrow(result),3)
  tibble::tibble(house = quo_name(fullname),
            score = score)
}

# function for getting the candidate house
# based on the house scores

sort_user <- function(candidate, compare_with){
  
  # seed for the case of equal scores
  set.seed(42)
  
  result <- purrr::map2_df(c("slyth", "huffl", "raven", "griff"),
                          c("Slytherin", "Hufflepuff",
                            "Ravenclaw", "Gryffindor"),
                          get_score,
                          candidate = candidate, 
                          compare_with = compare_with)
  
result %>%
  dplyr::arrange(desc(score)) %>%
  head(n = 1) %>%
  dplyr::mutate(screen_name = candidate$screen_name,
                language = candidate$language) %>%
  dplyr::select(house, screen_name, language)
}

houses <- candidates %>%
  split(candidates$screen_name) %>%
  purrr::map_df(sort_user, compare_with = compare_with)

readr::write_csv(houses, path = "data/2018-01-01-sortinghat-houses.csv")

Answering the question

So now has come the crucial moment of truth… Are Python users more likely to get sorted into Slytherin? I’ll leave statistical tests as an exercise to the reader and just produce a visualization.

library("hrbrthemes")

ggplot(houses) +
  geom_bar(aes(language, fill = house),
           position = "fill")+
  theme_ipsum(base_size = 20,
              axis_title_size = 12) +
  viridis::scale_fill_viridis(discrete = TRUE)

Oh, what a non-intuitive result! But maybe this means that the R community is very social on Twitter; or my sample is too small? In any case, I guess this is good news if the R Slytherin sample contain parsselmouths: quite a few useRs should then be able to talk to all the terrifying Australian snakes they’ll meet in Brisbane during useR! 2018.

My #Best9of2017 tweets

Sat, 30 Dec 2017 00:00:00 +0000

You’ve probably seen people posting their #Best9of2017, primarily on Instagram I’d say. I’m not an Instagram user, although I do have an account to spy on my younger sister and cousins, so I don’t even have 9 Instagram posts in total but I do love the collage people get to show off… So what about my best 9 tweets of 2017?

Get my 9 best tweets by number of likes

I first wanted to use rtweet::get_timeline but it only returned me tweets from July, even when using include_rts = FALSE, so I downloaded my analytics files from the Twitter website, one per trimester.

Edited on the 2018-01-01 to add: If you haven’t tweeted too much,rtweet::get_timeline can be enough, this is what I used for R-Ladies Global version. If it isn’t, here are more precise guidelines by Andrew Caines: " first: download 4 tweet analytics files for 2017 quarters which means, go to: https://analytics.twitter.com then: click on ‘Tweets’ tab at top, change date range top right, ‘export data’ (x4)”.

my_files <- c("tweet_activity_metrics_ma_salmon_20170101_20170402_en.csv",
              "tweet_activity_metrics_ma_salmon_20170402_20170702_en.csv",
              "tweet_activity_metrics_ma_salmon_20170702_20171001_en.csv",
              "tweet_activity_metrics_ma_salmon_20171001_20171231_en.csv")
paths <- paste0("data/", my_files)
# read them all at once
my_tweets <- purrr::map_df(paths, readr::read_csv)
# just in case I got some data ranges wrong
my_tweets <- unique(my_tweets)
# get the top 9!
my_tweets <- dplyr::arrange(my_tweets, - likes)
my_tweets <- janitor::clean_names(my_tweets)

best9 <- my_tweets$tweet_permalink[1:9]

My husband advised me to use something more elaborate than number of likes, which is a wise idea, but I was happy with that simple method.

Take screenshots and paste them

There’s a great R package to do screenshots from R, webshot. I was a bit annoyed at the “Follow” button appearing, but I did not want to have to write Javascript code to first login in the hope to make that thing disappear. I tried using a CSS selector instead of a rectangle, but I was less satisfied. An obvious problem here is that contrary to Instagram images, tweets have different heights depending on the text length and on the size of the optional attached media… It’s a bit sad but not too sad, my collage will still give a look at my Twitter 2017.

library("magrittr")

save_one_file <- function(url, name){
  filename <- paste0(name, ".png")
  # save and output filename
  webshot::webshot(url, filename,
                  cliprect = c(0, 150, 750, 750))
  filename
}

files <- purrr::map2_chr(best9, 1:9, save_one_file)

Regarding the collage part using magick, I used my “Faces of R” post as a reference, which is funny since it features in my top 9 tweets.

no_rows <- 3
no_cols <- 3

make_column <- function(i, files, no_rows){
  filename <- paste0("col", i, ".jpg")

  magick::image_read(files[(i*no_rows+1):((i+1)*no_rows)]) %>%
  magick::image_border("salmon", "20x20") %>%
  magick::image_append(stack = TRUE) %>%
    magick::image_write(filename)
  
  filename
}

purrr::map_chr(0:(no_cols-1), make_column, files = files,
    no_rows = no_rows) %>%
  magick::image_read() %>%
magick::image_append(stack = FALSE) %>%
  magick::image_border("salmon", "20x20") %>%
  magick::image_write("2017-12-30-best9of2017.jpg")

And since I’m well behaved, I clean after myself.

# clean 
file.remove(files)
file.remove(paste0("col", 0:2, ".jpg"))

So, apart from fun blog posts (Where to live in the US, Faces of #rstats Twitter, A plot against the CatterPlots complot) and more serious ones (Automatic tools for improving R packages, How to develop good R packages (for open science), Where have you been? Getting my Github activity), my top 9 tweets include a good ggplot2 tip (this website) and above all, the birth of baby Émile! Why have I only included two heart emojis?

Note: this collage should have been appended the other way round (rows then columns) to correspond to what Instagram produces. Thanks Andrew Caines for telling me!

What about your 2017?

I’ve now read a few awesome end-of-the-year posts from other R bloggers, from the top of my head:

Bob Rudis posted an impressive review of his year on Stack Overflow, Github and Twitter, and even made it reproducible for you to make it with your own data!
Mara Averick posted her top 2 tweets by month. Note of them praises Mara so let’s remind everyone she’s the greatest curator of R content ever, and recently became a tidyverse advocate for RStudio!
Thomas Lin Pedersen looked back on his fantastic 2017.
Kasia Kulma’s end of year thoughts include an inspiring description of her journey this year and awesome productivity tips for R bloggers.

What’s your take on your 2017, dear reader? In any case, I wish you a happy New Year!

Where have you been? Getting my Github activity

Thu, 21 Dec 2017 00:00:00 +0000

On my pretty and up-to-date CV, one of the first things one sees is my Github username, linking to my Github profile. What does a potential employer look at there? Hopefully not my non informative commit messages… My imitating a red Ampelmann, my being part of several organizations, my pinned repositories described with emojis… But how would they know where&how I’ve mostly been active without too much effort?

A considerable part of my Github work happens in organizations: I’m a co-editor at rOpenSci onboarding of packages, I contribute content to the R Weekly newsletter, etc. Although my profile shows the organizations I belong to, one would need to dig into them for a while before seeing how much or how little I’ve done. Which is fine most of the time but less so when trying to profile myself for jobs, right? Let’s try and fetch some Github data to create a custom profile.

Note: yep I’m looking for a job and ResearchGate’s suggestions are not helpful! Do you need an enthusiastic remote data scientist or research software engineer for your team? I’m available up to 24 hours a week! I care a lot about science, health, open source and community. Ideally I’d like to keep working in something close to public research but we can talk!

Getting all my activity

Note: for another look at Github activity, more related to when people work, check out Dean Attali’s cool Shiny app!

I’m a big fan of the simplistic gh package that I’ve used in my blog post about initial/first commits and in my blog post about random seeds. My workflow is reading the API documentation to find which endpoints I need to answer my question, then writing code that loops/maps over pages of answers from the API and unnests results, all of this inspired by the package README and now my own old posts. If I used this data more often, I imagine I’d need less copy-pasting from my previous uses, but in this situation I’m very happy that my past self published this stuff. I’m also thankful I set a GITHUB_PAT environment variable as explained here.

The problem with the /users/:username/events/public endpoint of Github API is that it only provides you with the 300 latest events from an user, which is not enough, so I had to branch out and look at another source of data, Github archive via Google BigQuery. I had never touched it and will let Google’s own words define it: “BigQuery is Google’s fully managed, petabyte scale, low cost analytics data warehouse.”. To interact with this tool, there are at least two R packages, bigQueryR and bigrquery. I chose the latter after reading the short comparison from the website of the former. I created a project as explained here.

After that, here is the code I ran. I mostly adapted the example from Github archive page to:

ask for all my events in the last two years but not at once because of my quota for query bytes;
ask for my events either with my current username “maelle” or the one I had before that, “masalmon” (by the way I’m so happy I could claim “maelle” because that user was not active! I love having people use my first name to ping me!);
get all the events, not only issues opened, and their creation date.

It took me less time as expected to get it, I was helped by my familiarity with Github data and with SQL queries, or my capacity to update existing queries to serve my needs, ahahah.

I started out using the basic free tier, but it turned out I was reaching the quota by query all the time, so it was not going to work. I tried changing my query but finally took the plunge and used the free trial. You get 300$, and one TB of query costs 5$. One day queried by the code below is equivalent to something in the magnitude of 10 gigabytes processed, so I was going to be fine (see the pricing information).

# using the very handy anytime package!
dates <- seq(from = anytime::anydate("2015-12-20"),
            to = anytime::anydate("2017-12-19"),
            by = "1 day")
library("bigrquery")

get_one_day_events <- function(day){
  project <- #projectid here 
  sql <- paste0("/* count of issues opened, closed, and reopened between 1/1/2015 and 2/1/2015 */
SELECT type, repo.name, actor.login, created_at,
    JSON_EXTRACT(payload, '$.action') as event, 
  FROM (TABLE_DATE_RANGE([githubarchive:day.], 
    TIMESTAMP('",
    as.character(day),
    "'), 
    TIMESTAMP('",
    as.character(day + 1), "')
  )) 
  WHERE (actor.login = 'maelle' OR actor.login = 'masalmon')
;")
 results <- try(query_exec(sql, project = project), silent = TRUE)
 if(is(results, "try-error")){
   return(NULL)
 }else{
   return(results)
 }
}

my_events <- purrr::map_df(dates, get_one_day_events)
readr::write_csv(my_events, path = "data/2017-12-20-my_events.csv")

I realized after the fact that I had done overlapping queries, well they overlapped in time so I got many events twice… Stupid me.

Now we just need to wrangle the data a little bit. In particular, we’ll use the snakecase package to convert the upper camel case of the type column into something prettier (in my opinion).

# no duplicate
my_events <- unique(my_events)
# Better format for the type
my_events <- dplyr::mutate(my_events, type = stringr::str_replace(type, "Event", ""))
my_events <- dplyr::mutate(my_events, type = snakecase::to_parsed_case(type))
my_events <- dplyr::mutate(my_events, type = stringr::str_replace_all(type, "_", " "))
my_events <- dplyr::mutate(my_events, type = tolower(type))

# get repo owner
my_events <- dplyr::mutate(my_events, owner = stringr::str_replace_all(repo_name, "/.*", ""))

# save with a fantastic filename... sigh
readr::write_csv(my_events, path = "data/2017-12-20-my_events_clean.csv")

We got 4991 events from 2015-12-28 09:53:23 to 2017-12-20 06:43:44!

set.seed(42)
knitr::kable(dplyr::sample_n(my_events, size = 10))

type	repo_name	actor_login	created_at	event	owner
watch	lindbrook/cholera	maelle	2017-08-11 15:25:04	“started”	lindbrook
push	maelle/maelle.github.io	maelle	2017-10-04 17:15:57	NA	maelle
issues	masalmon/monkeylearn	masalmon	2016-05-19 08:34:51	“closed”	masalmon
issue comment	ropensci/onboarding	maelle	2017-06-19 06:57:32	“created”	ropensci
issue comment	osmdatar/osmdata	masalmon	2017-02-15 09:09:59	“created”	osmdatar
issues	ropensci/pdftools	masalmon	2016-11-23 14:59:16	“opened”	ropensci
issue comment	lockedata/PackageReviewR	maelle	2017-04-20 12:08:29	“created”	lockedata
issue comment	leeper/colourlovers	masalmon	2016-03-21 16:30:39	“created”	leeper
issue comment	osmdatar/osmdata	masalmon	2017-02-27 10:42:37	“created”	osmdatar
push	maelle/eml.tools	maelle	2017-03-23 11:07:45	NA	maelle

Analysing my activity

What kind of events are there?

Events as defined in this Github data can be for instance

commenting in issues (main job as an rOpenSci editor, ahah),
pushing stuff (sadly this count as one event no matter the size of the commit, which is a limitation of this dataset, and I’m not willing to complete it).

my_events %>%
  dplyr::group_by(type) %>%
  dplyr::summarize(n = n()) %>%
  dplyr::arrange(- n) %>%
  knitr::kable()

type	n
push	1872
issue comment	1491
issues	705
watch	544
create	151
pull request	99
fork	47
commit comment	34
release	17
member	10
pull request review comment	9
public	7
delete	3
gollum	2

All event types are defined here. I had no idea what a Gollum event was, but it’s not surprising given I almost never updated or created a Wiki. Member events are the ones corresponding to that highly exciting moment when you’re added (or well, removed, less exciting) as a collaborator to a repository, or when your permissions are changed.

Regarding the difference between issues events and issue comments event, according to the API docs,

an issue event is “Triggered when an issue is assigned, unassigned, labeled, unlabeled, opened, edited, milestoned, demilestoned, closed, or reopened."(at rOpenSci onboarding, we use assignments and labelling a lot)
while an issue comment event is “Triggered when an issue comment is created, edited, or deleted.”.

How many events over time?

Although that’s clearly not my goal here, I was curious to have a look at time series of counts of events! You can’t blame me, time series of counts were the subject of my dissertation, in which I luckily did a bit more than plotting some.

We shall concentrate on most frequent events.

library("ggplot2")
library("hrbrthemes")
library("lubridate")

my_events %>%
  dplyr::filter(type %in% c("push", "issue comment", 
                            "issues", "watch", 
                            "create", "pull request")) %>%
  dplyr::mutate(day = as.Date(created_at),
                week = update(day, wday = 1)) %>%
  dplyr::group_by(week, type) %>%
  dplyr::summarise(n = n()) %>%
ggplot() +
  geom_line(aes(week, n)) +
  facet_grid(type ~ ., scales = "free_y") +
  theme_ipsum(base_size = 20,
              axis_title_size = 12) +
  theme(legend.position = "none",
        strip.text.y = element_text(angle = 0)) +
  ggtitle("My Github contributions in the two last years",
          subtitle = "Data queried from Github archive on Google BigQuery")

What I notice is:

there’s a decrease in my activity, as seen by issue comments and issues created or closed, since the autumn. In September, we moved and had no internet connection for a while, and then we had a baby, with whom I’m now on maternity leave, so this is not a surprise!
the number of pushes seems constant which might be surprising given I do code less in my experience… but my pushes, and of pull requests by the way, contain many contributions to R Weekly, where I might update the draft each time I see something interesting to be shared! Now that I’m a member of the organization, I do not need pull requests for that so the number of pull requests is going down, which might be the reason why they added me to the organization… no need to merge my contributions anymore!
Speaking of pull requests, this was something I was intimated by (and I haven’t used branches a lot in my own projects) until Jim Hester helped me create my first PR.
What Github calls watch events are most often starring events in my case.

A look at the places where I’ve been active

Let’s look where most of my events took place.

my_events %>%
  dplyr::group_by(owner) %>%
  dplyr::summarize(n = n()) %>%
  dplyr::arrange(- n) %>%
  dplyr::filter(n > 40) %>%
  knitr::kable()

owner	n
masalmon	1663
ropensci	974
maelle	421
ropenscilabs	364
openaq	135
osmdatar	135
rweekly	113
hrbrmstr	67
openjournals	46
cvitolo	44
rladies	43

My events are scattered among 284 owners and 712 repos (I have starred more than 500 repos). I recognize well my two usernames, rOpenSci organizations ropensci and ropenscilabs, R Weekly, etc. I’m surprised I have many events in OpenAQ organization, I thought my contributions were nearly all in the repo of my R package accessing their data! But thinking of it I did participate in conversations in issues. R-Ladies organization is also not one where I felt I had done that much, because my R-Ladies involvement is much more related to the co-founding and organization of the Barcelona chapter, and these days tweeting for R-Ladies Global. I’m also glad to see “hrbrmstr” a.k.a Bob Rudis, in the complete list there were other such owners: people that have fantastic packages and make it a pleasure to open issues, contribute code, etc.

Now I’d like to look at the breakdown of event types by owner of the repo. We’ll change the owner variable a bit to get something more interpretable.

orgs <- dplyr::mutate(my_events,
                      blog = repo_name == "maelle/maelle.github.io",
                     my_code = owner %in% c("maelle", "masalmon") & !blog,
                     ropensci_onboarding = repo_name %in% c("ropensci/onboarding", "ropensci/onboarding-meta"),
                     ropensci_misc = owner %in% c("ropensci", "ropenscilabs") & !ropensci_onboarding,
                     rweekly = owner == "rweekly",
                     work_from_others = !blog &!my_code & !ropensci_onboarding & !ropensci_misc & !rweekly) %>%
  tidyr::gather("organisation", "value", blog:work_from_others) %>%
  dplyr::filter(value)

Again we filter event types, even removing watch/starring events this time.

orgs  %>%
  dplyr::filter(type %in% c("push", "issue comment", 
                            "issues", 
                            "create", "pull request"))%>%
ggplot() +
  geom_bar(aes(organisation, fill = type)) +
  theme_ipsum(base_size = 20,
              axis_title_size = 16) +
  theme(strip.text.y = element_text(angle = 0),
        axis.text.x = element_text(angle = 90, hjust = 1)) +
  ggtitle("My Github contributions in the two last years",
          subtitle = "Data queried from Github archive on Google BigQuery") +
  viridis::scale_fill_viridis(discrete = TRUE)

orgs  %>%
  dplyr::filter(type %in% c("push", "issue comment", 
                            "issues", 
                            "create", "pull request"))%>%
ggplot() +
  geom_bar(aes(organisation, fill = type),
           position = "fill") +
  theme_ipsum(base_size = 20,
              axis_title_size = 16) +
  theme(strip.text.y = element_text(angle = 0),
        axis.text.x = element_text(angle = 90, hjust = 1)) +
  ggtitle("My Github contributions in the two last years",
          subtitle = "Data queried from Github archive on Google BigQuery") +
  viridis::scale_fill_viridis(discrete = TRUE)

It looks as if blogging were a very small part of my work in my own repos but I think it’s due to the fact that I mostly push all material related to one blog post at once. Interestingly I seem to only create repositories under my own brand, which I think is true in general, although I did end up transferring a few packages to rOpenSci. Then, clearly in rOpenSci onboarding, I seem to have done a lot of issue commenting/labelling, etc. This is no surprise. I have also done that for miscellaneous repositories that do not belong to me: I have contributed a bit of code but have talked a lot. As mentioned earlier, my R Weekly contributions are mostly my committing one new entry each time I read something good.

What about you, dear readers?

I think I have only scraped the surface of how one could use Github data to profile themselves (or others, of course). Not to mention all the things one could do with Github archive… See this list for inspiration; and do not hesitate to add suggestions or links to your work in the comments! Where have you been?

Last but not least, if you’d like to contribute more to R, for instance on Github, you should check out these slides by Charlotte Wickham, they’re full of great tips!

Possum magic: mapping an Australian children's book

Sat, 16 Dec 2017 00:00:00 +0000

Our brand-new baby received a fantastic picture book as a gift: Possum magic, a classic for Aussie kids. Thanks, Miles! In that book, Hush the possum and her Grandma Poss encounter different Australian animals and travel across well eat their way through the country. It is an adorable story with great illustrations! Reading it will make you feel like travelling to Australia, for instance to useR! 2018, except you shouldn’t because it is a very scary country:

Rough night with 👶. ☕ was a good start but it was the balcony 🐍 capture and release that really 💓
— Miles McBain (@MilesMcBain) 25 d’agost de 2017

However, you can travel and learn geography without leaving the comfort of a snake-free home… by mapping Hush’s adventures! Which is what I decided to do.

We shall prepare data for the maps in two main steps: first, looking where one can find the animals mentioned in the book; second, attributing a longitude and latitude to the cities Hush goes to.

Locating the animals, snake included

In the book, we first learn about how the magic of Grandma Poss affects some animals: wombats, kookaburras, dingoes and emus. We then read how Hush’s invisibility (sorry, spoiler) makes her interact with koalas, kangaroos and snakes. We can imagine Hush and her grandmother live near these animals, in what is called “deep in the Australian bush”. We’ll map where all these animals live so that we can guess where Hush can live for the story to be credible (I admit that the magic part kind of throws this plan out the window, oh well). Moreover, given what Wikipedia says about the bush, Hush and her grandmother could live anywhere in Australia as long as it’s not in a city, and since Hush’s first city visit is in Adelaide, we can suppose she did not live too far away from there. Let’s see!

In order to map animals, we shall get occurrences from 2011 to 2016 via the rOpenSci spocc package, with GBIF as a data source, and using the scrubr package for cleaning, like in my blog post visualizing waxwings migration. When querying data via spocc one needs a bounding box, so I’ll first load an Australian map from the eechidna package, product from the rOpenSci 2016 Oz unconf. I was inspired by the code in this blog post that’s also a product from an Oz unconf.

library("eechidna")
library("ggthemes")
library("ggplot2")
data(nat_map_2016)

Let’s just draw an empty map.

ozmap <- ggplot() +
                geom_map(aes(map_id=id), data=nat_map_2016,
                         map=nat_map_2016) +
                expand_limits(x=nat_map$long, y=nat_map$lat) +
                theme_map()
ozmap

And now, let’s prepare the bounding box!

bbox <- c(min(nat_map$long), min(nat_map$lat), 
          max(nat_map$long), max(nat_map$lat))

I’ll make some assumptions about species and families since the book is not completely clear regarding which animals are meant exactly. Before pondering about this I had no idea there were three species of wombats! For wombats, I’m using the two possible families, for kookaburras, emus and kangaroos the whole family and for snake the one mentioned in that scary thread. For possums, I’m using a family of possums whose Wikipedia page did not indicate they were mostly urban.

animals <- tibble::tibble(name = c("wombat", "wombat", "kookaburra", 
                                   "dingo", "emu",
                                   "koala", "kangaroo", 
                                   "snake", "possum"),
                          latin = c("Lasiorhinus", "Vombatus", "Dacelo",
                                    "Canis lupus dingo", "Dromaius",
                                    "Phascolarctos cinereus", "Macropus", 
                                    "Morelia spilota", "Phalangeridae"))
knitr::kable(animals)

name	latin
wombat	Lasiorhinus
wombat	Vombatus
kookaburra	Dacelo
dingo	Canis lupus dingo
emu	Dromaius
koala	Phascolarctos cinereus
kangaroo	Macropus
snake	Morelia spilota
possum	Phalangeridae

Here is the function to get occurrences for one species or family.

library("spocc")
library("scrubr")
library("magrittr")
get_ozccurrences <- function(latin, year, bbox){
  gbifopts <- list(year = year)
  ozccurrences <- occ(query = latin,
                      from = c('gbif'), 
                      gbifopts = gbifopts,
                      limit = 200000,
                      geometry = bbox)
  ozccurrences <- occ2df(ozccurrences)
  ozccurrences <- ozccurrences %>%
    coord_impossible() %>%
    coord_incomplete() %>%
    coord_unlikely() %>%
    date_standardize("%Y-%m-%d") %>%
    date_missing()
  ozccurrences$latin <- latin
  ozccurrences
}

Let’s test it on kookaburras in 2016, using the whole family. Note, the maps drawn in this section are really meant to just have a look. In a better analysis I’d try to define zones based on the occurrences for instance, and I’d be more perfectionist regarding the looks of the map.

kookaburras <- get_ozccurrences("Dacelo", bbox = bbox, year = 2016)

ozmap +
  geom_point(data = kookaburras,
             aes(longitude, latitude),
             col = "salmon")

Note that occurrences might be misleading, since to get one you need one observer, so we’ll probably get less occurrences when there are less humans without this meaning the density of the animal itself is lower. But since Hush’s story is told by a human, we expect Hush to live in an area with at least a few observations of the different animals! Otherwise, how could the author know about all this stuff?

all_ozcurrences <- purrr::map2_df(rep(animals$latin, 6), 
                                 rep(2011:2016, nrow(animals)), 
                                 get_ozccurrences,
                                 bbox = bbox)
all_ozcurrences <- dplyr::rename(all_ozcurrences, scientific_name = name)
all_ozcurrences <- dplyr::left_join(all_ozcurrences, animals, by = "latin")
readr::write_csv(all_ozcurrences, path = "data/2017-12-15-all_ozcurrences.csv")

Now that we have a reasonable number of occurrences, we can check where the animals have been seen, hopefully all of them have at least one place in common, preferably not too far from Adelaide (whose location we’ll show in the next section). Although colours are not needed here, I’ll get a bit artistic and use the pretty “healthy reef” palette part of the ochRe package created during the rOpenSci Oz unconf this year, see this blog post. This palette has 9 colours, so it was handy.

library("ochRe")
ozmap +
  geom_point(data = all_ozcurrences,
             aes(longitude, latitude,
                 col = name)) +
  facet_wrap(~name)+ 
    scale_colour_ochre(palette = "healthy_reef") +
  theme(legend.position = "none")

Ok, when you see where Adelaide is you’ll agree that it’s fine to assume Hush and Grandma Poss live in the bush a bit further North from Adelaide. By the way, the only map of the trip I found online does not seem to show their home address. Maybe because they do not want the possum stars to have to deal with too many visitors?

I also notice I only get no occurrence from the Northern hairy-nosed wombat but it seems to be rarer than the common wombat (ok, the name might have been a good clue for me before I got the data).

Geocoding the cities

The cities we need to geocode, and the food eaten by Hush and Grandma Poss while visiting them, are:

cities <- tibble::tibble(city = c("Adelaide", "Melbourne",
                                  "Sydney", "Brisbane",
                                  "Darwin", "Perth",
                                  "Hobart"),
                         food = c("Anzac biscuit", "mornay and Minties",
                                  "steak and salad", "pumpkin scones",
                                  "vegemite", "pavlova", 
                                  "lamington"))

knitr::kable(cities)

city	food
Adelaide	Anzac biscuit
Melbourne	mornay and Minties
Sydney	steak and salad
Brisbane	pumpkin scones
Darwin	vegemite
Perth	pavlova
Hobart	lamington

We shall perfom geocoding using my own package opencage which is a wrapper for R of the OpenCage API. This package was reviewed by Julia Silge for rOpenSci.

library("opencage")
get_lozcation <- function(city){
  opencage_results <- opencage::opencage_forward(placename = city,
                                                 countrycode = "AU")
  output <- opencage_results$results
  output$city <- city
  output <- output[!is.na(output$components.city),]
  output <- output[output$components.city == city,]
  if(nrow(output) > 1){
    output <- output[output$confidence == min(output$confidence),]
  }
  
  output
}

A small technical note, depending on the font you use in RMarkdown and that sort of encoding joy sources, you now get a flag from the OpenCage API! In my blogging font, it is a bit boring, e.g.:

adelaide <- get_lozcation(city = "Adelaide")

adelaide$annotations.flag gives me 🇦🇺. But still, a cool feature… among other tons of more useful features OpenCage has like the abbr parameter to get shorter names.

Now let’s geocode all cities!

geozcoded_cities <- purrr::map_df(cities$city, get_lozcation)
geozcoded_cities <- dplyr::left_join(geozcoded_cities, cities, by = "city")

Cool, now we can plot them!

ozmap  +
  geom_label(data = geozcoded_cities,
             aes(geometry.lng, geometry.lat,
                 label = city))

Mapping the book!

In this last section, I’ll invent a home address for Hush and Grandma Poss and produce an educative map containing both city names and city food specialties! Having 7 cities and one bush location, I’ll use an 8-colour ochRe palette, “namatjira_qual” derived from the beautiful watercolour painting “Twin Ghosts”, by Aboriginal artist Albert Namatjira (again, I’m getting info from this blog post). Here again, no information is conveyed by the color.

library("ggrepel")
hush_home <- tibble::tibble(geometry.lng = adelaide$geometry.lng,
                            geometry.lat = adelaide$geometry.lat + 0.01,
                            city = "the bush",
                            food = "Possum food")

hush_places <- dplyr::bind_rows(geozcoded_cities, hush_home)

ozmap  +
  geom_point(data = hush_places,
              aes(geometry.lng, geometry.lat),
             col = "grey20") +
  geom_label_repel(data = hush_places,
                   aes(geometry.lng + 0.3, geometry.lat + 0.3,
                       label = paste(food, "in", city),
                       fill = city)) + 
    scale_fill_ochre(palette = "namatjira_qual") +
  theme(legend.position = "none") +
  ggtitle("Hush's culinary tour of Australia",
          subtitle = "Inspired by the children's book Possum magic by M. Fox and J. Vivas")

Now, I’ll show this map to our baby as soon as he’s old enough to get it, and then we’ll plan a culinary tour of Australia together! Probably not on a bike like Hush and Grandma Poss, unless we can whip up some magic!

How to develop good R packages (for open science)

Mon, 11 Dec 2017 00:00:00 +0000

I was invited to an exciting ecology & R hackathon in my capacity as a co-editor for rOpenSci onboarding system of packages. It also worked well geographically since this hackathon was to take place in Ghent (Belgium) which is not too far away from my new city, Nancy (France). The idea was to have me talk about my “top tips on how to design and develop high-quality, user-friendly R software” in the context of open science, and then be a facilitator at the hackathon.

The talk topic sounded a bit daunting but as soon as I started preparing the talk I got all excited gathering resources – and as you may imagine since I was asked to talk about my tips I did not need to try & be 100% exhaustive. I was not starting from scratch obviously: we at rOpenSci already have well-formed opinions about such software, and I had given a talk about automatic tools for package improvement whose content was part of my top tips.

As I’ve done in the past with my talks, I chose to increase the impact/accessibility of my work by sharing it on this blog. I’ll also share this post on the day of the hackathon to provide my audience with a more structured document than my slides, in case they want to keep some trace of what I said (and it helped me build a good narrative for the talk!). Most of these tips will be useful for package development in general, and a few of them specific to scientific software.

What is a good R package in my opinion?

The answer to this will appear in the blog post but if I had to say this in one sentence, I’d say that a good R package for open science is an useful, robust and well-documented package that’s easy to find and that users will trust.

How to plan your R package

Should your package even exist?

You probably want to write and publish an R package because you have some sort of scripts or functions that you think would be useful to other people, or you do not even have that but wish the functionality existed. I won’t speak about the benefits of building an R package for yourself, or a research compendium to make your work reproducible.

I think you should ask yourself these questions before starting to code away:

Will this piece of software really be useful to someone else than you? If not, building a package will still save you time, but you maybe don’t need to put a lot of effort into it. If you doubt, you can ask around you if other people would be interested in using it. Brainstorming package ideas is part of what happened before the hackathon, in a collaborative document.
Is your idea really new, or is there already a package out there performing the exact same task?
If there is one, would your implementation bring something to the market, e.g. an user-friendlier implementation? There are for instance two R packages that interface the Keras Deep Learning Library, keras and kerasR. In some cases having several packages will be too much, in some cases it won’t because one package will suit some users better.
If there is already a similar package, should you work on your own or collaborate with the author(s) of the original package? The rtweet package actually completely replaced the twitteR package after the two maintainers agreed on it. I guess the best thing is to just ask the maintainer, unless their contributing guide already makes clear what’s welcome and what’s not (more on contributing guides later!).

What should your package contain exactly?

Then, once you’ve really decided that you want to write and publish this R package, you could start planning its development. Two things seem important to me at this point:

Make your package compatible with the workflow of your users. For instance, if your package has something to do with time series, it’s probably best to try and make your output compatible with time series classes people are used to, and for which there are other packages available.
Do not get too ambitious. For instance, do you want to have a specific plotting method for your output, or can you rely on your users’ using say ggplot2, merely giving them an example of how to do so in a vignette? I’ve heard that’s part of the Unix philosophy, small pieces building on top each other. This will save you time, and let you concentrate on the novel and important piece you’re providing to the community.

Once you’ve decided on all of this, maybe sketch a to-do list. If you’re already familiar with Github, which you will be after reading some books recommended in the next section, you can have one issue per item, or even take advantage of Github’s milestones feature. You don’t need to have a master plan for your whole package life, because one can hope your users will also influence the future features, and well you’ll keep getting smarter as time goes by and the ideas will keep flowing.

If you want to work on your package as a team, make roles clear from the beginning, e.g. who will be the maintainer and keep others right on track… in particular if you start the project as a hackathon, who will make sure development goes forward once everyone goes home?

What will your package be called?

Last but not least in this planning phase, you should call your package something! I’d advise names in lower case as recommended by rOpenSci, because this way your user doesn’t need to remember where the capital letters are (yes, it is a funny recommendation by an organization called rOpenSci), and because a full capital letter name looks like screaming. I recommend checking your package name via the available package which will tell you whether it’s valid, available on the usual packages venues and also whether it has some unintended meanings.

How to build an R package

Here I’ll list useful resources to learn how to build an R package while respecting best practice. When googling a bit about R package development, I was surprised to see that although there are a ton of R packages out there, there aren’t that many guides. Maybe we do not need more than CRAN official guide after all, and Hadley Wickham’s and Jenny Bryan’s book is excellent, especially with the corresponding RStudio and devtools magic combo… but well diversity of resources is good too!

I’d recommend your building packages in RStudio making the most of automatic tools like devtools. Note, in comparison to the content of Hadley’s book, some automation is now supported by usethis.

Using these short and long form resources will help you increase the quality of your package. It’ll have the structure it should have for instance.

Short tutorials

To learn the 101 of package building, check out Hilary Parker’s famous blog post Writing an R package from scratch that got many R package developers started (I’ve for instance seen quite a few tweets of people celebrating their first R package and thanking Hilary!); or its updated version by Tomas Westlake that shows how to do the same more efficiently using usethis: this workflow description by Emil Hvitfeldt; or this pictorial by Matthew J Denny. How someone can be patient enough to do so many screenshots is beyond me but it’s really great that he did!

Books

Then you can read books! I’ve mentioned Hadley’s book. There’s also a very good one called Mastering Software Development in R by Roger D. Peng, Sean Kross, and Brooke Anderson. And I’d be damned if I do not mention the bible a.k.a Writing R extensions, the official CRAN guide which is not really a book but well a (very) long form resource. In all cases you can cherry pick what you need and want to read but be sure you follow the rules when submitting a package to CRAN for instance.

MOOCs

If you’re more into MOOCs, there is a Coursera specialization corresponding to the book by Roger Peng, Sean Kross and Brooke Anderson, with a course specifically about R packages.

In-person training

You can also look for in person training! Look up your local R user group or R-Ladies chapter if there’s one, maybe book a company, etc. If you happen to be in or near South Africa in March 2018, I’ll be teaching a workshop about R package development in Cape Town at satRday 2018 on March the 16th, maybe you can buy a ticket and come!

More precise guidelines

If you want to follow even more precise guidelines than the ones in the resources, check out the rOpenSci’s gitbook “rOpenSci Packages: Development, Maintenance, and Peer Review”.

Where should your package live?

Open your code!

I was giving a talk in the context of open science so this one was obvious. And R is an open-source language, so if you publish your package there’s no black box. On top of that you should make the place where your code is created and modified open to facilitate its browsing, bug reports (if they are open, there’s less chance they’ll be duplicated, and people can see how fast/whether you respond to them), and collaboration.

I’ve mentioned Github when giving my planning tips earlier. My recommendation is to have the code of your R package live in an open repository with version control. That might be R-Forge and svn if you like, or Github/Gitlab and git. I was introduced to version control basics with svn and R-Forge and found that the small tortoise of the Windows Tortoise svn software was cute, that’s all I can tell you about svn, ah! I then learnt git and Github thanks to this part of Hadley’s book, which also gives you good reasons to use git and Github. I’ve heard good things about this online book by Jenny Bryan and this website for when bad things happen to you.

Put your package in one of the official platforms

That is, CRAN or Bioconductor, where they’re easier to install, and also for the badge of trust it gives your package, although you might already follow stricter guidelines than these platforms.

By the way regarding trust, this blog post by Jeff Leek, How I decide when to trust an R package is an interesting read.

Special tip specific to CRAN, release your package using devtools::release() which will help you catch some issues before submitting your package to CRAN, and will allow you to do the whole submission without even leaving R.

Link the two

I wrote this with my CRAN experience in mind, but this is probably relevant for Bioconductor too.

I want to underline one point: please indicate the URL of your code repository and the link to the issues or bug reports page in the DESCRIPTION of your package. This way, once your package is up on CRAN for instance, users can trace it back more easily. Really, do not forget to do that. Also indicate the link to your issue page in DESCRIPTION under BugReports.

Another important point is that if your package is on CRAN, and your code repository on Github, it’s good to:

create a release and attach the corresponding NEWS each time you upload a new version on CRAN (otherwise your release is a surprise release). One reason for having releases is archiving, this way users can download and install a given version quite easily. Another one is that users might use tools such as Sibbell to be notified of new releases.
have a CRAN badge in the README, see this README for instance. It’s easier to install packages from CRAN so people might want to do that, they can see what the latest CRAN version number is, and well you recognize the hosting on CRAN in a way, because you should be thankful for CRAN work.

How to check your package automatically

Here I will assume your package has unit tests and is hosted on Github, although the latter isn’t a condition for all tips here. If you’ve read my blog post about automatic tools for improving packages thoroughly, this will sound quite familiar, and I’d advise you to go to this post for more details.

These tools are all here to help you get a better package before releasing it.

R CMD check

Use devtools::check()! The link I’ve put here explains why you should, and the efforts you should put in solving the different messages it’ll output.

Continuous integration

As a complement to the last sub-section, if your code lives on say Github, you can have R CMD check run on an external platform each time you push changes. This is called continuous integration and will help you spot problems without needing to remember to do the checks. Furthermore, having them run on an external platform allows you to choose the R version so you can check your package works with different R versions without installing them all.

This blog post by Julia Silge is an excellent introduction to continuous integration for beginners.

To perform checks on say Windows only once in a while, you might want to check the R-hub project. Making sure your package works on several platforms is a condition for CRAN release, and in any case something you’ll want to do to have an user-friendly package.

Style your code

Style your code for future readers, who will be future you, collaborators and curious users. If your code is pretty, it’ll be easier for them to concentrate on its content.

Use lintr for static code analysis.

To prettify your code, use styler.

Get automatic advice about your package

Install and run goodpractice for “Advice (that) includes functions and syntax to avoid, package structure, code complexity, code formatting, etc.”.

Remove typos!

Use the devtools::spell_check() function. Typos in documentation are not the end of the world but can annoy the user, and it’s not hard to remove them thanks to this tool, so do it when you get a chance.

All these tools provide you with a list of things to change. This means they create a to-do list without your needing to think too much, so if you’re a bit tired but want to improve your package, you can let them fix the agenda!

Bring the package to its users and get feedback

These two aspects are linked because your users will help you make your package better for them, but for that to happen they need to know about your package and to understand it.

Documentation!

You know how your package works but the users don’t (and future you might forget as well!). Document your parameters, functions, add examples, and also write vignettes. At least one vignette will show your users how the different functions of your package interact.

At the beginning of this document I mentioned that when planning your package you should think about making it compatible with your users workflow. One of the vignettes can be an example of just that. Otherwise, it’ll be hard for users to see how useful your package is… and when writing the example workflow, you might find your package lacks some functionality.

Once you’ve written all these documentation parts, create a website for your package using pkgdown. This tutorial is excellent.

User-friendly programming

I’ll give four tips here, partly inspired by this essay by François Chollet. I want to share these two sentences from the essay with you: " In the long run, good design wins, because it makes its adepts more productive and more impactful, thus spreading faster than user-hostile undesign. Good design is infectious.”.

Have error and warning messages that are easy to understand. For instance, if communicating with an API, then translate http errors.
Check the input. For instance, If a parameter provided to a function should be a positive number, check it is and if it is not, return a friendly error message.
Offer shortcuts. For instance, if your function relies on an API key being provided, you can silently search for it in environment variables without the user doing anything, provided you explain how to store an environment variable in the documentation. This is how my opencage works if you want an example. Noam Ross made the PR that added this functionality, and I agree that it makes the package user-friendler. Likewise, have good default values of other parameters if possible.
Choose a good naming scheme that will allow your users to more easily remember the names of functions. You could choose to have a prefix for all the functions of your package, which is recommended by rOpenSci, and discussed in this very interesting and nuanced blog post by Jon Calder.

Promotion

Now that you have a well-documented error-friendly package… how are your end users going to learn that it exists? Note that you should have a good idea of your potential users when planning your package (or at least part of them, who knows who else might find it useful?). One way to define good venues for promotion is to think of how you learn about R packages, but as an R developer you’re probably more into R and R news than your future average user! For information you can also see the results of Julia Silge’s survey about how people discover packages here.

Obvious venues for promotion are social media, for instance Twitter with the #rstats hashtag in the form of a carefully crafted tweet including a link and a cool image like a figure or screenshot and maybe emojis, but that might not be enough for your niche package. If you write an ecology package and you have a well-recognized blog in that area blog about your package of course. You could also ask someone to let you guest post on their well-recognized niche field blog.

If your package is an interface to an existing tool or internet resource, try to have your package listed on that tool or resource webpage.

As a scientist, you can write a paper about your package and publish it to a journal well known in your field like Methods in Ecology and Evolution. By the way when you use R and packages in such a paper or any other paper do not forget to cite them (using the output of citation() for instance) to acknowledge their usefulness and help their authors feel their work is valued (furthermore if they’re academics having their work cited is quite crucial for their career advancement).

Review

As a scientist you’re used to having papers peer-reviewed. In general if you submit a paper about your package, your paper will be reviewed more than your package, but you can actually get the benefits of peer-review for your package itself! This is a fantastic way to improve the quality and user-friendliness of your package.

Where to have your package reviewed?

You don’t need anything formal, maybe you can ask someone you know and who could for instance be an user of your package? You can ask them to review the code or its documentation or both, depending on their skills and time of course.
The R Journal asks reviewers of papers to have a look at the package, as does The Journal of Statistical Software. That said, you’ll need to write a whole paper to go with your package which is maybe something you don’t want to.
The Journal of Open Science Software, JOSS asks reviewers to check some points in your package and you to write a very minimal paper about your package. The review process is minimal but you can still get useful feedback.
At rOpenSci we review packages that fit in our categories. Your package will be reviewed by two people. We have a partnership with the Journal of Open Source Software so that if your package fits in both venues, the review over at JOSS will be a breeze after the rOpenSci reviews. We also have a partnership with MEE, allowing a duo paper-package to be reviewed by MEE and rOpenSci respectively, see this post.

Needless to say, but I’ll write it anyway, all these review systems work both ways… Offer your reviewing services! You’ll actually learn so much that you’ll find ideas to improve your own packages.

Make your repo a friendly place

Another way to improve the quality of your package and make it more user friendly is to put people at ease when they have a question, a bug report, or a suggestion for improvement. Adding a code of conduct (use devtools::use_code_of_conduct as a starting point) and a contributing guide (with issue and PR templates if you want, in any case making clear what kind of contributions you welcome) to your package repository will help ensuring that. Also try to respond timeously to requests.

On top of having a contributing guide you could also add a status badge for your repository so that any visitor might see at a glance whether your package is still in an early development stage.

If people actually start contributing to the package, try being generous with acknowledgements. A selfish reason is that it’ll motivate people to keep contributing to your package. A less selfish reason is that it makes the whole R community more welcoming to new contributors. So you can add discoverers and solvers of bugs for instance as ctb in DESCRIPTION and add a line about their contribution in NEWS.md.

Package analytics

If you’re curious, you can get an idea of the number of downloads of your package via the cranlogs package. Using it you can compare your package popularity to similar packages, and check whether there was a peak after a course you’ve given for instance.

Similarly, if your package repository is hosted on Github, using the gh package you can do similar analyses, looking at the number of stars over time for instance.

The end

This is the end of my list of tips, do not hesitate to add suggestions in the comments!

Names of b.....s badder than Taylor Swift, a class in women's studies?

Tue, 05 Dec 2017 00:00:00 +0000

Once again, a Twitter trend sent me to my R prompt… Here is a bit of context. My summary: Taylor Swift apparently plays the bad girl in her new album and a fan of hers asked a question…

Name a bitch badder than Taylor Swift 😍😛😤 pic.twitter.com/AkSyQBUIME
— Nutella (@xnulz) 10 de novembre de 2017

The tweet was then quoted by many people mentioning badass women, and I decided to have a look at these heroes!

Name tweets badder than Nutella’s

I was a bit lazy and asked Mike Kearney, rtweet maintainer, how to find tweets quoting a tweet, to which Bob Rudis answered. Now that I even had the code, it was no trouble at all getting the data. I added the filtering steps myself, see, I’m not that lazy. I also removed the link to the quoted tweet that was at the end of each tweet.

question_tweet <- "928857792982781952"
badass <-  rtweet::search_tweets(question_tweet, n = 18000, include_rts = FALSE)
badass <- dplyr::filter(badass, is_quote_status, quote_status_id == question_tweet)
badass <- dplyr::mutate(badass, text = stringr::str_replace(text, "https://t\\.co/.*$", ""))
badass <- dplyr::mutate(badass, text = trimws(text))
readr::write_csv(badass, path = "data/2017-12-03-badderb_badass.csv")

I obtained 15653 tweets. Not bad!

library("magrittr")
set.seed(20171015)

indices <- sample.int(n = nrow(badass), size = 7)
badass$text[indices]

## [1] "Carmina Barrios"                                                                                                                                                                                                      
## [2] "Anyone"                                                                                                                                                                                                               
## [3] "Shirley Temple"                                                                                                                                                                                                       
## [4] "So a lot of people have shared some bad ass women in repsonse to the tweet below. I hope someone is compiling those responses for one kick ass book."                                                                 
## [5] "Mary Bowers was a slave with a photographic memory who pretended to be \"slow-witted\" in order to spy on confederate soldiers in the house worked, and pass intel she gathered to Union forces during the Civil War."
## [6] "Ramona Quimby, age 8"                                                                                                                                                                                                 
## [7] "Snow White. Sleeping Beauty. Cinderella, Elsa, Ariel bec the category example is obvs mediocre Disney caricatures allowed by the patriarchy, right?"

Unnamed badder b…..s: your mother and grand-mother

Out of 15653, 570 contained the word “mother” – I haven’t looked for the word “mum” and haven’t checked for the fact that it is someone from the family of the tweet author. Here a few of the personal stories (or not) identified with this quick and dirty method.

set.seed(20171015)
mothers <- dplyr::filter(badass, stringr::str_detect(text, "mother"))
indices <- sample.int(n = 15)
mothers$text[indices]

##  [1] "My grandmother struggled through poverty her entire life, in a family prone to depression, addiction, and suicide. She had so many Caesarian sections that when she reached menopause she didnt have a belly button anymore."                                                       
##  [2] "Mom married my dad. He had been married 2X before with a total of 10 kids. His first wife left him and the kid.  2nd wife died after their divorce.  He had custody of all 10, judge asked my mother if she would take them in, she did. Raised his 10 and had 3 with dad. 13 in all"
##  [3] "Each of my grandmothers raised 11 kids <f0><U+009F><U+0098><U+008F>"                                                                                                                                                                                                                 
##  [4] "My mother"                                                                                                                                                                                                                                                                           
##  [5] "My grandmother."                                                                                                                                                                                                                                                                     
##  [6] "My mother in law, who isnt a bitch in any way, shape, or form, and raised three girls on her own (and who are all bad asses in their own way) without the help of a deadbeat ex."                                                                                                 
##  [7] ".@RuPaul for starters. And my great grandmother who drove a car when she was 12. My dog, too. She is definitely a bad bitch."                                                                                                                                                        
##  [8] "My grandmother escaped from the Tsar with nothing but the clothes on her back."                                                                                                                                                                                                      
##  [9] "My mother"                                                                                                                                                                                                                                                                           
## [10] "Almost any woman alive today,@xnulz \n\nAnd - Heres another one, for sure (PLUS shes a good mother):\n\nTonya Harding; @ITonyaMovie \n\n<U+26F8><f0><U+009F><U+00A5><U+008A><U+26F8><f0><U+009F><U+00A5><U+008A><U+2764><U+FE0F><f0><U+009F><U+00A5><U+008A><U+26F8><f0><U+009F><U+00A5><U+008A><U+26F8>"
## [11] "all of the single mothers doing the most they can for their children"                                                                                                                                                                                                                
## [12] "My grandmother was married off at the age of 14 to an older man who had already been married once, dealt with an abusive marriage but stuck around for the kids and went back to school after having 5 children and worked for 20 years to support her family"                       
## [13] "my grandmother was an army nurse in WW2.\ntaught me how to tourniquet a leg and bandage it using only gauze"                                                                                                                                                                         
## [14] "My wife birthed a goddamn child. My mother and grandmother both birthed multiple. This is too easy."                                                                                                                                                                                 
## [15] "My grandmother worked for the OSS in London during WW2 as a code breaker."

Can we talk about that belly button thing?! I’m also happy to see a diversity of things they were recognized for.

Names of the badder b…..s

Quite a few of the tweets from this trend contained the name of someone. In order to extract these names, I resorted to a language processing method called entity extraction, the entity here being a person. For that, I could have used an extractor module of the Monkeylearn platform via my own monkeylearn package.

Instead, I chose to illustrate a different method: using the cleanNLP package that I know from the excellent R Journal paper presenting it. Among other things, it serves as an interface between R and the Python library spaCy and also as an interface between R and the coreNLP Java library. Installing these tools is the painful part of the setup, but 1) you only need to install one of them 2) there are detailed instructions here 3) once your tool is installed, using the package is a breeze (and well independent of any rate limit contrary to monkeylearn use). I am at that breeze stage, you can be jealous.

There were a few tweets with infuriating encoding issues, BOM or something like that, and I decided to just ignore them by using purrr::possibly. I obviously did this to illustrate the use of this purrr function, not out of laziness.

library("cleanNLP")
init_spaCy()
# we need to remove characters like "\u0098"
badass <- dplyr::mutate(badass, text = enc2native(text))

get_entities_with_text <- function(x){
  obj <- run_annotators(x, as_strings = TRUE)
  entities <- get_entity(obj)
  entities$text <- x
  entities
}

possibly_get_entities <- purrr::possibly(get_entities_with_text,
                                         otherwise = NULL)

entities <- purrr::map_df(badass$text, possibly_get_entities)

readr::write_csv(entities, path = "data/2017-12-03-badderb_entities.csv")

I got at least one entity for 7504 out of 15653 tweets, and at least one person for 4664. I am very satisfied with this.

So, who are you, badder b…..s?

We get this kind of entities: NORP, CARDINAL, LANGUAGE, GPE, DATE, ORG, PERSON, TIME, LOC, MONEY, WORK_OF_ART, EVENT, FAC, QUANTITY, LAW, PRODUCT, ORDINAL, PERCENT. I’m more interested in PERSON and no, I’m not shouting. I chose to look at the top 12 in order to get a top 10 excluding Taylor Swift herself.

entities %>%
  dplyr::filter(entity_type == "PERSON") %>%
  dplyr::group_by(entity) %>%
  dplyr::summarise(n = n()) %>%
  dplyr::arrange(- n) %>%
  head(n = 12) %>%
  knitr::kable()

entity	n
Taylor Swift	213
Taylor	145
Rosa Parks	140
Harriet Tubman	109
Dora	90
Rose West	85
Lyudmila Pavlichenko	77
Joan	71
Marie Curie	57
Myra Hindley	50
Nancy Wake	45
Hillary Clinton	41

At that point I did feel like bursting out laughing though. Dora! And I checked, we’re talking about Dora the explorer! Joan is Joan of arc. Interestingly in that top 10 we’re mixing really bad persons, e.g. Myra Hindley was a serial killer, and really badass persons, like Rosa Parks. My husband will be happy to see Marie Curie in this list, since he’s a big fan of hers, having even guided a few tours about her life in Paris.

Looking at the most frequently mentioned women obviously makes us loose well wrongly written names, and most importantly personal stories of badass mothers and the like, and of native women for instance, although I have the impression of having read about a few but probably because of my following Auriel Fournier.

Writing history?

I saw someone said they’d use the tweets as basis for history lessons. In order to get a view of a person, one could concatenate the tweets about them. Take Marie Curie for instance.

entities %>%
  dplyr::filter(entity_type == "PERSON", entity == "Marie Curie") %>%
  dplyr::summarise(text = toString(text)) %>%
  .$text

## [1] "Rachel Carson, Marie Curie, Ruth Bader Ginsberg, Madeleine Albright, Diane Fossey, Helen Keller, Gloria Steinem, Madonna, Aretha Franklin, Margot Lee Shetterly, Malala Yousafzai, and a whole lot more., Rosa Parks. Harriet Tubman. Anne Frank. Malala Yousafazi, Susan B. Anthony, Sally Ride, Marie Curie. Margaret Thatcher, Indira Ghandi, Golda Meier., ...its not bitch. Its woman.\n\nMarie Curie\n\nRosa Parks\n\nEleanor Roosevelt\n\nHedy Lamar\n\nSappho\n\nAbagail Adams\n\nFlorence Nightingale\n\nSally Ride\n\nMargaret Chase Smith\n\nAnne Frank\n\nMargaret Thatcher \n\nSandra Day OConnor\n\nOprah\n\nLilith\n\nMarilyn Monroe\n\nDita Von Teese, Ruby Bridges, Barbara Jordan, Marie Curie, Rosa Parks, Ida B Wells, Susan B Anthony, Harriet Tubman, ..., Lorde. Etc., I mean, off hand, Marie Curie was denied access to University because she was a woman, educated herself, served as a surgeon in WW1 and became the only person ever to earn two Nobel prizes, Emmeline Pankhurst, Amelia Earhart, Florence Nightingale, Rosa Parks, Joan of Arc, Marie Curie, Lilian Ngoyi, Helen Suzman, Katherine Johnson,  Frida Khalo, Catherine the Great, Eleanor Roosevelt, Mary Wollstonecraft, Valentina Tereshkova.., lmao Elizabeth Schuyler-Hamilton, Hedy Lamar, Amelia Earhart, Marie Curie, Sophie Scholl, Katherine Johnson, Dorothy Vaughn, Mary Jackson, Margaret Hamilton, Ada Lovelace, Anne Bonny, Calamity Jane, George Eliot, Mary Read, Mary Wolstncraft, Mary Shelley... need I go on?! <f0><U+009F><U+0098><U+00A1>, Marie Curie, Rose Parks\nFrancis Perkins\nJeanette Rankin\nSally Ride\nMarie Curie\nJK Rowling\n\nShall I go on?, Marie Curie: she invented radioactivity, got the Nobel prize twice and found polonium and radium., Well I would start with not calling a woman a bitch.\nAfter that I would say Hedy Lamarr, Susan B. Anthony, any of the @HiddenFigures, Marie Curie, Sally Ride, Molly Pitcher, Clara Barton  shall I continue?, Marie Curie literally died doing Nobel Prize winning research., Harriet Tubman, Shirley Chisholm, Rosa Parks, Ida B. Wells, Marie Curie, Maxine Waters, Lena Horne,  Ruby Dee, Angela Davis, My grandmama, my great-aunt Priscilla, my mama, my sister..., Marie Curie. Mother of radioactivity. First and only woman to win the nobel prize twice. Died due to radiation exposure., Rosa Parks, Freya Stark, Ida B Wells, Sally Ride, Marie Curie, Margaret Heafield, Amelia Earheart, Ruth Bader Ginsburg, Abigail Adams, Jane Goodall, Malala how much time you got?, Marie Curie made mobile x-ray unit, learned anatomy &amp; auto repairs, trained women, raised money, ran IRC's medical x-ray division in WWI., Marie Curie. No explanation needed., Rosa Parks, Katherine Johnson, Dorothy Vaughn, Mary Jackson, Bettie Page, Emma Watson, Malala Yousafzai, Emily Pankhurst, Anne Frank, Marie Curie, Joan of Arc, Boudicca... need I go on?, Marie Curie., Malala, Michelle Obama, Frida Kahlo, Emma Watson, Maya Angelou, Marie Curie, Lady Gaga, JK Rowling, Princess Diana, Carrie Fisher, Anne Frank, Laverne Cox...\n\n literally so many women... I could go on for weeks..., Marie Curie, Mary Anning, Dorothy Hodgkin, Jane Austin, Katherine Johnson, my GCSE geography and A Level geology teacher Sheila Tanner...., Marie Curie. She handled radioactive materials with barehand and won the Nobel Prize. How bout that? #RealTimeChem, Rosa Parks, Nellie Bly, Harriet Tubman, Jane Adams, Eleanor Roosevelt, Elizabeth Blackwell, Marie Curie, Amelia Earhart, Cleopatra, Queen Elizabeth I, Ella Fitzgerald, Grace Hopper, Dolores Huerta, Shirley Jackson, Joan of Arc, Helen Keller, Sacagawea, Sappho the Greek poet..., Marie Curie., Marie Curie., Marie Curie, Maya Angelou, Rosa Parks, Michelle Obama, my  grandma, my neuroscientist mum, Danica Patrick, the Queen of England, Angela Merkel,  @chrissyteigen @BrookeBCNN @KateBolduan - Me, Rosa Parks, Harriet Tubman, Shirley Chisholm, Ella Fitzgerald, Ruth Bader Ginsburg, Esperonza Spaulding, Sister Rossetta Tharpe, Sarah Vaughn, Aretha Franklin, Eartha Kitt, Carrie Fisher, Patti Smith, Jane Goodall, Marie Curie, Sally Ride, Leontyne Price, Princess Diana, etc..., Whoa hey now, Marie Curie won Nobel Prizes in two different sciences, only person to have done that. Probably had a lovely singing voice, too, before it got demolished by the radioactivity she was discovering for the betterment of humanity., Rosa Parks. Laura Secord. Hillary Rodham Clinton. Nellie McClung. Queen Victoria. Joan of Arc. Myrlie Evers-Williams. Tina Turner. Jane Goodall. Ruth Bader Ginsburg. Viola Desmond. Jennie Trout. Malala  Yousafzai. Margret Atwood. Ameila Earhart. J.K. Rowling. Marie Curie., Marie Curie. Won the Nobel Prize. Twice., Just off the top of my head: Elizabeth Cady Stanton, Clara Barton, Marie Curie, Amelia Earhart, Elizabeth I, Joan of Arc, Helen Keller., Amelia Earhart, Rosa Parks, Sacagawea, Pocahontas, Susan B. Anthony, Noor Inayat Khan, Marie Curie, Alice Paul, Margaret Thatcher, Harriet Tubman, Maya Angelou, Lucy Stone, Eleanor Roosevelt ..... just to name a few., Marie Curie., The kashariyot. Shirley Chisholm. bell hooks. Golda Meir. Edith Bunker. Hedy Lamar. Marie Curie. Bella Abzug. Michelle Obama. Judith Halberstam. Jo March. Hillary Rodham Clinton., Hedy Lamarr. Marie Curie. Mae Jemison. Sally Ride. Sonia Sotomayor. Ruth Bader Ginsburg. Beyoncé. Mulan. My wife., Marie Curie is the only person to win two Nobel Prizes in two separate scientific fields and died of leukaemia as a result of her research into radionuclides which are used in medicine to save thousands of lives a year, Marie Curie.\n2 Nobel Prizes., Marie Curie who died discovering two elements and isolated radioactive isotopes., Rosa Parks, Marie Curie, Charlize Theron, Noor Inayat Khan, Rihanna, Serena Williams, Ruth Bader Ginsberg, basically all these women on this list:, Marie Curie developed the theory of radioactivity, techniques for isolating isotopes, and discovered two chemical elements. She died of aplastic anemia from exposure to radiation over the course of her research and her work at field hospitals during WWI., Marie Curie won two Nobel prizes for two different fields of science and her notebook is so radioactive that exposure to it would be lethal., How much time have you got? How about Marie Curie, or Amelia Earhart, Nancy Hart? Arlene Limas? Tomoe Gozen? Lucretia Mott? Irina Sendler? Nothing against Ms. Swift, but we could do this all day, Marie Curie, Nobel prizes in Physics and Chemistry., Marie Curie? 1st woman in Europe to get a PhD and 1st woman to win a Nobel Prize (which she did twice, in Physics and Chemistry). She also developed a theory of a radioactivity &amp; found the elements radium and polonium. A pretty bad bitch., Marie Curie was a badass scientist who won two Nobel prizes and ate radium for breakfast. #thatlastpartwasntsogood, Rosa Parks? Harriet Tubman? Louisa May Alcott? Susan B. Anthony? Marie Curie? Amelia Earhart? Cleopatra? Frida Khalo? Mother Theresa? Sandra Day OConnor? Oprah Winfrey? Sally Ride? Margaret Sanger? Anne Frank? Joan of Arc? Sacajawea? Sappho? Sojourner Truth? Malala Yousafzai?, Cleopatra. Mata Hari. Marie Curie. Susan B. Anthony. Hannah Arendt. . . . \n\nIt's really not hard, 1. Malala Yousafzai, Marie Curie, Rosa Parks, Georgia O'Keeffe, Maria Mitchell, Tanya Tagaq, Billie Holiday, Valentina Tereshkova, Sally Ride, Anne Frank, Maya Angelou, Sacheen Littlefeather, Frida Kahlo, Harper Lee, Marsha P Johnson, Sylvia Rivera, Helen Keller, Margaret Sanger,, Marie Curie, Marie Curie., Ada Lovelace, Jane Austen, Helen Sharman, Margaret Atwood, Mary Beard, Rosalind Franklin, Hedy Lamarr, Marie Curie, Helen Keller, Emmeline Pankhurst, Rosa Parks, Florence Nightingale, Brenda Hale, QE1, Boudicca, Amy Johnson, Elizabeth Fry, Octavia Hill, Mary Wollstonecraft, &amp;c.<f0><U+009F><U+0096><U+00A4>, Marie Curie. Most people are lucky to have 1 Nobel prize, or share it. She has 2., Sojourner Truth, Khutulun, Marie Curie, Queen Lillikulani, Hedy Lamar, Empress Wu Zetain, Hellen Mirren, Raden Ajeng Kartini, Ida B. Wells, Artemisia Gentileschi, Nancy Wake, Tomoe Gozen, Yaa Asantewaa, Boudicca, Juana Inés de la Cruz..., Marie Curie, whose greatest discovery and lifes work also ended up being what killed her and who is the only person to win a Nobel Prize in two different sciences., Joan of Arc, Marie Curie, Queen Elizabeth, Cleopatra, my mom, the lady in front of me on line at the grocery store...\n\nIt's a pretty long list., Rosa Parks. All of the women from Hidden Figures. Marie Curie. Any woman in STEM fields., Marie Curie \nThat's where the list starts"

Doing this one also gets the name of many other women. Moreover, if writing history lessons, one should have several sources, right? What about Wikidata like in this other blog post of mine? It should have data for at least the most famous badass women.

# add a function for getting a silent answer
quietly_query <- purrr::quietly(WikidataQueryServiceR::query_wikidata)

# function for getting someone's data
get_wikidata <- function(name, pb = NULL){
  if (!is.null(pb)) pb$tick()$print()
  Sys.sleep(1)
  item <- WikidataR::find_item(name, language = "en")
  # sometimes people have no Wikidata entry so I need this condition
  if(length(item) > 0){
    entity_code <- item[[1]]$id
    query <-  paste0("PREFIX entity: <http://www.wikidata.org/entity/>
                     #partial results
                     
                     SELECT ?propUrl ?propLabel ?valUrl ?valLabel ?picture
                     WHERE
                     {
                     hint:Query hint:optimizer 'None' .
                     {	BIND(entity:",entity_code," AS ?valUrl) .
                     BIND(\"N/A\" AS ?propUrl ) .
                     BIND(\"identity\"@en AS ?propLabel ) .
                     }
                     UNION
                     {	entity:", entity_code," ?propUrl ?valUrl .
                     ?property ?ref ?propUrl .
                     ?property rdf:type wikibase:Property .
                     ?property rdfs:label ?propLabel
                     }
                     
                     ?valUrl rdfs:label ?valLabel
                     FILTER (LANG(?valLabel) = 'en') .
                     OPTIONAL{ ?valUrl wdt:P18 ?picture .}
                     FILTER (lang(?propLabel) = 'en' )
                     }
                     ORDER BY ?propUrl ?valUrl
                     LIMIT 200")
    results <- quietly_query(query) 
    results <- results$result
    results$name<- name
    results
  }else{
    NULL
  }
   
  }

Yes, I just had to replace all occurrences of “sv” with “en” to get a function for this post. I’d like to try to write an automatic text about badass women.

get_a_string <- function(prop, prep, wikidata){
  answer <- dplyr::filter(wikidata, propLabel == prop) %>%
    .$valLabel %>%
    unique() %>%
    toString
  if(answer == ""){
    return("")
  }else{
    return(paste(prep, answer))
  }
}

tell_me_about <- function(name){
  wikidata <- get_wikidata(name)
  questions <- c("occupation", "country of citizenship",
                 "field of work", "award received")
  
  words <- c("a", "from", 
             "known from her work in", "and who was awarded")
  
  strings <- purrr::map2_chr(questions, words, 
                            get_a_string,
                            wikidata = wikidata)
  
  strings <- strings[strings != ""]
  
  sentence <- paste(name, "was", toString(strings))
  sentence <- paste0(sentence, ".")
  return(sentence)
}

Ok, let’s try our automatic history writing function. It won’t work for Dora and Joan, sadly.

tell_me_about("Lyudmila Pavlichenko")

## [1] "Lyudmila Pavlichenko was a historian, sniper, military personnel, from Soviet Union, and who was awarded Medal \"For the Defence of Sevastopol\", Medal \"For the Defence of Odessa\", Hero of the Soviet Union, Order of Lenin, Medal \"For Battle Merit\", Gold Star, Medal \"For the Victory over Germany in the Great Patriotic War 19411945\"."

tell_me_about("Myra Hindley")

## [1] "Myra Hindley was a criminal, from United Kingdom."

tell_me_about("Harriet Tubman")

## [1] "Harriet Tubman was a writer, from United States of America, and who was awarded New Jersey Hall of Fame, National Women's Hall of Fame, Maryland Women's Hall of Fame."

Not many details clearly, but not too bad for a quickly written history hum bot, if I can call it so.

So, happy, Nutella?

This was my contribution to the meme following Nutella’s viral tweet. I am thankful for the badass women I did end up discovering thanks to the tweets, and am waiting for someone to replace the lyrics of all Taylor Swift’s songs with gems from this Twitter trend.

Where to live in the US

Thu, 16 Nov 2017 00:00:00 +0000

I was fascinated by this xkcd comic about where to live based on your temperature preferences. I also thought it’d be fun to try to make a similar one from my R session! Since I’m no meteorologist and was a bit unsure of how to define winter and summer, and of their relevance in countries like, say, India which has monsoon, I decided to focus on a single country located in one hemisphere only and big enough to offer some variety… the USA! So, dear American readers, where should you live based on your temperature preferences?

Defining data sources

Weather data

The data for the original xkcd graph comes from weatherbase. I changed sources because 1) I was not patient enough to wait for weatherbase to send me a custom dataset which I imagine is what xkcd author did and 2) I’m the creator of a cool package accessing airport weather data for the whole word including the US! My package is called “riem” like “R Iowa Environmental Mesonet” (the source of the data, a fantastic website) and “we laugh” in Catalan (at the time I wrote the package I was living in Barcelona and taking 4 hours of Catalan classes a week!). It’s a simple but good package which underwent peer-review at rOpenSci onboarding, thank you Brooke!

I based my graph on data from the last winter and the last summer. I reckon that one should average over more years, but nothing important is at stake here, right?

Cities sample

My package has a function for downloading weather data for a given airport based on its ID. For instance Los Angeles airport is LAX. At that point I just needed a list of cities in the US with their airport code. Indeed with my package you can get all airport weather networks, one per US state, and all the stations within that network… this is a lot of airports with no way to automatically determine how big they are! And to get the city name since it’d be so hard to parse the airport name I’d have resorted to geocoding with e.g. this package. A bit complicated for a simple fun graph!

So I went to the Wikipedia page of the busiest airports in the US and ended up getting this dataset from the US Department of Transportation (such a weird open data format by the way… I basically copy-pasted the first two columns in a spreadsheet!). This is not perfect, getting a list of major cities in every state would be more fair but hey reader I want you to live in a really big city so that I might know where it is.

Ok so let’s get the data

us_airports <- readr::read_csv("data/2017-11-16_wheretoliveus_airports.csv")
knitr::kable(us_airports[1:10,])

Name	Code
Atlanta, GA (Hartsfield-Jackson Atlanta International)	ATL
Los Angeles, CA (Los Angeles International)	LAX
Chicago, IL (Chicago O’Hare International)	ORD
Dallas/Fort Worth, TX (Dallas/Fort Worth International)	DFW
New York, NY (John F. Kennedy International)	JFK
Denver, CO (Denver International)	DEN
San Francisco, CA (San Francisco International)	SFO
Charlotte, NC (Charlotte Douglas International)	CLT
Las Vegas, NV (McCarran International)	LAS
Phoenix, AZ (Phoenix Sky Harbor International)	PHX

I first opened my table of 50 airports. Then, I made the call to the Iowa Environment Mesonet.

summer_weather <- purrr::map_df(us_airports$Code, riem::riem_measures,
                                date_start = "2017-06-01",
                                date_end = "2017-08-31")
winter_weather <- purrr::map_df(us_airports$Code, riem::riem_measures,
                                date_start = "2016-12-01",
                                date_end = "2017-02-28")

We then remove the lines that will be useless for further computations (The xkcd graph uses average temperature in the winter and average Humidex in the summer which uses both temperature and dew point).


summer_weather <- dplyr::filter(summer_weather,
                                !is.na(tmpf), !is.na(dwpf))

winter_weather <- dplyr::filter(winter_weather,
                                !is.na(tmpf))

I quickly checked that there was “nearly no missing data” so I didn’t remove any station nor day but if I were doing this analysis for something serious I’d do more checks including the time difference between measures for instance. Note that I end up with 48 airports only, there was no measure for Honolulu, HI (Honolulu International), San Juan, PR (Luis Munoz Marin International). Too bad!

Calculating the two weather values

I started by converting all temperatures to Celsius degrees because although I like you, American readers, Fahrenheit degrees do not mean anything to me which is problematic for checking results plausibility for instance. I was lazy and just used Brooke Anderson’s (yep the same Brooke who reviewed riem for rOpenSci) weathermetrics package.

summer_weather <- dplyr::mutate(summer_weather,
                                tmpc = weathermetrics::convert_temperature(tmpf, 
                                                                           old_metric = "f", new_metric = "c"),
                                dwpc = weathermetrics::convert_temperature(dwpf, 
                                                                           old_metric = "f", new_metric = "c")) 

winter_weather <- dplyr::mutate(winter_weather,
                                tmpc = weathermetrics::convert_temperature(tmpf, 
                                                                           old_metric = "f", new_metric = "c"))

Summer values

Summer values are Humidex values. The author explained Humidex “combines heat and dew point”. Once again I went to Wikipedia and found the formula. I’m not in a mood to fiddle with formula writing on the blog so please go there if you want to see it. There’s also a package with a function returning the Humidex. I was feeling adventurous and therefore wrote my own function and checked the results with the numbers from Wikipedia. It was first wrong because I wrote a “+” instead of a “-”… checking one’s code is crucial.

calculate_humidex <- function(temp, dewpoint){
  temp + 0.5555*(6.11 *exp(5417.7530*(1/273.16 - 1/(273.15 + dewpoint))) - 10)
}

calculate_humidex(30, 15)

## [1] 33.969

calculate_humidex(30, 25)

## [1] 42.33841

And then calculating the summer Humidex values by station was quite straightforward…

library("magrittr")
summer_weather <- dplyr::mutate(summer_weather,
                                humidex = calculate_humidex(tmpc, dwpc))
summer_values <- summer_weather %>%
  dplyr::group_by(station) %>%
  dplyr::summarise(summer_humidex = mean(humidex, na.rm = TRUE))

… as it was for winter temperatures.

winter_values <- winter_weather %>%
  dplyr::group_by(station) %>%
  dplyr::summarise(winter_tmpc = mean(tmpc, na.rm = TRUE))

Prepare data for plotting

I first joined winter and summer values

climates <- dplyr::left_join(winter_values, summer_values,
                             by = "station")

Then I re-added city airport names.

climates <- dplyr::left_join(climates, us_airports,
                             by = c("station" = "Code"))

I only kept the city name.

head(climates$Name)

## [1] "Atlanta, GA (Hartsfield-Jackson Atlanta International)"              
## [2] "Austin, TX (Austin - Bergstrom International)"                       
## [3] "Nashville, TN (Nashville International)"                             
## [4] "Boston, MA (Logan International)"                                    
## [5] "Baltimore, MD (Baltimore/Washington International Thurgood Marshall)"
## [6] "Cleveland, OH (Cleveland-Hopkins International)"

climates <- dplyr::mutate(climates, 
                          city = stringr::str_replace(Name, " \\(.*", ""))
head(climates$city)

## [1] "Atlanta, GA"   "Austin, TX"    "Nashville, TN" "Boston, MA"   
## [5] "Baltimore, MD" "Cleveland, OH"

Plotting!

When imitating an xkcd graph, one should use the xkcd package! I had already done that in this post.

library("xkcd")
library("ggplot2")
library("extrafont")
library("ggrepel")

xrange <- range(climates$summer_humidex)
yrange <- range(climates$winter_tmpc)

set.seed(42)

ggplot(climates,
       aes(summer_humidex, winter_tmpc)) +
  geom_point() +
  geom_text_repel(aes(label = city),
                   family = "xkcd", 
                   max.iter = 50000)+
  ggtitle("Where to live based on your temperature preferences",
          subtitle = "Data from airports weather stations, 2016-2017") +
  xlab("Summer heat and humidity via Humidex")+
  ylab("Winter temperature in Celsius degrees") +
  xkcdaxis(xrange = xrange,
           yrange = yrange)+
  theme_xkcd() +
  theme(text = element_text(size = 16, family = "xkcd"))

So, which city seems attractive to you based on this plot? Or would you use different weather measures, e.g. do you care about rain?

You beautiful, naïve, sophisticated newborn series

Sun, 05 Nov 2017 00:00:00 +0000

My husband and I recently started watching the wonderful series “Parks and recreation” which was recommended to me by my fellow R-Lady Jennifer Thompson in this very convincing thread. The serie was even endorsed by other R-Ladies. Jennifer told me the first two seasons are not as good as the following ones, but that it was worth it to make it through them. We actually started enjoying the humor and characters right away!

Then, this week while watching the show, one of the characters did a very basic text analysis that made me feel like imitating him for a blog post – my husband told me it was very Leslie of me to plan something while doing something else which made me very proud. I tested my idea on other Leslie fans, and they seemed to think it was a great idea… and that this post should be the beginning of a series of R-Ladies blog posts about Parks and recreation!

In this two-short-part blog post, I’ll therefore inaugurate this series, what an honor!

Less than stellar first seasons?

Jennifer told me the first two seasons were not the best ones. My brother who is a cruel man (just kidding) told me this was a very good joke idea: you tell someone a very bad TV series starts getting awesome after a few seasons that you need to watch in order to know the characters… therefore forcing them to loose their time and sanity in front of the screen, ah! Luckily Jennifer wasn’t doing that.

I said my husband and I were hooked on the series right from the first episode… what about other people? To answer this question, I downloaded IMDB ratings for the show! I first tried using R packages accessing the OMDB API, but this one was outdated and this up-to-date one was not but taught me the API no longer returns ratings by season… So I scraped the IMDB website using the rvest package, after checking I was allowed to do so thanks to the robotstxt package.

robotstxt::paths_allowed("http://www.imdb.com/title/tt1266020/eprate")

## [1] TRUE

library("rvest")
imdb_page <- html("http://www.imdb.com/title/tt1266020/eprate")
ratings <- imdb_page %>%
  html_nodes("table") %>%
  .[[1]] %>%
  html_table() %>%
  .[,1:4]

knitr::kable(ratings[1:5,])

#	Episode	UserRating	UserVotes
7.13	One Last Ride: Part 2	9.7	2,407
7.12	One Last Ride: Part 1	9.6	1,934
7.40	Leslie and Ron	9.6	1,842
6.22	Moving Up: Part 2	9.4	1,273
5.14	Leslie and Ben	9.3	1,205

I just had to wrangle this table a little bit.

names(ratings)[1] <- "number"
ratings <- tidyr::separate(ratings, col = number,
                           into = c("season", "episode"),
                           sep = "\\.")
ratings <- dplyr::mutate(ratings, UserVotes = stringr::str_replace(UserVotes, ",", ""))
ratings <- dplyr::mutate(ratings, UserVotes = as.numeric(UserVotes))
ratings <- dplyr::mutate(ratings, episode = as.numeric(episode))

And then I could at last plot the ratings and see whether the two first seasons were not that loved! Before that I quickly assessed whether there was an ok number of votes for all episodes, and that this did not vary widely over seasons, which was the case.

library("ggplot2")
library("hrbrthemes")
ggplot(ratings, aes(episode, UserRating,
                    color = season)) +
  geom_smooth()+
  geom_point()  +
  ylab("IMDB rating") +
  viridis::scale_color_viridis(discrete = TRUE) +
  theme_ipsum(base_size = 20,
              axis_title_size = 20)

Note that for the first season, there are too few points for actually doing a smoothing. Looking at the plot I guess we can say the first season with its only 6 episodes was slightly less enjoyed! I am very thankful that the first season was deemed good enough to continue the series though! Besides I am thankful that it was so simple to answer this crucial question.

Leslie’s wordcloud

Warning: the wordcloud produced here might be a spoiler if you have not watched the 4 first seasons!

In the 10th episode of season 4, Tom produces a wordcloud of Leslie’s emails and memos. I cannot really do the same, but was inspired to make a worldcloud of things said in the episodes by downloading all subtitles from the 4 first seasons and using the fantastic subtools package. Note that a worldcloud is probably not the coolest thing one can do with all these text data… but it is a start and I truly hope it inspires more people to use François Keck’s package which is really user-friendly since it can read all the subtitles of a serie at once from a folder where each sub-folder is a season. It also supports conversion to formats suitable for text analysis!

The single problem I experienced when doing this work was that I had to download subtitles from the internet and the websites where you find those are not cute places, they’re full of “women to date”…

To make the wordcloud I very simply followed this blog post of François Keck’s. It doesn’t get easier than that… which is great given I have very little time at the moment.

library("subtools")
library("tm")
library("SnowballC")
library("wordcloud")
parks_and_rec <- read.subtitles.serie(dir = paste0(getwd(), "/data/pr/"))

## Read: 4 seasons, 68 episodes

corpus <- tmCorpus(parks_and_rec)

corpus <- tm_map(corpus, content_transformer(tolower))
corpus <- tm_map(corpus, removePunctuation)
corpus <- tm_map(corpus, removeNumbers)
corpus <- tm_map(corpus, removeWords, stopwords("english"))
corpus <- tm_map(corpus, stripWhitespace)

TDM <- TermDocumentMatrix(corpus)
TDM <- as.matrix(TDM)
vec.season <- c(rep(1, 6), rep(2, 24), rep(3, 16), rep(4, 22))
TDM.season <- t(apply(TDM, 1, function(x) tapply(x, vec.season, sum)))
colnames(TDM.season) <- paste("S", 1:4)

set.seed(1)
comparison.cloud(TDM.season, title.size = 1, max.words = 100, random.order = T)

It doesn’t look anything like Leslie’s wordcloud but it made me happy because I had already forgotten about some words or names from the previous seasons!

Another thing I did with the subtitles was filtering those starting with “Oh Ann” in order to have a look at some of the terrific compliments Leslie made to Ann… And one of them inspired the title of this post, replacing “baby” by “series”!

The end of this post, the beginning a series!

Now that I’ve done my homework I’m truly looking forward to reading the other posts. I can’t wait! Stay tuned by following my fellow bloggers/Leslie fans who came out in this Twitter thread.

R-Ladies global tour

Fri, 06 Oct 2017 00:00:00 +0000

It was recently brought to my attention by Hannah Frick that there are now sooo many R-Ladies chapters around the world! R-Ladies is a world-wide organization to promote gender diversity in the R community, and I’m very grateful to be part of this community through which I met so many awesome ladies! Since we’re all connected, it has now happened quite a few times that R-Ladies gave talks at chapters outside of their hometowns. An R-Lady from Taiwan giving a talk in Madrid while on a trip in Europe and another one doing the same in Lisbon, an R-Lady from San Francisco presenting at the London and Barcelona chapters thanks to a conference on the continent, an R-Lady from Uruguay sharing her experience for the New York City and San Francisco chapters… It’s like rockstars tours!

Therefore we R-Ladies often joke about doing an exhaustive global tour. Hannah made me think about this tour again… If someone were to really visit all of the chapters, what would be the shortest itinerary? And could we do a cool gif with the results? These are the problems we solve here.

Getting the chapters

To find all chapters, I’ll use Meetup information about meetups whose topics include “r-ladies”, although it means forgetting a few chapters that maybe haven’t updated their topics yet. Thus, I’ll scrape this webpage because I’m too impatient to wait for the cool meetupr package to include the Meetup API topic endpoint and because I’m too lazy to include it myself. I did open an issue though. Besides, I was allowed to scrape the page:

robotstxt::paths_allowed("https://www.meetup.com/topics/")

## [1] TRUE

Yesss. So let’s scrape!

library("rvest")

link <- "https://www.meetup.com/topics/r-ladies/all/"
page_content <- read_html(link)
css <- 'span[class="text--secondary text--small chunk"]'

chapters <-  html_nodes(page_content, css) %>% html_text(trim = TRUE)
chapters <- stringr::str_replace(chapters, ".*\\|", "")
chapters <- trimws(chapters)
head(chapters)

## [1] "London, United Kingdom" "San Francisco, CA"     
## [3] "Istanbul, Turkey"       "Melbourne, Australia"  
## [5] "New York, NY"           "Madrid, Spain"

# Montenegro
chapters[stringr::str_detect(chapters, "Montenegro")] <- "Herceg Novi, Montenegro"

Geolocating the chapters

Here I decided to use a nifty package to the awesome OpenCage API. Ok, this is my own package. But hey it’s really a good geocoding API. And the package was reviewed for rOpenSci by Julia Silge! In the docs of the package you’ll see how to save your API key in order not to have to input it as a function parameter every time.

Given that there are many chapters but not that many (41 to be exact), I could inspect the results and check them.

geolocate_chapter <- function(chapter){
  # query the API
  results <- opencage::opencage_forward(chapter)$results
  # deal with Strasbourg
  if(chapter == "Strasbourg, France"){
    results <- dplyr::filter(results, components.city == "Strasbourg")
  }
  # get a CITY
  results <- dplyr::filter(results, components._type == "city")
  # sort the results by confidence score 
  results <- dplyr::arrange(results, desc(confidence))
  # choose the first line among those with highest confidence score
  results <- results[1,]
  # return only long and lat
  tibble::tibble(long = results$geometry.lng,
                 lat = results$geometry.lat,
                 chapter = chapter, 
                 formatted = results$formatted)
}

chapters_df <- purrr::map_df(chapters, geolocate_chapter)

# add an index variable
chapters_df <- dplyr::mutate(chapters_df, id = 1:nrow(chapters_df))

knitr::kable(chapters_df[1:10,])

long	lat	chapter	formatted	id
-0.1276473	51.50732	London, United Kingdom	London, United Kingdom	1
-122.4192362	37.77928	San Francisco, CA	San Francisco, San Francisco City and County, California, United States of America	2
28.9651646	41.00963	Istanbul, Turkey	Istanbul, Fatih, Turkey	3
144.9631608	-37.81422	Melbourne, Australia	Melbourne VIC 3000, Australia	4
-73.9865811	40.73060	New York, NY	New York City, United States of America	5
-3.7652699	40.32819	Madrid, Spain	Leganés, Community of Madrid, Spain	6
-77.0366455	38.89495	Washington, DC	Washington, District of Columbia, United States of America	7
-83.0007064	39.96226	Columbus, OH	Columbus, Franklin County, Ohio, United States of America	8
-71.0595677	42.36048	Boston, MA	Boston, Suffolk, Massachusetts, United States of America	9
-79.0232050	35.85030	Durham, NC	Durham, NC, United States of America	10

Planning the trip

I wanted to use the ompr package inspired by this fantastic use case, “Boris Johnson’s fully global itinerary of apology” – be careful, the code of this use case is slightly outdated but is up-to-date in the traveling salesperson vignette. The ompr package supports modeling and solving Mixed Integer Linear Programs. I got a not so bad notion of what this means by looking at this collection of use cases. Sadly, the traveling salesperson problem is a complicated problem and its solving time exponentially increases with the number of stops… in that case, it became really too long for plain mixed integer linear programming, as in “more than 24 hours later not done” too long.

Therefore, I decided to use a specific R package for traveling salesperson problems TSP. Dirk, ompr's maintainer, actually used it once as seen in this gist and then in this newspaper piece about how to go to all 78 Berlin museums during the night of the museums. Quite cool!

We first need to compute the distance between chapters. In kilometers and rounded since it’s enough precision.

convert_to_km <- function(x){
  round(x/1000)
}

distance <- geosphere::distm(as.matrix(dplyr::select(chapters_df, long, lat)), fun = geosphere::distGeo) %>% 
  convert_to_km()

I used methods that do not find the optimal tour. This means that probably my solution isn’t the best one, but let’s say it’s ok for this time. Otherwise, the best thing is to ask Concorde’s maintainer if one can use their algorithm which is the best one out there, see its terms of use here.

library("TSP")
set.seed(42)
result0 <- solve_TSP(TSP(distance), method = "nearest_insertion")
result <- solve_TSP(TSP(distance), method = "two_opt",
                    control = list(tour = result0))

And here is how to link the solution to our initial chapters data.frame.

paths <- tibble::tibble(from = chapters_df$chapter[as.integer(result)],
                        to = chapters_df$chapter[c(as.integer(result)[2:41], as.integer(result)[1])], 
                        trip_id = 1:41)
paths <- tidyr::gather(paths, "property", "chapter", 1:2)
paths <- dplyr::left_join(paths, chapters_df, by = "chapter")
knitr::kable(paths[1:3,])

trip_id	property	chapter	long	lat	formatted	id
1	from	Charlottesville, VA	-78.55676	38.08766	Charlottesville, VA, United States of America	38
2	from	Washington, DC	-77.03665	38.89495	Washington, District of Columbia, United States of America	7
3	from	New York, NY	-73.98658	40.73060	New York City, United States of America	5

Plotting the tour, boring version

I’ll start by plotting the trips as it is done in the vignette, i.e. in a static way. Note: I used Dirk’s code in the Boris Johnson use case for the map, and had to use a particular branch of ggalt to get coord_proj working.

library("ggplot2")
library("ggalt")
library("ggthemes")
library("ggmap")
world <- map_data("world") %>% dplyr::filter(region != "Antarctica")

ggplot(data = paths, aes(long, lat)) + 
  geom_map(data = world, map = world, aes(long, lat, map_id = region), 
           fill = "white", color = "darkgrey", alpha = 0.8, size = 0.2) + 
  geom_path(aes(group = trip_id), color = "#88398A") + 
  geom_point(data = chapters_df, color = "#88398A", size = 0.8) + 
  theme_map(base_size =20) + 
  coord_proj("+proj=robin +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs") + 
  ggtitle("R-Ladies global tour", 
          subtitle = paste0(tour_length(result), " km"))

Dirk told me the map would look better with great circles instead of straight lines so I googled a bit around, asked for help on Twitter before finding this post.

library("geosphere")

# find points on great circles between chapters
gc_routes <- gcIntermediate(paths[1:length(chapters), c("long", "lat")],
                            paths[(length(chapters)+1):(2*length(chapters)), c("long", "lat")],
                            n = 360, addStartEnd = TRUE, sp = TRUE, 
                            breakAtDateLine = TRUE)
gc_routes <- SpatialLinesDataFrame(gc_routes, 
                                   data.frame(id = paths$id,
                                              stringsAsFactors = FALSE))
gc_routes_df <- fortify(gc_routes)

p <- ggplot() + 
  geom_map(data = world, map = world, aes(long, lat, map_id = region), 
           fill = "white", color = "darkgrey", alpha = 0.8, size = 0.2) + 
  geom_path(data = gc_routes_df, 
            aes(long, lat, group = group), alpha = 0.5, color = "#88398A") + 
  geom_point(data = chapters_df, color = "#88398A", size = 0.8,
             aes(long, lat)) + 
  theme_map(base_size =20)+ 
  coord_proj("+proj=robin +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs")

p + 
  ggtitle("R-Ladies global tour", 
          subtitle = paste0(tour_length(result), " km"))

Ok this is nicer, it was worth the search.

Plotting the tour, magical version

And now I’ll use magick because I want to add a small star flying around the world. By the way if this global tour were to happen I reckon that one would need to donate a lot of money to rainforest charities or the like, because it’d have a huge carbon footprint! Too bad really, I don’t want my gif to promote planet destroying behaviours.

To make the gif I used code similar to the one shared in this post but in a better version thanks to Jeroen who told me to read the vignette again. Not saving PNGs saves time!

I first wanted to really show the emoji flying along the route and even created data for that, with a number of rows between chapters proportional to the distance between them. It’d have looked nice and smooth. But making a gif with hundreds of frames ended up being too long for me at the moment. So I came up with another idea, I’ll have to hope you like it!

library("emojifont")
load.emojifont('OpenSansEmoji.ttf')
library("magick")

plot_one_moment <- function(chapter, size, p,
                            chapters_df){
 
print(p + 
  ggtitle(paste0("R-Ladies global tour, ",
                 chapters_df[chapters_df$chapter == chapter,]$chapter), 
          subtitle = paste0(tour_length(result), " km"))+ 
    geom_text(data = chapters_df[chapters_df$chapter == chapter,], 
            aes(x = long, 
                y = lat,
                label = emoji("star2")),
            family="OpenSansEmoji",
                size = size))


}

img <- image_graph(1000, 800, res = 96)
out <- purrr::walk2(rep(chapters[as.integer(result)], each = 2),
                   rep(c(5, 10), length = length(chapters)*2),
                   p = p,
     plot_one_moment,
      chapters_df = chapters_df) 
dev.off()

## png 
##   2

image_animate(img, fps=1) %>%
  image_write("rladiesglobal.gif")

At least I made a twinkling star. I hope Hannah will be happy with the gif, because now I’d like to just dream of potential future trips! Or learn a bit of geography by looking at the gif.

The Guardian Experience: heavy or light topics?

Mon, 02 Oct 2017 00:00:00 +0000

I’ve recently been binge-reading The Guardian Experience columns. I’m a big fan of The Guardian life and style section regulars: the blind dates to which I dedicated a blog post, Oliver Burkeman’s This column will change your life, etc. Experience is another regular that I enjoy a lot. In each of the column, someone tells something remarkable that happened to them. It can really be anything.

I was thinking of maybe scraping the titles and get a sense of most common topics. The final push was my husband’s telling me about this article of Gabriella Paiella’s about the best Guardian Experience columns. She wrote “the “Experience” column does often touch on heavier topics”. Can one know what is the most prevalent “weight” of Experience columns scraping all their titles?

Experience: I downloaded all the titles of The Guardian Experience columns

I learnt a lot about responsible (and elegant) webscraping from Bob Rudis, and decided to use the tool he mentioned in this blog post, the robotstxt package which “makes it easy to check if bots (spiders, crawler, scrapers, …) are allowed to access specific resources on a domain.”

robotstxt::get_robotstxt("https://www.theguardian.com")

## # this is the robots.txt file for theguardian.com
## 
## User-agent: *
## Disallow: /sendarticle/
## Disallow: /Users/
## Disallow: /users/
## Disallow: /*/print$
## Disallow: /email/
## Disallow: /contactus/
## Disallow: /share/
## Disallow: /websearch
## Disallow: /*?commentpage=
## Disallow: /whsmiths/
## Disallow: /external/overture/
## Disallow: /discussion/report-abuse/*
## Disallow: /discussion/report-abuse-ajax/*
## Disallow: /discussion/comment-permalink/*
## Disallow: /discussion/report-abuse/*
## Disallow: /discussion/user-report-abuse/*
## Disallow: /discussion/handlers/*
## Disallow: /discussion/your-profile
## Disallow: /discussion/your-comments
## Disallow: /discussion/edit-profile
## Disallow: /discussion/search/comments
## Disallow: /discussion/*
## Disallow: /search
## Disallow: /music/artist/*
## Disallow: /music/album/*
## Disallow: /books/data/*
## Disallow: /settings/
## Disallow: /embed/
## Disallow: /*styles/js-on.css$
## Disallow: /sport/olympics/2008/events/*
## Disallow: /sport/olympics/2008/medals/*
## Disallow: /f/healthcheck
## Disallow: /sections
## Disallow: /top-stories
## Disallow: /most-read/sport
## Disallow: /articles
## Disallow: /podcasts
## Disallow: /global$
## Disallow: /*/feedarticle/*
## Disallow: /travel/2013/aug/22/been-there-readers-competition?*
## Disallow: /preference/*
## Disallow: /59666047/
## Disallow: /print/
## Disallow: /info/tech-feedback
## Disallow: /production-monitoring/
## 
## User-agent: Mediapartners-Google
## Disallow:
## 
## Sitemap: http://www.theguardian.com/sitemaps/news.xml
## Sitemap: http://www.theguardian.com/sitemaps/video.xml
## 
## User-agent: bingbot
## Crawl-delay: 1

robotstxt::paths_allowed("https://www.theguardian.com/lifeandstyle/series/experience")

## [1] TRUE

If I understand the above correctly, I’m allowed to scrape the titles of the columns, great!

I also noticed the crawl delay at the end of the robots.txt, of 1 second. Since I’ve decided to be a really nice scraper and also because I only have 29 pages to scrape in total, I’ll use a delay of 2 seconds between requests. In his post Bob says that if there is no indication, you should wait 5 seconds.

After these checks, I started working on the scraping itself.

library("rvest")

xtract_titles <- function(node) {
  css <- 'span[class="js-headline-text"]'
  html_nodes(node, css) %>% html_text(trim = TRUE)
}


get_titles_from_page <- function(page_number){
  Sys.sleep(2)
  link <- paste0("https://www.theguardian.com/lifeandstyle/series/experience?page=", page_number)
  page_content <- read_html(link)
  xtract_titles(page_content)
}

experience_titles <- purrr::map(1:29, get_titles_from_page) %>% unlist()
save(experience_titles, file = "data/2017-10-02-guardian-experience.RData")

set.seed(1)
sample(experience_titles, 10)

##  [1] "Experience: pregnancy sickness nearly killed me"            
##  [2] "Experience: I was a sperm donor for my friends"             
##  [3] "Experience: I was attacked in my front garden"              
##  [4] "I was brought up in the exclusive brethren"                 
##  [5] "Experience: I am Dancing Man"                               
##  [6] "The boy who missed the mainstream"                          
##  [7] "I still can't explain what I saw"                           
##  [8] "Experience: My twin rewrote my childhood"                   
##  [9] "Experience: I've renewed my wedding vows more than 50 times"
## [10] "Experience: I talk with my eyes"

See, these are really diverse topics! And I think this sample of 10 titles actually shows many heavy topics.

Experience: I computed the most frequent words

I’ll first remove the “Experience: " part of many titles, since it’s not exactly the most interesting word.

experience_titles <- stringr::str_replace(experience_titles, "^Experience: ", "")

I then unnested words. Interestingly in order to remember how to do this I went and read my Guardian blind dates post (the “So what did they talk about?” part).

library("tidytext")
library("rcorpora")

stopwords <- corpora("words/stopwords/en")$stopWords

words <- tibble::tibble(title = experience_titles) %>%
  unnest_tokens(word, title) %>%
  dplyr::filter(!word %in% stopwords) %>%
  dplyr::count(word, sort = TRUE)

knitr::kable(words[1:20,])

word	n
years	23
fell	21
lost	20
saved	20
life	19
man	19
baby	15
killed	13
survived	13
car	12
daughter	12
love	12
father	11
friend	11
husband	11
birth	9
dad	8
married	8
attacked	7
days	7

In my opinion this list of the most common words support my feeling topics are often heavy, but I also think it might be because there are many, many different words that can describe a light topic while well death will be primarily described by “killed”. Could sentiment analysis of the titles help me?

Experience: I computed the sentiment of titles

afinn <- get_sentiments("afinn")

sentiment <- tibble::tibble(title = experience_titles) %>%
  dplyr::mutate(saved_title = title) %>%
  unnest_tokens(word, title) %>%
  dplyr::inner_join(afinn) %>%
  dplyr::group_by(saved_title) %>%
  dplyr::summarize(sentiment = sum(score)) %>%
  dplyr::filter(!is.na(sentiment))

knitr::kable(sentiment[1:10,])

saved_title	sentiment
‘I stopped a terrorist attack’	-2
a coup interrupted our wedding	-2
A great white shark ate my leg	3
a head injury made me a musical prodigy	-2
a ladybird nearly killed me	-3
A machine keeps me alive	1
A six-metre wall collapsed on top of me	0
Becoming homeless helped me find love	3
Being obese made me feel like a social outcast	2
Blind date	-1

library("ggplot2")
library("hrbrthemes")

ggplot(sentiment) +
  geom_bar(aes(sentiment)) +
  theme_ipsum_rc()

Honestly, I think sentiment analysis didn’t help much here: the titles are too short, and the sample presented above is not very convincing. Moreoever, would the sentiment reveal the dramatic intensity of light vs. heavy, anyway?

Experience: I tried using machine learning to derive a topic from the title

In the following I’ll use my own monkeylearn package and in particular this topic classifier without too much hope since I’m feeding it a title, not a whole article.

topics <- monkeylearn::monkeylearn_classify(experience_titles,
                                            classifier_id = "cl_5icAVzKR")

titles <- tibble::tibble(title = experience_titles,
                         text_md5 = purrr::map_chr(experience_titles, digest::digest, algo = "md5"))

titles <- dplyr::inner_join(titles, topics, by = "text_md5")

Here’s a sample of the results after an arbitrary filtering based on probability:

titles <- dplyr::filter(titles, probability > 0.5)
set.seed(1)
dplyr::sample_n(titles, size = 20) %>% 
  dplyr::select(title, label, probability) %>%
  knitr::kable()

title	label	probability
my family was attacked by lions	Land Mammals	0.680
Muhammad Ali was my mentor	Religion & Spirituality	0.681
I’m a championship arm-wrestler	Entertainment & Recreation	0.873
I lit my father’s funeral pyre	Relationships	0.603
I have sudden death syndrome	Health & Medicine	0.816
One drink and I’m dead	Food & Drink	0.511
I was a compulsive gambler	Mental health	0.805
I flew the English Channel using a bunch of balloons	Aircraft	0.828
I crushed my £1m violin	Humanities	0.625
I crashed into the North Sea	Travel	0.549
I said yes to marriage the first time we met	Society	0.775
I can fly	Aircraft	0.930
We were told our son has cystic fibrosis – he hasn’t	Special Occasions	0.511
I found out I’m not my son’s father	Society	0.521
I became a famous artist at the age of 94	Music	0.673
I was impaled while pregnant	Mental health	0.548
A great white shark ate my leg	Animals	0.656
The holiday capsule wardrobe	Accommodation	0.761
I don’t wear shoes	Beauty & Style	0.798
I became a famous artist at the age of 94	Art	0.531

Note that after this filtering I had at least one topic for 288 titles. I don’t think this classification is really useful either but at least it’s fun to look at the proposed topic. What are the most frequent ones?

titles %>%
  dplyr::group_by(label) %>%
  dplyr::summarise(n = n(),
                   some = toString(title[1:3])) %>%
  dplyr::arrange(dplyr::desc(n)) %>%
  head(n = 10) %>%
  knitr::kable()

label	n	some
Transportation	45	I pulled a man from a burning car, I was hit by a car doing 101mph, a car crashed into me in the bath
Relationships	36	I’m a divorce party planner, a coup interrupted our wedding, my husband didn’t meet our daughter until she was 27
Society	32	my husband didn’t meet our daughter until she was 27, I first met my mother at a party, I was accused of having a sham marriage
Land Vehicles	30	I pulled a man from a burning car, I was hit by a car doing 101mph, a car crashed into me in the bath
Special Occasions	29	I fell in love through Airbnb, I made peace with my daughter’s killer, I’ve been protesting for more than 60 years
Animals	26	my dog rescues cats, I accidentally bought a giant pig, I was bitten by a shark
Parenting	19	I had a free birth, I saved a stranger’s life, We found a baby in a manger
Travel	16	a car crashed into me in the bath, I crashed into the North Sea, I saved my school bus from crashing
Land Mammals	15	my dog rescues cats, my cat saved me from a fire, I own the world’s ugliest dog
Health & Medicine	13	I have sudden death syndrome, I am afraid of pregnancy, my anti-malaria drugs made me psychotic

That, in a way, makes me more okay with the classification. I’ve always had the impression (you have to believe me) that many of the columns dealt with accidents, which corresponds to the transportation category, and families and relationships, and well animals, the ones that try to eat you or that steal your tractor. But now does it help me judge whether the Experience columns deal with rather light or heavy topics? Hum, no.

Experience: I could not really answer my initial question

So, it was fun, but I can’t really tell Gabriella Paiella whether she was right or wrong. One thing is sure, these columns are quite varied… so everyone can find what they’re looking for, either a dramatic story or a funny one?

The music of Les Mills Body Pump, with Spotify data

Sun, 01 Oct 2017 00:00:00 +0000

I am a runner but also a Body Pump enthusiast. Body Pump is a group fitness class of the Les Mills company, in which you train different muscle groups using a weighted bar – whose total weight you modulate with plates in order to adapt it to your fitness level and to the muscle group. Like R, Body Pump was created in New Zealand, what a wonderful country! Every three months, a new class is released, with new songs and choreographies. What doesn’t change is the muscle group trained in each of the 10 songs of each class.

I’ve thought of analysing Body Pump data for a long time now but could never find what I was looking for, which was a dataset of number of “reps” by song, e.g. how many squats do you do in each squats song. Then I realized I could also play with other data, like a quite comprehensive list of songs used in releases. I decided to cross this information with information about style of the corresponding artist in Spotify. Here is what I came up with!

Track 1: Track list cleaning

This I did listening to pop music, actually using my husband’s Deezer account. I downloaded data from this page which is quite exhaustive for many releases but well not formatted in a consistent way over releases, so I needed to be patient. A bit like when in the triceps track one does many half repetitions and feels one’s arms are going to fall off.

library("rvest")
pg <- "http://www.blogforumsupport.com/2008/bodypump2008.html"
pg_content <- read_html(pg, encoding = "latin1")
save(pg_content, file = "data/pg_content.RData")

Then I extracted the track lists, in a non very elegant way. Sometimes one just wants the job done…

library("magrittr")
xtract_nodes <- function(node, css) {
  html_nodes(node, css) %>% html_text(trim = TRUE)
}
tracks <- xtract_nodes(pg_content, 'p')
tracks <- tracks[grepl("1", tracks) & grepl("2", tracks) & grepl("9", tracks) & grepl("10", tracks)]
tracks <- unlist(tracks)

What follows is scripted manual correction of some mistakes in the data, or things that makes later extraction of title vs. artist a bit complicated.

tracks <- stringr::str_replace(tracks, 
                               "04. Campione 2012 - DJ Flow05. 50 Ways to Say Goodbye - Train",
                               "04. Campione 2012 - DJ Flow \n 05. 50 Ways to Say Goodbye - Train")
tracks <- stringr::str_replace(tracks, 
                               "Upion Havoc09. She's so Mean",
                               "Upion Havoc \n 09. She's so Mean")

tracks <- stringr::str_replace(tracks, "07. How Deep Is Your Love -", "07. How Deep Is Your Love - Calvin Harris")
tracks <- stringr::str_replace(tracks, "08 Bumble Bee -", "08 Bumble Bee - Zedd & Botnek")
tracks <- stringr::str_replace(tracks, "Young & Stupid -", "Young & Stupid - Travis Mills feat. T.I.")
tracks <- stringr::str_replace(tracks, "Bpn Jovi", " - Bon Jovi")
tracks <- stringr::str_replace(tracks, "Damn-R", "Damn R")
tracks <- stringr::str_replace(tracks, "Re-Dub", "Re Dub")
tracks <- stringr::str_replace(tracks, "Boom-Lay Boom-Lay Boom", "Boom Lay Boom Lay Boom")
tracks <- stringr::str_replace(tracks, "Hi-Tack", "Hi Tack")
tracks <- stringr::str_replace(tracks, "\\(Calvin Harris Remix\\) Fatboy Slim", "Fatboy Slim")
tracks <- stringr::str_replace(tracks, "B-Tastic", "B Tastic")
tracks <- stringr::str_replace(tracks, "E-Nergy", "E Nergy")
tracks <- stringr::str_replace(tracks, "In the Clear Six60", "In the Clear - Six60")
tracks <- stringr::str_replace(tracks, "Everybody Go! Vice", "Everybody Go! - Vice")
tracks <- stringr::str_replace(tracks, "My Love - feat. T.I. - Justin Timberlake",
                               "My Love - Justin Timberlake feat. T.I.")
 
tracks <- stringr::str_replace(tracks, "Original Release \\(PPCA\\)", "")
tracks <- tracks[!stringr::str_detect(tracks, "Get ready for what may be our fiercest release yet")]

After doing this I lied on the floor drenched in sweat / created a tibble.

tracks <- tibble::tibble(track = tracks)
tracks$release <- nrow(tracks):1

Note that the release variable isn’t the actual release number, it’s more a time variable.

At this stage I had one string by release, which I split in one string by track.

tracks$track <- purrr::map(tracks$track, stringr::str_split,
                     pattern = "\\\n", simplify = FALSE)
tracks$track <- purrr::map(tracks$track, unlist)
tracks <- tidyr::unnest(tracks, track)

The next step was keeping only the 10 first tracks by release, ignoring bonus and alternative tracks listed at the end. It also involved removing useless strings that were not actually songs.

tracks <- dplyr::filter(tracks, track != "")
tracks <- dplyr::filter(tracks, track != "              02. ")
tracks <- dplyr::filter(tracks, track != "              03. ")
tracks <- dplyr::filter(tracks, track != "              04. ")
tracks <- dplyr::filter(tracks, track != "              05. ")
tracks <- dplyr::filter(tracks, track != "              06. ")
tracks <- dplyr::filter(tracks, track != "              07. ")
tracks <- dplyr::filter(tracks, track != "              08. ")
tracks <- dplyr::filter(tracks, track != "            08. ")
tracks <- dplyr::filter(tracks, track != "              09. ")
tracks <- dplyr::filter(tracks, track != "            09. ")
tracks <- dplyr::filter(tracks, track != "              10. ")
tracks <- dplyr::filter(tracks, track != "            10. ")
tracks <- dplyr::filter(tracks, track != "          Calvin Harris")
tracks <- dplyr::filter(tracks, track != "          Zedd & Botnek")
tracks <- dplyr::filter(tracks, track != "          Travis Mills feat. T.I.")
tracks <- dplyr::group_by(tracks, release)
tracks <- dplyr::mutate(tracks, song = 1:n())
tracks <- dplyr::filter(tracks, song <= 10)
tracks <- dplyr::ungroup(tracks)
knitr::kable(tracks[1:10,])

release	track	song
42	01. Run Wild - Hardwell feat. Jake Reese	1
42	02. Go! (Vincent Price Remix) - Wolfpack & Avancada	2
42	03. Spoken Word - Chase & Status feat. George the Poet	3
42	04 .LRAD - Knife Party	4
42	05. Friday - PARRI	5
42	06. Jungle Youth - Young The Giant	6
42	07. Ocho Cinco - DJ Snake feat. Yellow Claw	7
42	08. Feel Your Love - Flux Pavilion & NGHTMRE feat. Jamie Lewis	8
42	09. Closer - The Chainsmokers feat. Halsey	9
42	10. Love On The Brain - Rihanna	10

After this I extracted the title and artist of each track. I’m not sure why I needed to change the encoding of the string for that, but before trying this solution I was bothered by there being different sorts of dashes and hyphens.

tracks <- dplyr::mutate(tracks, track = stringr::str_trim(track))

split_title_artist <- function(string, release){
  track <- string
  Encoding(string) <- "latin1"
  # avoid some issues later
  string <- stringr::str_replace(string, "â\u0080\u0099", "'")
  string <- stringr::str_replace(string, "ï¼\u0088", "(")
  string <- stringr::str_replace(string, "¼\u0089", ")")
  string <- stringr::str_replace(string, "`", "'")
  string <- stringr::str_replace(string, "â\u0080¦", "...")
  string <- stringr::str_replace(string, "DÃ©jÃ", "Déjà")
  string <- stringr::str_replace(string, "Bbn Jovi", "- Bon Jovi")
  string <- stringr::str_replace(string, "Marfia", "Mafia")
  string <- stringr::str_replace(string, ":", "")
  string <- stringr::str_replace(string, "Diamond Eyes (Boom-Lay Boom-Lay Boom)", "Diamond Eyes (Boom Lay Boom Lay Boom)")
  string <- stringr::str_replace(string, "Studio-X", "Studio X")
  
  # hyphens/dashes
  string <- stringr::str_replace(string, "â\u0080\u0093", "-")
  string <- stringr::str_replace(string, "–", "-")
  string <- stringr::str_replace(string, "-", "\u0096")
  both <- stringr::str_split(string, pattern = "\u0096",
                             simplify = TRUE)
  tibble::tibble(title = both[1],
                artist = both[2], 
                release = release,
                track = track)
}

split_a_release <- function(strings, release){
  purrr::map_df(strings, split_title_artist, release)
}

tracks_si <- purrr::map2_df(tracks$track, tracks$release, split_a_release)
tracks <- dplyr::left_join(tracks, tracks_si, by = c("release", "track"))
tracks <- dplyr::mutate(tracks, title = stringr::str_replace(title, "^[0-1]*[0-9]* *\\.*", ""))

When the artist was in fact two artists, I kept only the first one which I assumed to be more informative of the style of the song.

first_artist <- function(artist){
  artist <- stringr::str_replace(artist, "\\&.*$", "")
  first_artist <- stringr::str_replace(artist, "[fF]eat.*", "")
  if(stringr::str_detect(first_artist, "Colourbox")){
    first_artist <- "Colourbox"
  }
  first_artist
}

tracks <- dplyr::mutate(tracks, first_artist = first_artist(tracks$artist))

And I got there:

knitr::kable(tracks[1:10,])

release	track	song	title	artist	first_artist
42	01. Run Wild - Hardwell feat. Jake Reese	1	Run Wild	Hardwell feat. Jake Reese	Hardwell
42	02. Go! (Vincent Price Remix) - Wolfpack & Avancada	2	Go! (Vincent Price Remix)	Wolfpack & Avancada	Wolfpack
42	03. Spoken Word - Chase & Status feat. George the Poet	3	Spoken Word	Chase & Status feat. George the Poet	Chase
42	04 .LRAD - Knife Party	4	LRAD	Knife Party	Knife Party
42	05. Friday - PARRI	5	Friday	PARRI	PARRI
42	06. Jungle Youth - Young The Giant	6	Jungle Youth	Young The Giant	Young The Giant
42	07. Ocho Cinco - DJ Snake feat. Yellow Claw	7	Ocho Cinco	DJ Snake feat. Yellow Claw	DJ Snake
42	08. Feel Your Love - Flux Pavilion & NGHTMRE feat. Jamie Lewis	8	Feel Your Love	Flux Pavilion & NGHTMRE feat. Jamie Lewis	Flux Pavilion
42	09. Closer - The Chainsmokers feat. Halsey	9	Closer	The Chainsmokers feat. Halsey	The Chainsmokers
42	10. Love On The Brain - Rihanna	10	Love On The Brain	Rihanna	Rihanna

Track 2: Spotify data downloading

I dreamt of a database giving me the beats per minute, style, etc of each song. I didn’t get any information about the bpm but thanks to the Spotify API and this R package I was able to get information about the artists including their genres and popularity. For songs, the API does not give such information.

Note: the spotifyOAuth part of the following lines demands your first getting an Spotify API app client ID and secret as explained in the README of the package. I saved these as environment variables in my .Renviron.

library("Rspotify")
keys <- spotifyOAuth(app_id = Sys.getenv("SPOTIFY_APPID"),
                     client_id = Sys.getenv("SPOTIFY_CLIENTID"),
                     client_secret = Sys.getenv("SPOTIFY_SECRET"))
get_and_wait <- function(artist, keys){
  Sys.sleep(1)
  info <- searchArtist(artist, token = keys)
  if(!is.null(info)){
    info <- info[1,]
  }
  info$first_artist <- artist
  info
}

artists_info <- purrr::map_df(unique(tracks$first_artist), get_and_wait, keys)
readr::write_csv(artists_info, path = "data/artists_info.csv")

tracks <- dplyr::left_join(tracks, artists_info, by = "first_artist")
readr::write_csv(tracks, path = "data/tracks.csv")

knitr::kable(artists_info[1:5,])

display_name	id	popularity	followers	genres	type	first_artist
Hardwell	6BrvowZBreEkXzJQMpL174	78	1660846	big room,deep big room,dutch house,edm,electro house,progressive electro house,progressive house,tropical house	artist	Hardwell
Wolfpack	74ycentHh1YZrW5x4PLnox	54	8554	crust punk,deep big room,sky room	artist	Wolfpack
Chase & Status	3jNkaOXasoc7RsxdchvEVq	69	300912	brostep,deep groove house,drum and bass,edm,grime,house,tropical house	artist	Chase
Knife Party	2DuJi13MWHjRHrqRUwk8vH	64	529738	australian dance,big room,brostep,catstep,complextro,edm,electro house,electronic trap,moombahton,zapstep	artist	Knife Party
PARRI$	17L4AvcA7zS5xdnLhemEMM	40	7264	escape room	artist	PARRI

I got a genres information for 209 out of 313 artists.

Track 3: making sense of the genres of artists

As one sees above, each artist can have several genres and there are a ton of possible genres. When googling a bit about Spotify music genres, I found this gem of a webpage. Impressive! In my case, I wanted to have a few groups of musical genres in order to link them to the track, e.g. is there a musical genre more representative of the biceps track as opposed to other tracks?

I decided to cluster the artists based on the binary variables for each genre. For this I used the klaR::kmodes function because k-modes is a method for clustering based on categorical variables as opposed to k-means for instance. In work that I don’t present here, I played with 1) the number of clusters, looking at the resulting total sum of within-cluster differences and 2) the minimal number of artists that should have a genre for this genre to be included. For instance if a single artist has the style “francoton”, then I don’t want to use it for clustering. In the end, I was quite happy with 6 clusters and keeping only genres that occurred more than 10 times in the data. Doing this there were some artists I couldn’t cluster since they had only been described by very rare genres.

Before clustering I had to transform the data a bit.

artists_info <- readr::read_csv("data/artists_info.csv")
artists_info <- dplyr::filter(artists_info, !is.na(display_name))
artists_info <- dplyr::select(artists_info, - first_artist, - followers)
artists_info <- unique(artists_info)

# most important word?
split_the_genres <- function(genres){
  genres <- stringr::str_split(genres, ",")
  genres
}

artists_info <- dplyr::filter(artists_info, !is.na(genres))
artists_info <- dplyr::mutate(artists_info, genre = split_the_genres(genres))
artists_info <- tidyr::unnest(artists_info, genre)
artists_info <- dplyr::mutate(artists_info, value = TRUE)
artists_info <- tidyr::spread(unique(artists_info), genre, value, fill = FALSE)
# 
# keep only variables with more than 10 occurrence
sum_or_not <- function(x){
  if(is.logical(x)){
    sum(x) > 10
  }else{
    TRUE
  }
}
variables_info <- dplyr::summarise_all(artists_info, sum_or_not)
artists_info <- artists_info[,as.logical(t(variables_info))]

knitr::kable(artists_info[1:2,])

display_name	id	popularity	genres	type	alternative metal	big room	brostep	bubblegum dance	dance pop	deep big room	edm	electro house	eurodance	europop	modern rock	neo mellow	nu metal	pop	pop christmas	pop punk	pop rap	pop rock	post-grunge	post-teen pop	progressive electro house	progressive house	r&b	rock	tropical house
3 Doors Down	2RTUTCvo6onsAnheUk3aL9	76	alternative metal,alternative rock,heavy christmas,nu metal,pop punk,pop rock,post-grunge,rap metal,rock	artist	TRUE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE	TRUE	FALSE	FALSE	TRUE	FALSE	TRUE	TRUE	FALSE	FALSE	FALSE	FALSE	TRUE	FALSE
3OH!3	0FWzNDaEu9jdgcYTbcOa4F	68	dance pop,electropowerpop,modern rock,pop,pop punk,pop rap,post-teen pop	artist	FALSE	FALSE	FALSE	FALSE	TRUE	FALSE	FALSE	FALSE	FALSE	FALSE	TRUE	FALSE	FALSE	TRUE	FALSE	TRUE	TRUE	FALSE	FALSE	TRUE	FALSE	FALSE	FALSE	FALSE	FALSE

Then the clustering happened.

set.seed(1)
clustering <- klaR::kmodes(artists_info[, 6:ncol(artists_info)], modes = 6)
clustering$size

## cluster
##  1  2  3  4  5  6 
## 17 13 22 14 21 84

artists_info$cluster <- clustering$cluster

library("magrittr")
dplyr::filter(artists_info, cluster == 1) %>%
  dplyr::select(display_name, genres) %>%
  head() %>%
  knitr::kable()

display_name	genres
Armin van Buuren	big room,edm,progressive house,progressive trance,trance,tropical house
Avicii	big room,dance pop,edm,pop,tropical house
Bob Sinclar	disco house,edm,filter house,house,tropical house,vocal house
Calvin Harris	edm,electro house,house,pop,progressive house
Chase & Status	brostep,deep groove house,drum and bass,edm,grime,house,tropical house
David Guetta	dance pop,edm,pop,tropical house

I then tried to characterize each cluster by its most important genre, well one of them. It’s as arbitrary as the rest of the clustering but I needed to summarize information anyway. I’d be glad to hear about other ideas for categorizing artists!

get_most_important_genre <- function(df){
  counts <- as.numeric(t(dplyr::summarize_all(df[, 6:(ncol(df)-1)], sum)))
  names(df)[6:(ncol(df)-1)][which(counts == max(counts))][1]
}

styles <- artists_info %>%
  split(artists_info$cluster) %>%
  purrr::map_chr(get_most_important_genre)

cluster_music <- tibble::tibble(cluster = 1:6,
                                style = styles)

knitr::kable(cluster_music)

cluster	style
1	edm
2	big room
3	dance pop
4	alternative metal
5	pop
6	eurodance

Track 4: Plotting

Once I had my categories for some artists, I was impatient to look at their proportions depending on the muscle group. I first prepared the data in order to have meaningful names for clusters and tracks.

artists_info <- dplyr::select(artists_info, display_name, cluster, id, popularity,
                              genres)

tracks <- dplyr::left_join(tracks, artists_info,
                           by = c("display_name", "id", "popularity", "genres"))

library("ggplot2")
theme_set(theme_gray(base_size = 18))
library("viridis")
library("magrittr")

track_muscle <- tibble::tibble(song = 1:10,
                               muscle = c("warm-up", "squats", "chest",
                                          "back", "triceps", "biceps",
                                          "lunges", "shoulders", "core",
                                          "cool-down"))

tracks <- dplyr::left_join(tracks, track_muscle,
                           by = "song")

tracks <- dplyr::mutate(tracks,
                        muscle = forcats::fct_reorder(muscle, song))


tracks <- dplyr::left_join(tracks, cluster_music, by = "cluster")

I first made a figure of the number of missing values i.e. missing cluster by muscle group. Sadly some muscle groups seem to have more missing values, hopefully the sample I do have for them is representative. Note that in total I have 42 releases.

tracks %>%
  dplyr::group_by(muscle) %>%
  dplyr::summarise(prop_na = sum(is.na(cluster))/n()) %>%
  ggplot() +
  geom_point(aes(muscle, prop_na)) +
  ylab("Proportion of missing values")+
  theme(axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.5))

And then I looked at the musical genres by muscle group.

tracks %>%
  dplyr::filter(!is.na(cluster)) %>%
ggplot() +
  geom_bar(aes(muscle, fill = style),
           position = "fill") +
  scale_fill_viridis(discrete = TRUE) +
  theme(axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.5))

So, is this interesting? I’m not exactly sure, ah! It seems that the shoulders track is associated with less pop or dance than the other tracks, maybe because overhead presses demand more rythm? Cool down does seem more quite with no electronic music (edm and big room) which makes sense. All in all, I was happy to see this plot but would like to have a better categorization of songs and well less missing values. I’m not a good musical analyst yet! Oh and I’d love to get data about beats per minute… Maybe in a next release of this post, by me or one of you dear readers? Oh and if you want music that’s more quiet, check out this post of mine about Radio Swiss Classic program!

Is it wine o'clock?

Sat, 12 Aug 2017 00:00:00 +0000

Emojis were again quite en vogue this week on Twitter with Romain François doing some awesome stuff for the emo package, in particular this teeny tiny animated clock. It reminded me of my own emoji animated clock that I had done a while ago for representing time-use data. Time for me to present its genesis!

I’m actually not a Quantified Self person, but at my work time-use data was collected for an epidemiology project: information about people activities and locations throughout one day can help unraveling sources of exposure to air pollution. I’ve therefore spent some time thinking about how to represent such data. In particular, my colleague Margaux directed a fantastic video about our project. We introduced some real data from our project in it, including an animated clock that I made with emoji-coding of indoor-outdoor location. I’ll present code and data for producing a similar clock with my own agenda on some Wednesday evenings of last year.

Loading the time-use data

I logged data from my Wednesday in a rather ok format.

library("magrittr")
date <- "2016-03-10"
activities <- readr::read_csv2("data/2017-08-12-wineoclock-timeuse.csv",
                               col_types = "ccc") %>%
  dplyr::mutate(start = lubridate::ymd_hms(paste0(date, start, ":00")),
         end = lubridate::ymd_hms(paste0(date, end, ":00")))
knitr::kable(activities)

start	end	activity
2016-03-10 00:00:00	2016-03-10 06:30:00	sleeping
2016-03-10 06:30:00	2016-03-10 07:00:00	coffee
2016-03-10 07:00:00	2016-03-10 08:00:00	running
2016-03-10 08:00:00	2016-03-10 09:30:00	train
2016-03-10 09:30:00	2016-03-10 17:00:00	computer
2016-03-10 17:00:00	2016-03-10 18:00:00	train
2016-03-10 18:00:00	2016-03-10 20:00:00	school
2016-03-10 20:00:00	2016-03-10 20:30:00	pizza
2016-03-10 20:30:00	2016-03-10 22:00:00	dancer
2016-03-10 22:00:00	2016-03-10 22:30:00	wine_glass
2016-03-10 22:30:00	2016-03-10 23:59:00	sleeping

When I prepared this visualization ggimage didn’t exist so I knew I’d have to use emojis from emojifont to represent my activities, therefore I directly entered the activities as emojis. My Wednesdays at that time were quite varied, I started the day with breakfast (how original), then some time at the gym, followed by a rather short workday due to my taking a Catalan class at the end of the afternoon. The evening was spent enjoying a quick and cheap pizza dinner with my husband before our salsa class and sometimes treating ourselves to a wine glass at the small and rather shabby-looking bar at the end of our street, whose name really was Small bar.

Making the animated clock

I didn’t have to think about how to draw and animated a clock with ggplot2 because somebody already had: I used code from this gist of Drew Conway’s.

# Generate digitial clock face
first_nine <- c('00', '01', '02', '03', '04', '05', '06', '07', '08', '09')
hours <- c(first_nine, as.character(seq(10,23,1)))
mins <- c(first_nine, as.character(seq(10,59,1)))
time_chars_l <- lapply(hours, function(h) paste(h, ':', mins, sep=''))
time_chars <- do.call(c, time_chars_l)

# Generate analog clock face
hour_pos <- seq(0, 12, 12/(12*60))[1:720]
min_pos <-seq(0,12, 12/60)[1:60]
hour_pos <- rep(hour_pos, 2)
all_times <- dplyr::tbl_df(cbind(hour_pos, min_pos)) %>%
  dplyr::mutate(index = time_chars) %>%
  dplyr::mutate(time = lubridate::ymd_hms(paste0(date, index, ":00")))

I then needed to join the all_times table, containing many snapshots of the time, with my time-use table made of intervals (start and end of each activity). Another package born in the meantime, fuzzyjoin, would have allowed me to write more efficient code, and I was so sad when looking at the old code that I re-wrote it.

all_times <- dplyr::mutate(all_times, start = time, end = time)
all_times  <- fuzzyjoin::interval_left_join(all_times, activities)
knitr::kable(all_times[1:10,])

hour_pos	min_pos	index	time	start.x	end.x	start.y	end.y	activity
0.0000000	0.0	00:00	2016-03-10 00:00:00	2016-03-10 00:00:00	2016-03-10 00:00:00	2016-03-10	2016-03-10 06:30:00	sleeping
0.0166667	0.2	00:01	2016-03-10 00:01:00	2016-03-10 00:01:00	2016-03-10 00:01:00	2016-03-10	2016-03-10 06:30:00	sleeping
0.0333333	0.4	00:02	2016-03-10 00:02:00	2016-03-10 00:02:00	2016-03-10 00:02:00	2016-03-10	2016-03-10 06:30:00	sleeping
0.0500000	0.6	00:03	2016-03-10 00:03:00	2016-03-10 00:03:00	2016-03-10 00:03:00	2016-03-10	2016-03-10 06:30:00	sleeping
0.0666667	0.8	00:04	2016-03-10 00:04:00	2016-03-10 00:04:00	2016-03-10 00:04:00	2016-03-10	2016-03-10 06:30:00	sleeping
0.0833333	1.0	00:05	2016-03-10 00:05:00	2016-03-10 00:05:00	2016-03-10 00:05:00	2016-03-10	2016-03-10 06:30:00	sleeping
0.1000000	1.2	00:06	2016-03-10 00:06:00	2016-03-10 00:06:00	2016-03-10 00:06:00	2016-03-10	2016-03-10 06:30:00	sleeping
0.1166667	1.4	00:07	2016-03-10 00:07:00	2016-03-10 00:07:00	2016-03-10 00:07:00	2016-03-10	2016-03-10 06:30:00	sleeping
0.1333333	1.6	00:08	2016-03-10 00:08:00	2016-03-10 00:08:00	2016-03-10 00:08:00	2016-03-10	2016-03-10 06:30:00	sleeping
0.1500000	1.8	00:09	2016-03-10 00:09:00	2016-03-10 00:09:00	2016-03-10 00:09:00	2016-03-10	2016-03-10 06:30:00	sleeping

Then I could continue using the code from the gist, this step generating stuff for the two clock hands.

all_times <- all_times %>% 
  dplyr::select(- start.x, - end.x, - start.y, - end.y) %>%
  tidyr::gather(name, time.info, 1:2) %>%
  dplyr::arrange(index) %>%
  dplyr::mutate(hands = ifelse(name == "hour_pos", 0.5, 1))

I decided to only plot a subset of my exciting day.

all_times <- dplyr::filter(all_times, 
                           time > lubridate::ymd_hms("2016-03-10 15:00:00"),
                           time < lubridate::ymd_hms("2016-03-10 23:00:00") )

Last but not least, I plotted the clock. Nowadays I’d probably stick to magick for animating the plot but gganimate is awesome too anyway.

library("ggplot2")
emojifont::load.emojifont('OpenSansEmoji.ttf')
clock <- ggplot(all_times, aes(xmin = time.info,
                               xmax = time.info + 0.03, 
                               ymin = 0,
                               ymax = hands,
                              frame = index))
clock <- clock + geom_rect()
clock <- clock + theme_bw()
clock <- clock + coord_polar()
clock <- clock + scale_x_continuous(limits = c(0,12), 
                                    breaks = 0:11,
                     labels = c(12, 1:11))
clock <- clock + scale_y_continuous(limits=c(0,1.1)) 
clock <- clock + coord_polar()
clock <- clock + geom_text(aes(x = 0,
                               y = 0,
                               label = emojifont::emoji(activity)),
                          family="OpenSansEmoji", 
                          size=20)
clock <- clock + theme(legend.position="none",
                      axis.text.y = element_blank(),
                      axis.ticks.y = element_blank(),
                      axis.title.y = element_blank(),
                      axis.title.x = element_blank(),
                      panel.background = element_blank(),
                      panel.border = element_blank(),
                      panel.grid.major = element_blank(),
                      panel.grid.minor = element_blank(),
                      plot.background = element_blank())

animation::ani.options(interval = 0.025)
gganimate::gganimate(clock, "clock.gif")

I ended up with a very useful visualization as you can see… at least, I have no trouble finding out when wine o’clock is!

Pets or livestock? Naming your RMarkdown chunks

Tue, 08 Aug 2017 00:00:00 +0000

Today I made a confession on Twitter: I told the world I had spent my whole career not naming chunks in RMarkdown documents. Even if I had said one should name them when teaching RMarkdown. But it was also a tweet for showing off since I was working on the first manuscript with named chunks and loving it. I got some interesting reactions to my tweet, including one that made me feel better about myself (sorry Thomas), and other ones that made me feel like phrasing why one should name RMarkdown chunks.

Hadley Wickham asked whether chunks were pets or livestock, as in his analogy for models. Livestock chunks are identified by numbers, not names, in the case of chunks defined by position. I now think we have good reason to consider them as pets and here’s why…

My fur sibling does come if you call him

Navigating the RMarkdown document. You can yell a dog’s name and it’ll come to you. Well hopefully… In any case in RStudio you can actually navigate named chunks fairly easily.
Making your RMarkdown easier to understand. Even if you comment code, having an informative code chunk name will help your collaborators when they read your file. For that, the chunk names should reflect what’s being done in them, not your creativity, keep that for your real pets or kids. Naming the chunks has moreover made me divide my code into chunks doing one thing, e.g. preparing the data, then fitting a model. David Robinson suggested naming chunks such as to make dependencies more explicit, and the thread contains some nice tips about automatic dependencies between chunks, something I need to explore more myself (thanks Nick for providing me with this tweet link!). Don’t use spaces or dots in chunk labels, and Nick even says not to use “_” but rather “-” because it makes names navigation easier at least on Mac.

Chunk down! Don’t worry, Mowgli was just taking a nap.

Reading error reports or progress of knitting. It’s much easier to find which chunk exactly is down, or to see how well knitting is going, with chunk names.
Moving chunks around, adding some. In particular, cache results are saved by chunk name or position, so caching is more efficient with named chunks. With unnamed chunks you might end up unwillingly re-running that long slow chunk!

When you’re cool and you know it

Feeling organized (no shame, Pallavi!) and empowered. And boasting on Twitter.

Truth be told maybe I won’t always name my chunks, e.g. in blog posts, but for manuscripts and other bigger projects I think it’s the way to go. And now I need to go read more about autodep in knitr, at least to distract myself for missing my fur sibling too much!

Who are the Swedish radio P1 summer guests? Answer via Wikidata

Sun, 06 Aug 2017 00:00:00 +0000

This week, a very promising new R blog was launched, namely the blog of Eric Persson a.k.a as expersso on Twitter. I had really been looking forward to this because expersso’s code screenshots have always been quite cool, so seeing his no longer being limited to them is awesome! His first articles series is about a game, you should really check it out. (PSA: if you post screenshots of R code on Twitter, have a look at Sean Kross’ codefinch package!).

Because I’m a nosy person I asked Eric whether he was Swedish, his last name being quite Swedish-looking in my opinion. He is, which made me wonder about Swedish blog topics and actually decided to use one Swedish topic I came up with, the summer guests of the Swedish radio P1! Every summer since 1959, P1 selects a bunch of famous or interesting people and have them record a bit more than one hour program where they’re free to discuss what they want (important events of their life for instance) and to choose the musical breaks (which you don’t get to listen entirely to in the online version because of copyright stuff). The program is then broadcasted in the summer, one guest a day from the end of June to the beginning of August. There’s even a winter version now but I’ll ignore it because it’s too hot here in Barcelona to even think about winter.

It’s a very cool radio program in my opinion. I discovered it at the end of my 5-month research stay in Gothenburg in 2010 and decided it’d be one way to keep my Swedish skills up to date (my other methods include listening to ABBA in Swedish and reading Camilla Läckberg’s novels). I haven’t listened to that many guests but I really enjoy it when I do, and I like how diverse the list of guests is. In this post, I’ll actually try to have a look at the occupations of the guests via Wikidata!

How to get the data?

In order to answer my question I needed a list of all summer guests, and their occupations. I first thought I’d resort to webscraping P1 website, which would have been a good way to provide expersso with an opportunity to give me some constructive feedback. However after websurfing a bit to assess my options, I realized I could use Wikipedia data obtained through APIs rather than webscraping as in my posts about famous dead people. Here was my strategy.

Getting the list of all summer guests from 1959

I downloaded the table of all summer guests from 1959 from this page, which gave me their names and the dates of their episode(s) – yeah some people are invited back.

# all summer guests
sommargaester <- readr::read_csv("data/p1sommar.csv", col_names = FALSE, 
                                 locale = readr::locale(encoding = "latin1"))
knitr::kable(sommargaester[1:10,])

X1	X2	X3	X4	X5	X6	X7
Aaro,Lars-Eric	110714	NA	NA	NA	NA	NA
Abdelhadi,Magdi	070809	120716	NA	NA	NA	NA
Abdulle,Sherihan ‘Cherrie’	170706	NA	NA	NA	NA	NA
Abele,Anton	130725	NA	NA	NA	NA	NA
Abenius,Folke	800627	NA	NA	NA	NA	NA
Abrahamson,Kjell-Albin	880809	NA	NA	NA	NA	NA
Ackebo,Lena	970708	NA	NA	NA	NA	NA
Adami,Zanyar	060707	NA	NA	NA	NA	NA
Adamo,Amelia	860806	870718	080701	NA	NA	NA
Adams,Maud	920703	070805	NA	NA	NA	NA

# get their names
sommargaester_names <- unique(sommargaester$X1)

# for putting names in the right order for later queries
transform_name <- function(name){
  paste(stringr::str_split(name, ",",
                           simplify = TRUE)[2],
        stringr::str_split(name, ",",
                           simplify = TRUE)[1])
}

pretty_sommargaester_names <- purrr::map_chr(sommargaester_names, transform_name)
sommargaester_names <- tibble::tibble(name = pretty_sommargaester_names,
                                      X1 = sommargaester_names)
sommargaester <- dplyr::left_join(sommargaester, sommargaester_names,
                                  by = "X1")

sommargaester <- dplyr::select(sommargaester, - X1)
# transform the date
sommargaester <- tidyr::gather(sommargaester, "rep", "date", X2:X7)
sommargaester <- dplyr::group_by(sommargaester, name)
sommargaester <- dplyr::mutate(sommargaester, rep = 1:n())
sommargaester <- dplyr::ungroup(sommargaester)

sommargaester <- dplyr::filter(sommargaester, !is.na(date))
# remove winter
sommargaester <- dplyr::filter(sommargaester, !stringr::str_detect(date, "V"))
# remove repeat episodes
sommargaester <- dplyr::filter(sommargaester, !stringr::str_detect(date, "R"))


# transform the date to a format with a non ambiguous year
sommargaester <- dplyr::mutate(sommargaester, date = as.numeric(date))
sommargaester <- dplyr::mutate(sommargaester, date = ifelse(date > 180000, paste0("19", date),
                                                            ifelse(date < 100000,
                                                                   paste0("200", date),
                                                                   paste0("20", date))))
sommargaester <- dplyr::mutate(sommargaester, pretty_date = lubridate::ymd(date))

knitr::kable(sommargaester[1:10,])

name	rep	date	pretty_date
Lars-Eric Aaro	1	20110714	2011-07-14
Magdi Abdelhadi	1	20070809	2007-08-09
Sherihan ‘Cherrie’ Abdulle	1	20170706	2017-07-06
Anton Abele	1	20130725	2013-07-25
Folke Abenius	1	19800627	1980-06-27
Kjell-Albin Abrahamson	1	19880809	1988-08-09
Lena Ackebo	1	19970708	1997-07-08
Zanyar Adami	1	20060707	2006-07-07
Amelia Adamo	1	19860806	1986-08-06
Maud Adams	1	19920703	1992-07-03

Getting Wikidata about the summer guests

Then I read the list of Wikipedia R packages put together by Mikhail Popov, data scientist at the Wikimedia foundation with whom I exchanged a few messages after my Wikipedia deaths posts. These packages were either created by him or by Oliver Keyes and really give you access to plenty of structured data so I was happy to at last give the list a try!

I used WikidataQueryServiceR to access Wikidata via the query language SPARQL. Luckily I didn’t need to actually learn SPARQL, I used the first example of the documentation, getting Douglas Adams’ data and played with this online query tool to see how I could modify it to 1) obtain data in Swedish because Swedish Wikipedia is probably more complete about Swedish people and 2) obtain data about the summer guests, not Douglas Adams. For that second point I needed the item ID of each summer guests which I queried via another R package, WikidataR. I could have modified the SPARQL even more since I wouldn’t be using picture information but I wasn’t in a very perfectionist mood.

Doing all of this I only scraped the surface of the possibilities offered by the Wikipedia-related R packages and can already tell one could do a bunch of exciting analyses with them!

Note that I used Bob Rudis’ code as an example of how to insert a progress bar which made me feel quite cool. I also inserted a pause of 1 second between calls to the APIs in order to be a good person. My function name however shows how uninspired I was for naming things.

# function for getting someone's data
get_someone_data <- function(name, pb = NULL){
  if (!is.null(pb)) pb$tick()$print()
  Sys.sleep(1)
  item <- WikidataR::find_item(name, language = "sv")
  # sometimes people have no Wikidata entry so I need this condition
  if(length(item) > 0){
    entity_code <- item[[1]]$id
    query <-  paste0("PREFIX entity: <http://www.wikidata.org/entity/>
                     #partial results
                     
                     SELECT ?propUrl ?propLabel ?valUrl ?valLabel ?picture
                     WHERE
                     {
                     hint:Query hint:optimizer 'None' .
                     {	BIND(entity:",entity_code," AS ?valUrl) .
                     BIND(\"N/A\" AS ?propUrl ) .
                     BIND(\"identity\"@sv AS ?propLabel ) .
                     }
                     UNION
                     {	entity:", entity_code," ?propUrl ?valUrl .
                     ?property ?ref ?propUrl .
                     ?property rdf:type wikibase:Property .
                     ?property rdfs:label ?propLabel
                     }
                     
                     ?valUrl rdfs:label ?valLabel
                     FILTER (LANG(?valLabel) = 'sv') .
                     OPTIONAL{ ?valUrl wdt:P18 ?picture .}
                     FILTER (lang(?propLabel) = 'sv' )
                     }
                     ORDER BY ?propUrl ?valUrl
                     LIMIT 200")
    results <- WikidataQueryServiceR::query_wikidata(query)
    results$name<- name
    results
  }else{
    NULL
  }
   
  }

pb <- dplyr::progress_estimated(length(unique(sommargaester$name)))
sommargaester_wiki <- purrr::map_df(unique(sommargaester$name),
                                    get_someone_data, pb=pb)

sommargaester <- dplyr::left_join(sommargaester, sommargaester_wiki, by = "name")

readr::write_csv(sommargaester, path = "data/p1_wiki_data.csv")

knitr::kable(sommargaester[1:10,])

name	rep	pretty_date	propLabel	valLabel
Lars-Eric Aaro	1	2011-07-14	arbetsgivare	LKAB
Lars-Eric Aaro	1	2011-07-14	födelseort	Vittangi
Lars-Eric Aaro	1	2011-07-14	kön	man
Lars-Eric Aaro	1	2011-07-14	nationalitet	Sverige
Lars-Eric Aaro	1	2011-07-14	instans av	människa
Lars-Eric Aaro	1	2011-07-14	alma mater	Luleå tekniska universitet
Lars-Eric Aaro	1	2011-07-14	medlem av	Kungliga Ingenjörsvetenskapsakademien
Lars-Eric Aaro	1	2011-07-14	förnamn	Lars
Lars-Eric Aaro	1	2011-07-14	identity	Lars-Eric Aaro
Magdi Abdelhadi	1	2007-08-09	sysselsättning	journalist

I get one line per guest and propriety value, sometimes some propriety labels have more than one value for the same person, e.g. if the person has several occupations (occupation = sysselsättning).

Translation the occupations

This is the point at which I realized that maybe doing the search in English in the first place wouldn’t have made me get less data? Oh well. In any case for each possible occupation I can get an English translation which is awesome.

occupation <- dplyr::filter(sommargaester, propLabel == "sysselsättning")
occupations <- unique(occupation$valLabel)
translate_occupation <- function(occupation){
  job <- WikidataR::find_item(occupation, language = "sv")[[1]]$label
  if(is.null(job)){
    job <- ""
  }
  return(job)
}
english_occupations <- purrr::map_chr(occupations, translate_occupation)
translations <- data.frame(sysselsaettning = occupations,
                           occupation = english_occupations)
occupation <- dplyr::left_join(occupation, translations, by = c("valLabel" = "sysselsaettning"))

I guess it’s good to know that Wikidata has data in many languages because I can totally see it being useful as a sort of translation service.

Profiling the summer guests

How much data?

withoccupation <- dplyr::filter(sommargaester, !is.na(propLabel))
withoccupation <- dplyr::group_by(withoccupation, name)
withoccupation <- dplyr::summarise(withoccupation, withoccupation = any(propLabel == "sysselsättning"))

There are 2192 unique guests in the dataset. We got Wikidata information for 1587 of them which means 72% of them. We got at least one occupation for 1421 guests which is 65% of them. At this point I have no way to know whether the sample is representative. Maybe the people with more Wikidata output are the most famous ones and some occupations probably help getting more famous than other so I tend to think this makes my sample a bit limited.

How often do guests get invited

As I said earlier some guests were invited several times. I want to look at the distribution of the number of invitations per guest.

invites <- dplyr::select(sommargaester, name, rep)
invites <- dplyr::group_by(invites, name)
invites <- dplyr::summarize(invites, no_invites = max(rep))
invites <- unique(invites)
library("ggplot2")
 theme_set(theme_gray(base_size = 18))
ggplot(invites) +
  geom_histogram(aes(no_invites)) +
  xlab("Number of invitations per P1 summer guest")

Some people have been invited a lot! If you remember my initial tables, it had only a few columns for the episode dates but one guest could have several lines.

Who are the people that got invited that many times?

jobs <- dplyr::select(occupation, name, occupation) 
jobs <- unique(jobs)
invites <- dplyr::left_join(invites, occupation, by = "name")
invites <- dplyr::select(invites, name, no_invites, occupation)
invites <- unique(invites)
invites <- dplyr::group_by(invites, name, no_invites)
invites <- dplyr::summarise(invites, occupation = toString(occupation))
invites <- dplyr::ungroup(invites)
invites <- dplyr::arrange(invites, desc(no_invites))
knitr::kable(invites[1:20,])

name	no_invites	occupation
Torsten Ehrenmark	54	journalist, translator, writer
Jörgen Cederberg	53	NA
Gösta Knutsson	42	cartoonist, journalist, translator, children’s writer, writer
Maud Reuterswärd	35	writer
Georg Eliasson	29	screenwriter
Pekka Langer	29	television presenter, actor
Britt Edwall	24	writer
Arne Ericsson	23	musician
Bengt Feldreich	23	journalist, television presenter
Berndt Friberg	23	journalist
Carl-Olof Lång	23	director
Carl-Uno Sjöblom	23	television presenter
Lars Ulvenstam	23	journalist
Lars Widding	23	journalist, writer
Åke Falck	17	film director, director, television presenter, screenwriter, actor
Barbro Alving	17	journalist, screenwriter, writer
Bo Setterlind	17	writer, poet
Bo Strömstedt	17	NA
Elisabeth Söderström	17	singer, opera singer
Fredrik Burgman	17	journalist

Most frequent occupations (by episode)

everything <- dplyr::group_by(occupation, occupation)
everything <- dplyr::summarize(everything, n = n())
everything <- dplyr::filter(everything, n > 50)
library("magrittr")
everything %>%
  dplyr::arrange(n) %>%
  dplyr::mutate(occupation = factor(occupation, ordered = TRUE, levels = unique(occupation))) %>%
ggplot() +
  geom_col(aes(occupation, n)) +
  theme(axis.text.x = element_text(angle = 90, hjust = 1)) +
  ggtitle("Number of episodes by occupation",
subtitle = "for occupations represented more than 50 times")

So in terms of episodes I’d say the media and culture fields are quite well represented. That said it might also be due to the fact that a person doing sports for instance will be classified as either ice skater or sprinter, so there’s no chance for sports to appear here. I’ll leave the classification of occupations in categories as an exercise for the reader though.

Most frequent occupations (by unique guest)

everything <- dplyr::select(occupation, name, occupation)
everything <- unique(everything)
everything <- dplyr::group_by(everything, occupation)
everything <- dplyr::summarize(everything, n = n())
everything <- dplyr::filter(everything, n > 20)
library("magrittr")
everything %>%
  dplyr::arrange(n) %>%
  dplyr::mutate(occupation = factor(occupation, ordered = TRUE, levels = unique(occupation))) %>%
ggplot() +
  geom_col(aes(occupation, n)) +
  theme(axis.text.x = element_text(angle = 90, hjust = 1)) +
  ggtitle("Number of guests by occupation",
subtitle = "for occupations represented more than 20 times")

With this figure by guest rather than by episode, a bit more diversity has appeared, e.g. “university professor” and “association football player”. Moreover although this was a nice exercise I won’t pretend that someone’s occupation characterizes them fully! The P1 summer guest programm really presents a variety of episodes, and I’d tend to think everyone can find an episode that they appreciate. For instance I remember liking the baker Sébastien Boudet’s episode (yep he’s French) and the one featuring the former high jumper Kajsa Berqvist. Now all this data munging has left me more than ready to look at the list of episodes again to choose the next ones I’ll listen to!

What's in our internal chaimagic package at work

Sat, 22 Jul 2017 00:00:00 +0000

At my day job I’m a data manager and statistician for an epidemiology project called CHAI lead by Cathryn Tonne. CHAI means “Cardio-vascular health effects of air pollution in Telangana, India” and you can find more about it in our recently published protocol paper . At my institute you could also find the PASTA and TAPAS projects so apparently epidemiologists are good at naming things, or obsessed with food… But back to CHAI! This week Sean Lopp from RStudio wrote an interesting blog post about internal packages. I liked reading it and feeling good because we do have an internal R package for CHAI! In this blog post, I’ll explain what’s in there, in the hope of maybe providing inspiration for your own internal package!

As posted in this tweet, this pic represents the Barcelona contingent of CHAI, a really nice group to work with! We have other colleagues in India obviously, but also in the US.

The birth of `chaimagic`: help for data wrangling

Note: part of this post was submitted as an abstract for the useR! conference in Brussels, for which I received feedback from my whole team, as well as from François Michonneau, Dirk Schumacher and Jennifer Thompson. Thanks a lot! I got to present a lightning talk about rtimicropem instead.

Right from the beginning of my time here in October 2015 I had quite a lot of data to clean, which I started doing in R of course. One task in particular was parsing filenames because those were informative in our project, containing for instance a date and a participant ID. I wrote a function for that, batch_parse_filenames, which parses all filenames in a vector and returns a list composed of two data.frames: one with the information contained in the filename (e.g. participant ID, date of sampling) and one with possible parsing mistakes (e.g. a nonexistent ID) and an informative message. It was a very useful function, and I packaged it up together with the data containing the participant IDs for instance. This was the beginning of an internal package!

I decided to call it chaimagic because it sounded cool, and this despite knowing that there’s absolutely no magic in the code I write.

The growth of `chaimagic`: support for data analysis

chaimagic evolved with helper functions for analysis, e.g. a function returning the season in India from a date (e.g. monsoon, winter), or functions for common operations on variables from our questionnaire data. chaimagic also got one more contributor.

chaimagic also contains a Shiny dashboard for exploring personal monitoring data that were collected in the project: one selects a participant-day and gets interactive leaflet and rbokeh plots of GPS, air quality, and time-activity data from questionnaires and wearable cameras. I told my boss that the dashboard was brought to us by the biblical Magi but maybe I helped them, who knows.

The maturity of `chaimagic`: serving scientific communication

Now, our package also supports scientific communication thanks to an RMarkdown template for internal reports which fostered reproducibility of analyses by making the use of RMarkdown even more appealing. Having an RMarkdown template also supports consistent branding, which is discussed in the RStudio blog post.

chaimagic also includes a function returning affiliation information for any possible collaborator we have; and a function returning the agreed-upon acknowledgement sentences to be put in each manuscript. These are pieces of information we can be very happy not to have to go look for in a folder somewhere, we can get them right without leaving R!

Why is `chaimagic` a really cool package?

chaimagic has been a valuable tool for the work of two statisticians/data managers, one GIS technician, one post-doc and one PhD student. Discussing its development and using it was an occasion to share our R joy, thus fostering best practices in scientific computing in our project. Why use Stata or other statistical softwares when you have such a nice internal R package? We found that even if our package served a small team of users, an R package provided a robust infrastructure to ensure that everyone was using the same code and R coding best practices in our team.

So this is very good… but we all know from e.g. Airbnb (see also this post about Airbnb and R) that stickers are a very important aspect of an internal R package. I was over the moon when my colleague Carles handed me these fantastic chaimagic stickers! He used the hexSticker package. I’ll leave the team in September after everyone’s vacations, and this piece of art was the goodbye present I received at my goodbye lunch. Doesn’t it make our internal package completely cool now?

How I became a crolute i.e. an user of the crul package

Fri, 30 Jun 2017 00:00:00 +0000

A few months ago rOpenSci’s Scott Chamberlain asked me for feedback about a new package of his called crul, an http client like httr, so basically something you use for e.g. writing a package interfacing an API. He told me that a great thing about crul was that it supports asynchronous requests. I felt utterly uncool because I had no idea what this meant although I had already written quite a few API packages (for instance ropenaq, riem and opencage).

So I googled the concept, my mind was blown and I decided that I’d trust Scott’s skills (spoiler: you can always do that) and just replace the httr dependency of ropenaq by crul. Why? First of all note that Crul is a planet in Star Wars whose male inhabitants are called crolutes which sound quite cool (there are female ones as well, called gilliands which doesn’t sound like the package name) and which I now use as a synonym for “user of the crul package”. But I had other reasons to switch… that was the subject of my lightning talk today at the French R conference in Anglet. In this blog post I’ll tell the story again, with a bit more details, in the hope to make you curious about crul!

Pic by ThinkR, thanks Colin/Diane/Vincent!

Asynchronous requests for dummies like me

OpenAQ API, which is a fantastic source of open data about air quality (see my blog post about ropenaq on rOpenSci’s blog), has a pagination system. If you want say all ozone measures for one country, the API will produce several pages of results containing 10,000 measurements each, and for each of these pages you’ll need to make a request.

In theory, you can either do all these requests one after the other (synchronous requests, yes synchronous) or all of them at once or at least in bunches, which corresponds to asynchronous requests. The crul package supports asynchronous requests while httr doesn’t. As you can imagine, sending several requests at once can increase speed!

Of course, you can’t send too many requests at once to an API, either because it’s not allowed anyway, or at least because it’s not a really nice thing to do: you might put an API in trouble by asking too many things in one go and you probably don’t want to break things. I asked OpenAQ’s co-founder Joe Flasher whether I could use asynchronous requests, thinking he might say no. That was funny because I actually thought the API was frailer than it is, Joe told me it was fine to do up to 10 requests at once, so I started doing that.

A big change for `ropenaq`

So I started removing all calls to httr functions by calls to crul, which wasn’t too hard really because crul has two vignettes, in particular one about asynchronous requests. I like the style of the vignettes, I could recognize the style from R for cats. Besides, Scott had asked for feedback and I assumed that asking many questions would just help him make the documentation fool-proof. Thanks for your patience Scott!

You can see the crul calls of ropenaq here. I have no ambition to write a step by step but basically what happens now in ropenaq is:

if the user provides a page argument, then I do a simple request for that page thanks to an crul::HttpRequest object. In the past as an user of ropenaq for any data wish you had to first had to make a query to get the count of measurements and then write a loop over all pages. Now you don’t, see the following point.
if the user doesn’t provide a page, then I retrieve the number of measurements for the query in total, then count the pages and prepare one query by page, which I group in bunches of 10 queries. Each bunch, which is a list, is then used to create an AsyncVaried object via crul::AsyncVaried$new(.list = async), and then I use methods of that object to retrieve the status and the content.

The only methods I needed to use where new to create objects, request to actually make the requests, status_code to retrieve the request status, get to extract the content of the request output when it had worked and parse to well parse the contents. Their names are quite well chosen. For figuring all of that out I read the vignettes, the documentation of the classes and as mentioned previously asked questions to Scott. In particular at the beginning I wasn’t too sure how to make the 10 requests for different pages at once but as the async vignette of crul says,

“There are two interfaces to asynchronous requests in crul:

Simple async: any number of URLs, all treated with the same curl options, headers, etc., and only one HTTP method type at a time.
Varied request async: build any type of request and execute all asynchronously.”

and my case was the second one. I was also a bit puzzled as to how to pass my list of requests to the function creating the AsyncVaried object but it turned out you can just pass a list to the .list argument.

Now the CRAN version of ropenaq is a proud reverse dependency of crul (yes my packages have feelings), so if you do install.packages("ropenaq") you can take advantage of my hard work. In my talk I had the results of a small microbenchmark and long story short as you can imagine the total time needed to retrieve data is shorter when you use asynchronous requests.

Other arguments in favour of `crul`

So asynchronous requests are a big argument in my opinion to use crul instead of httr as an API package developer but I’ve also listed a few other reasons. Note that I don’t get any money for saying any of this, this is my most honest opinion.

R6 classes!

You might have noticed I talked about creating objects and then using methods. crul is based on R6 classes so uses object-oriented programming. You can do things with these objects thanks to the methods. In httr, everything is supported by functions. This is an argument for using crul if:

you like R6 classes or object-oriented programming in general.
like me you’ve not used a lot of object-oriented programming but you’re curious to discover them. It was really not scary. Besides I love the fact that once you’ve created an object, in RStudio thanks to auto-completion you can start typing “$”, then press tab and see all the methods of that object.

Furthermore, each time I called an httr function in my package code I wrote httr:: before the name of the function to specify the import. With crul you do that only when creating an object, after that when you use the methods of that object you don’t need to declare any more import. So now I can be a bit lazier!

The promise of mocking made easy quite soon!

The unit tests of an API package might fail because your code no longer works… or because the API changed its output or is just down. A solution to differentiate these failure reasons is to have tests calling the API, and tests using fake data, the latter being mocked tests. This is discussed in this very good thread. Right now crul supports mocking via webmockr, and if I understood correctly will soon support mocking in an easier way via vcr. When it does I’ll try to make myself wiser about mocking which I don’t use at all currently.

Conclusion

So overall if you’re an API package developper you should consider giving crul a try! I’ll keep using httr in my webscraping and well it is still a dependency of all my other API packages so I like both packages.

Pic by Aline Deschamps, on that slide I was merely advertising for rOpenSci and in particular its onboarding process for packages

At the French R conference I was happy to get to present the reasons why I became a crolute. I also enjoyed meeting new people, some of them I knew from Twitter (Colin Fay who actually gave a talk about API packages, Diane Beldame who lead a tutorial about the tidyverse, Vincent Guyader who made the use of Docker to encapsulate a Shiny app look really easy, all of them from ThinkR, Joël Gombin who gave a keynote about data journalism and Aline Deschamps from Dacta), many others I didn’t know at all, for instance the real guy behind The R graph gallery who made a cool project using tweets about surfing. I even got to see Nick Tierney again, who gave a talk about his awesome narnia package for handling missing data. The talks were great, some findings were shared via the #angletr2017 hashtag on Twitter.

We heard that the next French R conference will be in Rennes which is in Brittany, my beloved home region, so I’m already hoping I can go next year, as an human or a crolute or whatever!

Automatic tools for improving R packages

Sat, 17 Jun 2017 00:00:00 +0000

On Tuesday I gave a talk at a meetup of the R users group of Barcelona. I got to choose the topic of my talk, and decided I’d like to expand a bit on a recent tweet of mine. There are tools that help you improve your R packages, some of them are not famous enough yet in my opinion, so I was happy to help spread the word! I published my slides online but thought that a blog post would be nice as well.

During my talk at RUG BCN, for each tool I gave a short introduction and then applied it to a small package I had created for the occasion. In that post I’ll just shortly present each tool. Most of them are only automatic because they automatically provide you with a list of things to fix, but they won’t do the work for you, sorry. If you have an R package you develop at hand, I’d really advise you to apply them on it and see what you get! I concentrated on tools improving the coding style, the package structure, testing, the documentation, but not features and performance.

Note: before my talk Luca Cerone gave a short introduction to package development with devtools and roxygen2. This is how I develop packages now although the first package I contributed to, surveillance, has separate man pages, a NAMESPACE maintained by hand, etc. I gave its maintainer a hard time with all my stupid beginner mistakes! Thankfully Sebastian was always patient and nice. In any case, if you want to start developping packages you should definitely read Hadley Wickham’s book, it’s a good one! I read it all once I started upping my package game – this upping being a continuous process of course.

The basics: run R CMD check!

In his talk Luca said that when he develops a package he often doesn’t have time to check it passes R CMD check, which made me die inside a little bit. I totally understand the lack of time but I think that R CMD check complains about important stuff, and most often unknown NOTE, WARNING or ERROR messages can be googled and solved, and one learns something. So please do run R CMD check!

The R-Lady that uploaded this picture to the Meetup event told me that this is my “Run R CMD check, you fools!” face (she actually said Usad el R CMD check, malditos!!!).

`lintr`, Static Code Analysis for R

I then presented Jim Hester’s lintr package which allows you to check some stuff in your code without running it. It will for instance tell you if you used camel case instead of snake case for a variable name, or if you defined a variable without using it afterwards. The README of the Github repository of the package provides a list of available linters, each linter being one of these annoying, hum useful, checks.

How does one use lintr? Just by typing lintr::lint_package() at the root of the package folder. This week I applied it to a package of mine and spent some time adding white space that made my code prettier. In my opinion using lintr helps you maintain a code that is nice to read, which is cool because then you, future you or another human can concentrate on important stuff when reading it.

I mentioned two small tips as well.

How to customize the `lintr` linters?

For that one needs to create a .lintr file at the root of the package (and add it to .Rbuildignore). Here is an example of the content of such a file:

linters: with_defaults(camel_case_linter = NULL, # you prefer camel case 
              snake_case_linter, # so flag snake case
              line_length_linter(120)) # you love long lines

Refer to the “Project Configuration” section of lintr README.

How to never ever ignore `lintr` output

Edit on 2019-10-02: the lintrdocs recommend using the package in combination with continuoua integration, not to set up an unit test.

You can add an unit test for lintr! Why would one ever want to do that? Well, you might want to force yourself, or contributors to your package, to respect some rules. The test would be:

if (requireNamespace("lintr", quietly = TRUE)) {
  context("lints")
  test_that("Package Style", {
    lintr::expect_lint_free()
  })
}

After I published my slides someone added such a test to one of his packages which made me very proud.

One tip I forgot to mention during my talk is how to add exceptions, I was far too strict!

How to exclude lines from `lintr` checks

For that either put # nolint at the end of the line or # nolint start and # nolint end before and after a block of lines you’d like to exclude. A good example is for instance a very long line that you can’t break because it contains a very long URL.

`goodpractice`, Advice on R Package Building

This package by Gabor Csardi can only be installed from Github. It integrates a variety of tools, including lintr, for providing “Advice (that) includes functions and syntax to avoid, package structure, code complexity, code formatting, etc.”. For instance, goodpractice uses cyclocomp to check whether any of your functions is too complex. A participant of the meetup told me she got a lot of messages when running goodpractice::gp() on her first package, which is great because it means goodpractice will really have provided her with useful guidance. Some things that goodpractice::gp() will flag might be necessary features of your package, but in general it’s a nice list to read. Note: you can only run goodpractice::gp() once R CMD check passes.

Romain François who came to the Meetup from Montpellier (thanks, Romain!) asked me if I had ever run goodpractice::gp() on famous packages like dplyr. I haven’t but since the default advice of goodpractice seems quite consistent with the advice in Hadley Wickham’s book, I doubt I’d find anything interesting.

At rOpenSci onboarding, we run goodpractice::gp() on every submitted package as part of the editorial checks, which makes me like this tool a bit more every week. It’s actually tunable but the defaults are so great that I never needed to tune it!

I was asked when to run goodpractice::gp() and I’d say once in a while, e.g. before submitting a package to CRAN for the first time, or after creating its first working version.

`devtools::spell_check()`

This function was written by Jeroen Ooms and is supported by his hunspell package. It will look for possible typos in the documentation of a package. In the output, there might be quite a few false positives, e.g. the name of an API, but I strongly believe that its helping you find typos in the documentation compensates the effort of reading the whole list of potential typos.

I’ve noticed that devtools::release(), which one can use when submitting a package to CRAN, now includes the output of devtools::spell_check(). It helped me find a typo at the last minute in one of my packages! Typos might not be that bad, but a typo-free documentation looks much better.

`pkgdown`, generate static html documentation for an R package

This last tool was the only one that’s truly automatic. This package was created by Hadley Wickham and is currently only on Github. It allows you to create a pretty website for your package without any big effort.

How does one use it? After installing it, run pkgdown::init() and pkgdown::build_site() at the root of your package. This will create a docs/ folder containing the static html documentation, that you can browse locally.

Now, how do you present it to the world? In theory I guess you could publish it anywhere but the really handy thing is to have it as a Github page, if you develop your package on Github. For this you just need to change the settings of the repository to make master/docs the source for Github pages.

Unless you typically redirect all links from your Github account to your domain, your package will live at account_name.github.io/package_name. See for instance this nice package of Dirk Schumacher’s whose website is here.

How to customize the website?

This part is not automatic, and demands creating a _pkgdown.yml file at the root of your package. I’ve only mentioned two aspects:

One can change the style of the website, e.g. in this config file I chose another Boostrap theme than the default one. To choose one Boostrap theme one can browse them online or look at a pkgdown website that one likes and look at how it was created. A participant of the meetup told me that there is an RStudio addin for choosing a theme but I couldn’t locate it.
One can change the order of vignettes, or of the functions, and even group them, which might be extremely useful when a package has many functions. Usually the alphabetical order makes little sense. See here the config file of this website, the grouping is really well done.

Conclusion

Since this was a talk, not a long workshop, and also because I don’t know everything, I have not mentioned every useful tool in the world. I’d be grateful for suggestions in the comments, and in the meantime here are some tools I briefly mentioned at the end of my talk:

Continuous integration, with this well written post by Julia Silge
The R-hub project
The CRAN task view about package development
The available package to check whether the name you want to use for your package is a) available and b) appropriate.

A very important tip I gave was to have your code read by humans. The automatic tools will flag some things, but the feedback of an human about your code and your documentation will be crucial. And when you review code, you’ll learn a lot! If you want to do that, you can volunteer to become a reviewer for rOpenSci, see more info here and fill the short form here.

After my talk I got a nice RMarkdown t-shirt that has the same colour as my face! Thanks a lot to the organizers, in particular Luca Cerone!

Who is talking about the French Open?

Sat, 10 Jun 2017 00:00:00 +0000

I don’t think rOpenSci’s Jeroen Ooms can ever top the coolness of his magick package but I have to admit other things he’s developped are not bad at all. He’s recently been working on interfaces to Google compact language detectors 2 and 3 (the latter being more experimental). I saw this cool use case and started thinking about other possible applications of the packages.

I was very sad when I realized it was too late to try and download tweets about the Eurovision song context but then I also remembered there’s this famous tennis tournament going on right now, about which people probably tweet in various languages. I don’t follow the French Open myself, but it seemed interesting to find out which languages were the most prevalent, and whether the results from the cld2 and cld3 packages are similar and whether they’re similar to the language detection results from Twitter itself.

Getting the tweets

I’m using my usual rtweet recipe. I no longer need to open my eyes when downloading tweets.

rg_tweets <- rtweet::search_tweets(q = "#RolandGarros2017",
                                   include_rts = FALSE,
                                   n = 18000)
save(rg_tweets, file = "data/2017-06-07-rolandgarros.RData")

I got 18000 tweets.

Using the language detectors

I decided to first clean the tweets a bit, removing hashtags, mentions, and at least part of the links.

rg_tweets <- dplyr::mutate(rg_tweets,
                           text = stringr::str_replace_all(text, "#.*$", ""),
                           text = stringr::str_replace_all(text, "https.*$", ""),
                           text = stringr::str_replace_all(text, "@.*$", ""),
                           text = stringr::str_replace_all(text, "#.* ", ""),
                           text = stringr::str_replace_all(text, "https.* ", ""),
                           text = stringr::str_replace_all(text, "@.* ", ""))

Today I’m a happy naive user of language detectors, but more technical details can be found in their respective README’s. It’d be difficult to have an easier interface than the two cld2 and cld3 packages. They’re also fast, although I haven’t timed the following chunk so you’ll have to believe me or test the packages yourselves.

Note that cld2 and cld3 both have functions for outputting several languages instead of one, with the associated reliability, but I won’t use them since I want a direct comparison with the Twitter output of one language per tweet.

rg_tweets <- dplyr::mutate(rg_tweets, 
                           cld2_language = cld2::detect_language(text,
                                                                 lang_code = TRUE), 
                           cld3_language = cld3::detect_language(text))

Before analysing the results, I’ll transform the Twitter detected language a bit: it’s not NA for undertemined language, it’s “und”.

rg_tweets <- dplyr::mutate(rg_tweets, 
                           lang = ifelse(lang == "und", NA, lang))

Looking at detected languages

Twitter output a language for 96% of the tweets, cld2 for 59% and cld3 for 84% of them.

Let’s see a few cases in which Twitter outputs a language whereas the other don’t.

As a side note, I learnt how to insert a DT::datatable from Daniela Vázquez after admiring one she had put in this cool blog post. This was an adventure in htmlwidgets hell. I started looking at Daniela’s Github blog repo, then talked with her on the R-Ladies slack, and I told her I’d do more the next day. I woke up to a PR solving all problems! Thanks a lot Daniela, and also thanks to your husband! Update: Daniela’s husband Gervasio came up with a fix not involving custom Jekyll plugins after I realized my site couldn’t be built on Github pages… I am so thankful for their help!

library("magrittr")
rg_tweets %>%
  dplyr::filter(is.na(cld2_language), is.na(cld3_language)) %>%
  dplyr::select(text, lang, cld2_language,
         cld3_language)%>%
  head(n = 50) %>%
  DT::datatable()

It seems that the languages with undetermined languages via cld2 and cld3 are quite short. The Twitter language detector might be more focused at short sentences which well given the length of tweets wouldn’t be surprising. Moreover, maybe it often inputs a language even when uncertain. If we take the example of the word “Merci”, it’s French but also used in Catalan at least in Barcelona, so to me that seems uncertain. Some other tweets to which Twitter but not the other language detectors associated a language are a mix of languages.

Let’s have a look at lines with disagreements when no language information is missing.

rg_tweets %>%
  dplyr::filter(!is.na(lang),
                !is.na(cld2_language),
                !is.na(cld3_language)) %>%
  dplyr::group_by(text) %>%
  dplyr::filter(length(unique(c(lang, cld2_language,
                              cld3_language))) != 1) %>%
  head(n = 50) %>%
  dplyr::select(text, lang, cld2_language,
         cld3_language)%>%
  DT::datatable()

Similarly I think these tweets are quite short. Moreover, languages seem to often be different but not that different, e.g. “es” (Spanish) and “ca” (Catalan) or “es” and “gl” (Galician). I sometimes make similar mistakes, when I say I’ve heard a “Nordic language”” because I couldn’t identify it further than “not Swedish” (which I should be able to recognize).

I can also look at dissimilarities, computed on tweets with determined language.

dplyr::select(rg_tweets, lang, cld2_language, cld3_language) %>%
  dplyr::mutate_all(dplyr::funs(as.factor(.))) %>%
  t() %>%
  as.data.frame() %>%
cluster::daisy()

## Dissimilarities :
##                     lang cld2_language
## cld2_language 0.09870999              
## cld3_language 0.41158430    0.16547982
## 
## Metric :  mixed ;  Types = N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N, N 
## Number of objects : 3

Unsurprisingly there’s a worse agreement between cld3 results and the other two ones than between cld2 and Twitter. I say I’m not surprised because cld3 is a still experimental language detector.

Representing languages

I’m going to assume that if Twitter and cld2 agree on the language assigned to a tweet, then it’s quite reliable.

agreed <- dplyr::filter(rg_tweets,
                        lang == cld2_language)

I’m therefore only considering 9432 tweets out of the original 18000 tweets.

agreed <- dplyr::group_by(agreed, lang)
agreed <- dplyr::summarize(agreed, tweets_count = n())
agreed <- dplyr::ungroup(agreed)
agreed <- dplyr::arrange(agreed, desc(tweets_count))
agreed <- dplyr::mutate(agreed, lang = factor(lang, ordered = TRUE, levels = unique(lang)))

Let’s plot the results.

library("ggplot2")
library("hrbrthemes")
ggplot(agreed) +
  geom_col(aes(lang, tweets_count)) +
  theme_ipsum(base_size = 20,
              axis_title_size = 20) +
  xlab("Detected language") +
  ylab("No. of tweets in the sample")

The most represented languages, English, French and Spanish are surely a result of who’s on Twitter, what are the most spoken languages on Earth, who’s interested in tennis (or the tennis players) and who’s awake when the tournament happens. One way to control for the timezone of the tournament would be to stream tweets during each of the tournaments of the tennis Grand Slam. Another extension of this small blog post would be to look for players names in tweets and to then see if one can find an association between the most mentioned players in a language and the nationality of these players. This could even be coupled to a sentiment analysis (you could support one player and criticize the other players). Then again, that’s something that’d be even more interesting in my opinion if applied to Eurovision contestants instead! Next year maybe…

Which science is all around? #BillMeetScienceTwitter

Sat, 20 May 2017 00:00:00 +0000

I’ll admit I didn’t really know who Bill Nye was before yesterday. His name sounds a bit like Bill Nighy’s, that’s all I knew. But well science is all around and quite often scientists on Twitter start interesting campaigns. Remember the #actuallylivingscientists whose animals I dedicated a blog post? This time, the Twitter campaign is the #BillMeetScienceTwitter hashtag with which scientists introduce themselves to the famous science TV host Bill Nye. Here is a nice article about the movement.

Since I like surfing on Twitter trends, I decided to download a few of these tweets and to use my own R interface to the Monkeylearn machine learning API, monkeylearn (part of the rOpenSci project!), to classify the tweets in the hope of finding the most represented science fields. So, which science is all around?

Getting the tweets

It might sound a bit like trolling by now, but if you wanna get Twitter data, I recommend using rtweet because it’s a good package and because it’s going to replace twitteR which you might know from other blogs.

I only keep tweets in English, and moreover original ones, i.e. not retweets.

library("rtweet")
billmeet <- search_tweets(q = "#BillMeetScienceTwitter", n = 18000, type = "recent")
billmeet <- unique(billmeet)

billmeet <- dplyr::filter(billmeet, lang == "en")
billmeet <- dplyr::filter(billmeet, is_retweet == FALSE)

I’ve ended up with 2491 tweets.

Classifying the tweets

I’ve chosen to use this taxonomy classifier which classifies text according to generic topics and had quite a few stars on Monkeylearn website. I don’t think it was trained on tweets, and well it wasn’t trained to classify science topics in particular, which is not optimal, but it had the merit of being readily available. I’ve still not started training my own algorithms, and anyway, if I did I’d start by creating a very crucial algorithm for determining animal fluffiness on pictures, not text mining stuff. This was a bit off topic, let’s go back to science Twitter!

When I decided to use my own package I had forgotten it took charge of cutting the request vector into groups of 20 tweets, since the API only accept 20 texts at a time. I thought I’d have to do that splitting myself, but no, since I did it once in the code of the package, I’ll never need to write that code ever again. Great feeling! Look at how easy the code is after cleaning up the tweets a bit! One just needs to wait a bit before getting all results.

output <- monkeylearn::monkeylearn_classify(request = billmeet$text,
                                            classifier_id = "cl_5icAVzKR")
str(output)

## Classes 'tbl_df', 'tbl' and 'data.frame':	4466 obs. of  4 variables:
##  $ category_id: int  64638 64640 64686 64696 64686 64687 64689 64692 64648 64600 ...
##  $ probability: num  0.207 0.739 0.292 0.784 0.521 0.565 0.796 0.453 0.301 0.605 ...
##  $ label      : chr  "Computers & Internet" "Internet" "Humanities" "Religion & Spirituality" ...
##  $ text_md5   : chr  "f7b28f45ea379b4ca6f34284ce0dc4b7" "f7b28f45ea379b4ca6f34284ce0dc4b7" "b95429d83df2cabb9cd701a562444f0b" "b95429d83df2cabb9cd701a562444f0b" ...
##  - attr(*, "headers")=Classes 'tbl_df', 'tbl' and 'data.frame':	0 obs. of  0 variables

In the output, the package creator decided not to put the whole text corresponding to each line but its digested form itself, digested by the MD5 algorithm. So to join the output to the tweets again, I’ll have to first digest the tweet, which I do just copying the code from the package. After all I wrote it. Maybe it was the only time I successfully used vapply in my whole life.

billmeet <- dplyr::mutate(billmeet, text_md5 = vapply(X = text,
                                                    FUN = digest::digest,
                                                    FUN.VALUE = character(1),
                                                    USE.NAMES = FALSE,
                                                    algo = "md5"))
billmeet <- dplyr::select(billmeet, text, text_md5)
output <- dplyr::left_join(output, billmeet, by = "text_md5")

Looking at this small sample, some things make sense, other make less sense, either because the classification isn’t good or because the tweet looks like spam. Since my own field isn’t text analysis, I’ll consider myself happy with these results, but I’d be of course happy to read any better version of it.

As in my #first7jobs, I’ll make a very arbitrary decision and filter the labels to which a probability higher to 0.5 was attributed.

output <- dplyr::filter(output, probability > 0.5)

This covers 0.45 of the original tweets sample. I can only hope it’s a representative sample.

How many labels do I have by tweet?

dplyr::group_by(output) %>%
  dplyr::summarise(nlabels = n()) %>%
  dplyr::group_by(nlabels) %>%
  dplyr::summarise(n_tweets = n()) %>%
  knitr::kable()

nlabels	n_tweets
1415	1

Perfect, only one.

Looking at the results

I know I suck at finding good section titles… At least I like the title of the post, which is a reference to the song Bill Nighy, not Bill Nye, sings in Love Actually. My husband assumed that science Twitter has more biomedical stuff. Now, even if my results were to support this fact, note that this could as well be because it’s easier to classify biomedical tweets.

I’ll first show a few examples of tweets for given labels.

dplyr::filter(output, label == "Chemistry") %>%
  head(n = 5) %>%
  knitr::kable()

category_id	probability	label	text_md5	text
64701	0.530	Chemistry	e82fc920b07ea9d08850928218529ca9	Hi @billnye I started off running BLAST for other ppl but now I have all the money I make them do my DNA extractions #BillMeetScienceTwitter
64701	0.656	Chemistry	d21ce4386512aae5458565fc2e36b686	.@uw’s biochemistry dept - home to Nobel Laureate Eddy Fischer & ZymoGenetics co founder Earl Davie… https://t.co/0nsZW3b3xu
64701	0.552	Chemistry	1d5be9d1e169dfbe2453b6cbe07a4b34	Yo @BillNye - I’m a chemist who plays w lasers & builds <U+009F><U+0094><U+00AC> to study protein interactions w materials #BillMeetScienceTwitter
64701	0.730	Chemistry	1b6a25fcb66deebf35246d7eeea34b1f	Meow @BillNye! I’m Zee and I study quantum physics and working on a Nobel prize. #BillMeetScienceTwitter https://t.co/oxAZO5Y6kI
64701	0.873	Chemistry	701d8c53e3494961ee7f7146b28b9c8c	Hi @BillNye, I’m a organic chemist studying how molecules form materials like the liquid crystal shown below.… https://t.co/QNG2hSG8Fw

dplyr::filter(output, label == "Aquatic Mammals") %>%
  head(n = 5) %>%
  knitr::kable()

category_id	probability	label	text_md5	text
64609	0.515	Aquatic Mammals	f070a05b09d2ccc85b4b1650139b6cd0	Hi Bill, I am Anusuya. I am a palaeo-biologist working at the University of Cape Town. @BillNye #BillMeetScienceTwitter
64609	0.807	Aquatic Mammals	bb06d18a1580c28c255e14e15a176a0f	Hi @BillNye! I worked with people at APL to show that California blue whales are nearly recovered #BillMeetScienceTwitter
64609	0.748	Aquatic Mammals	1ca07aad8bc1abe54836df8dd1ff1a9d	Hi @BillNye! I’m researching marine ecological indicators to improve Arctic marine monitoring and management… https://t.co/pJv8Om4IeI
64609	0.568	Aquatic Mammals	a140320fcf948701cfc9e7b01309ef8b	More like as opposed to vaginitis in dolphins or chimpanzees or sharks #BillMeetScienceTwitter https://t.co/gFCQIASty1
64609	0.520	Aquatic Mammals	06d1e8423a7d928ea31fd6db3c5fee05	Hi @BillNye I study visual function in ppl born w/o largest connection between brain hemispheres #callosalagenesis… https://t.co/WSz8xsP38R

dplyr::filter(output, label == "Internet") %>%
  head(n = 5) %>%
  knitr::kable()

category_id	probability	label	text_md5	text
64640	0.739	Internet	f7b28f45ea379b4ca6f34284ce0dc4b7	@BillNye #AskBillNye @BillNye join me @AllendaleCFD. More details at https://t.co/nJPwWARSsa

Based on that, and on the huge number of internet-labelled tweets, I decided to remove those.

library("ggplot2")
library("viridis")

label_counts <- output %>% 
  dplyr::filter(label != "Internet") %>%
  dplyr::group_by(label) %>% 
  dplyr::summarise(n = n()) %>% 
  dplyr::arrange(desc(n))

label_counts <- label_counts %>%
  dplyr::mutate(label = ifelse(n < 5, "others", label)) %>%
  dplyr::group_by(label) %>%
  dplyr::summarize(n = sum(n)) %>%
  dplyr::arrange(desc(n))

label_counts <- dplyr::mutate(label_counts,
                        label = factor(label,
                                        ordered = TRUE,
                                        levels = unique(label)))

ggplot(label_counts) +
  geom_bar(aes(label, n, fill = label), stat = "identity")+
  scale_fill_viridis(discrete = TRUE, option = "plasma")+
    theme(axis.text.x = element_text(angle = 90,
                            hjust = 1,
                            vjust = 1),
          text = element_text(size=25),
          legend.position = "none")

In the end, I’m always skeptical when looking at the results of such classifiers, and well at the quality of my sample to begin with – but then I doubt there ever was a hashtag that was perfectly used to only answer the question and not spam it and comment it (which is what I’m doing). I’d say it seems to support my husband’s hypothesis about biomedical stuff.

I’m pretty sure Bill Nye won’t have had the time to read all the tweets, but I think he should save them, or at least all the ones he can get via the Twitter API thanks to e.g. rtweet, in order to be able to look through them next time he needs an expert. And in the random sample of tweets he’s read, let’s hope he was exposed to a great diversity of science topics (and of scientists), although, hey, the health and life related stuff is the most interesting of course. Just kidding. I liked reading tweets about various scientists, science rocks! And these last words would be labelled with “performing arts”, perfect way to end this post.

How not to make an evergreen review graph

Sun, 14 May 2017 00:00:00 +0000

In this post I am inspired by two tweets, mainly this one and also this one. Since the total number of articles every year is increasing, no matter which subject you choose, the curve representing number of articles as a function of year of publication will probably look exponential, so one should not use such graphs to impress readers. At least I’m not impressed, I’m more amused by such graphs now that there’s a hashtag for them.

I shall use an rOpenSci package for getting some data about number of articles about a query term, and to do a graph that’s not an evergreen review graph!

Getting some numbers of articles

For this I’ll use one of the numerous rOpenSci packages simplifying your programmatic access to scientific literature, europepmc which is an interface to Europe PMC, “Search worldwide, life-sciences literature”. The package includes a cool function called epmc_profile that gives you the number of hits for a given query. Perfect!

Now my goal is to get the number of hits by year for a search term but also the total number of hits in order to put things into perspective. I had no idea how to write a proper Europe PMC query so I went to the advanced search page and made queries there in order to learn enough words of this language to get what I needed. Indeed, when you create a query by clicking there and then hit search, the URL contains the query as you need it to be written for using the europepmc package.

For each query epmc_profile returns hits by source and type, but I’m interested in all hits, including e.g. books, so that’s what I’ll use.

Let’s write functions!

library("europepmc")
find_hits <- function(query_term){
  years <- 1975:2016
  results <- purrr::map(years, find_hits_by_year, query_term = query_term)
  dplyr::bind_rows(results)
}

find_hits_by_year <- function(year, query_term){
  queryforall <- paste0('(FIRST_PDATE:[', year, '-01-01+TO+', year, '-12-31])')
  all_hits <- as.numeric(epmc_profile(queryforall)$pubType[1, 2])
  
  queryforterm <- paste0('(FIRST_PDATE:[', year, '-01-01+TO+', year, '-12-31]) AND "', query_term, '"')
  
  term_hits <- as.numeric(epmc_profile(queryforterm)$pubType[1, 2])
  
  return(tibble::tibble(year = year, all_hits = all_hits, term_hits = term_hits))
  
}

Now let’s apply the function to one term. I’ll choose Malaria because it’s the term used in one of europepmc examples.

malaria_fame <- find_hits("malaria")
library("magrittr")
head(malaria_fame) %>%
  knitr::kable()

year	all_hits	term_hits
1975	252386	394
1976	259044	425
1977	266538	432
1978	279160	475
1979	294491	516
1980	295471	443

Obviously in a review about malaria research one might have a more precise query, and one might want to compare the number of hits for that query to other precise queries, but in this simple example I’m just comparing the number of hits of one term to the number of all hits, Note that one could also make no graph about number of hits over time, shocking I now! I’ll most certainly won’t do no graph here though, now that I collected the data.

Making nice plots

I’ll do as Lucy in this blog post and use the xkcd package. As noted by Lucy, its vignette is great. I might have done something wrong but after downloading the file with the xkcd font I had to copy-paste it myself in the fonts folder of Windows. The syntax of the xkcdman is a bit long but following the examples I could get what I wanted.

library("xkcd")
library("ggplot2")
library("extrafont")
xrange <- range(malaria_fame$year)
yrange <- range(malaria_fame$term_hits)
ratioxy <- diff(xrange)/diff(yrange)
mapping <- aes(x=x,
               y=y,
               scale=scale,
               ratioxy=ratioxy,
               angleofspine = angleofspine,
               anglerighthumerus = anglerighthumerus,
               anglelefthumerus = anglelefthumerus,
               anglerightradius = anglerightradius,
               angleleftradius = angleleftradius,
               anglerightleg =  anglerightleg,
               angleleftleg = angleleftleg,
               angleofneck = angleofneck)
dataman <- data.frame(x = 1985, y = 4000,
                  scale = 1000,
                  ratioxy = ratioxy,
                  angleofspine =  - pi / 2,
                  anglerighthumerus = 4*pi/6,
                  anglelefthumerus = pi/6,
                  anglerightradius = pi/2,
                  angleleftradius = pi/2,
                  angleleftleg = 3*pi/2  + pi / 12 ,
                  anglerightleg = 3*pi/2  - pi / 12,
                  angleofneck =  3 * pi / 2 - pi/10)

ggplot(malaria_fame) +
  geom_point(aes(year, term_hits)) +
  xlab("Time in years") +
  ylab("Number of hits for Malaria") +
  xkcdaxis(xrange = xrange,
           yrange = yrange) +
  theme(text = element_text(size = 16, family = "xkcd")) +
  xkcdman(mapping,dataman) +
  annotate("text", family = "xkcd", x = 1985, y = 4800, 
           label = "Oops!\n I made an evergreen review graph!", size = 6)

Ok that one is not really the one I’d advise you to submit to any journal. Let’s do better.

malaria_fame <- dplyr::mutate(malaria_fame, 
                              prop = term_hits/all_hits)
xrange <- range(malaria_fame$year)
yrange <- range(malaria_fame$prop)
ratioxy <- diff(xrange)/diff(yrange)

dataman <- data.frame(x = 1985, y = 0.005,
                  scale = 0.001,
                  ratioxy = ratioxy,
                  angleofspine =  - pi / 2,
                  anglerighthumerus = -pi/6,
                  anglelefthumerus = pi + pi/6,
                  anglerightradius = 0,
                  angleleftradius = 0,
                  angleleftleg = 3*pi/2  + pi / 12 ,
                  anglerightleg = 3*pi/2  - pi / 12,
                  angleofneck =  3 * pi / 2 - pi/10)

ggplot(malaria_fame) +
  geom_point(aes(year, prop)) +
  xlab("Time in years") +
  ylab("No. of hits for Malaria divided by total no. of hits") +
  xkcdaxis(xrange = xrange,
           yrange = yrange) +
  theme(text = element_text(size = 16, family = "xkcd")) +
  xkcdman(mapping,dataman) +
  annotate("text", family = "xkcd", x = 1985, y = 0.006, 
           label = "Now that's more interesting!", size = 6)

Okay friends so that one is a bit more interesting, and I’m more convinced my its shape. I’d like to know whether there really was a malaria research peak, but only waiting for a few years will bring me the answer! It’d also be interesting to compare the fate of other diseases, and to put that in context with some domain knowledge. But that’d be a subject for another study, now at least you know how to not make an evergreen review graph!

A tribute to Lucy D'Agostino McGowan's git commit emoji game

Wed, 03 May 2017 00:00:00 +0000

Do you know Lucy? She is a very talented biostatistics PhD candidate that I had the chance to e-meet thanks to R-Ladies. One maybe superficial reason to admire her, on top of her other achievements, is her emoji game in git commits. Looking at Lucy’s git history (find her on Github), one wants to start using version control because she makes it look fun!

In this post, I will download many git commit messages of Lucy’s from Github’s API via the gh package, and have a look at the emojis she uses the most frequently.

Getting the commit messages

As I said I’ll use the gh package as in my post about first commits and as in the one about random seeds.

The first step is to get the names of Lucy’s repositories. I get all her public repositories, and also a few of the repositories of the LFOD organization where she blogs, that I selected by hand for not getting forks from the public repositories I already had. Hopefully her commit messages in these repositories, obviously excluding her private repositories, is representative of her typical emoji use.

library("gh")
library("magrittr")
lucy_repos <- gh("/users/:username/repos", 
                 type = "public", 
                 username = "LucyMcGowan")
lucy_repos <- vapply(lucy_repos, "[[", "", "name")
lucy_repos <- tibble::tibble(repos = lucy_repos,
                             who = "LucyMcGowan")
lucy_repos_org <- tibble::tibble(repos = c("LFOD.github.io",
                                           "blog",
                                           "real-blog"),
                                 who = "LFOD")

lucy_repos <- dplyr::bind_rows(lucy_repos,
                               lucy_repos_org)

head(lucy_repos) %>%
  knitr::kable()

repos	who
2017-03-18_hackathon	LucyMcGowan
branchingExample	LucyMcGowan
comps-survival-guide	LucyMcGowan
contributr	LucyMcGowan
cowsay	LucyMcGowan
datasauRus	LucyMcGowan

Then I got the commit messages, but also their authors in order to filter commits by Lucy herself, and the time and date at which they were made because Lucy told me she started using emojis last October. gh is a minimal package so it outputs lists of lists, so each time I use it I have a look at their structure to find how to extract exactly what I need, see below.

get_all_lucy_commits <- function(repo, who){
  
  # Since I don't know the number of pages
  # I grow the vectors of messages, dates and authors
  messages <- NULL
  authors <- NULL
  dates <- NULL
  
  page <- 1
  is_still_ok <- TRUE
  # loop while page still full of content
  while(is_still_ok[1]){
    commits <- try(gh("/repos/:owner/:repo/commits",
                      owner = who,
                      repo = repo,
                      page = page))
    
    if(class(commits)[1] != "try-error"){
      is_still_ok <- commits != ""
    }else{
      is_still_ok <- FALSE
    }
    
    if(is_still_ok[1]){
      # get information about commits
      commit_info <- lapply(commits, "[[", "commit")
      
      # commit messages
      now_messages <- lapply(commit_info, "[[", "message")
      messages <- c(messages, unlist(now_messages))
      
      # date (and time)
      committer <- lapply(commit_info, "[[", "committer")
      now_dates <- lapply(committer, "[[", "date")
      dates <- c(dates, unlist(now_dates))
      
      # who committed
      now_authors <- lapply(commit_info, "[[", "author")
      now_authors <- lapply(now_authors, "[[", "name")
      authors <- c(authors, unlist(now_authors))
      
      page <- page + 1
    }
  }
  
  results <- tibble::tibble(commit = messages,
                           repo = repo,
                           owner = who,
                           author = authors,
                           datetime = dates)
  # I only want commits from Lucy herself
  dplyr::filter_(results, 
                 lazyeval::interp(~ stringr::str_detect(author, "[Ll]ucy")))
}

commits <- purrr::map2(lucy_repos$repos,
                       lucy_repos$who, 
                       get_all_lucy_commits)
commits <- dplyr::bind_rows(commits)
commits <- dplyr::mutate(commits, 
                         encoding = Encoding(commit))

commits <- dplyr::mutate(commits, 
                         datetime = lubridate::ymd_hms(datetime))


head(commits) %>%
  knitr::kable()

commit	repo	owner	author	datetime	encoding
Rename park_dat_json.JSON to park_dat_json.json	2017-03-18_hackathon	LucyMcGowan	Lucy D’Agostino McGowan	2017-03-18 19:59:44	unknown
update json	2017-03-18_hackathon	LucyMcGowan	Lucy	2017-03-18 19:59:00	unknown
update json	2017-03-18_hackathon	LucyMcGowan	Lucy	2017-03-18 19:57:29	unknown
update json	2017-03-18_hackathon	LucyMcGowan	Lucy	2017-03-18 19:57:16	unknown
fix json	2017-03-18_hackathon	LucyMcGowan	Lucy	2017-03-18 19:31:56	unknown
first commit <U+009F><U+00A4><U+00A1>	2017-03-18_hackathon	LucyMcGowan	Lucy	2017-03-18 19:23:06	UTF-8

Looking at emojis

I used this awesome emoji dictionary created and presented in this blog.

dico <- readr::read_csv2("https://raw.githubusercontent.com/today-is-a-good-day/emojis/master/emDict.csv")
head(dico) %>%
  knitr::kable()

Description	Native	Bytes	R-encoding
AERIAL TRAMWAY	<U+009F><U+009A><U+00A1>	\xF0\x9F\x9A\xA1
AIRPLANE	<U+2708>	\xE2\x9C\x88	<9c><88>
ALARM CLOCK	<U+23F0>	\xE2\x8F\xB0	<8f>
ALIEN MONSTER	<U+009F><U+0091><U+00BE>	\xF0\x9F\x91\xBE
AMBULANCE	<U+009F><U+009A><U+0091>	\xF0\x9F\x9A\x91	<91>
AMERICAN FOOTBALL	<U+009F><U+008F><U+0088>	\xF0\x9F\x8F\x88	<88>

As the Github username of the person who created, today is a good day! Like in my last post I’ll use fuzzyjoin to identify emojis.

library("fuzzyjoin")

commits_decoded <- regex_inner_join(commits, dico,
                            by = c(commit = "Native"))

Yes! Doing this, I got 63 emojis!

table(commits_decoded$Description) %>%
  broom::tidy() %>%
  dplyr::arrange(desc(Freq)) %>%
  head(n = 10) %>%
  knitr::kable()

Var1	Freq
SEE-NO-EVIL MONKEY	10
ROOSTER	8
BUG	4
FACE WITH OK GESTURE	4
NAIL POLISH	3
PARTY POPPER	3
FACE SCREAMING IN FEAR	2
INFORMATION DESK PERSON	2
RUNNER	2
SUNFLOWER	2

Seeing the frequency of the “see no evil” emoji… Is Lucy ashamed of her commits? Why, Lucy? Then, rooster is a chicken, and Lucy is trying to make the “#RChickenLadies” hashtag happen (note to self: Lucy probably has puns in her commit messages). One important point though, on top of these cool most frequent emojis, is the diversity of emojis, 33 unique emojis for 63 emojis in total. It’s a real language! I can’t wait for Lucy to commit and push more in her public repositories to try and decode this language! And also to see the awesome work behind each commit, of course.

Radio edit: an improved scraping of and look at Radio Swiss classic program

Sun, 30 Apr 2017 00:00:00 +0000

Last week I published a post about scraping Radio Swiss Classic program. After that, Bob Rudis wrote an extremely useful post improving my code a lot and teaching me cool stuff. I don’t know why I forgot to add pauses between requests… Really bad behaviour! I will use his code today for re-scraping the data.

Why re-scrape the data? I mentioned broken links in my post. In fact, each time I hit a broken page, Radio Swiss Classic webmaster received an email. That person received a lot of emails because of me. They repaired the bug explaining these broken pages and contacted me because someone had turned me in (I feel super famous or spied on now), very kindly mentioning they had fixed all pages, and not holding any grudge against me. So let’s scrape everything again!

Scraping the program

So this is mostly Bob Rudis’ code with my clumsy comments alongside his comments. His are always in-line. I really recommend you to read his post!

library("rvest")
library("purrr")
library("stringi")
library("lubridate")
library("tidyverse")

# using purrr::safely is cool because if the page 
# is broken you get NULL as an output
s_read_html <- purrr::safely(read_html)

# helper for brevity
xtract_nodes <- function(node, css) {
  html_nodes(node, css) %>% html_text(trim = TRUE)
}

get_one_day_program <- function(date=Sys.Date(),
                                base_url="http://www.radioswissclassic.ch/en/music-programme/search/%s",
                                pb=NULL) {
  # progress bar magic!
  if (!is.null(pb)) pb$tick()$print()

  # that's the part where you're nice towards the website
  Sys.sleep(sample(seq(0,1,0.25), 1)) # ideally, make this sample(5,1)

  date <- ymd(date) # handles case where input is character ISO date

  pg <- s_read_html(sprintf(base_url, format(date, "%Y%m%d")))

  if (!is.null(pg$result)) {
# go read Bob Rudis' post, in particular to read about
# the extra selector for "playlist"
    dplyr::data_frame(

      date = date,
      time = xtract_nodes(pg$result, 'div[class="playlist"] *
                                            span[class="time hidden-xs"]') %>% hm(),
      datetime = update(date, hour = hour(time), minute = minute(time)),
      artist = xtract_nodes(pg$result, 'div[class="playlist"] * span[class="titletag"]'),
      title = xtract_nodes(pg$result, 'div[class="playlist"] * span[class="artist"]')

    )

  } else {
    closeAllConnections()
    NULL
  }

}

search_dates <- seq(from = ymd("2008-09-01"), to = ymd("2017-04-22"), by = "1 day")

# how you can make the progress bar work,
# estimate the time necessary for 5 requests
pb <- dplyr::progress_estimated(length(search_dates))
programs_df <- map_df(search_dates, get_one_day_program, pb=pb)
programs_df <- programs_df %>%
   dplyr::select(- time) %>%
   dplyr::mutate(datetime = force_tz(datetime, tz = "Europe/Zurich"))
programs_df

save(programs_df, file = "data/radioswissclassic_programs_radioedit.RData")

load("data/radioswissclassic_programs_radioedit.RData")
search_dates <- seq(from = lubridate::ymd("2008-09-01"), to = lubridate::ymd("2017-04-22"), by = "1 day")

This time I got 100% of programs!

Now I can improve my analysis of the data: My colleague Carles (who miiight have been receiving the link to each of my posts because I feel he needs to know about all of them… poor Carles) left a comment on my last post with two nice suggestions.

Applying quantile regression to the duration of pieces

What Carles said: “Regarding differences throughout the day, I agree that there are no major differences on the mean, but there are in the upper quantiles. I’d suggest quantile regression or quantile smoothing in case you want to try ;)”. Let’s not comment on the absence of nose of this smiley, my colleague is a nice guy so I’ll try to not freak out and to concentrate on the statistics.

library("magrittr")
programs_df <- dplyr::arrange(programs_df, datetime)
programs_df <- dplyr::mutate(programs_df,
                          duration = difftime(lead(datetime, 1),
                                       datetime,
                                       units = "min"))

programs_df <- dplyr::mutate(programs_df,
                          duration = ifelse(duration > 60,
                                            NA, duration))
programs_df <- dplyr::mutate(programs_df,
                          hour = as.factor(lubridate::hour(datetime)))

programs_df <- dplyr::mutate(programs_df,
                             night = (lubridate::hour(datetime) <= 4 | lubridate::hour(datetime) >= 20))


quantiles <- function(x){
  probs <- seq(from = 0.5, to = 0.95, by = 0.05)
  quantiles_names <- paste0("q", probs*100)
  quantiles <- purrr::map_dbl(probs, function(p, v = x){
    quantile(v, probs = p, na.rm = TRUE)
  })
  df <- tibble::as.tibble(t(quantiles))
  names(df) <- quantiles_names
  return(df)
}

programs_df %>%
  dplyr::group_by(night) %>%
  dplyr::summarize(duration_q = list(quantiles(duration))) %>%
  tidyr::unnest(duration_q) %>%
  knitr::kable()

night	q50	q55	q60	q65	q70	q75	q80	q85	q90	q95
FALSE	7	8	8	9	9	10	10	11	13	16
TRUE	8	8	9	9	10	11	13	16	20	26

Looking at this table above, it’s true that the upper quantiles of piece duration seem to be longet at night.

library("ggplot2")
library("hrbrthemes")
ggplot(programs_df) +
  geom_histogram(aes(duration)) +
  facet_grid(night ~ .) +
  theme_ipsum() +
  xlab("Duration (minutes)")

And I think these histograms are actually clearer than the boxplots of last post.

So now let’s try quantile regression, which I admit I had never used. The vignette of the quantreg package is really nice, it feels a bit old school with a real introduction and a section called “What is a vignette?”. The vignette starts with this sentence “Beran’s (2003) provocative definition of statistics as ‘the study of algorithms for data analysis’ elevates computational considerations to the forefront of the field.”. Roger Koenker also writes " Although there is now some functionality for quantile regression in most of the major commercial statistical packages, I have a natural predilection for the R environment and the software that I have developed for R.” which made me smile. Last nice point about the vignette I noticed, it encourages a contributor mindset by saying “When such modifications appear to be of general applicability, it is desirable to communicate them to the package author, so they could be shared with the larger community.” Ok a real last point about that package, the list of authors includes Brian D Ripley with this note from Roger Koenker “Initial (2001) R port from S (to my everlasting shame – how could I have been so slow to adopt R!”.

In the following I will just look at the effect of night on the duration for various quantiles, nothing fancy.

library("quantreg")

model <- rq(duration ~ night, 
            tau = seq(from = 0.5, to = 0.95, by = 0.05), 
           data = programs_df,
           method = "pfn")
summaries <- summary(model, se = "ker")

transform_summary <- function(summ){
  coeffs <- tibble::as.tibble(summ$coefficients)
  coeffs <- dplyr::mutate_(coeffs, tau = ~summ$tau)
  coeffs[2,]
}

purrr::map(summaries, transform_summary) %>%
  dplyr::bind_rows() %>%
  knitr::kable()

Value	Std. Error	t value	Pr(>\|t\|)	tau
1	0.0082075	121.83907	0	0.50
0	0.0082689	0.00000	1	0.55
1	0.0094450	105.87571	0	0.60
0	0.0093682	0.00000	1	0.65
1	0.0106465	93.92756	0	0.70
1	0.0122657	81.52811	0	0.75
3	0.0216791	138.38211	0	0.80
5	0.0298266	167.63566	0	0.85
7	0.0428316	163.43057	0	0.90
10	0.0605325	165.20043	0	0.95

This seems to support the idea that upper quantiles of duration are bigger at night. This shift of the distribution of duration at night compared to its distribution during daytime means that there are more long pieces at night… which is what the website claims, so we’re good. Just for info, we are looking at 171668 pieces played at night and 355487 pieces played during the day, that is to say a lot.

Using keywords to identify the speed of pieces

Carles said: “For romantic music the type of composition “Scherzo”/“Waltz”/“Nocturne” may tell you something about the mood of the composition”. Okay, time to analyse some text data then! Instead of looking at the mood of the music as suggested by Carles, I’ll look at the tempo of the pieces instead (from Wikipedia page, see “Italian tempo markings”).

To identify the tempo of each piece indicating one, I shall use fuzzyjoin, a very handy package for joining tables based on say, close geographical coordinates or regular expressions.

tempos <- tolower(c("Larghissimo", "Grave", "Largo",
    "Lento", "Larghetto", "Adagio",
    "Adagietto", "Andante", "Andantino",
    "Marcia moderato", "Andante moderato",
    "Moderato", "Allegretto", "Allegro moderato",
    "Allegro", "Vivace", "Vivacissimo",
    "Allegrissimo", "Presto", "Prestissimo"))
tempos <- tibble::tibble(tempo = tempos,
                         speed = 1:length(tempos))

programs <- dplyr::mutate(programs_df,
                          title = tolower(title))

library("fuzzyjoin")
programs <- regex_inner_join(programs, tempos,
                            by = c(title = "tempo"))

Now I have to take care of pieces that got several tempos, for instance because their title is “grave and allegro blabla” or “something moderato” thus yielding “something” and “something moderato” as tempos. For the first case I’ll randomly sample one of the tempos, for the second case I’ll keep the most precise tempo.

set.seed(1)
nrow(programs)

## [1] 120052

programs <- split(programs, programs$datetime)

clean_tempos <- function(df){
  if(nrow(df) > 1){
    if(any(stringr::str_detect(df$tempo, " moderato"))){
      df <- dplyr::filter_(df, lazyeval::interp(~ stringr::str_detect(df$tempo, " moderato")))
    }else{
      df <- dplyr::sample_n(df, size = 1)
    }
    
  }
  return(df)
  
}

programs <- purrr::map(programs, clean_tempos)
programs <- dplyr::bind_rows(programs)
nrow(programs)

## [1] 102439

Now that we have 102439 pieces with a known speed out of 527168, let’s look at speed distribution depending on the hour of the day.

programs %>%
  dplyr::group_by(night) %>%
  dplyr::summarize(mean_speed = mean(speed),
            median_speed = median(speed)) %>%
  knitr::kable()

night	mean_speed	median_speed
FALSE	10.453073	9
TRUE	9.888101	8

programs <- dplyr::mutate(programs, speed = as.factor(speed))

library("viridis")

ggplot(programs) +
  geom_bar(aes(hour, fill = speed),
           position = "fill") +
  scale_fill_viridis(discrete = TRUE) +
  theme_ipsum()

ggplot(programs) +
  geom_bar(aes(night, fill = speed),
           position = "fill") +
  scale_fill_viridis(discrete = TRUE) +
  theme_ipsum()

Honestly, I don’t really see a change of speed at night, or only a really small one, so my method was probably not a very good way of estimating the liveliness of pieces.

Conclusion

I’d like to say thanks again to Bob Rudis, to Radio Swiss Classic webmaster and to my colleague Carles. I was also surprised and happy to receive an email from a university music theory teacher, who told me he could use such data in class to look at popular music pieces. I really like how this blogging thing makes me discover new fields!

A classical analysis (Radio Swiss classic program)

Sun, 23 Apr 2017 00:00:00 +0000

I am not a classical music expert at all, but I happen to have friends who are, and am even married to someone who plays the cello (and the ukulele!). I appreciate listening to such music from time to time, in particular Baroque music. A friend made me discover Radio Swiss classic, an online radio playing classical music all day and all night long, with a quite nice variety, and very little speaking between pieces, with no ads (thank you, funders of the radio!). Besides, the voices telling me which piece has just been played are really soothing, so Radio Swiss classic is a good one in my opinion.

Today, instead of anxiously waiting for the results of the French presidential elections, I decided to download the program of the radio in the last years and have a quick look at it, since after all, the website says that the radio aims at relaxing people.

Scraping the program

My webscraping became a bit more elegant because I followed the advice of EP alias expersso, who by the way should really start blogging. I started downloading programs since September 2008 because that’s when I met the friend who told me about Radio Swiss Classic.

dates <- seq(from = lubridate::ymd("2008-09-01"),
             to = lubridate::ymd("2017-04-22"),
             by = "1 day")


base_url <- "http://www.radioswissclassic.ch/en/music-programme/search/"

get_one_day_program <- function(date, base_url){
  # in order to see progress
  message(date)
  
  # build URL
  date_as_string <- as.character(date)
  date_as_string <- stringr::str_replace_all(date_as_string, "-", "")
  url <- paste0(base_url, date_as_string)
  
  # read page
  page <- try(xml2::read_html(url),
              silent = TRUE)
  if(is(page, "try-error")){
    message("horribly wrong")

    closeAllConnections()
    return(NULL)
  }else{
    
    # find all times, artists and pieces
    times <- xml2::xml_text(xml2::xml_find_all(page, 
                                               xpath="//span[@class='time hidden-xs']//text()"))
    artists <- xml2::xml_text(xml2::xml_find_all(page, 
                                                 xpath="//span[@class='titletag']//text()"))
    pieces <- xml2::xml_text(xml2::xml_find_all(page, 
                                                xpath="//span[@class='artist']//text()"))
    # the last artist and piece are the current ones
    artists <- artists[1:(length(artists) - 1)]
    pieces <- pieces[1:(length(pieces) - 1)]
    
    # get a timedate from each time
    timedates <- paste(as.character(date), times)
    timedates <- lubridate::ymd_hm(timedates)
    timedates <- lubridate::force_tz(timedates, tz = "Europe/Zurich")
    
    # format the output
    program <- tibble::tibble(time = timedates,
                              artist = artists,
                              piece = pieces)
    
    return(program)
  }
  
}

programs <- purrr::map(dates, get_one_day_program, 
                       base_url = base_url)

programs <- dplyr::bind_rows(programs)

save(programs, file = "data/radioswissclassic_programs.RData")

There were some days without any program on the website, for which the website said something was horribly wrong with the server.

load("data/radioswissclassic_programs.RData")
wegot <- length(unique(lubridate::as_date(programs$time)))
wewanted <- length(seq(from = lubridate::ymd("2008-09-01"),
                       to = lubridate::ymd("2017-04-22"),
                       by = "1 day"))

However, I got a program for approximately 0.96 of the days.

Who are the most popular composers?

library("magrittr")
table(programs$artist) %>%
  broom::tidy() %>%
  dplyr::arrange(desc(Freq)) %>%
  head(n = 20) %>%
  knitr::kable()

Var1	Freq
Wolfgang Amadeus Mozart	37823
Ludwig van Beethoven	20936
Joseph Haydn	18140
Franz Schubert	15596
Antonio Vivaldi	14947
Johann Sebastian Bach	12003
Felix Mendelssohn-Bartholdy	11541
Antonin Dvorak	10265
Gioachino Rossini	9591
Frédéric Chopin	8470
Piotr Iljitsch Tchaikowsky	8092
Georg Friedrich Händel	7935
Tomaso Albinoni	6175
Gaetano Donizetti	5945
Giuseppe Verdi	5639
Johannes Brahms	5526
Johann Nepomuk Hummel	5439
Camille Saint-Saëns	5395
Luigi Boccherini	5130
Johann Christian Bach	4976

I’ll have to admit that I don’t even know all the composers in this table but they’re actually all famous according to my live-in classical music expert. Radio Swiss classic allows listeners to rate pieces, so the most popular ones are programmed more often, and well I guess the person making the programs also tend to program famous composers quite often.

library("ggplot2")
library("hrbrthemes")
table(programs$artist) %>%
  broom::tidy() %>%
  ggplot() +
  geom_histogram(aes(Freq)) +
  scale_x_log10() +
  theme_ipsum(base_size = 14)

Interestingly, but not that surprisingly I guess given the popularity of, say, Mozart, the distribution of occurrences by composers seems to be log-normally distributed.

How long are pieces?

On the website of Radio Swiss classic it is stated that pieces are longer in the evening than during the day, which I wanted to try and see. Because the program of the radio was not corrected for time changes (i.e. on 25 hour-days there are only 24 hours of music according to the online program), I shall only look at pieces whose duration is smaller than 60 minutes, which solves the issue of missing days at the same time.

programs <- dplyr::arrange(programs, time)
programs <- dplyr::mutate(programs,
                          duration = difftime(lead(time, 1),
                                       time,
                                       units = "min"))

programs <- dplyr::mutate(programs,
                          duration = ifelse(duration > 60,
                                            NA, duration))
programs <- dplyr::mutate(programs,
                          hour = as.factor(lubridate::hour(time)))

programs %>%
ggplot() +
  geom_boxplot(aes(hour, duration))+
  theme_ipsum(base_size = 14)

I don’t find the difference between day and night that striking, maybe I could try to define day and night to have a prettier figure (but I won’t do any test, I soon need to go watch TV).

programs %>%
  dplyr::mutate(night = (lubridate::hour(time) <= 4 | lubridate::hour(time) >= 20)) %>%
ggplot() +
  geom_boxplot(aes(night, duration))+
  theme_ipsum(base_size = 14)

Conclusion

The website also states that the pieces are more lively in the morning, but I have no data to which to match the titles of the pieces to investigate that claim. Well I have not even looked for such data. Another extension that I would find interesting would be to match each composer’s name to a style and then see how often each style is played. Now I’ll stop relaxing and go stuff my face with food in front of the election results!

Sow the seeds, know the seeds

Wed, 12 Apr 2017 00:00:00 +0000

When you do simulations, for instance in R, e.g. drawing samples from a distribution, it’s best to set a random seed via the function set.seed in order to have reproducible results. The function has no default value. I think I mostly use set.seed(1). Last week I received an R script from a colleague in which he used a weird number in set.seed (maybe a phone number? or maybe he let his fingers type randomly?), which made me curious about the usual seed values. As in my blog post about initial commit messages I used the Github API via the gh package to get a very rough answer (an answer seedling from the question seed?).

From Github API search endpoint you can get up to 1,000 results corresponding to a query which in the case of set.seed occurrences in R code isn’t the whole picture but hopefully a good sample. I wrote a function to treat the output of a query to the API where I take advantage of the stringr package. I just want the thing inside set.seed() from the text matches returned by the API.

get_seeds_from_matches <- function(item){
  url <- item$html_url
  matches <- item$text_matches
  matches <- unlist(lapply(matches, "[[", "fragment"))
  matches <- stringr::str_split(matches, "\\\n", simplify = TRUE)
  matches <- stringr::str_extract(matches, "set\\.seed\\(.*\\)")
  matches <- stringr::str_replace(matches, "set\\.seed\\(", "")
  seeds <- stringr::str_replace(matches, "\\).*", "")
  seeds <- seeds[!is.na(seeds)]
  tibble::tibble(seed = seeds,
             url = rep(url, length(seeds)))
}

After that I made the queries themselves, pausing every 30 pages because of the rate limiting, and adding a try around the call in order to stop as soon as I reached the 1,000 results. Not a very elegant solution but I wasn’t in a perfectionnist mood.

Note that the header "Accept" = 'application/vnd.github.v3.text-match+json' is very important, without it you wouldn’t get the text fragments in the results.

library("gh")
seeds <- NULL

ok <- TRUE
page <- 1
while(ok){
  matches <- try(gh("/search/code", q = "set.seed&language:r",
                    .token = Sys.getenv("GITHUB_PAT"),
                    .send_headers = c("Accept" = 'application/vnd.github.v3.text-match+json'),
                    page = page), silent = TRUE)
  ok <- !is(matches, "try-error")
 
  if(ok){
    seeds <- bind_rows(seeds, bind_rows(lapply(matches$items, 
                                               get_seeds_from_matches)))
  }

  page <- page + 1
  # wait 2 minutes every 30 pages
  if(page %% 30 == 1 & page > 1){
    Sys.sleep(120)
  }
  
}

save(seeds, file = "data/2017-04-12-seeds.RData.RData")

library("magrittr")
load("data/2017-04-12-seeds.RData")
head(seeds) %>%
  knitr::kable()

seed	url
1	https://github.com/berndbischl/ParamHelpers/blob/9d374430701d94639cc78db84f91a0c595927189/tests/testthat/helper_zzz.R
1	https://github.com/TypeFox/R-Examples/blob/d0917dbaf698cb8bc0789db0c3ab07453016eab9/ParamHelpers/tests/testthat/helper_zzz.R
1	https://github.com/cran/ParamHelpers/blob/92a49db23e69d32c8ae52585303df2875d740706/tests/testthat/helper_zzz.R
4.0	https://github.com/ACP-KR/AsanAdvR/blob/0517e88efce94266997d680e8b5a7c2a97c9277d/R-Object-Oriented-Programming-master/chapter4/chapter_4_ex11.R
4.0	https://github.com/ACP-KR/AsanAdvR/blob/0517e88efce94266997d680e8b5a7c2a97c9277d/R-Object-Oriented-Programming-master/chapter4/chapter_4_ex11.R
4.0	https://github.com/KellyBlack/R-Object-Oriented-Programming/blob/efbb0b81063baa30dd9d56d5d74b3f73b12b4926/chapter4/chapter_4_ex11.R

I got 984 entries, not 1,000 so maybe I lost some seeds in the process or the results weren’t perfect. The reason why I also added the URL of the script to the results was to be able to go and look at the code around surprising seeds.

Let’s have a look at the most frequent seeds in the sample.

table(seeds$seed) %>%
  broom::tidy() %>%
  dplyr::arrange(desc(Freq)) %>%
  head(n = 12) %>%
  knitr::kable()

Var1	Freq
seed	312
1	134
123	60
iseed	48
10	47
13121098	28
ss	24
20	21
1234	18
42	18
123456	15
0	14

So the most prevalent seed is a mystery because I’m not motivated enough to go scrape the code to find if the seed gets assigned a value before, like in that tweet I saw today. I was happy that 1 was so popular, maybe it means I belong?

I was surprised by two values. First, 13121098.

dplyr::filter(seeds, seed == "13121098") %>%
  head(n = 10) %>% 
  knitr::kable()

seed	url
13121098	https://github.com/DJRumble/Swirl-Course/blob/4e2771141e579904eb6dd32bce51ff6e0d840d44/Regression_Models/Residuals_Diagnostics_and_Variation/initLesson.R
13121098	https://github.com/swirldev/swirl_courses/blob/b3d432bfdf480c865af1c409ee0ee927c1fdbda0/Regression_Models/Residuals_Diagnostics_and_Variation/initLesson.R
13121098	https://github.com/1vbutkus/swirl/blob/310874100536e1e7c66861eced9ecb52939a3e0a/Regression_Models/Residuals_Diagnostics_and_Variation/initLesson.R
13121098	https://github.com/gotitsingh13/swirldev/blob/b7369b974ba76716fbcf6101bcbdc2db2f774d18/Regression_Models/Residuals_Diagnostics_and_Variation/initLesson.R
13121098	https://github.com/pauloramazza/swirl_courses/blob/4e2771141e579904eb6dd32bce51ff6e0d840d44/Regression_Models/Residuals_Diagnostics_and_Variation/initLesson.R
13121098	https://github.com/ildesoft/Swirl_Courses/blob/3e7f43cecbeb41e92e4f5972658f9b293e0e4b84/Regression_Models/Residuals_Diagnostics_and_Variation/initLesson.R
13121098	https://github.com/hrdg/Regression_Models/blob/22f47ecf2ae62f553aa132d3d948cc6b4e1599cc/Residuals_Diagnostics_and_Variation/initLesson.R
13121098	https://github.com/Rizwanabro/Swirl-Course/blob/3e7f43cecbeb41e92e4f5972658f9b293e0e4b84/Regression_Models/Residuals_Diagnostics_and_Variation/initLesson.R
13121098	https://github.com/mkgiitr/Data-Analytics/blob/1d659db1e9137b1fe595a6ef3356887de431b1be/win-library/3.1/swirl/Courses/Regression_Models/Residuals_Diagnostics_and_Variation/initLesson.R
13121098	https://github.com/Jutair/R-programming-Coursera/blob/4faeed6ca780ee7f14b224c293cae77293146f37/Swirl/Rsubversion/branches/Writing_swirl_Courses/Regression_Models/Residuals_Diagnostics_and_Variation/initLesson.R

I went and had a look and it seems most repositories correspond to code learnt in a Coursera course. I have taken a few courses from that specialization and loved it but I don’t remember learning about the special seed, too bad. Well I guess everyone used it to reproduce results but what does this number mean in the first place? Who typed it? A cat walking on the keyboard?

The other number that surprised me was 42 but then I remembered it is the “Answer to the Ultimate Question of Life, the Universe, and Everything” . I’d therefore say that this might be the coolest random seed. Now I can’t tell you whether it produces better results. Maybe it helps when your code actually tries to answer the Ultimate Question of Life, the Universe, and Everything?

Mapping waxwings annual migration without Twitter

Sat, 08 Apr 2017 00:00:00 +0000

Recently a reader left a comment on this blog mentioning his cool blog post in which he mapped the spread of a migratory bird using Twitter. His data source was the Waxwings UK account which reports sightings of Bohemian waxwings in the UK. I decided to try reproducing and extending his work using the rOpenSci spocc package that interfaces different sources of species occurrence data.

Getting the occurrence data

As mentioned above I got the data via spocc. I read the README of the Github repo and learnt that it’s called spocc like species occurrence data. So now I should never forget how many “c” there are in the word occurrence. Now please send help for my remembering it has two “r”.

The spocc package interacts with so many data sources that I felt a bit overwhelmed. I guess ecologists are not often blessed with so much data. Note that I don’t limit my query to the UK. Indeed, since seeing this map my interest for the Bohemian waxwing became global. I even decided to get data for the two other species of waxwings, although it only worked for the Bohemian and Cedar waxwings in the end.

I decided to use only data from GBIF. You’ll find more information about GBIF for instance, and other data sources for species occurrences, in this blog post from rOpenSci blog. Since I wanted to get data for different years and all 3 species of waxwings, which meant a lot of data so for getting it I sliced the Earth (I like how evil this sentence makes me sound, ah!). Note that I used overlapping slices because otherwise I didn’t get data at the limits between slices. I guess I could have made my slices slightly less overlapping though.

library("spocc")
library("dplyr")
library("purrr")
library("scrubr")
get_slice <- function(longitude, year){
  print(paste(year, longitude))
  gbifopts <- list(limit = 200000,
                   year = year)
  waxwings <- occ(query = "Bombycilla",
                  from = c('gbif'), 
                  gbifopts = gbifopts,
                  geometry = c(longitude - 0.5, - 90, 
                               longitude + 0.5, 90))
  waxwings <- occ2df(waxwings)
}
longitudes <- seq( -180, 179, by = 0.5)
years <- rep(2011:2016, length(lo ngitudes))
longitudes <- rep(longitudes,  6)
waxwings <- map2(longitudes, years, get_slice)
for (i in 1:length(waxwings)){
  waxwings[[i]]$latitude <- as.numeric(waxwings[[i]]$latitude)
  waxwings[[i]]$longitude <- as.numeric(waxwings[[i]]$longitude)
}

waxwings <- bind_rows(waxwings)
waxwings <- unique(waxwings)

waxwings <- filter(waxwings, !is.na(name))
readr::write_csv(waxwings, path = "uncleaned_waxwings.csv")

Cleaning the occurrence data

Now because my MSc of ecology is far behind me (but still close to my heart!) I have no idea how to assess the quality of species occurrence data. And even if I did I would have been delighted by the discovery of another rOpenSci package, scrubr, whose aim is to clean species occurrence records. Because I had so much data, I cleaned each day separately, otherwise it was just too long and hard on my poor computer. Don’t start thinking scrubr is slow, it isn’t and it’s getting faster by the minute thanks to work by its maintainer.

cleanup <- function(df){
  print(df$date[1])
  df <- df %>%
    coord_impossible() %>%
    coord_incomplete() %>%
    coord_unlikely() 

    if(nrow(df) > 1){
    
      df <- dedup(df)
    }
  df <- df %>%
    date_standardize("%Y-%m-%d") %>%
    date_missing()
    return(df)
} 
waxwings <- readr::read_csv("uncleaned_waxwings.csv")


waxwings <- split(waxwings, 
                  waxwings$date)

waxwings <- lapply(waxwings, cleanup)

waxwings <- bind_rows(waxwings)
waxwings <- unique(waxwings)
waxwings <- dplyr::filter(waxwings, name != "Bombycilla japonica",
                   longitude < 50)
readr::write_csv(waxwings, path = "waxwings.csv")

I removed the records with impossible, incomplete or unlikely coordinates (unlikely being e.g. a political centroid, impossible coordinates with a too high longitude), I also removed duplicate records and records without a data. This was so easy! I’d like the scrubr package to come clean my flat, as a team with the janitor package. Now, in real life, I’d probably be even stricter with data cleaning but for the scope of this blog post, using that package without any additionnal check was enough.

I also removed occurrences of the Japanese waxwing because they were too few of them, and occurrences with a longitude higher than 50 because it seemed weird to have non Japanese waxwings in Asia. After all this glorious data cleaning, I had 1015603 records.

Exploring the data

Let the fun begin!

Number of occurrences by species over time

library("ggplot2")
library("hrbrthemes")
library("dplyr")
library("lubridate")
waxwings <- mutate(waxwings,
                   week = update(date, wday = 1))
waxwings %>%
  group_by(week, name) %>%
  summarize(n = n()) %>%
  ggplot() +
  geom_point(aes(week, n)) +
  facet_grid(name ~ ., scales = "free_y") +
   theme(strip.text.y = element_text(angle = 0, size = 6)) +
  xlab("Time (weeks)") +
  ylab("No. of occurrences")

I have no idea why I have no data in 2016 for one of the two species. I decided to not investigate it further since I had enough birds for my primary goal which was mapping migration. The number of occurrences of Cedar waxwing increases over time before 2016, maybe because of more birders reporting sightings? For both species there is a clear seasonality, probably because these birds tend to breed in places where less people can observe them as we’ll see later in the post.

Day-of-the-week effects

waxwings <- mutate(waxwings, 
                   wday = lubridate::wday(date, label = TRUE))
waxwings %>%
  group_by(update(date, wday = 1), wday) %>%
  summarize(n = n()) %>%
ggplot() +
  geom_boxplot(aes(wday, n))+
  labs(x="Day of the week", y="No. of reported occurrences",
       title="No. of occurrences of waxwings by day-of-the week",
       caption="Data from GBIF accessed via rOpenSci package spocc") + 
  theme_ipsum()

So, more birds are reported on week-ends than on weekdays which I assume is due to a difference in human rather than bird behaviour (but who knows?). Note that for finer characterization of days where more people are birding, I could have used the bizdays package, but then I’d have limited my observations for one country only, because mixing holidays from different countries doesn’t sound like fun. Another thing that might influence sightings, beside people having the day off, might be the weather. For getting weather data in R I can recommend a few packages.

Mapping the migrations!

I first decided to plot the occurrences themselves on maps, by month, and to make a gif out of it. I then had to choose the colour and shape of the points used to represent the birds. Shape? Bird emojis of course! Regarding the colour, I was quite glad when someone I follow on Twitter posted about the birdcolourbot account created by David Lawrence Miller, because I looked and found this tweet with colours for the Bohemian waxwing, Bombycilla garrulus, the one present in both America and Europe. For the Cedar waxwing I had to create a palette myself, which I’d never thought of doing if I hadn’t seen the birdcolourbot account, I’m not a colour artist and often stick to viridis. I used a pretty pic from the internet. In both cases, I uploaded images on this website to get colour codes. A bit of copy-paste work but much easier than having to mix paint colours for real.

bohemian_palette <- c("#1A1A1A", "#878787",
                    "#B15929", "#E21A1C",
                    "#FEFF99")

cedar_palette <- c("#050608", "#5D5C7A",
                   "#AF5F2A", "#8E2F49",
                   "#F3DD31")

To me both species look quite similar so I didn’t expect the palettes to be reallt different. I decided the colours would all have the same probability, instead of weighing them according them to their presence in the usual patterns of each species.

set.seed(1)
library("dplyr")
waxwings <- mutate(waxwings,
                   colour = factor(sample(1:5, size = nrow(waxwings), replace = TRUE)))
waxwings <- mutate(waxwings,
                   month = lubridate::month(date),
                   month_name = lubridate::month(date, label = TRUE, abbr = FALSE))
bohemian <- filter(waxwings,
                   name == "Bombycilla garrulus")


cedar <- filter(waxwings,
                   name == "Bombycilla cedrorum")

After this preparation of the two data.frames I created a function for plotting a month of data for one species. A point I’d like to work on in the future for not being so ashamed of each of my maps are projections.

library("ggalt") 
library("magick")
library("ggmap")
library("ggthemes")
library("emojifont")
load.emojifont('OpenSansEmoji.ttf')
wax_map <- map_data("world")
wax_map <- wax_map[wax_map$region != "Antarctica",]

plot_month_species <- function(df, species,
                               name, palette){
  p <- ggplot()
  p <- p + geom_map(data = wax_map,
                    map = wax_map,
                    aes(x = long, y = lat, map_id = region),
                    color = "white", fill = "#7f7f7f",
                    size = 0.05, alpha = 1/4)
  p <- p + theme_map() 
  p <- p + geom_text(aes(longitude, latitude,
                col = colour),
                label = emoji("bird"),
                      data = df,
            family="OpenSansEmoji", 
                      size = 5)
  p <- p + scale_colour_manual(values = palette)
  p <- p + ylim(min(species$latitude), 
                max(species$latitude))
  p <- p + xlim(min(species$longitude), 
                max(species$longitude))
  p <- p + theme(legend.position = "none")
  outfil <- paste0("fig_waxwings/", name, "_", df$month[1], ".png")
  ggsave(outfil, p, width=5, height=5)
  image_read(outfil) %>%
    image_annotate(text = as.character(df$month_name[1]),
                   size = 100) %>%
    image_write(outfil)
  outfil
  
}

I chose to annotate the graph after creating it in order not to have to use the emoji font for the month name.

I first applied the function to the Bohemian waxwings.

bohemian_l <- split(bohemian, bohemian$month)


bohemian_l %>%
  purrr::map(plot_month_species, species = bohemian, name = "bohemian", palette = bohemian_palette) %>%
  purrr::map(image_read) %>%
  image_join() %>%
  image_animate(fps=1) %>%
  image_write("bohemian.gif")

First observation: I think the colours are pretty, but I also think I’ve just invented the concept of a confetti map and I’m not sure I’m proud of that. Then, regarding the underlying bird movements, Bohemian waxwings breed in the spring in regions farther up North than the regions where they winter and this can be seen on the map. There seem to be less sightings in the breeding season, probably because less people live up there, and apparently Santa’s helpers don’t even help.

Then it was the turn of the Cedar waxwing for which I had many more observations (871310 vs. 144293). Brace yourself for many confetti!

cedar_l <- split(cedar, cedar$month)


cedar_l %>%
  purrr::map(plot_month_species, species = cedar, name = "cedar", palette = cedar_palette) %>%
  purrr::map(image_read) %>%
  image_join() %>%
  image_animate(fps=1) %>%
  image_write("cedar.gif")

I’ve had a lot of fun making my confetti gifs. I could extend the analysis by using spatial smoothing to draw the zone where waxwings are mostly seen each month. I think the general pattern is clear enough now, but there are some outliers like one Bohemian waxwing in Spain. If there really is a Bohemian waxwing in Spain I’d appreciate its visiting me because I’ve never seen a waxwing!

Concluding

I was impressed by rOpenSci tools for getting and cleaning occurrence data, and quite thankful for the species occurrence data provided by GBIF. I liked plotting migration thanks to species occurrence, although I guess the best method for knowing migration patterns is tracking birds. But then I’ll let professionals do this and keep my bird migration confetti map making as a nice hobby.

Faces of #rstats Twitter

Sun, 19 Mar 2017 00:00:00 +0000

This week I was impressed by this tweet where Daniel Pett, Digital Humanities Lead at the British Museum, presented a collage of Twitter profile pics of all his colleagues. He made this piece of art using R (for collecting the usernames) and Python. I’m a bit of an R fanatic (or a Python dummy…) so I decided to write a code in R only to make a collage of profile pics of Twitter #rstats users.

Note: if you’re interested in Daniel Pett’s code, the data gathering is here and the montage code is here.

Getting the users

Once again I’ll use rtweet! In case you’ve been using twitteR, note that twitteR maintainer is currently passing the torch to rtweet so you’d better switch packages.

library("rtweet")
library("dplyr")
library("magick")
library("httr")
library("stringr")
library("purrr")

users <- search_users(q= '#rstats',
                      n = 1000,
                      parse = TRUE)
users <- unique(users)

I think that doing this I got only profiles whose description includes “rstats” or “#rstats”. Moreover, although I got 1,000 results, many of them were duplicates, so in the end I had about 450 #rstats Twitter users.

Gathering profile pics

This is the part where I felt slightly creepy downloading pictures of so many people.

I first had to create the link to each person’s profile page.

users <- mutate(users,
                where = paste0("https://twitter.com/",
                               screen_name))

Then I wrote a function for getting the profile pic for each user. I worked for everyone except 1) eggs and 2) a person whose profile pic had no extension, so I was quite satisfied with the result.

Update: Daniel Pett told me I could have used the column “profile_image_url” of the data.frame obtained from rtweet::search_users… I had completely missed that column! Thanks a lot!

get_piclink <- function(df){
  content <- httr::GET(df$where)
  content <- httr::content(content, as = "text")
  image <- str_extract(content,
                       "class=\"ProfileAvatar-image \" src=\"https://pbs.twimg.com/profile_images/.*\\..*\" alt")
  image <- str_replace(image, "class=\"ProfileAvatar-image \" src=\"", "")
  image <- str_replace(image, "..alt", "")
  return(image)
}

users <- by_row(users, get_piclink,
                .to = "piclink", .collate = "cols")

readr::write_csv(users, path = "users.csv")

After getting the links I could start saving the pictures, in a small 50px x 50px format. At that point I started using the magick package, a real game changer for image manipulation in R!

save_image <- function(df){
  image <- try(image_read(df$piclink), silent = TRUE)
  if(class(image)[1] != "try-error"){
    image %>%
      image_scale("50x50") %>%
      image_write(paste0("data/2017-03-19-facesofr_users_images/", df$screen_name,".jpg"))
  }
  
}

users <- filter(users, !is.na(piclink))
users <- split(users, 1:nrow(users))
walk(users, save_image)

I got 448 images.

Preparing the collage itself

I want to take a minute thinking about myself in elementary school. I am left-handed and I had scissors for right-handed people that I used with my left hand because no one in my family or school knew any better. I was therefore very bad at collages because of the cutting part. Had I known that I’d do collages with a computer as a grown-up, I would have been less sad to have to skip the break to keep on slowly cutting pieces of paper! (Note: I actually had a very happy childhood.)

Back to R talk now! For doing the collage I had to know how many columns and rows I wanted. For this I factorized the number of pics I had. Note that I also re-sampled the images, because I didn’t want all the R-Ladies chapters and their beautiful purple logo to end up near each other in the collage.

files <- dir("data/2017-03-19-facesofr_users_images/", full.names = TRUE)
set.seed(1)
files <- sample(files, length(files))
gmp::factorize(length(files))

## Big Integer ('bigz') object of length 7:
## [1] 2 2 2 2 2 2 7

Based on this I chose this format:

no_rows <- 28
no_cols <- 16

In magick there’s a function for appending images, image_append, from which you get either a row or a column depending on the stack argument. I first created all columns.

make_column <- function(i, files, no_rows){
  image_read(files[(i*no_rows+1):((i+1)*no_rows)]) %>%
  image_append(stack = TRUE) %>%
    image_write(paste0("cols/", i, ".jpg"))
}

walk(0:(no_cols-1), make_column, files = files,
    no_rows = no_rows)

And finally, I appended the columns!

image_read(dir("cols/", full.names = TRUE)) %>%
image_append(stack = FALSE) %>%
  image_write("2017-03-19-facesofr.jpg")

The end

Here is the result, where many, many R users are most certainly missing. Besides, can we talk about these accounts with the old R logo? They make me feel old, while the R-Ladies logo make me happy. Too bad Forwards wasn’t included, because their logo is really pretty.

Now you can try doing the same with your followers, the people you follow, or making a doormat with base plot advocates.

Maëlle's R blog on Maëlle Salmon's personal website

More crochet/programming thoughts

New Git analogy! The crochet lifeline

Communities of practice

Usefulness of seeing others’ work

Conclusion

Git worktrees are now cool

Git worktrees are for any work, not only reading

Creating a worktree for a brand-new branch is easy

Conclusion

Better Git diff with difftastic

Installing difftastic

difftastic on two files

difftastic with Git

Cool features of difftastic

Ignoring formatting changes

Matching delimiters in wrappers

Will I use difftastic?

Zut, Git ! Porting {saperlipopette} to the terminal with Claude

What my CLI already had

What I wanted for the CLI

The state of zut

Conclusion: future work

Git commits: please mark your stitches!

Don’t forget your stitch markers!

Don’t forget your Git commits!

Conclusion

Parallel strands between learning crochet and learning programming

Importance of instruction quality

Frustration is part of the process

Thanks for tech

The importance of practice

The importance of scrap yarn

Silly conventions

So many pitfalls

Conclusion

How to open files, folders, websites in R

Open any file in the editor: utils::file.edit() or styling with {cli}

Open the script or test file of a script-test pair: usethis::use_r() and usethis::use_test()

Open a project: positron, project launcher, {usethis}

Open an URL in the browser: browseURL() or styling with {cli}

Open only in interactive sessions: interactive(), rlang::is_interactive()

Conclusion

Cover and modify, some tips for R package development

“Cover and modify” with “characterization tests”

How to characterize the output

How to check coverage

Conclusion

Create and use a custom roxygen2 tag

Context: need for a custom tag in the igraph R package

Minimal example

Create our minimal R package

Create a package holding the custom tag

Create a roclet for sharing the custom tag

Build the package holding the custom tag

Register the roclet in our minimal package

Try it

Do I need an external package?

Sources

Conclusion

Does my test really validate a bug fix? Check it with git cherry-pick

Scenario: you fix a bug in a branch

Problem: you add a test after the fix, not before the fix

Solution: a throwaway branch and git cherry-pick

Conclusion

Get your codebase lint-free forever with lintr

Step 1: Add a lintr configuration file

Step 2: Run lintr, fix or skip or configure until done

Fix the problem

Skip a linter for a line

Configure lintr

Repeat until a clean output

Step 3: Set up lintr on continuous integration

Conclusion

Extracting names of functions defined in a script with treesitter

My use case: I won’t copy-paste these function names

Get the whole tree from the script

Find the node of the parent function of interest

Find the functions defined within the parent

Conclusion

Open any file in the editor: `utils::file.edit()` or styling with `{cli}`

Open the script or test file of a script-test pair: `usethis::use_r()` and `usethis::use_test()`

Open a project: `positron`, project launcher, {usethis}

Open an URL in the browser: `browseURL()` or styling with {cli}

Open only in interactive sessions: `interactive()`, `rlang::is_interactive()`

Extract and format links from `index.qmd`

“The repeated amend"™️: `git commit --amend`

“Start from scratch”: `git reset --soft` + `git add` (`--patch`)

“Mix and match your commits”: `git rebase -i`

One can negate patterns in `.gitignore`

`git log` options