TENTH PERCENTILE

The magic of paste0 & sqldf

TENTH PERCENTILE — Fri, 27 Jan 2023 23:50:30 +0000

When I was first transitioning from SAS to R, I really wanted to figure out what the R equivalent to SAS macros would be. While there aren’t explicit “macros” in R, values and functions are very similar and there are many ways to finesse values into getting the job done. One of my favorite ways to do this is utilizing the paste0 function to paste values into a string and then pass that string into another function. This example uses sqldf to paste the value into a SQL query, but this can easily be done with any function that uses a character string input.

This is a game changer if you have multiple SQL queries that use the same start/end dates in one long script, and say part of your workflow is to to change the dates every month to get an output with updated information. Instead of manually sifting through the script and changing out all of the dates, you can change them once at the beginning and have a flawless run through.

Steps

First, we’ll load the 2 packages needed for the tutorial: sqldf and openxlsx. Sqldf allows us to use SQL syntax on R datasets, and openxlsx allows us to import an XLSX file from a URL so we can pull in some sample data. This example uses a dataset on US greenhouse gas emissions by state from 1990 – 2018 from the World Resources Institute. Let’s say we want to pull all of the emissions data from the year 1995 to 2002 for the state of Alabama.

After loading the packages, we’ll set up our values so we can pass them into the sqldf statement:

#load packages 
library(sqldf) 
library(openxlsx)

#setup start/end dates & the state we want to look at
start <- '1995'
end <- '2002'
state <- 'Alabama'

Next, we can import the data directly from the World Resources Institute website using the read.xlsx function from the openxlsx package. I chose to separate out the URL from the read.xlsx function, but you can keep them altogether if you prefer.

#import example data
url <- "http://datasets.wri.org/dataset/005e156f-cf77-42e2-a151-8f1f53e4f009/resource/2b403bc7-0874-482b-a1a4-2ead1d5452a9/download/climatewatch-usemissions"
emissions <- read.xlsx(url, startRow=4)

And finally, we can combine the paste0 and sqldf functions to query our emissions dataset with the values we set at the beginning of the script.

emissions_query <- sqldf(paste0("
SELECT * FROM emissions 
WHERE year BETWEEN ",start," AND ",end,"
AND state = '",state,"'
"))

It is important to note the syntax used in the query above. Paste0 basically smushes everything you list between the parentheses (separated with commas) together into one string, so if you forget a quotation mark or comma there will be errors. For reference, the output of the paste0/SQL statement looks like this:

"SELECT * FROM emissions 
WHERE year BETWEEN 1995 AND 2002 
AND state = 'Alabama'"

So that is what is passed into sqldf() to give us the emissions_query output we want.

Full Example

The full code can be found below, try it for yourself and report back with your own fun use cases for this powerful duo!

#the magic of paste0 & sqldf 

#load packages 
library(sqldf) 
library(openxlsx) 

#setup start/end dates & the state we want to 
#look at 
start <- '1995' 
end <- '2002' 
state <- 'Alabama' 

#import example data from the World Resources 
#Institute on U.S. States Greenhouse Gas Emissions 
url <- "http://datasets.wri.org/dataset/005e156f-cf77-42e2-a151-8f1f53e4f009/resource/2b403bc7-0874-482b-a1a4-2ead1d5452a9/download/climatewatch-usemissions" 
emissions <- read.xlsx(url, startRow=4) 

#use paste0 & sqldf to utilize values that were 
#set at the beginning of the script 
emissions_query <- sqldf(paste0(" 
SELECT * FROM emissions 
WHERE year BETWEEN ",start," AND ",end," 
AND state = '",state,"' "))

Check out the related posts below for more tips!

The post The magic of paste0 & sqldf appeared first on TENTH PERCENTILE.

Using SQL in R

TENTH PERCENTILE — Fri, 27 Jan 2023 22:18:22 +0000

Thanks to the sqldf package, using SQL in R to analyze & transform data is seamless. First, install & load the package:

#install
install.packages("sqldf")
#load
library(sqldf)

This package allows you to use the SQL programming language on your current R datasets. You can create new datasets, transform existing ones, incorporate functions…the possibilities are endless. Here is the documentation guide for a complete rundown of the package & list of supported SQL versions. There are so many awesome ways to incorporate SQL statements into your R code. Simply wrap your SELECT statement in double quotation marks & sqldf will do the rest.

Example

Below are two basic examples to give the gist, but there are more detailed articles on our SQL page with more elaborate strategies to completely transform your R code.

#create example data:
ex_data <- tibble(id = c(1, 2, 3, 4),
                  license_num = c(65, 76, 09, 29),
                  month = c('1', '8', '3', '11'))
#example 1 - Isolate rows where id > 2
ex1 <- sqldf("SELECT * FROM ex_data WHERE id > 2")
#example 2 - How many ID's are in the dataset?
ex2 <- sqldf("SELECT count(id) AS id_count FROM ex_data")

Check out the related posts below for more coding tips!

The post Using SQL in R appeared first on TENTH PERCENTILE.

Remove decimal from column in R

TENTH PERCENTILE — Fri, 27 Jan 2023 19:30:35 +0000

To remove decimals from a single column or the entire data frame, we can use str_remove from the tidyverse.

Basic syntax of str_remove:

str_remove(string, pattern)
#example:
str_remove(var, '\\.')

Since we want to remove the decimal in every row in the entire column, we can put the column name (var) in the string position and the expression with the period in the pattern position. In regular expressions (regex), the period by itself “.” indicates “any character,” so we’ll need to add the double backslash in front to specify that we want to remove the decimal. For more information on regular expressions, check out this article on the stringr package straight from the source — the Tidyverse creators.

Example

Here are 3 examples using some sample data to show how to use str_remove on 1 column, 2+ columns, and the entire data frame.

#remove decimal from a column or entire data frame 

#load packages
library(tidyverse)

#create example data
ex_data <- tibble(id = c('1.w', '2.j', '3.e', '4.o'),
                  license_num = c('65.j8', '76.s3', '09.i0', '29.l2'))

#using str_remove on 1 column
ex1 <- ex_data %>% mutate(license_num = str_remove(license_num, '\\.'))

#using str_remove on 2+ columns
ex2 <- ex_data %>% mutate(across(1:2, ~ str_remove(., '\\.')))

#using str_remove on entire data frame
ex3 <- ex_data %>% mutate(across(everything(), ~ str_remove(., '\\.')))

Check out the related posts below for more tips!

The post Remove decimal from column in R appeared first on TENTH PERCENTILE.

Convert variable class by index in R

TENTH PERCENTILE — Fri, 07 Oct 2022 19:23:39 +0000

To convert to a different variable class by index in R quickly and efficiently, we can use the mutate(across( functions from dplyr (the mutate_at function has been superseded by this):

mutate(across(4:5, as.numeric))

The example above would convert columns 4 and 5 to numeric. You can also use the “c” function if your columns are not in order. That would look like:

mutate(across(c(4, 5, 8), as.character))

This would convert columns 4, 5, and 8 to character. This is really helpful if you have complicated variable names or simply don’t want to mess with typing them all out. The tidyverse blog has an in depth description of all of the variations of the mutate function and a few more useful tips and examples on how to use them.

Example

#convert variables by index
#load packages
suppressPackageStartupMessages(library(pacman))
pacman::p_load(tidyverse, sqldf)
#create example data
ex_data <- tibble(
  id = c(1, 2, 3, 4),
  license_num = c(65, 76, 09, 29),
  month = c('1', '8', '3', '11'))
#convert columns 1 - 2 from numeric to character
conv <- ex_data %>% mutate(across(1:2, as.character))
#convert all back to numeric
conv_back <- conv %>% mutate(across(c(1, 3), as.numeric))

The post Convert variable class by index in R appeared first on TENTH PERCENTILE.

Use double quotes within double quotes in R

TENTH PERCENTILE — Wed, 05 Oct 2022 19:01:39 +0000

To use double quotes within double quotes in R, use the backslash in front of each quote surrounding your text. It will look like:

\"Type\"

While you can typically get the same result by using single quotation marks instead of double, there may be situations where single doesn’t cut it. I ran into this issue recently using the package sqldf to utilize SQL code in R. The specific code I was using broke if I used single quotation marks, so I used this workaround instead.

An example of what this looks like in a SQL SELECT statement:

my_data <- sqldf("SELECT * FROM table WHERE month = \"January\" AND id = 1")

Example

Here is a full example using test data with the statement above. The output my_data includes the double quotes within double quotes, and my_data2 shows the same result with single quotes.

#Double quotes within double quotes

#load packages
suppressPackageStartupMessages(library(pacman))
pacman::p_load(tidyverse, sqldf)

#create example data
ex_data <- tibble(
  id = c(1, 2, 3, 4),
  month = c("January", "February", "January", "March"))

my_data <- sqldf("SELECT * FROM ex_data WHERE month = \"January\" AND id = 1")
my_data2 <- sqldf("SELECT * FROM ex_data WHERE month = 'January' AND id = 1")

The post Use double quotes within double quotes in R appeared first on TENTH PERCENTILE.

Getting Started with R – Download R & RStudio

TENTH PERCENTILE — Mon, 12 Sep 2022 20:10:25 +0000

Getting started with a new programming platform can be daunting! Follow the steps below to download R & RStudio, and you’ll be coding like a pro in no time.

Side Note: R is an open-source software so everything is free! The R software is free, the RStudio IDE is free, and you’ll find lots of great free tutorials online. There are some paid courses that may be worthwhile, but my advice is to get as far as you can with the free resources since they are so widely available.

Download R

So first thing’s first, you can download R directly from R’s website. It can be a little complicated to go through some of the options, so here are the direct links to download on Windows and MacOS. If you go this route, you can skip to the next part.

If you have another operating system or would prefer more detailed options, you can select “download R” from the main page and choose your “CRAN mirror.” This will be the link of the location closest to you in the list. For instance, if you’re closest to Texas and click on Revolution Analytics, it brings you to a page where you can download the software for Windows, macOS, or Linux. Click on the option you need and follow the download instructions.

Download & Setup an IDE (RStudio)

After you have downloaded R and gone through the installation steps, you’ll want to download an IDE (integrated development environment) as well. This is just a console you can use with R to make your life a little easier. It has a user-friendly setup, adds customization options, allows you to integrate other programming languages…the list goes on. There are many options out there, but I prefer the tried and true RStudio. Here are the options to download RStudio, and I suggest installing the desktop version. I haven’t used the server version so I can’t vouch for it, but the RStudio folks run an excellent service so I’m sure it’s great if that better suits your needs.

Above is an example of your RStudio interface. Some people will start practicing directly in the console, but I prefer opening a script that I can save if I need to (shown below). You never know when you’ll output a brilliant, reusable code chunk you’ll want to hold onto forever! And losing important work is the fuel of nightmares. I had a junior high computer teacher who would yell at us to “save and save often!” so much that it’s permanently etched into my brain.

Walk-through of RStudio

Editor: On the top left you have the editor. This is where you can type your code so that you can save it and use it again. It’s best practice to use a script vs. just the Console so you can reuse pieces of code and come back and make adjustments if needed. As a bonus, you’ll also have documentation of things that have worked AND (sometimes more importantly) things that haven’t.
Environment: Your environment is where your data frames and functions are stored. When you import data and make adjustments, you can save them here by using the “<-” operator. More on that here.
Console: The console shows the progress of the code as it runs. If there are errors, it will stop running and give a quick description of the error. You may be prompted to type a “yes”, “no”, “1”, “2”, etc. for a package install or update. You can also run a quick fragment here that you don’t care about saving like view(df) or install.package(“package”).
Files/Plots/Packages: This pane shows the files in your directory, the most recent plot that you’ve created, and all of your packages. There’s a button to install/update packages here as well. If you click on a package, the Help tab will open with an overview, tips, and a list of all of the functions in the package (which really comes in handy while learning!).

Now you’re all set up! If you’re stuck on what to do next, check out w3schools for some excellent, beginner-friendly tutorials. (That is not an ad, I genuinely LOVE their website.) And please comment below if you have any questions about the setup process or advice for newbies!

Happy coding!

The post Getting Started with R – Download R & RStudio appeared first on TENTH PERCENTILE.

How to work with the %YM%m (2022M01) date format in R

TENTH PERCENTILE — Mon, 22 Aug 2022 23:55:01 +0000

Working with data from different institutional or governmental sources, I sometimes have data in weird formats where I need to turn it into something a programming language can process. NHSN dates in particular make me want to scream. If you work with NHSN (the CDC’s National Healthcare Safety Network) and you export data by month, you’re probably familiar with the date format 2022M01. (Why, CDC, why?!) Now you may be wondering, “Why dedicate an entire post to this weird, obscure format?” When I was trying to find some answers searching the ole Google-machine, I found little to nothing to go off of. I tested quite a few strategies out before finding something that works. And hopefully this will save someone else a future headache.

Convert %YM%m to a standard date format

First, let’s break this format apart. The first 4 digits are the year, the M stands for month, and the last two digits are the month number (i.e. January = 01). As you can imagine, thinking of this as a date will break R’s brain, so it reads it as a character variable instead. Since there isn’t a “day” number, we will have to finesse it a little bit and then re-format it into a usable date. We can use the paste0 function to tack on an “01” at the end (or whatever day of the month you need) and set the format to “%YM%m%d”. If you don’t do this, R will read your month number as the day number and you’ll get wonky dates in your output.

new_data <- data_table %>% 
mutate(
date = as.Date(paste0(as.character(date_var), '01'), "%YM%m%d"), 
new_var = as.Date(date, "new_date_format"))

^{*Blue text indicates info you’ll want to swap out for your own}

Example:

#libraries
library(tidyverse)
#using May 2022 as an example
mydata <- tibble(summaryym = "2022M05")

new_dates <- mydata %>% mutate(date = as.Date(paste0(as.character(summaryym), '01'), "%YM%m%d"), 
       new_summaryym = as.Date(date, "%Y-%m-%d"))
#original summaryym = 2022M05
#new_summaryym = 2022-05-01

Convert a standard date format to %YM%m

To do the reverse, we’ll need to use the strptime function to strip apart the date and specify which parts make up the month & year. You may be thinking, “WHY why why would you want to murk up a perfectly good date into this god awful format?” In my day job, we have a pre-existing process that will break if the dates are not in this format. So in order for that process to work, it is necessary to convert it back.

data_table %>% mutate(date = strptime(as.character(date_var), "date_var_format", 
       new_var = format(date, "%YM%m"))

^{date_var = your original date variable}

Example:

#librarieslibrary(tidyverse)

#using May 2022 as an example
mydata <- tibble(summaryym = as.Date("2022-05-01"))
                      
new_dates <- mydata %>% mutate(date = strptime(as.character(summaryym), "%Y-%m-%d"), 
       new_summaryym = format(date, "%YM%m"))

#original summaryym = 2022-01-01
#new_summaryym = 2022M01

I’ve tried a few different ways to convert 2022M01 to and from a standard date format, but these two seem to work most consistently. A simple format() or as.Date() returns NA. And using strptime in the first conversion (from 2022M01 to standard) was also a no-go for me. I would love to hear strategies from others who have used this format before…please comment below if you have one to share!

Also let us know of any other weird date formats you’ve come across in the wild! Bonus points for random non-numeric characters.

The post How to work with the %YM%m (2022M01) date format in R appeared first on TENTH PERCENTILE.

Convert dates from POSIXct to Date in R

TENTH PERCENTILE — Mon, 22 Aug 2022 21:17:27 +0000

Working with dates in any programming language can be incredibly stressful. Luckily for us, there is an easy way to convert dates from a Date/Time class like POSIXct to a Date class in R using modify_if from the Tidyverse.

new_data <- orig_data %>% modify_if(is.POSIXct, as.Date)

^{*Blue text is information you’ll need to sub out for your own}

There two date/time classes in R, POSIXct and POSIXlt, have subtle differences that I won’t go into here. If you’d like to know more, this article from UC Berkeley gives a great overview. The main thing to note is that they store dates in the yyyy-mm-dd hh:mm:ss format, which can be a problem if you need your dates in a format like yyyy-mm-dd or mm/dd/yyyy. There’s a super quick way to update all of the Date/Time variables in a data frame using the purrr modify_if function:

Example

To provide a visual, we’ll walk through an example of converting dates from Date/Time to Date below.

First, we’ll load the packages we want to work with:

#load packages
suppressPackageStartupMessages(library(pacman))
pacman::p_load(tidyverse, lubridate)

Then, we can create some example data. We can create ‘birthday’ as class POSIXct and ‘party’ as class POSIXlt.

#example data
birthday <- as.POSIXct("1990-01-09 01:30:00")
party <- as.POSIXlt("2023-01-09 20:00:00")

dt_dates <- tibble(birthday = birthday,
                party = party)

Finally, we’ll use the modify_if function to convert one or both classes to Dates.

#convert only POSIXct
new_dates1 <- dt_dates %>% modify_if(is.POSIXct, as.Date)

#convert both POSIXct and POSIXlt
new_dates2 <- dt_dates %>% modify_if(is.POSIXct, as.Date) %>% modify_if(is.POSIXlt, as.Date)

When we select the dropdown arrow on our dataset, we can see that both variables have now been converted to a date class:

The post Convert dates from POSIXct to Date in R appeared first on TENTH PERCENTILE.

Blue Ombré Banner

TENTH PERCENTILE — Fri, 19 Aug 2022 22:50:18 +0000

^{Created in R using the tidyverse, aRt, and MetBrewer packages.}

Thanks to the incredible R packages aRt by nrennie and MetBrewer by BlakeRMills, this blue ombré banner is easy to recreate with a few simple lines of code. MetBrewer is my go-to for color palettes because it’s inspired by pieces of art from the Met…how cool! I used the palette Hokusai2, which is based on the painting Lake Suwa in Shinano Province (Shinshū Suwako) by Katsushika Hokusai.

^{Image source: BlakeRMills}

Blue Ombré Banner Code

#Tenth Percentile Blue Ombré Banner

#libraries
suppressPackageStartupMessages(library(pacman))
p_load(tidyverse, aRt, MetBrewer)

#generate blue ombré image
stripes(perc=1, n=1, col_palette = met.brewer("Hokusai2", n=7), alpha = 1, s=1234)

#export image
#ggsave("filepath/filename.png")
ggsave("figure%03d.png")

The pacman package is a game changer. It conveniently allows you to load all your packages at once so you can avoid multiple lines of code. Another tip, if you don’t specify a file path and only include the file name, ggsave will save your image to the file path that your directory is set to (example below).

This banner on the Console shows the path of your file directory in R.

Rainbow Gradient Banner

You can swap out the colors by replacing Hokusai2 in the code above with another MetBrewer palette. The pretty reds and rich blues in the Nizami palette make for a really beautiful rainbow gradient.

This image is created by replacing the Hokusai2 palette with the Nizami palette and increase n to 10.

#generate rainbow gradient image
stripes(perc=1, n=1, col_palette = met.brewer("Nizami", n=10), alpha = 1, s=1234)

More aRt

I also love the National Parks palettes from kevinsblake, which was based on the MetBrewer package. And aRt has a lot of different functions to play around with, with outputs ranging from sunsetty gradients to Rorschach fractals to retro patterns. You can spend hours trying out different palette and function combinations to create unique designs. The following image is generated using the fractals function from aRt and the Glacier palette from NatParksPalettes.

#libraries
suppressPackageStartupMessages(library(pacman))
p_load(tidyverse, aRt, MetBrewer, NatParksPalettes)

#generate glacier fractal image
fractals(N = 25, col_palette = natparks.pals("Glacier", n = 25),
shift = 0, left = -1, right = 1,
y_param = 3, resolution = 0.005, dist_max = 4)

These packages are so fun to play around with, and generating aRt is such a great way to practice coding skills and error troubleshooting. Please share your creations by tagging @tenthpercentile on Instagram, I would love to see them!

Happy coding!

The post Blue Ombré Banner appeared first on TENTH PERCENTILE.

Welcome to Tenth Percentile

TENTH PERCENTILE — Wed, 17 Aug 2022 20:46:52 +0000

The idea behind Tenth Percentile is that we all start from scratch when learning a new skill. In statistics, the tenth percentile is the lowest 10% of a group/population/cohort. It’s not typically where you would aim to be. But when it comes to learning something new, the tenth percentile is the best place to be because it’s where the highest potential for growth is.

No one is born with the inherent knowledge of how to code or analyze data. And more importantly, no one is predisposed to be “better at it” than anyone else. There may be many factors at play including our different strengths and weaknesses, but I would argue that the gatekeeping in the coding/programming/data analytics world is the primary factor holding most people back from being empowered to learn the skill.

It took me a few years of piddling here and there to learn R after I decided that was something I wanted to pursue. Even with all the resources available, I was hard-pressed to find certain beginner-level tutorials that weren’t difficult for me to jump into or full-blown books. I wasn’t motivated enough to spend hours reading a textbook over something I couldn’t even begin to understand. Then, when I had a job where it was not necessarily required but would be helpful for my day-to-day process, I picked it up in a matter of a few months.

I was thinking about why it took me so long to commit, and I realized that I needed my learning environment to be simpler and all in one place. I didn’t want to spend a million hours googling how to’s and errors until I absolutely had to. That’s why I wanted to create this website, to have an outlet to document my findings in one place and share what I’m learning along the way. If even just one other person finds it useful I will consider it a success.

That being said, please reach out if you have a question we could figure out together, a suggestion for a post, or simply want to geek out about a new data discovery!

Thanks for stopping by,

Chandler

The post Welcome to Tenth Percentile appeared first on TENTH PERCENTILE.