A quick summary of selection syntax in {datawizard}

This vignette can be referred to by citing the following:

Patil et al., (2022). datawizard: An R Package for Easy Data Preparation and Statistical Transformations. Journal of Open Source Software, 7(78), 4684, https://doi.org/10.21105/joss.04684

Selecting variables

Quoted names

This is the most simple way to select one or several variables. Just use a character vector containing variables names, like in base R.

data_select(iris, c("Sepal.Length", "Petal.Width"))
#>    Sepal.Length Petal.Width
#> 1           4.3         0.1
#> 2           5.0         0.2
#> 3           7.7         2.2
#> 4           4.4         0.2
#> 5           5.9         1.8
#> 6           6.5         2.0
#> 7           5.5         1.3
#> 8           5.5         1.2
#> 9           5.8         1.9
#> 10          6.1         1.4

Unquoted names

It is also possible to use unquoted names. This is useful if we use the tidyverse and want to be consistent about the way variable names are passed.

iris %>%
  group_by(Species) %>%
  standardise(Petal.Length) %>%
  ungroup()
#> # A tibble: 10 × 5
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width Species   
#>           <dbl>       <dbl>        <dbl>       <dbl> <fct>     
#>  1          4.3         3         -1.09          0.1 setosa    
#>  2          5           3.3        0.873         0.2 setosa    
#>  3          7.7         3.8        1.50          2.2 virginica 
#>  4          4.4         3.2        0.218         0.2 setosa    
#>  5          5.9         3         -0.542         1.8 virginica 
#>  6          6.5         3         -0.414         2   virginica 
#>  7          5.5         2.5       -1.09          1.3 versicolor
#>  8          5.5         2.6        0.218         1.2 versicolor
#>  9          5.8         2.7       -0.542         1.9 virginica 
#> 10          6.1         3          0.873         1.4 versicolor

Positions

In addition to variable names, select can also take indices for the variables to select in the dataframe.

data_select(iris, c(1, 2, 5))
#>    Sepal.Length Sepal.Width    Species
#> 1           4.3         3.0     setosa
#> 2           5.0         3.3     setosa
#> 3           7.7         3.8  virginica
#> 4           4.4         3.2     setosa
#> 5           5.9         3.0  virginica
#> 6           6.5         3.0  virginica
#> 7           5.5         2.5 versicolor
#> 8           5.5         2.6 versicolor
#> 9           5.8         2.7  virginica
#> 10          6.1         3.0 versicolor

Functions

We can also pass a function to the select argument. This function will be applied to all columns and should return TRUE or FALSE. For example, if we want to keep only numeric columns, we can use is.numeric.

data_select(iris, is.numeric)
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width
#> 1           4.3         3.0          1.1         0.1
#> 2           5.0         3.3          1.4         0.2
#> 3           7.7         3.8          6.7         2.2
#> 4           4.4         3.2          1.3         0.2
#> 5           5.9         3.0          5.1         1.8
#> 6           6.5         3.0          5.2         2.0
#> 7           5.5         2.5          4.0         1.3
#> 8           5.5         2.6          4.4         1.2
#> 9           5.8         2.7          5.1         1.9
#> 10          6.1         3.0          4.6         1.4

Note that we can provide any custom function to select, provided it returns TRUE or FALSE when applied to a column.

my_function <- function(i) {
  is.numeric(i) && mean(i, na.rm = TRUE) > 3.5
}

data_select(iris, my_function)
#>    Sepal.Length Petal.Length
#> 1           4.3          1.1
#> 2           5.0          1.4
#> 3           7.7          6.7
#> 4           4.4          1.3
#> 5           5.9          5.1
#> 6           6.5          5.2
#> 7           5.5          4.0
#> 8           5.5          4.4
#> 9           5.8          5.1
#> 10          6.1          4.6

Patterns

With larger datasets, it would be tedious to write the names of variables to select, and it would be fragile to rely on variable positions as they may change later. To this end, we can use four select helpers: starts_with(), ends_with(), contains(), and regex(). The first three can take several patterns, while regex() takes a single regular expression.

data_select(iris, starts_with("Sep", "Peta"))
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width
#> 1           4.3         3.0          1.1         0.1
#> 2           5.0         3.3          1.4         0.2
#> 3           7.7         3.8          6.7         2.2
#> 4           4.4         3.2          1.3         0.2
#> 5           5.9         3.0          5.1         1.8
#> 6           6.5         3.0          5.2         2.0
#> 7           5.5         2.5          4.0         1.3
#> 8           5.5         2.6          4.4         1.2
#> 9           5.8         2.7          5.1         1.9
#> 10          6.1         3.0          4.6         1.4

data_select(iris, ends_with("dth", "ies"))
#>    Sepal.Width Petal.Width    Species
#> 1          3.0         0.1     setosa
#> 2          3.3         0.2     setosa
#> 3          3.8         2.2  virginica
#> 4          3.2         0.2     setosa
#> 5          3.0         1.8  virginica
#> 6          3.0         2.0  virginica
#> 7          2.5         1.3 versicolor
#> 8          2.6         1.2 versicolor
#> 9          2.7         1.9  virginica
#> 10         3.0         1.4 versicolor

data_select(iris, contains("pal", "ec"))
#>    Sepal.Length Sepal.Width    Species
#> 1           4.3         3.0     setosa
#> 2           5.0         3.3     setosa
#> 3           7.7         3.8  virginica
#> 4           4.4         3.2     setosa
#> 5           5.9         3.0  virginica
#> 6           6.5         3.0  virginica
#> 7           5.5         2.5 versicolor
#> 8           5.5         2.6 versicolor
#> 9           5.8         2.7  virginica
#> 10          6.1         3.0 versicolor

data_select(iris, regex("^Sep|ies"))
#>    Sepal.Length Sepal.Width    Species
#> 1           4.3         3.0     setosa
#> 2           5.0         3.3     setosa
#> 3           7.7         3.8  virginica
#> 4           4.4         3.2     setosa
#> 5           5.9         3.0  virginica
#> 6           6.5         3.0  virginica
#> 7           5.5         2.5 versicolor
#> 8           5.5         2.6 versicolor
#> 9           5.8         2.7  virginica
#> 10          6.1         3.0 versicolor
Note: these functions are not exported by `datawizard` but are detected and applied internally. This means that they won't be detected by autocompletion when we write them.
Note #2: because these functions are not exported, they will not create conflicts with the ones that come from the `tidyverse` and that have the same name. So we can still use `dplyr` and its friends, it won't change anything for selection in `datawizard` functions!

Excluding variables

What if we want to keep all variables except for a few ones? There are two ways we can invert our selection.

The first way is to put a minus sign "-" in front of the select argument.

data_select(iris, -c("Sepal.Length", "Petal.Width"))
#>    Sepal.Width Petal.Length    Species
#> 1          3.0          1.1     setosa
#> 2          3.3          1.4     setosa
#> 3          3.8          6.7  virginica
#> 4          3.2          1.3     setosa
#> 5          3.0          5.1  virginica
#> 6          3.0          5.2  virginica
#> 7          2.5          4.0 versicolor
#> 8          2.6          4.4 versicolor
#> 9          2.7          5.1  virginica
#> 10         3.0          4.6 versicolor

data_select(iris, -starts_with("Sep", "Peta"))
#>       Species
#> 1      setosa
#> 2      setosa
#> 3   virginica
#> 4      setosa
#> 5   virginica
#> 6   virginica
#> 7  versicolor
#> 8  versicolor
#> 9   virginica
#> 10 versicolor

data_select(iris, -is.numeric)
#>       Species
#> 1      setosa
#> 2      setosa
#> 3   virginica
#> 4      setosa
#> 5   virginica
#> 6   virginica
#> 7  versicolor
#> 8  versicolor
#> 9   virginica
#> 10 versicolor

Note that if we use numeric indices, we can’t mix negative and positive values. This means that we have to use select = -(1:2) if we want to exclude the first two columns; select = -1:2 will not work:

data_select(iris, -(1:2))
#>    Petal.Length Petal.Width    Species
#> 1           1.1         0.1     setosa
#> 2           1.4         0.2     setosa
#> 3           6.7         2.2  virginica
#> 4           1.3         0.2     setosa
#> 5           5.1         1.8  virginica
#> 6           5.2         2.0  virginica
#> 7           4.0         1.3 versicolor
#> 8           4.4         1.2 versicolor
#> 9           5.1         1.9  virginica
#> 10          4.6         1.4 versicolor

Same thing for variable names:

data_select(iris, -(Petal.Length:Species))
#>    Sepal.Length Sepal.Width
#> 1           4.3         3.0
#> 2           5.0         3.3
#> 3           7.7         3.8
#> 4           4.4         3.2
#> 5           5.9         3.0
#> 6           6.5         3.0
#> 7           5.5         2.5
#> 8           5.5         2.6
#> 9           5.8         2.7
#> 10          6.1         3.0

The second way is to use the argument exclude. This argument has the same possibilities as select. Although this may not be required in most contexts, if we wanted to, we could use both select and exclude arguments at the same time.

data_select(iris, exclude = c("Sepal.Length", "Petal.Width"))
#>    Sepal.Width Petal.Length    Species
#> 1          3.0          1.1     setosa
#> 2          3.3          1.4     setosa
#> 3          3.8          6.7  virginica
#> 4          3.2          1.3     setosa
#> 5          3.0          5.1  virginica
#> 6          3.0          5.2  virginica
#> 7          2.5          4.0 versicolor
#> 8          2.6          4.4 versicolor
#> 9          2.7          5.1  virginica
#> 10         3.0          4.6 versicolor

data_select(iris, exclude = starts_with("Sep", "Peta"))
#>       Species
#> 1      setosa
#> 2      setosa
#> 3   virginica
#> 4      setosa
#> 5   virginica
#> 6   virginica
#> 7  versicolor
#> 8  versicolor
#> 9   virginica
#> 10 versicolor

Programming with selections

Since datawizard 0.6.0, it is possible to pass function arguments and loop indices in select and exclude arguments. This makes it easier to program with datawizard.

For example, if we want to let the user decide the selection they want to use:

my_function <- function(data, selection) {
  find_columns(data, select = selection)
}
my_function(iris, "Sepal.Length")
#> [1] "Sepal.Length"
my_function(iris, starts_with("Sep"))
#> [1] "Sepal.Length" "Sepal.Width"

my_function_2 <- function(data, pattern) {
  find_columns(data, select = starts_with(pattern))
}
my_function_2(iris, "Sep")
#> [1] "Sepal.Length" "Sepal.Width"

It is also possible to pass these values in loops, for example if we have a list of patterns and we want to relocate columns based on these patterns, one by one:

new_iris <- iris
for (i in c("Sep", "Pet")) {
  new_iris <- new_iris %>%
    data_relocate(select = starts_with(i), after = -1)
}
new_iris
#>       Species Sepal.Length Sepal.Width Petal.Length Petal.Width
#> 1      setosa          4.3         3.0          1.1         0.1
#> 2      setosa          5.0         3.3          1.4         0.2
#> 3   virginica          7.7         3.8          6.7         2.2
#> 4      setosa          4.4         3.2          1.3         0.2
#> 5   virginica          5.9         3.0          5.1         1.8
#> 6   virginica          6.5         3.0          5.2         2.0
#> 7  versicolor          5.5         2.5          4.0         1.3
#> 8  versicolor          5.5         2.6          4.4         1.2
#> 9   virginica          5.8         2.7          5.1         1.9
#> 10 versicolor          6.1         3.0          4.6         1.4

In the loop above, all columns starting with "Sep" are moved at the end of the data frame, and the same thing was made with all columns starting with "Pet".

Useful to know

Ignore the case

In every selection that uses variable names, we can ignore the case in the selection by applying ignore_case = TRUE.

data_select(iris, c("sepal.length", "petal.width"), ignore_case = TRUE)
#>    Sepal.Length Petal.Width
#> 1           4.3         0.1
#> 2           5.0         0.2
#> 3           7.7         2.2
#> 4           4.4         0.2
#> 5           5.9         1.8
#> 6           6.5         2.0
#> 7           5.5         1.3
#> 8           5.5         1.2
#> 9           5.8         1.9
#> 10          6.1         1.4

data_select(iris, ~ Sepal.length + petal.Width, ignore_case = TRUE)
#>    Sepal.Length Petal.Width
#> 1           4.3         0.1
#> 2           5.0         0.2
#> 3           7.7         2.2
#> 4           4.4         0.2
#> 5           5.9         1.8
#> 6           6.5         2.0
#> 7           5.5         1.3
#> 8           5.5         1.2
#> 9           5.8         1.9
#> 10          6.1         1.4

data_select(iris, starts_with("sep", "peta"), ignore_case = TRUE)
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width
#> 1           4.3         3.0          1.1         0.1
#> 2           5.0         3.3          1.4         0.2
#> 3           7.7         3.8          6.7         2.2
#> 4           4.4         3.2          1.3         0.2
#> 5           5.9         3.0          5.1         1.8
#> 6           6.5         3.0          5.2         2.0
#> 7           5.5         2.5          4.0         1.3
#> 8           5.5         2.6          4.4         1.2
#> 9           5.8         2.7          5.1         1.9
#> 10          6.1         3.0          4.6         1.4

Formulas

It is also possible to use formulas to select variables:

data_select(iris, ~ Sepal.Length + Petal.Width)
#>    Sepal.Length Petal.Width
#> 1           4.3         0.1
#> 2           5.0         0.2
#> 3           7.7         2.2
#> 4           4.4         0.2
#> 5           5.9         1.8
#> 6           6.5         2.0
#> 7           5.5         1.3
#> 8           5.5         1.2
#> 9           5.8         1.9
#> 10          6.1         1.4

This made it easier to use selection in custom functions before datawizard 0.6.0, and is kept available for backward compatibility.