---
title: "Feature selection: crabs"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Feature selection: crabs}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---





Filters against a wrapper, on five measurements that are almost perfectly correlated and
two different targets.

The other vignettes are listed by `vignette (package = "fdm2id")`; they use the same
handful of functions on other data, and can be read in any order.


``` r
library (fdm2id)
```

# The data

The `crabs` dataset of the `MASS` package, on *Leptograpsus variegatus* crabs collected at
Fremantle, Australia. Two colour forms, orange and blue, and both sexes. Five measurements were
taken on each individual (in mm): the size of the frontal lobe (`FL`), the rear width (`RW`),
the length of the carapace (`CL`), its width (`CW`) and the depth of the body (`BD`).

There are two targets rather than one: the species (`sp`, `O` or `B`) and the sex (`sex`, `F`
or `M`), with 50 males and 50 females of each species.


``` r
data (crabs, package = "MASS")
summary (crabs)
#>  sp      sex         index            FL              RW              CL       
#>  B:100   F:100   Min.   : 1.0   Min.   : 7.20   Min.   : 6.50   Min.   :14.70  
#>  O:100   M:100   1st Qu.:13.0   1st Qu.:12.90   1st Qu.:11.00   1st Qu.:27.27  
#>                  Median :25.5   Median :15.55   Median :12.80   Median :32.10  
#>                  Mean   :25.5   Mean   :15.58   Mean   :12.74   Mean   :32.11  
#>                  3rd Qu.:38.0   3rd Qu.:18.05   3rd Qu.:14.30   3rd Qu.:37.23  
#>                  Max.   :50.0   Max.   :23.10   Max.   :20.20   Max.   :47.60  
#>        CW              BD       
#>  Min.   :17.10   Min.   : 6.10  
#>  1st Qu.:31.50   1st Qu.:11.40  
#>  Median :36.80   Median :13.90  
#>  Mean   :36.41   Mean   :14.03  
#>  3rd Qu.:42.00   3rd Qu.:16.60  
#>  Max.   :54.60   Max.   :21.60
```


``` r
plotdata (crabs [, 4:8], crabs [, 1], type = "pairs")
```

<div class="figure" style="text-align: center">
<img src="figure/v08-feature-selection-crabs-unnamed-chunk-4-1.png" alt="plot of chunk unnamed-chunk-4"  />
<p class="caption">plot of chunk unnamed-chunk-4</p>
</div>


``` r
plotdata (crabs [, 4:8], crabs [, 2], type = "pairs")
```

<div class="figure" style="text-align: center">
<img src="figure/v08-feature-selection-crabs-unnamed-chunk-5-1.png" alt="plot of chunk unnamed-chunk-5"  />
<p class="caption">plot of chunk unnamed-chunk-5</p>
</div>

The five measurements are almost perfectly correlated with each other -- a crab is simply
bigger or smaller -- so most of what any one of them says, the others say too. That is what
makes the *selection* the interesting question here rather than the classifier.


``` r
round (cor (crabs [, 4:8]), 3)
#>       FL    RW    CL    CW    BD
#> FL 1.000 0.907 0.979 0.965 0.988
#> RW 0.907 1.000 0.893 0.900 0.889
#> CL 0.979 0.893 1.000 0.995 0.983
#> CW 0.965 0.900 0.995 1.000 0.968
#> BD 0.988 0.889 0.983 0.968 1.000
```

# Question 1. Selecting for the species

Using a ranking algorithm and Fisher's index as the univariate criterion, which multivariate
criterion -- the F statistic, mRMR, or a wrapper -- gives the best predictions of the species
with a naive Bayes classifier?


``` r
# The two filters are deterministic; the wrapper is not -- it judges each subset by fitting a
# naive Bayes classifier under a bootstrap, so without 'seed' it can stop at a different
# subset from one run to the next. That is the criterion's own variance, not the data's.
s.fstat1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                           unieval = "fisher", multieval = "fstat", seed = 0)
s.mrmr1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                          unieval = "fisher", multieval = "mrmr", seed = 0)
s.wrap1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                          unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
                          seed = 0)
s.fstat1
#> Feature selection
#>   algorithm              : ranking
#>   univariate criterion   : fisher
#>   multivariate criterion : fstat
#>   features kept          : 1: FL
#>   score                  : 46.99
s.mrmr1
#> Feature selection
#>   algorithm              : ranking
#>   univariate criterion   : fisher
#>   multivariate criterion : mrmr
#>   features kept          : 1: FL
#>   score                  : 0.1974
s.wrap1
#> Feature selection
#>   algorithm              : ranking
#>   univariate criterion   : fisher
#>   multivariate criterion : wrapper
#>   features kept          : 1: FL
#>   score                  : 0.6779
```


``` r
performance (NB, crabs [, 4:8], crabs [, 1], nruns = 100, seed = 0)
#>  accuracy 
#> 0.6191188
performance (NB, crabs [, 4:8] [, s.fstat1$selection], crabs [, 1], nruns = 100, seed = 0)
#>  accuracy 
#> 0.6626326
performance (NB, crabs [, 4:8] [, s.mrmr1$selection], crabs [, 1], nruns = 100, seed = 0)
#>  accuracy 
#> 0.6626326
performance (NB, crabs [, 4:8] [, s.wrap1$selection], crabs [, 1], nruns = 100, seed = 0)
#>  accuracy 
#> 0.6626326
```

**Answer.** *All three criteria stop at the same subset and therefore give the same
performance -- which is better than using all five variables. Adding correlated measurements
to a naive Bayes classifier, whose whole assumption is that they are independent, costs
accuracy.*

# Question 2. Selecting for the sex

The same question, for the sex of the crabs.


``` r
s.fstat2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                           unieval = "fisher", multieval = "fstat", seed = 0)
s.mrmr2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                          unieval = "fisher", multieval = "mrmr", seed = 0)
s.wrap2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                          unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
                          seed = 0)
s.fstat2
#> Feature selection
#>   algorithm              : ranking
#>   univariate criterion   : fisher
#>   multivariate criterion : fstat
#>   features kept          : 1: RW
#>   score                  : 18.4
s.mrmr2
#> Feature selection
#>   algorithm              : ranking
#>   univariate criterion   : fisher
#>   multivariate criterion : mrmr
#>   features kept          : 1: RW
#>   score                  : 0.1264
s.wrap2
#> Feature selection
#>   algorithm              : ranking
#>   univariate criterion   : fisher
#>   multivariate criterion : wrapper
#>   features kept          : 3: RW, CL, BD
#>   score                  : 0.7358
```


``` r
performance (NB, crabs [, 4:8], crabs [, 2], nruns = 100, seed = 0)
#>  accuracy 
#> 0.6624966
performance (NB, crabs [, 4:8] [, s.fstat2$selection], crabs [, 2], nruns = 100, seed = 0)
#>  accuracy 
#> 0.6236062
performance (NB, crabs [, 4:8] [, s.mrmr2$selection], crabs [, 2], nruns = 100, seed = 0)
#>  accuracy 
#> 0.6236062
performance (NB, crabs [, 4:8] [, s.wrap2$selection], crabs [, 2], nruns = 100, seed = 0)
#>  accuracy 
#> 0.7129453
```

**Answer.** *Only the wrapper improves on the full set of variables. The two filter criteria
stop at a single variable and lose accuracy -- which is the trade-off of the family: a filter
judges a subset without ever fitting the model it is selecting for, and here that judgement is
wrong.*

# Question 3. Which variables, in the end?


``` r
colnames (crabs) [4:8] [s.fstat1$selection]
#> [1] "FL"
colnames (crabs) [4:8] [s.wrap2$selection]
#> [1] "RW" "CL" "BD"
```

**Answer.** *The frontal lobe alone is what distinguishes the two species; distinguishing the
sexes takes three measurements -- the rear width, the length of the carapace and the depth of
the body. Two different questions asked of the same five measurements, and two different
answers.*
