# Feature selection: crabs

Filters against a wrapper, on five measurements that are almost perfectly correlated and two different targets.

The other vignettes are listed by `vignette (package = "fdm2id")`; they use the same handful of functions on other data, and can be read in any order.

    library (fdm2id)

# The data

The `crabs` dataset of the `MASS` package, on Leptograpsus variegatus crabs collected at Fremantle, Australia. Two colour forms, orange and blue, and both sexes. Five measurements were taken on each individual (in mm): the size of the frontal lobe (`FL`), the rear width (`RW`), the length of the carapace (`CL`), its width (`CW`) and the depth of the body (`BD`).

There are two targets rather than one: the species (`sp`, `O` or `B`) and the sex (`sex`, `F` or `M`), with 50 males and 50 females of each species.

    data (crabs, package = "MASS")
    summary (crabs)
    #>  sp      sex         index            FL              RW              CL
    #>  B:100   F:100   Min.   : 1.0   Min.   : 7.20   Min.   : 6.50   Min.   :14.70
    #>  O:100   M:100   1st Qu.:13.0   1st Qu.:12.90   1st Qu.:11.00   1st Qu.:27.27
    #>                  Median :25.5   Median :15.55   Median :12.80   Median :32.10
    #>                  Mean   :25.5   Mean   :15.58   Mean   :12.74   Mean   :32.11
    #>                  3rd Qu.:38.0   3rd Qu.:18.05   3rd Qu.:14.30   3rd Qu.:37.23
    #>                  Max.   :50.0   Max.   :23.10   Max.   :20.20   Max.   :47.60
    #>        CW              BD
    #>  Min.   :17.10   Min.   : 6.10
    #>  1st Qu.:31.50   1st Qu.:11.40
    #>  Median :36.80   Median :13.90
    #>  Mean   :36.41   Mean   :14.03
    #>  3rd Qu.:42.00   3rd Qu.:16.60
    #>  Max.   :54.60   Max.   :21.60

    plotdata (crabs [, 4:8], crabs [, 1], type = "pairs")

plot of chunk unnamed-chunk-4

    plotdata (crabs [, 4:8], crabs [, 2], type = "pairs")

plot of chunk unnamed-chunk-5

The five measurements are almost perfectly correlated with each other – a crab is simply bigger or smaller – so most of what any one of them says, the others say too. That is what makes the selection the interesting question here rather than the classifier.

    round (cor (crabs [, 4:8]), 3)
    #>       FL    RW    CL    CW    BD
    #> FL 1.000 0.907 0.979 0.965 0.988
    #> RW 0.907 1.000 0.893 0.900 0.889
    #> CL 0.979 0.893 1.000 0.995 0.983
    #> CW 0.965 0.900 0.995 1.000 0.968
    #> BD 0.988 0.889 0.983 0.968 1.000

# Question 1. Selecting for the species

Using a ranking algorithm and Fisher’s index as the univariate criterion, which multivariate criterion – the F statistic, mRMR, or a wrapper – gives the best predictions of the species with a naive Bayes classifier?

    # The two filters are deterministic; the wrapper is not -- it judges each subset by fitting a
    # naive Bayes classifier under a bootstrap, so without 'seed' it can stop at a different
    # subset from one run to the next. That is the criterion's own variance, not the data's.
    s.fstat1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                               unieval = "fisher", multieval = "fstat", seed = 0)
    s.mrmr1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                              unieval = "fisher", multieval = "mrmr", seed = 0)
    s.wrap1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                              unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
                              seed = 0)
    s.fstat1
    #> Feature selection
    #>   algorithm              : ranking
    #>   univariate criterion   : fisher
    #>   multivariate criterion : fstat
    #>   features kept          : 1: FL
    #>   score                  : 46.99
    s.mrmr1
    #> Feature selection
    #>   algorithm              : ranking
    #>   univariate criterion   : fisher
    #>   multivariate criterion : mrmr
    #>   features kept          : 1: FL
    #>   score                  : 0.1974
    s.wrap1
    #> Feature selection
    #>   algorithm              : ranking
    #>   univariate criterion   : fisher
    #>   multivariate criterion : wrapper
    #>   features kept          : 1: FL
    #>   score                  : 0.6779

    performance (NB, crabs [, 4:8], crabs [, 1], nruns = 100, seed = 0)
    #>  accuracy
    #> 0.6191188
    performance (NB, crabs [, 4:8] [, s.fstat1$selection], crabs [, 1], nruns = 100, seed = 0)
    #>  accuracy
    #> 0.6626326
    performance (NB, crabs [, 4:8] [, s.mrmr1$selection], crabs [, 1], nruns = 100, seed = 0)
    #>  accuracy
    #> 0.6626326
    performance (NB, crabs [, 4:8] [, s.wrap1$selection], crabs [, 1], nruns = 100, seed = 0)
    #>  accuracy
    #> 0.6626326

Answer. All three criteria stop at the same subset and therefore give the same performance – which is better than using all five variables. Adding correlated measurements to a naive Bayes classifier, whose whole assumption is that they are independent, costs accuracy.

# Question 2. Selecting for the sex

The same question, for the sex of the crabs.

    s.fstat2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                               unieval = "fisher", multieval = "fstat", seed = 0)
    s.mrmr2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                              unieval = "fisher", multieval = "mrmr", seed = 0)
    s.wrap2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                              unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
                              seed = 0)
    s.fstat2
    #> Feature selection
    #>   algorithm              : ranking
    #>   univariate criterion   : fisher
    #>   multivariate criterion : fstat
    #>   features kept          : 1: RW
    #>   score                  : 18.4
    s.mrmr2
    #> Feature selection
    #>   algorithm              : ranking
    #>   univariate criterion   : fisher
    #>   multivariate criterion : mrmr
    #>   features kept          : 1: RW
    #>   score                  : 0.1264
    s.wrap2
    #> Feature selection
    #>   algorithm              : ranking
    #>   univariate criterion   : fisher
    #>   multivariate criterion : wrapper
    #>   features kept          : 3: RW, CL, BD
    #>   score                  : 0.7358

    performance (NB, crabs [, 4:8], crabs [, 2], nruns = 100, seed = 0)
    #>  accuracy
    #> 0.6624966
    performance (NB, crabs [, 4:8] [, s.fstat2$selection], crabs [, 2], nruns = 100, seed = 0)
    #>  accuracy
    #> 0.6236062
    performance (NB, crabs [, 4:8] [, s.mrmr2$selection], crabs [, 2], nruns = 100, seed = 0)
    #>  accuracy
    #> 0.6236062
    performance (NB, crabs [, 4:8] [, s.wrap2$selection], crabs [, 2], nruns = 100, seed = 0)
    #>  accuracy
    #> 0.7129453

Answer. Only the wrapper improves on the full set of variables. The two filter criteria stop at a single variable and lose accuracy – which is the trade-off of the family: a filter judges a subset without ever fitting the model it is selecting for, and here that judgement is wrong.

# Question 3. Which variables, in the end?

    colnames (crabs) [4:8] [s.fstat1$selection]
    #> [1] "FL"
    colnames (crabs) [4:8] [s.wrap2$selection]
    #> [1] "RW" "CL" "BD"

Answer. The frontal lobe alone is what distinguishes the two species; distinguishing the sexes takes three measurements – the rear width, the length of the carapace and the depth of the body. Two different questions asked of the same five measurements, and two different answers.