The exametrika package provides comprehensive Test Data
Engineering tools for analyzing educational test data. Based on the
methods described in Shojima (2022), this package enables researchers
and practitioners to:
The package implements both traditional psychometric approaches and advanced statistical methods, making it suitable for various assessment and research purposes.
The package implements various psychometric models and techniques:
The package implements three complementary approaches to modeling local dependencies in test data:
Exametrika was originally developed and published as a Mathematica and Excel Add-in. For additional information about Exametrika, visit:
The development version of Exametrika can be installed from GitHub:
```{r install, eval=FALSE} # Install devtools if not already installed if (!require(“devtools”)) install.packages(“devtools”)
devtools::install_github(“kosugitti/exametrika”)
### Dependencies
The package requires:
+ R (>= 4.1.0)
+ igraph (for network analysis)
+ Other dependencies are automatically installed
## Data Format and Usage
### Basic Usage
```{r setup-library, message=FALSE, warning=FALSE}
library(exametrika)Exametrika accepts both binary and polytomous response data:
The package accepts data in several formats with the following features:
Note: Some analysis methods may have specific data type requirements. Please refer to each function’s documentation for detailed requirements.
The dataFormat function preprocesses input data for
analysis:
Example:
{r example-data-format} # Format raw data for analysis data <- dataFormat(J15S500) # Using sample dataset str(data) # View structure of formatted data
The package includes various sample datasets from Shojima (2022) for testing and learning:
Available datasets:
{r results-test-statistics, message=FALSE, warning=FALSE} TestStatistics(J15S500)
{r results-item-statistics, message=FALSE, warning=FALSE} ItemStatistics(J15S500)
{r results-ctt, message=FALSE, warning=FALSE} CTT(J15S500)
The IRT function estimates the number of parameters using a logistic
model, which can be specified using the model option. It
supports 2PL, 3PL, and 4PL models.
{r model-irt, message=FALSE, warning=FALSE} result.IRT <- IRT(J15S500, model = 3) result.IRT
The estimated population of subjects is included in the returned
object.
{r results-irt-ability, message=FALSE, warning=FALSE} head(result.IRT$ability)
The plots offer options for Item Characteristic Curves (ICC), Item
Information Curves (IIC), and Test Information Curves (TIC), which can
be specified through options. Items can be specified using the
items argument, and if not specified, plots will be drawn
for all items. The number of rows and columns for dividing the plotting
area can be specified using nr and nc,
respectively.
{r plot-irt-curves, fig.width=7, fig.height=5, message=FALSE, warning=FALSE} plot(result.IRT, type = "ICC", items = 1:6, nc = 2, nr = 3) plot(result.IRT, type = "IIC", items = 1:6, nc = 2, nr = 3) plot(result.IRT, type = "TIC")
Latent Class Analysis requires specifying the dataset and the number of classes.
{r model-lca, message=FALSE, warning=FALSE} LCA(J15S500, ncls = 5)
The returned object contains the Class Membership Matrix, which indicates which latent class each subject belongs to. The Estimate includes the one with the highest membership probability.
{r results-lca-membership, message=FALSE, warning=FALSE} result.LCA <- LCA(J15S500, ncls = 5) head(result.LCA$Students)
The plots offer options for IRP, CMP, TRP, and LCD. For more details on each, please refer to Shojima (2022).
{r plot-lca, message=FALSE, warning=FALSE} plot(result.LCA, type = "IRP", items = 1:6, nc = 2, nr = 3) plot(result.LCA, type = "CMP", students = 1:9, nc = 3, nr = 3) plot(result.LCA, type = "TRP") plot(result.LCA, type = "LCD")
Latent Rank Analysis requires specifying the dataset and the number of classes.
{r model-lra, message=FALSE, warning=FALSE} LRA(J15S500, nrank = 6)
The estimated subject rank membership probabilities and plots are almost the same as those in LCA (Latent Class Analysis). Since a ranking is assumed for the latent classes, rank-up odds and rank-down odds are calculated.
{r results-lra-membership, message=FALSE, warning=FALSE} result.LRA <- LRA(J15S500, nrank = 6) head(result.LRA$Students)
{r plot-lra, message=FALSE, warning=FALSE} plot(result.LRA, type = "IRP", items = 1:6, nc = 2, nr = 3) plot(result.LRA, type = "RMP", students = 1:9, nc = 3, nr = 3) plot(result.LRA, type = "TRP") plot(result.LRA, type = "LRD")
Biclustering and Ranklustering algorithms are almost the same,
differing only in whether they include a filtering matrix or not. The
difference is specified using the method option in the
Biclustering() function. For more details, please refer to
the help documentation.
Biclustering(J35S515, nfld = 5, ncls = 6, method = "B")result.Ranklustering <- Biclustering(J35S515, nfld = 5, ncls = 6, method = "R")
plot(result.Ranklustering, type = "Array")
plot(result.Ranklustering, type = "FRP", nc = 2, nr = 3)
plot(result.Ranklustering, type = "RMP", students = 1:9, nc = 3, nr = 3)
plot(result.Ranklustering, type = "LRD")To find the optimal number of classes and the optimal number of fields, the Infinite Relational Model is available.
result.IRM <- IRM(J35S515, gamma_c = 1, gamma_f = 1, verbose = TRUE)
plot(result.IRM, type = "Array")
plot(result.IRM, type = "FRP", nc = 3)
plot(result.IRM, type = "TRP")Additionally, supplementary notes on the derivation of the Infinite Relational Model with Chinese restaurant process is here.
The Bayesian network model is a model that represents the conditional probabilities between items in a network format based on the pass rates of the items. By providing a Directed Acyclic Graph (DAG) between items externally, it calculates the conditional probabilities based on the specified graph. The igraph package is used for the analysis and representation of the network.
There are three ways to specify the graph. You can either pass a matrix-type DAG to the argument adj_matrix, pass a DAG described in a CSV file to the argument adj_file, or pass a graph-type object g used in the igraph package to the argument g.
The methods to create the matrix-type adj_matrix and the graph object g are as follows:
{r setup-igraph, message=FALSE, warning=FALSE} library(igraph) DAG <- matrix( c( "Item01", "Item02", "Item02", "Item03", "Item02", "Item04", "Item03", "Item05", "Item04", "Item05" ), ncol = 2, byrow = T ) ## graph object g <- igraph::graph_from_data_frame(DAG) g ## Adjacency matrix adj_mat <- as.matrix(igraph::get.adjacency(g)) print(adj_mat)
A CSV file with the same information as the graph above in the following format. The first line contains column names (headers) and will not be read as data.
{r print-dag, echo=FALSE, message=FALSE, warning=FALSE} cat("From,To\n") for (i in 1:nrow(DAG)) { cat(sprintf("%s,%s\n", DAG[i, 1], DAG[i, 2])) }
While only one specification is sufficient, if multiple specifications are provided, they will be prioritized in the order of file, matrix, and graph object.
An example of executing BNM by providing a graph structure (DAG) is as follows:
{r model-bnm, message=FALSE, warning=FALSE} result.BNM <- BNM(J5S10, adj_matrix = adj_mat) result.BNM
The function searches for a DAG suitable for the data using a genetic algorithm. A best DAG is not necessarily identified. Instead of exploring all combinations of nodes and edges, only the space topologically sorted by the pass rate, namely the upper triangular matrix of the adjacency matrix, is explored. For interpretability, the number of parent nodes should be limited. A null model is not proposed. Utilize the content of the items and the experience of the questioner to aid in interpreting the results. For more details, please refer to Section 8.5 of the text(Shojima,2022).
Please note that the GA may take a considerable amount of time, depending on the number of items and the size of the population.
{r model-ga-bnm, message=FALSE, warning=FALSE} StrLearningGA_BNM(J5S10, population = 20, Rs = 0.5, Rm = 0.002, maxParents = 2, maxGeneration = 100, crossover = 2, elitism = 2 )
The method of Population-Based incremental learning proposed by Fukuda (2014) can also be used for learning. This method has several variations for estimating the optimal adjacency matrix at the end, which can be specified as options. See help or text Section 8.5.2.
{r model-pbil-bnm, message=FALSE, warning=FALSE} StrLearningPBIL_BNM(J5S10, population = 20, Rs = 0.5, Rm = 0.005, maxParents = 2, alpha = 0.05, estimate = 4 )
LD-LRA is an analysis that combines LRA and BNM, and it is used to analyze the network structure among items in the latent rank. In this function, structural learning is not performed, so you need to provide item graphs for each rank as separate files.
For each class, it is necessary to specify a graph, and there are three ways to do so. You can either pass a matrix-type DAG for each class or a list of graph-type objects used in the igraph package to the arguments adj_list or g_list, respectively, or you can provide a DAG described in a CSV file. The way to specify it in a CSV file is as follows.
```{r setup-dag-data, message=FALSE, warning=FALSE} DAG_dat <- matrix(c( “From”, “To”, “Rank”, “Item01”, “Item02”, 1, “Item04”, “Item05”, 1, “Item01”, “Item02”, 2, “Item02”, “Item03”, 2, “Item04”, “Item05”, 2, “Item08”, “Item09”, 2, “Item08”, “Item10”, 2, “Item09”, “Item10”, 2, “Item08”, “Item11”, 2, “Item01”, “Item02”, 3, “Item02”, “Item03”, 3, “Item04”, “Item05”, 3, “Item08”, “Item09”, 3, “Item08”, “Item10”, 3, “Item09”, “Item10”, 3, “Item08”, “Item11”, 3, “Item02”, “Item03”, 4, “Item04”, “Item06”, 4, “Item04”, “Item07”, 4, “Item05”, “Item06”, 4, “Item05”, “Item07”, 4, “Item08”, “Item10”, 4, “Item08”, “Item11”, 4, “Item09”, “Item11”, 4, “Item02”, “Item03”, 5, “Item04”, “Item06”, 5, “Item04”, “Item07”, 5, “Item05”, “Item06”, 5, “Item05”, “Item07”, 5, “Item09”, “Item11”, 5, “Item10”, “Item11”, 5, “Item10”, “Item12”, 5 ), ncol = 3, byrow = TRUE)
edgeFile <- tempfile(fileext = “.csv”) write.csv(DAG_dat, edgeFile, row.names = FALSE, quote = TRUE)
Here, it is shown an example of specifying with matrix-type and graph objects using the aforementioned CSV file. While only one specification is sufficient, if multiple specifications are provided, they will be prioritized in the order of file, matrix, and graph object.
```{r setup-graph-conversion, message=FALSE, warning=FALSE}
g_csv <- read.csv(edgeFile)
colnames(g_csv) <- c("From", "To", "Rank")
adj_list <- list()
g_list <- list()
for (i in 1:5) {
adj_R <- g_csv[g_csv$Rank == i, 1:2]
g_tmp <- igraph::graph_from_data_frame(adj_R)
adj_tmp <- igraph::get.adjacency(g_tmp)
g_list[[i]] <- g_tmp
adj_list[[i]] <- adj_tmp
}
## Example of graph list
g_list{r results-adj-list, message=FALSE, warning=FALSE} ### Example of adjacency list adj_list
The example of running the LDLRA function using this CSV file would look like this.
{r model-ldlra, message=FALSE, warning=FALSE} result.LDLRA <- LDLRA(J12S5000, ncls = 5, adj_file = edgeFile ) result.LDLRA
Of course, it also supports various types of plots.
{r plot-ldlra, message=FALSE, warning=FALSE} plot(result.LDLRA, type = "IRP", nc = 4, nr = 3) plot(result.LDLRA, type = "TRP") plot(result.LDLRA, type = "LRD")
# Clean up temporary file
unlink(edgeFile)You can learn item-interaction graphs for each rank using the PBIL algorithm. In addition to various options, the learning process requires a very long computation time. It’s also important to note that the result is merely one of the feasible solutions, and it’s not necessarily the optimal solution.
{r model-pbil-ldlra, message=FALSE, warning=FALSE, eval=T} result.LDLRA.PBIL <- StrLearningPBIL_LDLRA(J35S515, seed = 123, ncls = 5, method = "R", elitism = 1, successiveLimit = 15 ) result.LDLRA.PBIL
Local Dependence Biclustering combines biclustering and Bayesian network models. The model requires three main components:
Here’s an example implementation:
```{r setup-ldb} # Create field configuration vector (assign items to fields) conf <- c(1, 6, 6, 8, 9, 9, 4, 7, 7, 7, 5, 8, 9, 10, 10, 9, 9, 10, 10, 10, 2, 2, 3, 3, 5, 5, 6, 9, 9, 10, 1, 1, 7, 9, 10)
edges_data <- data.frame( “From Field (Parent) >>>” = c( 6, 4, 5, 1, 1, 4, # Class/Rank 2 3, 4, 6, 2, 4, 4, # Class/Rank 3 3, 6, 4, 1, # Class/Rank 4 7, 9, 6, 7 # Class/Rank 5 ), “>>> To Field (Child)” = c( 8, 7, 8, 7, 2, 5, # Class/Rank 2 5, 8, 8, 4, 6, 7, # Class/Rank 3 5, 8, 5, 8, # Class/Rank 4 10, 10, 8, 9 # Class/Rank 5 ), “At Class/Rank (Locus)” = c( 2, 2, 2, 2, 2, 2, # Class/Rank 2 3, 3, 3, 3, 3, 3, # Class/Rank 3 4, 4, 4, 4, # Class/Rank 4 5, 5, 5, 5 # Class/Rank 5 ) )
edgeFile <- tempfile(fileext = “.csv”) write.csv(edges_data, file = edgeFile, row.names = FALSE)
```{r setup-ldb-conf, include=FALSE, message=FALSE, warning=FALSE}
# Fit Local Dependence Biclustering model
result.LDB <- LDB(
U = J35S515,
ncls = 5, # Number of latent classes
conf = conf, # Field configuration vector
adj_file = edgeFile # Network structure file
)
# Display model results
print(result.LDB)Additionally, as mentioned in the text (Shojima, 2022), it is often the case that seeking the network structure exploratively does not yield appropriate results, so it has not been implemented.
result.LDB <- LDB(U = J35S515, ncls = 5, conf = conf, adj_file = edgeFile)
result.LDB# Clean up temporary file
unlink(edgeFile)Of course, it also supports various types of plots.
{r plot-ldb, message=FALSE, warning=FALSE} # Show bicluster structure plot(result.LDB, type = "Array") # Test Response Profile plot(result.LDB, type = "TRP") # Latent Rank Distribution plot(result.LDB, type = "LRD") # Rank Membership Profiles for first 9 students plot(result.LDB, type = "RMP", students = 1:9, nc = 3, nr = 3) # Field Reference Profiles plot(result.LDB, type = "FRP", nc = 3, nr = 2)
In this model, you can draw a Field PIRP Profile that visualizes the correct answer count for each rank and each field.
{r plot-ldb-fieldpirp, fig.width=7, fig.height=5, message=FALSE, warning=FALSE} plot(result.LDB, type = "FieldPIRP")
Bicluster Network Model: BINET is a model that combines the Bayesian network model and Biclustering. BINET is very similar to LDB and LDR.
The most significant difference is that in LDB, the nodes represent the fields, whereas in BINET, they represent the class. BINET explores the local dependency structure among latent classes at each latent field, where each field is a locus.
To execute this analysis, in addition to the dataset, the same field correspondence file used during exploratory Biclustering is required, as well as an adjacency matrix between classes.
```{r setup-binet} # Create field configuration vector for item assignment conf <- c(1, 5, 5, 5, 9, 9, 6, 6, 6, 6, 2, 7, 7, 11, 11, 7, 7, 12, 12, 12, 2, 2, 3, 3, 4, 4, 4, 8, 8, 12, 1, 1, 6, 10, 10)
edges_data <- data.frame( “From Class (Parent) >>>” = c( 1, 2, 3, 4, 5, 7, # Dependencies in various fields 2, 4, 6, 8, 10, 6, 6, 11, 8, 9, 12 ), “>>> To Class (Child)” = c( 2, 4, 5, 5, 6, 11, # Target classes 3, 7, 9, 12, 12, 10, 8, 12, 12, 11, 13 ), “At Field (Locus)” = c( 1, 2, 2, 3, 4, 4, # Field locations 5, 5, 5, 5, 5, 7, 8, 8, 9, 9, 12 ) )
edgeFile <- tempfile(fileext = “.csv”) write.csv(edges_data, file = edgeFile, row.names = FALSE)
The model requires three components:
1. Field assignments for items (conf vector)
2. Network structure between classes for each field
3. Number of classes and fields
```{r model-binet, message=FALSE, warning=FALSE}
# Fit Bicluster Network Model
result.BINET <- BINET(
U = J35S515,
ncls = 13, # Maximum class number from edges (13)
nfld = 12, # Maximum field number from conf (12)
conf = conf, # Field configuration vector
adj_file = edgeFile # Network structure file
)
# Display model results
print(result.BINET)# Clean up temporary file
unlink(edgeFile)Of course, it also supports various types of plots.
{r plot-binet, message=FALSE, warning=FALSE} # Show bicluster structure plot(result.BINET, type = "Array") # Test Response Profile plot(result.BINET, type = "TRP") # Latent Rank Distribution plot(result.BINET, type = "LRD") # Rank Membership Profiles for first 9 students plot(result.BINET, type = "RMP", students = 1:9, nc = 3, nr = 3) # Field Reference Profiles plot(result.BINET, type = "FRP", nc = 3, nr = 2)
LDPSR plot shows all Passing Student Rates for all locally dependent classes compared with their respective parents.
{r plot-binet-ldpsr, message=FALSE, warning=FALSE} # Locally Dependent Passing Student Rates plot(result.BINET, type = "LDPSR", nc = 3, nr = 2)
| Model/Type | IIC | ICC | TIC | IRP | FRP | TRP | LCD/LRD | CMP/RMP | Array | FieldPIRP | LDPSR |
|---|---|---|---|---|---|---|---|---|---|---|---|
| IRT | ◯ | ◯ | ◯ | ||||||||
| LCA | ◯ | ◯ | ◯ | ◯ | ◯ | ||||||
| LRA | ◯ | ◯ | ◯ | ◯ | ◯ | ||||||
| Biclustering | ◯ | ◯ | ◯ | ◯ | ◯ | ◯ | |||||
| IRM | ◯ | ◯ | ◯ | ||||||||
| LDLRA | ◯ | ◯ | ◯ | ||||||||
| LDB | ◯ | ◯ | ◯ | ◯ | ◯ | ◯ | |||||
| BINET | ◯ | ◯ | ◯ | ◯ | ◯ | ◯ |
We welcome community involvement and feedback to improve
exametrika. Here’s how you can participate and get
support:
If you encounter bugs or have suggestions for improvements:
sessionInfo())Join our GitHub Discussions:
We appreciate contributions from the community:
Please check our existing Issues and Discussions before posting to avoid duplicates.
Shojima, Kojiro (2022) Test Data Engineering: Latent Rank Analysis, Biclustering, and Bayesian Network (Behaviormetrics: Quantitative Approaches to Human Behavior, 13),Springer.
Follow our GitHub repository and join the Discussions to stay updated on development progress and provide feedback on desired features.