Analyze rater agreement, disagreement, and severity-direction networks

Usage

rater_network_analysis(
  fit,
  diagnostics = NULL,
  rater_facet = NULL,
  context_facets = NULL,
  mode = c("agreement", "disagreement", "severity_direction"),
  weight_metric = NULL,
  min_pair_n = 1,
  min_weight = 0,
  score_diff_tolerance = 0,
  severity_continuity = 0.5,
  exact_warn = 0.5,
  corr_warn = 0.3,
  include_graph = FALSE
)

Arguments

fit: Output from fit_mfrm().
diagnostics: Optional output from diagnose_mfrm().
rater_facet: Name of the rater-like facet. If omitted, mfrmr uses the same heuristic as interrater_agreement_table().
context_facets: Facets defining shared scoring contexts. By default, the person facet and all non-rater facets are used.
mode: Network definition. "agreement" builds an undirected network whose edge weights represent observed agreement. "disagreement" builds an undirected network whose edge weights represent observed disagreement. "severity_direction" builds a directed network: an edge from rater A to rater B means A assigned higher scores than B in shared contexts and is therefore relatively more lenient under the usual higher-score-is-better rating convention.
weight_metric: Pair-level weight used for "agreement" or "disagreement" networks. Defaults to Exact for agreement and MAD for disagreement. Available pair columns include Exact, Adjacent, Corr, MAD, OneMinusExact, and AbsMeanDiff.
min_pair_n: Minimum number of shared contexts required for a rater pair to contribute an edge.
min_weight: Minimum edge weight retained in the graph. This is an analysis threshold: it changes graph topology and all graph-derived centrality summaries, not only the displayed edges.
score_diff_tolerance: Score-difference tolerance for directed severity networks. With the default 0, any higher score contributes to the outgoing leniency edge. This is an analysis tolerance: increasing it changes directional counts, strengths, and SeverityIndex; it is not a plot-only filter.
severity_continuity: Continuity constant added to incoming and outgoing strengths before computing the finite severity index -log((OutStrength + c) / (InStrength + c)). The default 0.5 is a package finite-value correction. Setting c = 0 gives the uncorrected published form but can produce non-finite values for zero strengths.
exact_warn, corr_warn: Passed to interrater_agreement_table() to keep pair flags consistent with the tabular agreement view.
include_graph: If TRUE, include the underlying igraph object in the returned bundle.

Value

A bundle of class mfrm_rater_network containing:

summary: One-row graph summary.
node_metrics: Rater-level degree, strength, graph-theoretic centrality, and severity-direction summaries.
edge_metrics: Retained rater-pair network edges.
pair_metrics: All estimated pairwise agreement and directional comparison metrics, including EligiblePair, before min_weight filtering.
caveats: Interpretation notes and sparse-design warnings.
source_interrater: The underlying interrater_agreement_table() output used for agreement statistics.

Details

This function implements a package-native rater-effect network view complementary to MFRM output. It follows the pairwise-network logic used in Lamprianou's rater-effect network work: nodes are raters, edges summarize pairwise relationships among raters in shared scoring contexts, and directed disagreement edges can be interpreted as relative leniency/severity indicators. These network summaries are descriptive diagnostics, not Rasch logit estimates and not formal fit statistics. They describe score relationships conditional on observed shared contexts; they do not test the assignment graph's connectedness or establish a common measurement scale.

Degree, Strength, Betweenness, and Closeness are graph-theoretic quantities computed after min_pair_n, min_weight, and (for directed networks) score_diff_tolerance are applied. They are not rating-scale central tendency or restriction-of-range measures. Use mfrm_network_analysis() for assignment/co-observation connectedness.

For mode = "severity_direction", outgoing strength means the rater more often assigned higher scores than comparison raters; incoming strength means comparison raters more often assigned higher scores than this rater. The reported SeverityIndex is positive for relatively severe raters and negative for relatively lenient raters, but it is on a network-analysis scale and should not be read as an MFRM severity logit. A rater without any retained directional comparisons has an unavailable index, not a balanced index. Zero or negative undirected weights remain in the pair table but do not form graph edges. Graph distances summarize reachable pairs only, so they do not describe distance across disconnected components. Recreate older network results from the existing fit and matching diagnostics with the original settings before summary, plotting or export.

References

Lamprianou, I. (2018). Investigation of rater effects using Social Network Analysis and Exponential Random Graph Models. Educational and Psychological Measurement, 78(3), 430-459.

Lamprianou, I. (2025). Network Analysis for the investigation of rater effects in language assessment: A comparison of ChatGPT vs human raters. Research Methods in Applied Linguistics, 4, 100205.

Examples

# \donttest{
toy <- load_mfrmr_data("example_core")
fit <- fit_mfrm(toy, "Person", c("Rater", "Criterion"), "Score",
  method = "JML", maxit = 300
)
if (requireNamespace("igraph", quietly = TRUE)) {
  rn <- rater_network_analysis(fit, mode = "severity_direction")
  rn$summary
  head(rn$node_metrics)
  plot(rn, type = "severity", draw = FALSE)
}
# }