Skip to contents

Compute diagnostics for an mfrm_fit object

Source: R/api-estimation.R
diagnose_mfrm.Rd

Compute diagnostics for an mfrm_fit object

Usage

diagnose_mfrm(
  fit,
  interaction_pairs = NULL,
  top_n_interactions = 20,
  whexact = FALSE,
  residual_pca = c("none", "overall", "facet", "both"),
  pca_max_factors = 10L
)

Arguments

fit

Output from fit_mfrm().

interaction_pairs

Optional list of facet pairs.

top_n_interactions

Number of top interactions.

whexact

Use exact ZSTD transformation.

residual_pca

Residual PCA mode: "none", "overall", "facet", or "both".

pca_max_factors

Maximum number of PCA factors to retain per matrix.

Value

An object of class mfrm_diagnostics including:

  • obs: observed/expected/residual-level table

  • measures: facet/person fit table (Infit, Outfit, ZSTD, PTMEA)

  • overall_fit: overall fit summary

  • fit: element-level fit diagnostics

  • reliability: facet-level model/real separation and reliability

  • precision_profile: one-row summary of the active precision tier and its recommended use

  • precision_audit: package-native checks for SE, CI, and reliability

  • facet_precision: facet-level precision summary by distribution basis and SE mode

  • facets_chisq: fixed/random facet variability summary

  • interactions: top interaction diagnostics

  • interrater: inter-rater agreement bundle (summary, pairs) including agreement and rater-severity spread indices

  • unexpected: unexpected-response bundle

  • fair_average: adjusted-score reference bundle

  • displacement: displacement diagnostics bundle

  • approximation_notes: method notes for SE/CI/reliability summaries

  • residual_pca_overall: optional overall PCA object

  • residual_pca_by_facet: optional facet PCA objects

Details

This function computes a diagnostic bundle used by downstream reporting. It calculates element-level fit statistics, approximate facet separation/reliability summaries, residual-based QC diagnostics, and optionally residual PCA for exploratory residual-structure screening.

Key fit statistics computed for each element:

  • Infit MnSq: information-weighted mean-square residual; sensitive to on-target misfitting patterns. Expected value = 1.0.

  • Outfit MnSq: unweighted mean-square residual; sensitive to off-target outliers. Expected value = 1.0.

  • ZSTD: Wilson-Hilferty cube-root transformation of MnSq to an approximate standard normal deviate.

  • PTMEA: point-measure correlation (item-rest correlation in MFRM context); positive values confirm alignment with the latent trait.

Misfit flagging guidelines (Bond & Fox, 2015):

  • MnSq < 0.5: overfit (too predictable; may inflate reliability)

  • MnSq 0.5–1.5: productive for measurement

  • MnSq > 1.5: underfit (noise degrades measurement)

  • \(|\mathrm{ZSTD}| > 2\): statistically significant misfit (5\

When Infit and Outfit disagree, Infit is generally more informative because it downweights extreme observations. Large Outfit with acceptable Infit typically indicates a few outlying responses rather than systematic misfit.

interaction_pairs controls which facet interactions are summarized. Each element can be:

  • a length-2 character vector such as c("Rater", "Criterion"), or

  • omitted (NULL) to let the function select top interactions automatically.

Residual PCA behavior:

  • "none": skip PCA (fastest; recommended for initial exploration)

  • "overall": compute overall residual PCA across all facets

  • "facet": compute facet-specific residual PCA for each facet

  • "both": compute both overall and facet-specific PCA

Overall PCA examines the person \(\times\) combined-facet residual matrix; facet-specific PCA examines person \(\times\) facet-level matrices. These summaries are exploratory screens for residual structure, not standalone proofs for or against unidimensionality. Facet-specific PCA can help localise where a stronger residual signal is concentrated.

Reading key components

Practical interpretation often starts with:

  • overall_fit: global infit/outfit and degrees of freedom.

  • reliability: facet-level model/real separation and reliability. MML uses model-based ModelSE values where available; JML keeps these quantities as exploratory approximations.

  • fit: element-level misfit scan (Infit, Outfit, ZSTD).

  • unexpected, fair_average, displacement: targeted QC bundles.

  • approximation_notes: method notes for SE/CI/reliability summaries.

Interpreting output

Start with overall_fit and reliability, then move to element-level diagnostics (fit) and targeted bundles (unexpected, fair_average, displacement, interrater, facets_chisq).

Consistent signals across multiple components are typically more robust than a single isolated warning. For example, an element flagged for both high Outfit and high displacement is more concerning than one flagged on a single criterion.

SE is kept as a compatibility alias for ModelSE. RealSE is a fit-adjusted companion defined as ModelSE * sqrt(max(Infit, 1)). Reliability tables report model and fit-adjusted bounds from observed variance, error variance, and true variance; JML entries should still be treated as exploratory.

Typical workflow

  1. Run diagnose_mfrm(fit, residual_pca = "none") for baseline diagnostics.

  2. Inspect summary(diag) and targeted tables/plots.

  3. If needed, rerun with residual PCA ("overall" or "both").

See also

fit_mfrm(), analyze_residual_pca(), build_visual_summaries(), mfrmr_visual_diagnostics, mfrmr_reporting_and_apa

Examples

toy <- load_mfrmr_data("example_core")
fit <- fit_mfrm(toy, "Person", c("Rater", "Criterion"), "Score", method = "JML", maxit = 25)
diag <- diagnose_mfrm(fit, residual_pca = "none")
s_diag <- summary(diag)
s_diag$overview[, c("Observations", "Facets", "Categories")]
#> # A tibble: 1 × 3
#>   Observations Facets Categories
#>          <int>  <int>      <int>
#> 1          768      2          4
p_qc <- plot_qc_dashboard(fit, diagnostics = diag, draw = FALSE)
class(p_qc)
#> [1] "mfrm_plot_data" "list"          

# Optional: include residual PCA in the diagnostic bundle
diag_pca <- diagnose_mfrm(fit, residual_pca = "overall")
pca <- analyze_residual_pca(diag_pca, mode = "overall")
head(pca$overall_table)
#>   Component Eigenvalue Proportion Cumulative
#> 1         1   2.113455 0.13209091  0.1320909
#> 2         2   1.833187 0.11457417  0.2466651
#> 3         3   1.706146 0.10663410  0.3532992
#> 4         4   1.519040 0.09494001  0.4482392
#> 5         5   1.289522 0.08059513  0.5288343
#> 6         6   1.157161 0.07232254  0.6011568

# Reporting route:
prec <- precision_audit_report(fit, diagnostics = diag)
summary(prec)
#> mfrmr Precision Audit Summary 
#>   Class: mfrm_precision_audit
#>   Components (4): profile, checks, approximation_notes, settings
#> 
#> Precision overview
#>  Method PrecisionTier SupportsFormalInference Checks ReviewOrWarn NoteRows
#>     JML   exploratory                   FALSE      7            1        4
#> 
#> Audit checks: checks
#>                     Check Status
#>            Precision tier review
#>     Optimizer convergence   pass
#>      ModelSE availability   pass
#>  Fit-adjusted SE ordering   pass
#>      Reliability ordering   pass
#>  Facet precision coverage   pass
#>          SE source labels   pass
#>                                                                                                            Detail
#>  This run uses the package's exploratory precision path; prefer MML for formal SE, CI, and reliability reporting.
#>                                                                               The optimizer reported convergence.
#>                                                          Finite ModelSE values were available for 100.0% of rows.
#>                                         Fit-adjusted SE values were not smaller than their paired ModelSE values.
#>                                      Conservative reliability values were not larger than the model-based values.
#>                              Each facet had sample/population summaries for both model and fit-adjusted SE modes.
#>                                                JML SE labels consistently identify observation-table information.
#> 
#> Settings
#>         Setting       Value
#>           model         RSM
#>          method        JMLE
#>  precision_tier exploratory
#> 
#> Notes
#>  - Exploratory precision path detected; use this run for screening and calibration triage, not as the package's primary inferential summary.