Bootstrap prediction intervals for observed random raters

Regenerate persons, shared raters and scores on the observed assignment, refit the same RSM, and retain prediction errors for every planned replicate.

Usage

mfrm_random_rater_intervals(object, nsim = 499L, seed, level = 0.95)

# S3 method for class 'mfrm_random_rater_intervals'
plot(
  x,
  level = x$settings$level,
  method = c("studentized", "error"),
  comparison = TRUE,
  draw = TRUE,
  sort = c("input", "estimate", "uncertainty"),
  palette = c("accessible", "mono"),
  title = NULL,
  caption = NULL,
  show_title = FALSE,
  show_notes = FALSE,
  show_labels = TRUE,
  reference = 0,
  text_scale = 1,
  point_size = 2.5,
  ...
)

# S3 method for class 'mfrm_random_rater_intervals'
confint(
  object,
  parm = NULL,
  level = 0.95,
  method = c("studentized", "error"),
  ...
)

# S3 method for class 'mfrm_random_rater_intervals'
summary(object, ...)

# S3 method for class 'mfrm_random_rater_intervals'
print(x, ...)

Arguments

object

A numerically ready fit_mfrm_random_rater() result with positive information, positive population SD and finite positive rater prediction SEs. Estimated zero-variance sources are not supported.

nsim

Number of planned bootstrap datasets; default 499, minimum 2. Each requires a full refit. Small values are useful for checking a workflow, not for stable tail quantiles.

seed

Required nonnegative integer simulation seed. The caller's random number state is restored. Per-replicate seeds and RNG kind are retained.

level

For mfrm_random_rater_intervals(): Pointwise nominal coverage; default 0.95.

For plot(): Nominal pointwise coverage; defaults to the saved level.

x

For mfrm_random_rater_intervals(): A bootstrap result.

For plot(): A result from mfrm_random_rater_intervals().

method

For mfrm_random_rater_intervals(): "studentized" (default) uses generated-minus-estimated rater effects divided by each refit's calibration-adjusted PredictionSE. "error" uses unscaled prediction errors as a comparison. Changing method or level through confint() reuses saved draws without new fitting.

For plot(): "studentized" or "error"; see mfrm_random_rater_intervals().

comparison

Show ordinary normal intervals from the same source fit.

draw

FALSE returns plotted data without opening a device.

sort

Row order for interval plots: input order, increasing estimate, or increasing interval width. Missing values come last; ties retain input order. Sorting is descriptive, not a test of differences.

palette

"accessible" uses blue and dark orange with filled/open symbols; "mono" uses dark grey with the same symbol distinctions.

title, caption

NULL uses the default text; a string replaces it; "" removes it. Interpretation notes remain in the saved plot data.

show_title, show_notes

Show the title and caption, respectively. FALSE hides text without deleting its metadata.

show_labels

Show IDs on interval and precision plots. Distribution plots summarize estimates and do not label individual IDs. For crowded precision plots use FALSE and consult the retained table.

reference

Vertical reference line in logits; NULL omits it. This is an orientation aid, not a quality threshold.

text_scale

Positive multiplier for text size.

point_size

Positive point size in ggplot millimetres; base graphics use the corresponding relative size (2.5 is the default).

...

Unused.

parm

Rater IDs for confint(); NULL returns all fitted raters.

Value

An mfrm_random_rater_intervals object containing the source rater table, intervals, aligned error/studentized matrices, generated effects, refitted estimates/SEs, per-trial checks, warnings/errors, seeds, model settings and analysis data. confint() returns a two-column matrix with availability, method and Monte Carlo resolution attributes. No native pointers are stored; save with saveRDS().

For plot(): Invisibly, a ggplot object. plot_data() extracts exact bounds, availability, settings and the source estimates.

Details

Every generated dataset draws one ability per person and one severity per rater, shared across their observed rows. Generating fixed facets, steps and both population SDs are the source estimates. Each Person ability is drawn from the fitted normal population. The refit re-estimates calibration and each population SD unless originally fixed. Earlier saved fits retain their known N(0,1) ability population. Per-trial records include the refitted ability SD and its zero-boundary status. The target is a realized effect relative to the population mean, not a centered effect, new-rater score or fixed rater coefficient. Coverage is marginal over new persons, rater effects and scores at this assignment; it is not conditional coverage for each fixed true severity.

Studentized endpoints are the source estimate plus its PredictionSE times the empirical tail quantiles (type 1) of bootstrap errors divided by refitted PredictionSE. Unscaled endpoints add error quantiles directly. All planned replicates remain in the denominator. Failed fits and unavailable studentizers are unresolved, including studentizers at estimated variance boundaries. Lower-tail calculations place unresolved roots at minus infinity; upper-tail calculations place them at plus infinity. The resulting limits enclose empirical bootstrap limits for any completion of those roots. They may be unbounded; failures are never silently removed or replaced. A numerically ready boundary refit still supplies an unscaled prediction error.

This is a model-based bootstrap candidate, not a general finite-sample coverage guarantee. Linear mixed-model bootstrap theory motivates the construction but does not establish its accuracy for this crossed ordinal RSM, few raters, variance boundaries or Laplace approximation. Basic error intervals and studentized intervals need separate studies of their coverage. The fit's normal-population and assignment assumptions remain essential. Omitted scores stay omitted; this conditions on analyzed rows and does not simulate a missingness mechanism or impute assigned scores. Rater contrasts, familywise intervals and simultaneous rater classification are not provided.

plot() method: Compare bootstrap and ordinary random-rater intervals

For plot(): Arrows identify unbounded bootstrap endpoints. Their displayed ends are plot limits, not finite interval limits. Ordinary intervals never replace an unbounded bootstrap interval. These are pointwise prediction intervals for realized rater effects, not a rater-quality classification.

Review unresolved refits

Inspect $trials before interpreting the intervals. FitReady combines numerical and information checks; it does not certify interval coverage. New results also retain OptimizerCode, NumericalReady, InformationPositive, PersonQuadratureStable, QuadraturePoints, CheckPoints, LogLikDifference, GradientDifference, EstimatedVarianceBoundary and PersonVarianceUpperBoundary from each refit. Together with the recorded gradient and Person-variance boundary, these distinguish unresolved numerical calculations from variance boundaries and missing regular studentizers. A failed refit has missing additional check values and its recorded error; missing checks are not passes. Older saved trial tables lack these additional fields and cannot acquire them without rerunning the corresponding refits. Failed checks must not be removed, selectively retried until successful or relabeled as adequate interval coverage. Increasing Person quadrature can address integration precision but does not qualify the rater Laplace approximation, the interval method or the population assumptions.

Increasing nsim alone does not resolve unavailable prediction errors. For example, with 499 planned refits at 95%, 13 unresolved studentized errors for one rater make both of that rater's limits infinite under the type-1 completion rule. Twelve unresolved errors leave finite empirical limits when all other errors and the source estimate/SE are finite. This describes the calculation, not a threshold establishing accurate coverage. If the unresolved fraction remains above the nominal tail probability, a larger run still has unbounded limits. Inspect causes in $trials before committing to more refits; do not discard unresolved draws.

References

Chatterjee, S., Lahiri, P. and Li, H. (2008). Parametric bootstrap approximation to the distribution of EBLUP and related prediction intervals in linear mixed models. Annals of Statistics, 36, 1221–1245. doi:10.1214/07-AOS512 .

Examples

example <- readRDS(system.file("examples", "extended-models.rds", package = "mfrmr"))
fit <- example$random_rater$fit
intervals <- example$random_rater$intervals
# To regenerate (requires RTMB >= 2.0 and repeated model fitting):
# intervals <- mfrm_random_rater_intervals(fit, nsim = 19, seed = 923701)
# These 19 saved trials illustrate mechanics, not accurate 2.5% tails.
# Every trial is retained; unresolved trials can give unbounded intervals.
summary(intervals)
#> $intervals
#>     Lower Upper
#> R01  -Inf   Inf
#> R02  -Inf   Inf
#> R03  -Inf   Inf
#> R04  -Inf   Inf
#> attr(,"availability")
#>     Rater Planned Known Unresolved Finite
#> R01   R01      19    17          2  FALSE
#> R02   R02      19    17          2  FALSE
#> R03   R03      19    17          2  FALSE
#> R04   R04      19    17          2  FALSE
#> attr(,"method")
#> [1] "studentized"
#> attr(,"level")
#> [1] 0.95
#> attr(,"expected_tail_draws")
#> [1] 0.475
#> attr(,"note")
#> [1] "Pointwise model-based bootstrap; unresolved roots widen limits, possibly to infinity. Coverage and Monte Carlo tail accuracy require separate assessment."
#> 
#> $availability
#>     Rater Planned Known Unresolved Finite
#> R01   R01      19    17          2  FALSE
#> R02   R02      19    17          2  FALSE
#> R03   R03      19    17          2  FALSE
#> R04   R04      19    17          2  FALSE
#> 
#> $trials
#>           Planned          FitReady EstimatedBoundary 
#>                19                18                 1 
#> 
#> $settings
#> $settings$nsim
#> [1] 19
#> 
#> $settings$seed
#> [1] 923701
#> 
#> $settings$replicate_seeds
#>  [1] 1289759734 1889042303   97805746  219806802 1816506017  464639873
#>  [7] 1916682954  487847372 1641835732   26525370 1507912874 1457296021
#> [13] 1325607976 1595872271 1288178159  782172590 1853828702 1266267398
#> [19] 1539409418
#> 
#> $settings$rng_kind
#> [1] "Mersenne-Twister" "Inversion"        "Rejection"       
#> 
#> $settings$person_sd
#> [1] 0.9686791
#> 
#> $settings$fixed_person_sd
#> NULL
#> 
#> $settings$level
#> [1] 0.95
#> 
#> $settings$target
#> [1] "Realized rater effects relative to the population mean"
#> 
#> $settings$sampling
#> [1] "New persons, shared raters and responses on analyzed rows"
#> 
#> $settings$quantile_type
#> [1] 1
#> 
#> 
confint(intervals, method = "error")
#>     Lower Upper
#> R01  -Inf   Inf
#> R02  -Inf   Inf
#> R03  -Inf   Inf
#> R04  -Inf   Inf
#> attr(,"availability")
#>     Rater Planned Known Unresolved Finite
#> R01   R01      19    18          1  FALSE
#> R02   R02      19    18          1  FALSE
#> R03   R03      19    18          1  FALSE
#> R04   R04      19    18          1  FALSE
#> attr(,"method")
#> [1] "error"
#> attr(,"level")
#> [1] 0.95
#> attr(,"expected_tail_draws")
#> [1] 0.475
#> attr(,"note")
#> [1] "Pointwise model-based bootstrap; unresolved roots widen limits, possibly to infinity. Coverage and Monte Carlo tail accuracy require separate assessment."
# Changing the level reuses the same draws, with no further fitting:
confint(intervals, level = .90)
#>     Lower Upper
#> R01  -Inf   Inf
#> R02  -Inf   Inf
#> R03  -Inf   Inf
#> R04  -Inf   Inf
#> attr(,"availability")
#>     Rater Planned Known Unresolved Finite
#> R01   R01      19    17          2  FALSE
#> R02   R02      19    17          2  FALSE
#> R03   R03      19    17          2  FALSE
#> R04   R04      19    17          2  FALSE
#> attr(,"method")
#> [1] "studentized"
#> attr(,"level")
#> [1] 0.9
#> attr(,"expected_tail_draws")
#> [1] 0.95
#> attr(,"note")
#> [1] "Pointwise model-based bootstrap; unresolved roots widen limits, possibly to infinity. Coverage and Monte Carlo tail accuracy require separate assessment."