Skip to contents

Review a peer-assessment assignment design

Source: R/api-reports.R
review_peer_design.Rd

Review existing assignment metadata for self-review, reviewer/submission load and shared submissions. This function does not create assignments, simulate scores or fit a measurement model. In 0.2.4 it replaces build_peer_review_design_review() with the same arguments and result class.

Usage

review_peer_design(peer_review_design, top_n = 10)

# S3 method for class 'mfrm_peer_review_design_review'
summary(object, digits = 3, top_n = 10, ...)

Arguments

peer_review_design

A generated data frame carrying the mfrm_peer_review_design attribute, the attribute itself, or its overview data frame.

top_n

For review_peer_design(): Number of reviewer-load, submission-load, common-link, and reciprocal-pair rows to keep in compact summary tables.

For summary(): Number of rows to keep in compact follow-up tables.

object

Output from review_peer_design().

digits

Number of digits for printed numeric values.

...

Reserved for generic compatibility.

Value

A bundle of class mfrm_peer_review_design_review.

For summary(): An object of class summary.mfrm_peer_review_design_review.

Details

review_peer_design() converts peer-review simulation metadata into a reportable design-review object. The review summarizes self-review checks, reviewer and submission load, common submissions per reviewer pair, and reciprocal review pairs. These rows are assignment-design diagnostics: they do not replace MFRM estimates, fit, separation, reliability, or substantive review-quality evidence.

Peer-review use of MFRM follows studies that model peer/self/teacher rater severity and leniency. Common-link anchor interpretation follows sparse rater-mediated design work; the review status is therefore descriptive and conservative rather than a literature-derived universal adequacy cutoff.

summary() method: Summarize a peer-review design review

References

  • Farrokhi, F., Esfandiari, R., & Schaefer, E. (2012). A many-facet Rasch measurement of differential rater severity/leniency in three types of assessment. JALT Journal, 34(1), 79-102. doi:10.37546/JALTJJ34.1-3.

  • Uto, M., & Ueno, M. (2020). A generalized many-facet Rasch model and its Bayesian estimation using Hamiltonian Monte Carlo. Behaviormetrika, 47, 469-496. doi:10.1007/s41237-020-00115-7.

  • DeMars, C. E., Shapovalov, Y. A., & Hathcoat, J. D. (2023). Many-Facet Rasch Designs: How Should Raters be Assigned to Examinees? NCME presentation.

See also

build_peer_review_sim_spec(), simulate_mfrm_data(), build_mfrm_network_review(), build_summary_table_bundle()

review_peer_design()

Examples

peer_spec <- build_peer_review_sim_spec(
  n_submission = 12,
  n_criterion = 3,
  reviewers_per_submission = 2,
  anchor_submissions = 2
)
peer_sim <- simulate_mfrm_data(sim_spec = peer_spec, seed = 123)
review <- review_peer_design(peer_sim)
summary(review)$overview
#> # A tibble: 1 × 26
#>   Active Scenario    Submissions Reviewers Criteria  Rows ReviewPairs
#>   <lgl>  <chr>             <int>     <int>    <int> <int>       <int>
#> 1 TRUE   peer_review          12        12        3   126          42
#> # ℹ 19 more variables: PossibleReviewPairs <int>, DesignDensity <dbl>,
#> #   SelfReviews <int>, AvoidSelfReview <lgl>, ReciprocalPairs <int>,
#> #   AnchorSubmissions <int>, OrdinaryReviewersPerSubmission <int>,
#> #   AnchorReviewersPerSubmission <int>, MinReviewersPerSubmission <int>,
#> #   MeanReviewersPerSubmission <dbl>, MaxReviewersPerSubmission <int>,
#> #   MinSubmissionsPerReviewer <int>, MeanSubmissionsPerReviewer <dbl>,
#> #   MaxSubmissionsPerReviewer <int>, …