MFRM Desktop User Guide — Windows & macOS

日本語ガイド · Project Overview · Why MFRM? · Get & Open · Sample Walkthrough · Prepare Your CSV · Choose an Analysis · Interpret Every Plot · Save & Reopen · Troubleshooting

Why Use MFRM? Start with a Concrete Example

Suppose you are analyzing university essay ratings (or L2 speaking interviews, music juries, or clinical OSCE stations) for a thesis or departmental assessment. Four trained raters (R1–R4) score 30 students across two writing prompts (Task1, Task2) and four rubric criteria (Content, Organization, Language, Mechanics) on a 0–5 scale.

In realistic testing programs, you rarely want to rely on raw score averages alone:

• The "severe rater / hard prompt" trap: Suppose Student A writes Task2 (an abstract policy argument) and is graded by R4 (the most severe rater), earning a raw average of 3.2 / 5.0. Student B writes Task1 (a familiar personal topic) for R1 (the most lenient rater) and averages 3.6 / 5.0. Raw averages make Student B look stronger—even if Student A actually demonstrated higher ability under tougher conditions.
• Why correlation or Kappa isn't enough: Even if R1 and R4 correlate at r = 0.90, R4 might consistently award scores one point lower than R1. High rank-order agreement does not remove a constant severity gap.
• Why rubric steps need checking: Writing "0 = Emerging … 5 = Exemplary" on a rubric sheet does not guarantee raters actually use all six categories cleanly or treat Content and Mechanics with equal severity.

What MFRM & G/D Studies Give You Instead

1. One shared "logit" ruler (MFRM): MFRM converts ordinal rubric scores into a common log-odds (logit) scale centered at 0.00 for non-person facets:
   • Examinees (Person): Higher logits = higher ability (adjusted for the raters, tasks, and criteria involved).
   • Raters (Rater): Higher logits = greater severity; lower (negative) logits = greater leniency.
   • Tasks & Criteria (Task, Criterion): Higher logits = harder to earn high scores on; lower logits = easier.
2. Quality diagnostics (Fit, Category Curves, Bias): Checks whether a severe rater is predictably severe (Infit MnSq ≈ 1.00) vs. erratic (Infit >> 1.00), whether every rubric score (0–5) has its own clear zone, and whether specific pairs (like R1 × Content) behave unusually.
3. Practical assessment design (G/D Study): Uses observed scores to show how reliability (G for relative ranking, Phi for pass/fail cutoffs) changes if you assign 2 vs. 3 raters or 2 vs. 4 tasks.

All of these run offline in one three-step desktop window (Data → Settings → Results), with an instant English / 日本語 switch in the top bar.

1. Get and Open the App

Why macOS & Windows show an "unverified" notice — and how to open in 3 steps

MFRM Desktop is distributed 100% free of charge as open-source academic software without paid commercial Apple Developer Program notarization or commercial Windows code-signing subscriptions. Because no commercial signing certificate is attached, macOS Gatekeeper ("MFRM-Desktop" Not Opened — Apple could not verify "MFRM-Desktop" is free of malware) and Microsoft Defender SmartScreen (Windows protected your PC) display a standard first-launch confirmation—not because malicious code was detected, but because the archive is unnotarized / unsigned.

Three transparent safeguards protect your work:

1. 100% open source: Every line of estimation and GUI code is publicly auditable in the GitHub repository.
2. 100% offline analysis (zero data upload or telemetry): All CSV reading, estimation, plotting, and .mfrm / .html / .csv saving run locally on your machine—you can turn off Wi-Fi before launching and run every analysis workflow offline (only clicking Help → Web theory and help opens online documentation in your browser, without uploading desktop data).
3. Official GitHub Releases & build provenance: Both ZIP archives are hosted directly on GitHub Releases (TOK0) with verified SHA-256 digests and GitHub Actions Sigstore provenance on Windows.

macOS (Apple Silicon, M1 or Later) — Standalone Preview ZIP

MFRM Desktop uses the Data → Settings → Results interface on Apple Silicon Macs (arm64, bundled libraries targeting macOS 14.0+; Python is bundled inside MFRM-Desktop.app, while Intel Macs can run from Python source):

1. Download TOK0 (release notes).
2. Double-click the ZIP in Finder to expand MFRM-Desktop.app, and move it to a writable folder such as Applications.
3. Double-click MFRM-Desktop.app and complete the one-time Apple TOK1 approval (macOS Sequoia 15+ / Sonoma 14+):
   • When "MFRM-Desktop" Not Opened appears, click Done — do NOT click the blue Move to Trash button.
   • Open System Settings → Privacy & Security and scroll down to the Security section.
   • Next to "MFRM-Desktop" was blocked to protect your Mac, click Open Anyway and authenticate with Touch ID or your Mac password (needed once only; opens normally every time after that).

Windows 11 (x64) — Standalone Preview ZIP

Python is bundled inside the download—no Python, R, FACETS, or ConQuest installation is needed.

1. Download TOK0 (release notes & sample projects).
2. In File Explorer, right-click MFRM-Desktop-Windows-x64.zip and select Extract All. Keep MFRM-Desktop.exe and the entire _internal folder together in the same folder (never run the EXE from inside the unextracted ZIP or move the EXE alone).
3. Double-click MFRM-Desktop.exe. If Microsoft Defender SmartScreen (Windows protected your PC) appears, click More info → Run anyway (needed once only; if institutional policy blocks execution, record the exact message for support).

Want to try in a browser first?

Open the Web App with synthetic or properly de-identified data.

2. Try the Sample — Walkthrough with Actual Numbers

Let's run the built-in synthetic essay dataset (30 examinees × 4 raters × 2 tasks × 4 criteria = 960 ratings) and read the exact numbers that appear on screen:

1. In Data, click Sample (Writing sample (synthetic data)), inspect the 10-row preview, and click Next.
2. In Settings → Columns / rubric, confirm the pre-filled values (Person = Person, Score = Score, Facets = Rater, Task, Criterion, Minimum / Maximum = 0 / 5). Leave Model / estimation on the defaults (RSM and JMLE).
3. Click Analyze (and confirm the 960-row prompt). Within seconds, Results opens.

Check what you just discovered in Table → Measures and Fit

Switch to Table and look at the non-person rows of Measures and Fit:

(Note: In JMLE, the table's Conditional SE / SD and bounds hold other fitted parameters fixed as screening approximations—always read the note directly below each table!)

Now switch to Observed Category Use: all six rubric categories (0 through 5) are well populated—from 72 ratings (7.5%) at score 0 and 96 (10.0%) at score 5 up to 230 (24.0%) at score 2 and 225 (23.4%) at score 3. Finally, try Save project (.mfrm) and Save report (.html).

3. Prepare and Import Your Own CSV

Reshape wide spreadsheets into "one row = one rating"

Spreadsheets exported from gradebooks often put raters side by side (wide format). Before importing into MFRM Desktop, stack them so every row is one single rating event:

❌ Wide format (do NOT import directly):
Student, Task, Criterion, Rater1_Score, Rater2_Score
E01,     T1,   Content,   4,            5

✅ Long format (ready for MFRM Desktop):
Person,Rater,Task,Criterion,Score
E01,R1,T1,Content,4
E01,R2,T1,Content,5

(Click CSV format → Save example CSV in the app or inspect TOK0 for a complete template.)

Four data-checks before clicking Analyze

1. Save as UTF-8 CSV: In Excel, pick CSV UTF-8 (Comma delimited) (*.csv). Japanese and accented column/level names are supported; leading zeros in IDs (like 007) are kept intact.
2. Use individual integer ratings, not averages: MFRM models discrete rubric categories (0, 1, 2, 3, 4, 5). Fractional scores (like 3.5) will stop validation with an explanation. Every row receives equal (unit) weight.
3. Enter the full rubric range—and never code "missing" as 0: On a 0–5 rubric, type 0 and 5 in Settings even if nobody scored 0. If a rater skipped an essay, omit that row or leave Score blank (0 means an earned score of zero!).
4. Make sure raters and tasks overlap ("connectedness"): If Raters A & B only grade Group 1 and Raters C & D only grade Group 2 with zero shared students or tasks, no statistical model can tell whether Group 1 is stronger or Raters A & B are simply more lenient! Design rating assignments with shared examinees/tasks across raters.

4. Choose the Right Analysis for Your Research Question

In Settings, select MFRM (tabs: Columns / rubric, Model / estimation, Bias / intervals, G/D plan) or standalone G/D Study:

5. Interpret Every Plot — What Patterns Mean in Practice

Here is how to read each visual output in Results, including both healthy patterns and warning signs you might encounter in your own thesis or assessment data:

5.1 Wright Map — Scale Targeting & Facet Spread

5.2 Pathway Map — Separating Severity from Consistency (Infit MnSq)

One of the most useful ideas in Rasch measurement for graduate research is that vertical position (severity/difficulty) and horizontal position (Infit MnSq consistency) ask two completely different questions:

5.3 Category Probability Curves — Is Your Rubric Working?

5.4 Conditional Bias Heatmap vs. Specified Intervals & Joint Interaction

• Reading the Conditional Bias Heatmap (Rater × Criterion above):
  • Numbers near 0.00 mean that pairing behaves just as expected from the rater's overall severity plus the criterion's overall difficulty.
  • Non-zero shifts (e.g., R1 × Content = +0.24 vs. R1 × Organization = -0.22) describe how that cell's observed scores deviate when main calibration is held fixed; gray cells mark pairs without an interior estimate (always check cell counts and status in the table!).
  • Thesis writing caution: Conditional bias heatmaps are exploratory screening maps, NOT statistical proof of bias or fairness (Tests / intervals withheld). Write "Cell R1 × Content showed a descriptive shift of +0.24 logits with main calibration fixed, prompting qualitative review of rubric exemplars"—not "Rater 1 was significantly biased."
• When you need model-based uncertainty intervals:
  • Use Specified intervals (fixed-SD MML) for a structural facet's levels or a prespecified A − B contrast, or Joint interaction (fixed-N(0,1) MML) to refit main effects, steps, and one interaction together.
  • If a target passes numerical checks, pointwise approximate intervals are plotted; if withheld (blank in table or reason shown on chart), report it as unavailable rather than zero—and remember numerical screens do not guarantee finite-sample coverage or simultaneous intervals.

5.5 D-Study Curves — Planning Rater and Task Counts (G and Phi)

(Reminder: G/D requires complete, balanced 1–2 random-facet observed-score designs with 1 observation per cell and unit weights; G and Phi are point projections without confidence intervals, and benchmarks like 0.80 depend on assessment stakes rather than acting as universal rules. See Analyses, interpretation and reporting limits for full details.)

6. Save, Export Manuscript Figures, and Reopen Projects

Using Manuscript Figure Mode Properly

• Explanatory (default): Keeps the figure title and caution footer inside the image so slides and working notes stay self-contained.
• Manuscript: Moves the overall title and long caution prose outside the graphic while retaining axis labels, legends, data labels, and any withheld-output notice. Use Title and note to copy the text, and always carry the saved .caption.txt content into your thesis or manuscript figure note so statistical conditions are never lost. (Check your target journal's or APA's specific layout guidelines.)

Safe Project Reopening & Version Updates

• Open saved .mfrm projects through Data → Open project (one project at a time; portable Windows builds do not register .mfrm file associations in Windows). Images and HTML reports cannot restore an interactive session.
• Editing controls in Settings does not overwrite displayed results—a notice reminds you that shown outputs belong to the previous analysis until you click Analyze.
• Backups across versions: This version reads older basic .mfrm projects without altering them, whereas older applications cannot open extended (schema 2) projects saved by this version, and reopened interval records check matching source/runtime conditions. Keep untouched copies of your previous app folder and original .mfrm files before testing a new build in a separate folder. Review .mfrm files and reports for student/rater IDs before sharing.

7. Troubleshooting

For bug reports, please email the author, Ryuya Komuro, at ryuya.komuro.c4@tohoku.ac.jp with your OS, app version (0.2.19-beta), archive filename, steps to reproduce, and exact error message—never include confidential or identifiable examinee/rater data in emails, files, or screenshots.