Supplement B - Simulation Atlas

Extended sensitivity analyses for The Link Function Problem in Psychological Interaction Testing

Authors

Margherita Calderan

Filippo Gambarota

Laura Sità

Tommaso Feraco

Enrico Toffalini

1 Purpose and scope

Supplement B extends the three manuscript simulations by varying selected parameters while keeping the true interaction equal to zero on the generating scale. It uses one-dimensional sensitivity slices and small main-effect surfaces rather than a full Cartesian design.

Reference scenarios retain the generating settings of the corresponding manuscript simulations. In every scenario, a rejection under the matched generating model estimates the nominal generating-scale rejection rate; under another scale, it is a pseudo-interaction detection rate. For forced choice, these primary rates use finite tests from fits passing the minimal lme4 fit-quality checks. Apparent rejection rates among flagged fits are reported separately. Publication runs request 3,000 Monte Carlo replications.

1.1 How to read the figures

All rejection-rate figures use the same interpretation. The horizontal dashed line at .05 marks nominal alpha. Rates close to .05 under the generating scale provide a calibration check. Elevated rates under an alternative fitted scale show how often the scale difference becomes a statistically significant product term at the stated sample size and design. Error bars are shown for the reference scenarios. The sensitivity plots omit error bars to keep the larger set of scenarios readable; Wilson 95% intervals are available in the numerical tables and in data/atlas-summary.csv.

The deterministic pseudo-interaction reported in the tables is calculated from expected values before sampling. For forced choice and the other mixed-model family, it is evaluated at random intercept 0, not as a marginal population-average probability. It describes the interaction contrast implied by evaluating the generating pattern on the fitted scale. The Monte Carlo rejection rate adds sampling variability, standard errors, and sample size to that deterministic discrepancy.

2 Atlas design

Table 1: Structure of the simulation atlas. Each family contains reference scenarios plus prespecified sensitivity slices.
Manuscript case Generating scenarios Sensitivity slices
Simulation 1: Forced-choice accuracy 24 Main effects; sample size; intercept; trials; chance level
Simulation 2: Logit vs probit 46 Main effects; sample size; trials; intercept; ICC
Simulation 3: Sum scores 23 Main effects; sample size; scale location; item count; latent dispersion

The one-dimensional slices change one named parameter while holding the remaining settings at the central reference scenario. The main-effect surfaces vary the two additive main-effect coefficients jointly. The actual coefficients are reported on the figure axes and in the table below.

Table 2: Generating-scale coefficients used in the main-effect surfaces. Coefficients are reported on the scale that generated each scenario.
Manuscript case Generating link/scale First main effect First coefficient values Second main effect Second coefficient values
Simulation 1: Forced-choice accuracy Chance-corrected logit Age coefficient 0.30, 0.60, 0.90 Group coefficient -1.35, -0.90, -0.45
Simulation 2: Logit vs probit Logit Group coefficient -2.475, -1.650, -0.825 Condition coefficient -2.475, -1.650, -0.825
Simulation 2: Logit vs probit Probit Group coefficient -1.50, -1.00, -0.50 Condition coefficient -1.50, -1.00, -0.50
Simulation 3: Sum scores Latent generating scale Predictor coefficient 0.425, 0.850, 1.275 Group coefficient -1.35, -0.90, -0.45

For Simulation 2, logit and probit coefficients are on different generating-link scales. As in the manuscript simulation, the logit coefficients are scaled by 1.65 relative to the probit reference coefficients so that the two generating scenarios have similar cell probabilities. The figures therefore report the actual coefficient values for each generating link instead of a common relative index.

3 Simulation 1: Forced-choice accuracy and the non-zero chance floor

Forced-choice accuracy is generated as trial-level Bernoulli responses from a chance-corrected binomial logit mixed model with a .50 floor, a subject random intercept, and no age-by-group product term on the conditional chance-corrected-logit scale. The subject random-intercept SD corresponds to latent-scale ICC \(.30\) under the usual logistic convention. The fitted models are a matched chance-corrected-logit GLMM using psyphy::mafc.logit, standard binomial-logit and binomial-probit GLMMs, and a Gaussian identity LMM. All four use the same (1 | id) structure.

3.1 Reference scenarios

Figure 1: Forced-choice reference scenarios. The product term is zero on the conditional chance-corrected-logit generating scale. Rates use fits passing the minimal fit-quality checks. The dashed line marks alpha = .05.
Table 3: Forced-choice reference scenarios. The deterministic pseudo-interaction is evaluated on each fitted scale before sampling.
Scenario Generating link/scale Fitted model/scale Rate type Rejection rate 95% CI Deterministic pseudo-interaction Flagged fits Fit-problem rate Apparent rejection rate among flagged fits
Lower performance chance-corrected logit Chance-corrected binomial logit Nominal generating-scale rejection rate 4.6% [3.7%, 5.6%] 0.000 1209 40.3% 90.1%
Lower performance chance-corrected logit Gaussian identity Pseudo-interaction detection rate 12.2% [11.0%, 13.4%] -0.077 0 0.0% –
Lower performance chance-corrected logit Standard binomial logit Pseudo-interaction detection rate 20.2% [18.8%, 21.6%] -0.453 0 0.0% –
Lower performance chance-corrected logit Standard binomial probit Pseudo-interaction detection rate 19.0% [17.6%, 20.4%] -0.262 0 0.0% –
Middle performance chance-corrected logit Chance-corrected binomial logit Nominal generating-scale rejection rate 5.8% [4.8%, 7.0%] 0.000 1270 42.3% 91.0%
Middle performance chance-corrected logit Gaussian identity Pseudo-interaction detection rate 6.4% [5.6%, 7.3%] -0.036 0 0.0% –
Middle performance chance-corrected logit Standard binomial logit Pseudo-interaction detection rate 16.2% [14.9%, 17.5%] -0.473 0 0.0% –
Middle performance chance-corrected logit Standard binomial probit Pseudo-interaction detection rate 13.7% [12.5%, 15.0%] -0.243 69 2.3% 95.7%
Higher performance chance-corrected logit Chance-corrected binomial logit Nominal generating-scale rejection rate 5.9% [4.8%, 7.1%] 0.000 1308 43.6% 95.5%
Higher performance chance-corrected logit Gaussian identity Pseudo-interaction detection rate 5.8% [5.0%, 6.7%] 0.028 0 0.0% –
Higher performance chance-corrected logit Standard binomial logit Pseudo-interaction detection rate 11.7% [10.6%, 12.9%] -0.396 1 0.0% 100.0%
Higher performance chance-corrected logit Standard binomial probit Pseudo-interaction detection rate 9.5% [8.5%, 10.6%] -0.153 105 3.5% 81.9%

The forced-choice atlas must be regenerated before numerical ranges are reported for the revised mixed-model design. The saved atlas results from the earlier fixed-effect design are not valid for this specification.

3.2 One-dimensional sensitivity slices

Figure 2: Forced-choice sensitivity slices. One parameter varies at a time around the central reference scenario. The dashed line marks alpha = .05.
Table 4: Range of rejection rates across each forced-choice sensitivity slice.
Slice Generating link/scale Fitted model/scale Minimum rejection rate Maximum rejection rate
Chance level chance-corrected logit Chance-corrected binomial logit 5.6% 5.8%
Intercept chance-corrected logit Chance-corrected binomial logit 4.6% 6.0%
Sample size chance-corrected logit Chance-corrected binomial logit 5.2% 6.0%
Trials per participant chance-corrected logit Chance-corrected binomial logit 4.9% 6.0%
Chance level chance-corrected logit Standard binomial logit 11.2% 16.2%
Intercept chance-corrected logit Standard binomial logit 9.8% 20.3%
Sample size chance-corrected logit Standard binomial logit 8.7% 44.2%
Trials per participant chance-corrected logit Standard binomial logit 9.0% 19.9%
Chance level chance-corrected logit Standard binomial probit 10.6% 13.7%
Intercept chance-corrected logit Standard binomial probit 7.0% 19.9%
Sample size chance-corrected logit Standard binomial probit 8.0% 38.3%
Trials per participant chance-corrected logit Standard binomial probit 8.7% 17.2%
Chance level chance-corrected logit Gaussian identity 6.0% 6.6%
Intercept chance-corrected logit Gaussian identity 5.8% 16.3%
Sample size chance-corrected logit Gaussian identity 4.6% 9.3%
Trials per participant chance-corrected logit Gaussian identity 4.5% 6.4%

The slices separate distinct reasons why a deterministic scale discrepancy may or may not become a statistical interaction. The intercept and the chance level change the curvature relevant to the fitted link. Sample size and number of trials change how precisely that discrepancy is estimated. The figure therefore distinguishes the existence of a pseudo-interaction on the fitted scale from the probability that a product-term test detects it.

3.3 Main-effect surface

Figure 3: Forced-choice main-effect surface. Axes report the age and group coefficients on the chance-corrected logit generating scale. Cell labels are product-term rejection rates.

The surface directly targets the mechanism emphasized in the manuscript. Two additive main effects can jointly move observations through regions where equal differences on the generating scale map to unequal differences on the fitted scale. Larger main effects can therefore increase the deterministic pseudo-interaction even though the generating age-by-group term remains zero.

5 Simulation 3: Sum scores

Sum-score data are generated on a latent continuous scale with no x-by-group interaction, converted to ordinal item responses, and summed. Analyses compare the latent generating scale, observed sum-score identity model, and Gaussian-probit bounded-score sensitivity model; the figures and Table 1 show the sensitivity design.

5.1 Reference scenarios

Figure 7: Sum-score reference scenarios. The product term is zero on the latent generating scale. The dashed line marks alpha = .05.
Table 7: Sum-score reference scenarios. The latent generating scale is available only in simulation.
Scenario Generating link/scale Fitted model/scale Rate type Rejection rate 95% CI Deterministic pseudo-interaction
Lower range graded logistic latent scale Gaussian-probit bounded score Pseudo-interaction detection rate 7.9% [7.0%, 9.0%] -0.056
Lower range graded logistic latent scale Latent generating scale Nominal generating-scale rejection rate 4.9% [4.2%, 5.8%] 0.000
Lower range graded logistic latent scale Observed sum-score identity Pseudo-interaction detection rate 72.8% [71.2%, 74.4%] -2.034
Middle range graded logistic latent scale Gaussian-probit bounded score Pseudo-interaction detection rate 3.5% [2.9%, 4.3%] -0.010
Middle range graded logistic latent scale Latent generating scale Nominal generating-scale rejection rate 5.6% [4.9%, 6.5%] 0.000
Middle range graded logistic latent scale Observed sum-score identity Pseudo-interaction detection rate 8.6% [7.6%, 9.6%] -0.696
Upper range graded logistic latent scale Gaussian-probit bounded score Pseudo-interaction detection rate 4.8% [4.1%, 5.6%] 0.027
Upper range graded logistic latent scale Latent generating scale Nominal generating-scale rejection rate 4.5% [3.8%, 5.3%] 0.000
Upper range graded logistic latent scale Observed sum-score identity Pseudo-interaction detection rate 37.4% [35.7%, 39.1%] 1.570

Across the atlas, the latent generating scale rejects from 4.2% to 6.0%. The observed sum-score identity model ranges from 2.4% to 82.1%, while the Gaussian-probit bounded-score sensitivity model ranges from 3.1% to 11.9%. The latter remains a model of the manifest total. It does not recover the latent construct and should be read as a sensitivity analysis for the scale on which additivity is imposed.

5.2 One-dimensional sensitivity slices

Figure 8: Sum-score sensitivity slices. One parameter varies at a time around the central reference scenario. The dashed line marks alpha = .05.
Table 8: Range of rejection rates across each sum-score sensitivity slice.
Slice Generating link/scale Fitted model/scale Minimum rejection rate Maximum rejection rate
Latent dispersion graded logistic latent scale Latent generating scale 4.2% 5.6%
Number of items graded logistic latent scale Latent generating scale 4.7% 5.6%
Sample size graded logistic latent scale Latent generating scale 4.5% 5.6%
Scale location graded logistic latent scale Latent generating scale 4.5% 6.0%
Latent dispersion graded logistic latent scale Gaussian-probit bounded score 3.3% 3.9%
Number of items graded logistic latent scale Gaussian-probit bounded score 3.5% 3.7%
Sample size graded logistic latent scale Gaussian-probit bounded score 3.1% 4.3%
Scale location graded logistic latent scale Gaussian-probit bounded score 3.5% 11.9%
Latent dispersion graded logistic latent scale Observed sum-score identity 5.5% 13.4%
Number of items graded logistic latent scale Observed sum-score identity 7.4% 9.1%
Sample size graded logistic latent scale Observed sum-score identity 3.6% 12.1%
Scale location graded logistic latent scale Observed sum-score identity 8.6% 82.1%

The scale-location slice changes how strongly the bounded sum score compresses the latent dimension. Item count and latent dispersion change the mapping between the latent variable and the manifest total. Sample size changes the probability that any resulting fitted-scale interaction is detected. These are distinct components of the sum-score problem and should not be reduced to a generic floor-or-ceiling warning.

5.3 Main-effect surface

Figure 9: Sum-score main-effect surface. Axes report the predictor and group coefficients on the latent generating scale. Cell labels are product-term rejection rates.

The surface shows how the mapping from latent values to a bounded total can turn two additive latent effects into a non-additive pattern on the manifest score. The latent benchmark remains the calibration reference because its product term is zero by construction.

6 Diagnostic atlas

The diagnostic analyses cover the forced-choice chance-floor case and the probit-generated, logit-fitted repeated-trial case. The forced-choice diagnostic uses the same trial-level random-intercept DGP and paired standard-logit and chance-corrected-logit GLMMs as the revised manuscript scenario. They examine how pseudo-interaction risk and diagnostic detection change with sample size and intercept. The forced-choice diagnostic atlas must be regenerated before its numerical values are interpreted.

Diagnostics are applied to the misspecified interaction model, matching the manuscript’s post-estimation question: once the fitted model has produced the product-term finding, does that same model show enough evidence of misspecification to warn the analyst? AIC compares prespecified candidate mean models with the same structural interaction formula. Positive delta AIC, defined as AIC of the misspecified model minus AIC of the generating model, favors the generating link.

Table 9: Diagnostics used for each diagnostic case.
Diagnostic case Diagnostics used
Forced-choice chance-floor case AIC favors generating link; DHARMa uniformity; DHARMa dispersion; DHARMa residual quantiles over fitted values; DHARMa residual quantiles over focal predictor; Pregibon-style added-term link check
Probit-generated, logit-fitted case AIC favors generating link; DHARMa uniformity; DHARMa dispersion; DHARMa residual distribution across design cells; Pregibon-style added-term link check

Checks not listed were not applicable to the design. In particular, the design-cell residual check is used only for the categorical repeated-trial case, whereas residual quantile checks over a focal predictor require a continuous predictor.

6.1 Detection across sensitivity slices

Figure 10: Diagnostic atlas. Pseudo-interaction detection is shown together with each diagnostic or candidate-model comparison. Sample-size and scale-location slices are shown separately.

The rates in this figure answer different questions. Pseudo-interaction detection is the probability that the misspecified fitted scale yields a significant product term. Pregibon-style detection is the probability that the added-term link check flags a departure. AIC reports how often the generating link has the lower AIC within the specified candidate pair. The curves therefore need not be complements of one another. Diagnostic misalignment occurs when the interaction conclusion changes more readily than the diagnostic workflow signals a problem.

6.2 Strength of the AIC comparison

Table 10: AIC comparison across the diagnostic atlas. Positive delta AIC favors the generating link. The within-two column reports the proportion of finite comparisons with absolute delta AIC below 2.
Case Setting Pseudo-interaction detection AIC favors generating link Median delta AIC Delta AIC IQR Absolute delta AIC < 2
Forced-choice chance-floor case Sample size = 100 11.6% 88.7% 5.24 [2.14, 8.62] 18.3%
Forced-choice chance-floor case Sample size = 250 20.7% 96.7% 13.17 [8.36, 18.41] 4.0%
Forced-choice chance-floor case Sample size = 500 32.1% 99.5% 27.70 [20.61, 34.85] 0.4%
Forced-choice chance-floor case Sample size = 1000 58.6% 100.0% 55.51 [44.85, 65.60] 0.0%
Probit-generated, logit-fitted case Sample size = 200 7.8% 91.6% 3.92 [1.99, 5.77] 23.1%
Probit-generated, logit-fitted case Sample size = 300 10.3% 95.4% 6.04 [3.72, 8.46] 11.1%
Probit-generated, logit-fitted case Sample size = 700 18.5% 99.3% 13.76 [10.02, 17.48] 1.3%
Probit-generated, logit-fitted case Sample size = 1200 26.9% 100.0% 23.54 [18.83, 28.28] 0.1%
Forced-choice chance-floor case Intercept = -0.80 20.7% 96.7% 13.17 [8.36, 18.41] 4.0%
Forced-choice chance-floor case Intercept = -1.50 20.6% 94.9% 10.42 [5.75, 15.57] 7.2%
Forced-choice chance-floor case Intercept = 0.00 16.2% 97.3% 13.18 [8.77, 18.02] 3.3%
Forced-choice chance-floor case Intercept = 0.80 11.4% 94.2% 9.28 [5.46, 13.10] 8.2%
Forced-choice chance-floor case Intercept = 1.50 40.6% 88.4% 5.31 [2.45, 8.16] 16.4%
Probit-generated, logit-fitted case Intercept = 1.50 10.3% 95.4% 6.04 [3.72, 8.46] 11.1%
Probit-generated, logit-fitted case Intercept = 0.00 19.8% 95.3% 6.68 [4.03, 9.34] 10.3%
Probit-generated, logit-fitted case Intercept = 0.75 6.1% 95.9% 5.56 [3.34, 7.70] 12.9%
Probit-generated, logit-fitted case Intercept = 2.25 22.1% 93.9% 6.50 [3.58, 9.41] 12.3%

The AIC win rate is reported together with the magnitude of the difference because these describe different situations. A generating link that wins frequently by a very small margin is weaker evidence than one that wins by a large margin. As in the manuscript, these comparisons rank a limited candidate set. They do not identify the substantively correct scale by themselves.

7 Scope and interpretation

The sensitivity results show whether the reference scenarios reflect broader patterns or depend on a narrow configuration. They also distinguish parameters that change the deterministic scale discrepancy, such as the intercept and main-effect magnitude, from parameters that mainly change detectability, such as sample size and trial count. The diagnostic analyses place pseudo-interaction detection and model-checking performance on the same sensitivity slices.

The atlas does not estimate how often link misspecification occurs in psychological research. It does not cover every plausible outcome family, link, random-effects structure, measurement model, or predictor distribution. It also does not turn model diagnostics into a decision rule for the scientifically correct scale. Its role is narrower: to show how the mechanisms developed in the paper change across a declared set of nearby scenarios.

8 Numerical appendix

The tables below provide scenario-level results for the core sensitivity analyses. Publication runs request 3,000 Monte Carlo replications.

8.1 Forced-choice accuracy

Table 11: Scenario-level results for the forced-choice atlas.
ID Slice Scenario Generating link/scale Fitted model/scale Rejection rate Deterministic pseudo-interaction Fit-problem rate Apparent rejection rate among flagged fits
FC-001 paper anchor Lower performance chance-corrected logit Chance-corrected binomial logit 4.6% 0.000 40.3% 90.1%
FC-001 paper anchor Lower performance chance-corrected logit Gaussian identity 12.2% -0.077 0.0% –
FC-001 paper anchor Lower performance chance-corrected logit Standard binomial logit 20.2% -0.453 0.0% –
FC-001 paper anchor Lower performance chance-corrected logit Standard binomial probit 19.0% -0.262 0.0% –
FC-002 paper anchor Middle performance chance-corrected logit Chance-corrected binomial logit 5.8% 0.000 42.3% 91.0%
FC-002 paper anchor Middle performance chance-corrected logit Gaussian identity 6.4% -0.036 0.0% –
FC-002 paper anchor Middle performance chance-corrected logit Standard binomial logit 16.2% -0.473 0.0% –
FC-002 paper anchor Middle performance chance-corrected logit Standard binomial probit 13.7% -0.243 2.3% 95.7%
FC-003 paper anchor Higher performance chance-corrected logit Chance-corrected binomial logit 5.9% 0.000 43.6% 95.5%
FC-003 paper anchor Higher performance chance-corrected logit Gaussian identity 5.8% 0.028 0.0% –
FC-003 paper anchor Higher performance chance-corrected logit Standard binomial logit 11.7% -0.396 0.0% 100.0%
FC-003 paper anchor Higher performance chance-corrected logit Standard binomial probit 9.5% -0.153 3.5% 81.9%
FC-004 main effect surface Age effect x 0.50; group effect x 0.50 chance-corrected logit Chance-corrected binomial logit 6.7% 0.000 51.4% 93.3%
FC-004 main effect surface Age effect x 0.50; group effect x 0.50 chance-corrected logit Gaussian identity 5.4% -0.007 0.0% –
FC-004 main effect surface Age effect x 0.50; group effect x 0.50 chance-corrected logit Standard binomial logit 6.4% -0.124 0.0% –
FC-004 main effect surface Age effect x 0.50; group effect x 0.50 chance-corrected logit Standard binomial probit 6.2% -0.063 0.7% 81.8%
FC-005 main effect surface Age effect x 1.00; group effect x 0.50 chance-corrected logit Chance-corrected binomial logit 6.6% 0.000 43.3% 92.9%
FC-005 main effect surface Age effect x 1.00; group effect x 0.50 chance-corrected logit Gaussian identity 5.2% -0.009 0.0% –
FC-005 main effect surface Age effect x 1.00; group effect x 0.50 chance-corrected logit Standard binomial logit 8.0% -0.232 0.0% –
FC-005 main effect surface Age effect x 1.00; group effect x 0.50 chance-corrected logit Standard binomial probit 6.8% -0.113 13.1% 94.9%
FC-006 main effect surface Age effect x 1.50; group effect x 0.50 chance-corrected logit Chance-corrected binomial logit 5.3% 0.000 38.5% 92.0%
FC-006 main effect surface Age effect x 1.50; group effect x 0.50 chance-corrected logit Gaussian identity 3.9% -0.009 0.0% –
FC-006 main effect surface Age effect x 1.50; group effect x 0.50 chance-corrected logit Standard binomial logit 9.6% -0.313 0.0% –
FC-006 main effect surface Age effect x 1.50; group effect x 0.50 chance-corrected logit Standard binomial probit 8.9% -0.145 43.4% 92.2%
FC-007 main effect surface Age effect x 0.50; group effect x 1.00 chance-corrected logit Chance-corrected binomial logit 5.5% 0.000 49.5% 92.9%
FC-007 main effect surface Age effect x 0.50; group effect x 1.00 chance-corrected logit Gaussian identity 4.7% -0.024 0.0% –
FC-007 main effect surface Age effect x 0.50; group effect x 1.00 chance-corrected logit Standard binomial logit 8.0% -0.255 0.0% –
FC-007 main effect surface Age effect x 0.50; group effect x 1.00 chance-corrected logit Standard binomial probit 7.5% -0.136 0.1% 100.0%
FC-008 main effect surface Age effect x 1.50; group effect x 1.00 chance-corrected logit Chance-corrected binomial logit 6.4% 0.000 36.0% 89.5%
FC-008 main effect surface Age effect x 1.50; group effect x 1.00 chance-corrected logit Gaussian identity 6.1% -0.034 0.0% –
FC-008 main effect surface Age effect x 1.50; group effect x 1.00 chance-corrected logit Standard binomial logit 26.8% -0.635 0.0% –
FC-008 main effect surface Age effect x 1.50; group effect x 1.00 chance-corrected logit Standard binomial probit 22.3% -0.309 25.3% 94.5%
FC-009 main effect surface Age effect x 0.50; group effect x 1.50 chance-corrected logit Chance-corrected binomial logit 5.1% 0.000 48.1% 93.6%
FC-009 main effect surface Age effect x 0.50; group effect x 1.50 chance-corrected logit Gaussian identity 7.2% -0.048 0.0% –
FC-009 main effect surface Age effect x 0.50; group effect x 1.50 chance-corrected logit Standard binomial logit 12.7% -0.380 0.0% –
FC-009 main effect surface Age effect x 0.50; group effect x 1.50 chance-corrected logit Standard binomial probit 11.5% -0.210 0.1% 100.0%
FC-010 main effect surface Age effect x 1.00; group effect x 1.50 chance-corrected logit Chance-corrected binomial logit 5.8% 0.000 38.5% 91.2%
FC-010 main effect surface Age effect x 1.00; group effect x 1.50 chance-corrected logit Gaussian identity 11.0% -0.073 0.0% –
FC-010 main effect surface Age effect x 1.00; group effect x 1.50 chance-corrected logit Standard binomial logit 32.9% -0.706 0.0% –
FC-010 main effect surface Age effect x 1.00; group effect x 1.50 chance-corrected logit Standard binomial probit 28.6% -0.376 1.4% 97.7%
FC-011 main effect surface Age effect x 1.50; group effect x 1.50 chance-corrected logit Chance-corrected binomial logit 6.3% 0.000 34.6% 90.8%
FC-011 main effect surface Age effect x 1.50; group effect x 1.50 chance-corrected logit Gaussian identity 14.5% -0.073 0.0% –
FC-011 main effect surface Age effect x 1.50; group effect x 1.50 chance-corrected logit Standard binomial logit 52.8% -0.949 0.0% –
FC-011 main effect surface Age effect x 1.50; group effect x 1.50 chance-corrected logit Standard binomial probit 45.1% -0.481 12.9% 98.4%
FC-012 sample size N = 50 chance-corrected logit Chance-corrected binomial logit 5.5% 0.000 33.7% 84.9%
FC-012 sample size N = 50 chance-corrected logit Gaussian identity 4.6% -0.036 0.0% –
FC-012 sample size N = 50 chance-corrected logit Standard binomial logit 8.7% -0.473 0.1% 0.0%
FC-012 sample size N = 50 chance-corrected logit Standard binomial probit 8.0% -0.243 6.4% 91.7%
FC-013 sample size N = 100 chance-corrected logit Chance-corrected binomial logit 6.0% 0.000 38.2% 88.3%
FC-013 sample size N = 100 chance-corrected logit Gaussian identity 5.4% -0.036 0.0% –
FC-013 sample size N = 100 chance-corrected logit Standard binomial logit 10.1% -0.473 0.0% –
FC-013 sample size N = 100 chance-corrected logit Standard binomial probit 9.1% -0.243 6.9% 92.8%
FC-014 sample size N = 150 chance-corrected logit Chance-corrected binomial logit 5.2% 0.000 40.3% 89.7%
FC-014 sample size N = 150 chance-corrected logit Gaussian identity 5.0% -0.036 0.0% –
FC-014 sample size N = 150 chance-corrected logit Standard binomial logit 11.9% -0.473 0.0% –
FC-014 sample size N = 150 chance-corrected logit Standard binomial probit 10.3% -0.243 5.3% 94.3%
FC-015 sample size N = 200 chance-corrected logit Chance-corrected binomial logit 5.6% 0.000 43.7% 91.4%
FC-015 sample size N = 200 chance-corrected logit Gaussian identity 5.4% -0.036 0.0% –
FC-015 sample size N = 200 chance-corrected logit Standard binomial logit 14.3% -0.473 0.0% –
FC-015 sample size N = 200 chance-corrected logit Standard binomial probit 11.8% -0.243 3.5% 96.2%
FC-016 sample size N = 500 chance-corrected logit Chance-corrected binomial logit 5.8% 0.000 0.0% –
FC-016 sample size N = 500 chance-corrected logit Gaussian identity 7.0% -0.036 0.0% –
FC-016 sample size N = 500 chance-corrected logit Standard binomial logit 25.5% -0.473 0.0% –
FC-016 sample size N = 500 chance-corrected logit Standard binomial probit 21.9% -0.243 0.0% –
FC-017 sample size N = 1000 chance-corrected logit Chance-corrected binomial logit 5.5% 0.000 0.0% –
FC-017 sample size N = 1000 chance-corrected logit Gaussian identity 9.3% -0.036 0.0% –
FC-017 sample size N = 1000 chance-corrected logit Standard binomial logit 44.2% -0.473 0.0% –
FC-017 sample size N = 1000 chance-corrected logit Standard binomial probit 38.3% -0.243 0.0% –
FC-018 scale location Intercept = -1.50 chance-corrected logit Chance-corrected binomial logit 4.9% 0.000 37.8% 84.4%
FC-018 scale location Intercept = -1.50 chance-corrected logit Gaussian identity 16.3% -0.079 0.0% –
FC-018 scale location Intercept = -1.50 chance-corrected logit Standard binomial logit 20.3% -0.365 0.0% –
FC-018 scale location Intercept = -1.50 chance-corrected logit Standard binomial probit 19.9% -0.222 0.0% –
FC-019 scale location Intercept = 1.50 chance-corrected logit Chance-corrected binomial logit 6.0% 0.000 34.8% 95.5%
FC-019 scale location Intercept = 1.50 chance-corrected logit Gaussian identity 11.7% 0.071 0.0% –
FC-019 scale location Intercept = 1.50 chance-corrected logit Standard binomial logit 9.8% -0.286 36.3% 95.3%
FC-019 scale location Intercept = 1.50 chance-corrected logit Standard binomial probit 7.0% -0.053 0.0% –
FC-020 trial count Trials = 5 chance-corrected logit Chance-corrected binomial logit 4.9% 0.000 21.0% 82.8%
FC-020 trial count Trials = 5 chance-corrected logit Gaussian identity 4.5% -0.036 0.7% 0.0%
FC-020 trial count Trials = 5 chance-corrected logit Standard binomial logit 9.0% -0.473 4.8% 78.3%
FC-020 trial count Trials = 5 chance-corrected logit Standard binomial probit 8.7% -0.243 3.9% 62.7%
FC-021 trial count Trials = 10 chance-corrected logit Chance-corrected binomial logit 5.3% 0.000 33.3% 89.8%
FC-021 trial count Trials = 10 chance-corrected logit Gaussian identity 5.2% -0.036 0.0% –
FC-021 trial count Trials = 10 chance-corrected logit Standard binomial logit 12.2% -0.473 0.0% –
FC-021 trial count Trials = 10 chance-corrected logit Standard binomial probit 11.8% -0.243 13.6% 87.0%
FC-022 trial count Trials = 50 chance-corrected logit Chance-corrected binomial logit 6.0% 0.000 0.0% –
FC-022 trial count Trials = 50 chance-corrected logit Gaussian identity 6.3% -0.036 0.0% –
FC-022 trial count Trials = 50 chance-corrected logit Standard binomial logit 19.9% -0.473 0.0% –
FC-022 trial count Trials = 50 chance-corrected logit Standard binomial probit 17.2% -0.243 0.0% –
FC-023 chance level Chance = 0.25 chance-corrected logit Chance-corrected binomial logit 5.8% 0.000 40.5% 94.6%
FC-023 chance level Chance = 0.25 chance-corrected logit Gaussian identity 6.0% -0.054 0.0% –
FC-023 chance level Chance = 0.25 chance-corrected logit Standard binomial logit 11.2% -0.411 0.0% –
FC-023 chance level Chance = 0.25 chance-corrected logit Standard binomial probit 10.6% -0.228 0.0% –
FC-024 chance level Chance = 0.3333 chance-corrected logit Chance-corrected binomial logit 5.6% 0.000 41.0% 94.5%
FC-024 chance level Chance = 0.3333 chance-corrected logit Gaussian identity 6.6% -0.048 0.0% 0.0%
FC-024 chance level Chance = 0.3333 chance-corrected logit Standard binomial logit 13.6% -0.444 0.0% –
FC-024 chance level Chance = 0.3333 chance-corrected logit Standard binomial probit 12.6% -0.240 0.0% –

8.2 Logit versus probit

Table 12: Scenario-level results for the within-family logit-probit atlas.
ID Slice Scenario Generating link/scale Fitted model/scale Rejection rate Deterministic pseudo-interaction
WF-001 paper anchor Logit-generated manuscript scenario logit Binomial logit 5.6% 0.000
WF-001 paper anchor Logit-generated manuscript scenario logit Binomial probit 16.5% -0.112
WF-002 paper anchor Probit-generated manuscript scenario probit Binomial logit 17.6% 0.216
WF-002 paper anchor Probit-generated manuscript scenario probit Binomial probit 5.0% 0.000
WF-003 main effect surface logit generated: group x 0.50; condition x 0.50 logit Binomial logit 4.5% 0.000
WF-003 main effect surface logit generated: group x 0.50; condition x 0.50 logit Binomial probit 7.4% -0.048
WF-004 main effect surface logit generated: group x 1.00; condition x 0.50 logit Binomial logit 4.9% 0.000
WF-004 main effect surface logit generated: group x 1.00; condition x 0.50 logit Binomial probit 10.3% -0.080
WF-005 main effect surface logit generated: group x 1.50; condition x 0.50 logit Binomial logit 4.7% 0.000
WF-005 main effect surface logit generated: group x 1.50; condition x 0.50 logit Binomial probit 11.7% -0.080
WF-006 main effect surface logit generated: group x 0.50; condition x 1.00 logit Binomial logit 5.3% 0.000
WF-006 main effect surface logit generated: group x 0.50; condition x 1.00 logit Binomial probit 11.8% -0.080
WF-007 main effect surface logit generated: group x 1.50; condition x 1.00 logit Binomial logit 5.6% 0.000
WF-007 main effect surface logit generated: group x 1.50; condition x 1.00 logit Binomial probit 11.2% -0.080
WF-008 main effect surface logit generated: group x 0.50; condition x 1.50 logit Binomial logit 5.2% 0.000
WF-008 main effect surface logit generated: group x 0.50; condition x 1.50 logit Binomial probit 10.2% -0.080
WF-009 main effect surface logit generated: group x 1.00; condition x 1.50 logit Binomial logit 5.2% 0.000
WF-009 main effect surface logit generated: group x 1.00; condition x 1.50 logit Binomial probit 10.6% -0.080
WF-010 main effect surface logit generated: group x 1.50; condition x 1.50 logit Binomial logit 5.1% 0.000
WF-010 main effect surface logit generated: group x 1.50; condition x 1.50 logit Binomial probit 5.8% 0.000
WF-011 main effect surface probit generated: group x 0.50; condition x 0.50 probit Binomial logit 8.5% 0.107
WF-011 main effect surface probit generated: group x 0.50; condition x 0.50 probit Binomial probit 5.7% 0.000
WF-012 main effect surface probit generated: group x 1.00; condition x 0.50 probit Binomial logit 12.4% 0.162
WF-012 main effect surface probit generated: group x 1.00; condition x 0.50 probit Binomial probit 5.2% 0.000
WF-013 main effect surface probit generated: group x 1.50; condition x 0.50 probit Binomial logit 13.2% 0.162
WF-013 main effect surface probit generated: group x 1.50; condition x 0.50 probit Binomial probit 5.3% 0.000
WF-014 main effect surface probit generated: group x 0.50; condition x 1.00 probit Binomial logit 11.2% 0.162
WF-014 main effect surface probit generated: group x 0.50; condition x 1.00 probit Binomial probit 4.9% 0.000
WF-015 main effect surface probit generated: group x 1.50; condition x 1.00 probit Binomial logit 12.1% 0.162
WF-015 main effect surface probit generated: group x 1.50; condition x 1.00 probit Binomial probit 5.1% 0.000
WF-016 main effect surface probit generated: group x 0.50; condition x 1.50 probit Binomial logit 10.9% 0.162
WF-016 main effect surface probit generated: group x 0.50; condition x 1.50 probit Binomial probit 4.8% 0.000
WF-017 main effect surface probit generated: group x 1.00; condition x 1.50 probit Binomial logit 10.2% 0.162
WF-017 main effect surface probit generated: group x 1.00; condition x 1.50 probit Binomial probit 5.2% 0.000
WF-018 main effect surface probit generated: group x 1.50; condition x 1.50 probit Binomial logit 4.4% 0.000
WF-018 main effect surface probit generated: group x 1.50; condition x 1.50 probit Binomial probit 4.7% 0.000
WF-019 sample size logit generated: subjects = 100 logit Binomial logit 5.5% 0.000
WF-019 sample size logit generated: subjects = 100 logit Binomial probit 7.4% -0.112
WF-020 sample size logit generated: subjects = 200 logit Binomial logit 5.6% 0.000
WF-020 sample size logit generated: subjects = 200 logit Binomial probit 9.3% -0.112
WF-021 sample size logit generated: subjects = 300 logit Binomial logit 5.7% 0.000
WF-021 sample size logit generated: subjects = 300 logit Binomial probit 11.9% -0.112
WF-022 sample size logit generated: subjects = 400 logit Binomial logit 4.4% 0.000
WF-022 sample size logit generated: subjects = 400 logit Binomial probit 11.8% -0.112
WF-023 sample size logit generated: subjects = 1200 logit Binomial logit 5.3% 0.000
WF-023 sample size logit generated: subjects = 1200 logit Binomial probit 25.9% -0.112
WF-024 sample size logit generated: subjects = 2000 logit Binomial logit 4.7% 0.000
WF-024 sample size logit generated: subjects = 2000 logit Binomial probit 38.5% -0.112
WF-025 trial count logit generated: trials per cell = 5 logit Binomial logit 4.8% 0.000
WF-025 trial count logit generated: trials per cell = 5 logit Binomial probit 8.2% -0.112
WF-026 trial count logit generated: trials per cell = 30 logit Binomial logit 4.4% 0.000
WF-026 trial count logit generated: trials per cell = 30 logit Binomial probit 31.0% -0.112
WF-027 latent location logit generated: reference intercept = 0 logit Binomial logit 5.4% 0.000
WF-027 latent location logit generated: reference intercept = 0 logit Binomial probit 38.2% 0.175
WF-028 latent location logit generated: reference intercept = 0.75 logit Binomial logit 5.1% 0.000
WF-028 latent location logit generated: reference intercept = 0.75 logit Binomial probit 8.2% 0.059
WF-029 latent location logit generated: reference intercept = 2.25 logit Binomial logit 5.6% 0.000
WF-029 latent location logit generated: reference intercept = 2.25 logit Binomial probit 44.1% -0.186
WF-030 icc logit generated: target ICC = 0 logit Binomial logit 4.9% 0.000
WF-030 icc logit generated: target ICC = 0 logit Binomial probit 78.7% -0.112
WF-031 icc logit generated: target ICC = 0.15 logit Binomial logit 5.5% 0.000
WF-031 icc logit generated: target ICC = 0.15 logit Binomial probit 37.0% -0.112
WF-032 icc logit generated: target ICC = 0.45 logit Binomial logit 5.9% 0.000
WF-032 icc logit generated: target ICC = 0.45 logit Binomial probit 9.2% -0.112
WF-033 sample size probit generated: subjects = 100 probit Binomial logit 6.6% 0.216
WF-033 sample size probit generated: subjects = 100 probit Binomial probit 5.3% 0.000
WF-034 sample size probit generated: subjects = 200 probit Binomial logit 8.8% 0.216
WF-034 sample size probit generated: subjects = 200 probit Binomial probit 5.5% 0.000
WF-035 sample size probit generated: subjects = 300 probit Binomial logit 11.5% 0.216
WF-035 sample size probit generated: subjects = 300 probit Binomial probit 5.6% 0.000
WF-036 sample size probit generated: subjects = 400 probit Binomial logit 11.6% 0.216
WF-036 sample size probit generated: subjects = 400 probit Binomial probit 4.5% 0.000
WF-037 sample size probit generated: subjects = 1200 probit Binomial logit 28.1% 0.216
WF-037 sample size probit generated: subjects = 1200 probit Binomial probit 5.5% 0.000
WF-038 sample size probit generated: subjects = 2000 probit Binomial logit 41.5% 0.216
WF-038 sample size probit generated: subjects = 2000 probit Binomial probit 5.6% 0.000
WF-039 trial count probit generated: trials per cell = 5 probit Binomial logit 7.9% 0.216
WF-039 trial count probit generated: trials per cell = 5 probit Binomial probit 4.8% 0.000
WF-040 trial count probit generated: trials per cell = 30 probit Binomial logit 32.2% 0.216
WF-040 trial count probit generated: trials per cell = 30 probit Binomial probit 4.6% 0.000
WF-041 latent location probit generated: reference intercept = 0 probit Binomial logit 39.9% -0.424
WF-041 latent location probit generated: reference intercept = 0 probit Binomial probit 5.7% 0.000
WF-042 latent location probit generated: reference intercept = 0.75 probit Binomial logit 8.0% -0.108
WF-042 latent location probit generated: reference intercept = 0.75 probit Binomial probit 4.7% 0.000
WF-043 latent location probit generated: reference intercept = 2.25 probit Binomial logit 46.2% 0.520
WF-043 latent location probit generated: reference intercept = 2.25 probit Binomial probit 4.7% 0.000
WF-044 icc probit generated: target ICC = 0 probit Binomial logit 82.3% 0.216
WF-044 icc probit generated: target ICC = 0 probit Binomial probit 5.0% 0.000
WF-045 icc probit generated: target ICC = 0.15 probit Binomial logit 40.9% 0.216
WF-045 icc probit generated: target ICC = 0.15 probit Binomial probit 4.8% 0.000
WF-046 icc probit generated: target ICC = 0.45 probit Binomial logit 8.6% 0.216
WF-046 icc probit generated: target ICC = 0.45 probit Binomial probit 5.4% 0.000

8.3 Sum scores

Table 13: Scenario-level results for the sum-score atlas.
ID Slice Scenario Generating link/scale Fitted model/scale Rejection rate Deterministic pseudo-interaction
SS-001 paper anchor Lower range graded logistic latent scale Gaussian-probit bounded score 7.9% -0.056
SS-001 paper anchor Lower range graded logistic latent scale Latent generating scale 4.9% 0.000
SS-001 paper anchor Lower range graded logistic latent scale Observed sum-score identity 72.8% -2.034
SS-002 paper anchor Middle range graded logistic latent scale Gaussian-probit bounded score 3.5% -0.010
SS-002 paper anchor Middle range graded logistic latent scale Latent generating scale 5.6% 0.000
SS-002 paper anchor Middle range graded logistic latent scale Observed sum-score identity 8.6% -0.696
SS-003 paper anchor Upper range graded logistic latent scale Gaussian-probit bounded score 4.8% 0.027
SS-003 paper anchor Upper range graded logistic latent scale Latent generating scale 4.5% 0.000
SS-003 paper anchor Upper range graded logistic latent scale Observed sum-score identity 37.4% 1.570
SS-004 main effect surface x effect x 0.50; group effect x 0.50 graded logistic latent scale Gaussian-probit bounded score 4.7% -0.001
SS-004 main effect surface x effect x 0.50; group effect x 0.50 graded logistic latent scale Latent generating scale 4.9% 0.000
SS-004 main effect surface x effect x 0.50; group effect x 0.50 graded logistic latent scale Observed sum-score identity 4.8% -0.096
SS-005 main effect surface x effect x 1.00; group effect x 0.50 graded logistic latent scale Gaussian-probit bounded score 3.3% -0.002
SS-005 main effect surface x effect x 1.00; group effect x 0.50 graded logistic latent scale Latent generating scale 4.3% 0.000
SS-005 main effect surface x effect x 1.00; group effect x 0.50 graded logistic latent scale Observed sum-score identity 2.6% -0.180
SS-006 main effect surface x effect x 1.50; group effect x 0.50 graded logistic latent scale Gaussian-probit bounded score 3.2% -0.004
SS-006 main effect surface x effect x 1.50; group effect x 0.50 graded logistic latent scale Latent generating scale 4.8% 0.000
SS-006 main effect surface x effect x 1.50; group effect x 0.50 graded logistic latent scale Observed sum-score identity 2.4% -0.242
SS-007 main effect surface x effect x 0.50; group effect x 1.00 graded logistic latent scale Gaussian-probit bounded score 4.6% -0.005
SS-007 main effect surface x effect x 0.50; group effect x 1.00 graded logistic latent scale Latent generating scale 4.5% 0.000
SS-007 main effect surface x effect x 0.50; group effect x 1.00 graded logistic latent scale Observed sum-score identity 6.4% -0.370
SS-008 main effect surface x effect x 1.50; group effect x 1.00 graded logistic latent scale Gaussian-probit bounded score 3.5% -0.015
SS-008 main effect surface x effect x 1.50; group effect x 1.00 graded logistic latent scale Latent generating scale 5.4% 0.000
SS-008 main effect surface x effect x 1.50; group effect x 1.00 graded logistic latent scale Observed sum-score identity 10.5% -0.940
SS-009 main effect surface x effect x 0.50; group effect x 1.50 graded logistic latent scale Gaussian-probit bounded score 4.9% -0.011
SS-009 main effect surface x effect x 0.50; group effect x 1.50 graded logistic latent scale Latent generating scale 5.6% 0.000
SS-009 main effect surface x effect x 0.50; group effect x 1.50 graded logistic latent scale Observed sum-score identity 13.3% -0.786
SS-010 main effect surface x effect x 1.00; group effect x 1.50 graded logistic latent scale Gaussian-probit bounded score 4.5% -0.022
SS-010 main effect surface x effect x 1.00; group effect x 1.50 graded logistic latent scale Latent generating scale 5.1% 0.000
SS-010 main effect surface x effect x 1.00; group effect x 1.50 graded logistic latent scale Observed sum-score identity 30.0% -1.481
SS-011 main effect surface x effect x 1.50; group effect x 1.50 graded logistic latent scale Gaussian-probit bounded score 3.5% -0.036
SS-011 main effect surface x effect x 1.50; group effect x 1.50 graded logistic latent scale Latent generating scale 4.9% 0.000
SS-011 main effect surface x effect x 1.50; group effect x 1.50 graded logistic latent scale Observed sum-score identity 42.9% -2.008
SS-012 sample size N = 75 graded logistic latent scale Gaussian-probit bounded score 3.1% -0.010
SS-012 sample size N = 75 graded logistic latent scale Latent generating scale 5.3% 0.000
SS-012 sample size N = 75 graded logistic latent scale Observed sum-score identity 3.6% -0.696
SS-013 sample size N = 150 graded logistic latent scale Gaussian-probit bounded score 4.0% -0.010
SS-013 sample size N = 150 graded logistic latent scale Latent generating scale 5.1% 0.000
SS-013 sample size N = 150 graded logistic latent scale Observed sum-score identity 4.1% -0.696
SS-014 sample size N = 225 graded logistic latent scale Gaussian-probit bounded score 3.4% -0.010
SS-014 sample size N = 225 graded logistic latent scale Latent generating scale 5.4% 0.000
SS-014 sample size N = 225 graded logistic latent scale Observed sum-score identity 4.5% -0.696
SS-015 sample size N = 300 graded logistic latent scale Gaussian-probit bounded score 4.3% -0.010
SS-015 sample size N = 300 graded logistic latent scale Latent generating scale 4.8% 0.000
SS-015 sample size N = 300 graded logistic latent scale Observed sum-score identity 5.4% -0.696
SS-016 sample size N = 450 graded logistic latent scale Gaussian-probit bounded score 3.5% -0.010
SS-016 sample size N = 450 graded logistic latent scale Latent generating scale 5.2% 0.000
SS-016 sample size N = 450 graded logistic latent scale Observed sum-score identity 7.1% -0.696
SS-017 sample size N = 1200 graded logistic latent scale Gaussian-probit bounded score 3.4% -0.010
SS-017 sample size N = 1200 graded logistic latent scale Latent generating scale 4.5% 0.000
SS-017 sample size N = 1200 graded logistic latent scale Observed sum-score identity 12.1% -0.696
SS-018 scale location Threshold shift = -2.00 graded logistic latent scale Gaussian-probit bounded score 6.2% 0.038
SS-018 scale location Threshold shift = -2.00 graded logistic latent scale Latent generating scale 6.0% 0.000
SS-018 scale location Threshold shift = -2.00 graded logistic latent scale Observed sum-score identity 56.8% 1.877
SS-019 scale location Threshold shift = 2.00 graded logistic latent scale Gaussian-probit bounded score 11.9% -0.072
SS-019 scale location Threshold shift = 2.00 graded logistic latent scale Latent generating scale 4.6% 0.000
SS-019 scale location Threshold shift = 2.00 graded logistic latent scale Observed sum-score identity 82.1% -1.993
SS-020 item count Items = 5 graded logistic latent scale Gaussian-probit bounded score 3.5% -0.013
SS-020 item count Items = 5 graded logistic latent scale Latent generating scale 4.7% 0.000
SS-020 item count Items = 5 graded logistic latent scale Observed sum-score identity 7.4% -0.384
SS-021 item count Items = 15 graded logistic latent scale Gaussian-probit bounded score 3.7% -0.008
SS-021 item count Items = 15 graded logistic latent scale Latent generating scale 5.2% 0.000
SS-021 item count Items = 15 graded logistic latent scale Observed sum-score identity 9.1% -1.162
SS-022 latent dispersion Latent SD = 0.70 graded logistic latent scale Gaussian-probit bounded score 3.9% -0.013
SS-022 latent dispersion Latent SD = 0.70 graded logistic latent scale Latent generating scale 4.2% 0.000
SS-022 latent dispersion Latent SD = 0.70 graded logistic latent scale Observed sum-score identity 13.4% -0.831
SS-023 latent dispersion Latent SD = 1.30 graded logistic latent scale Gaussian-probit bounded score 3.3% -0.007
SS-023 latent dispersion Latent SD = 1.30 graded logistic latent scale Latent generating scale 4.7% 0.000
SS-023 latent dispersion Latent SD = 1.30 graded logistic latent scale Observed sum-score identity 5.5% -0.563