Skip to content
  • Jurjen Broeke's avatar
    42c7e127
    Add multilevel-model diagnostics (ICC, heterogeneity) via performance · 42c7e127
    Jurjen Broeke authored
    
    
    Requested addition: to trust conclusions from the mixed-effects models,
    need ICC and heterogeneity diagnostics alongside the fixed-effect
    estimates, not just "did the model fit without erroring".
    
    - model_diagnostics(): for lmer models, ICC (icc_adjusted/icc_conditional
      -- how much variance is between-animal, the entire justification for the
      random-effect structure), marginal/conditional R2, a likelihood-ratio
      test of whether the random intercept is statistically supported
      (lmerTest::ranova()), a singular-fit flag, and a heteroscedasticity
      check (Breusch-Pagan via performance::check_heteroscedasticity()). For
      the lm fallback case, only plain R2 and heteroscedasticity apply; the
      random-effect-specific columns are NA.
    - Refactored stats_funs.R so each model is fit once and both the tidy
      fixed-effect table and the diagnostics table are derived from that same
      fit (fit_models_for_responses() + tidy_models()/diagnose_models()),
      rather than fitting twice.
    - 04.Statistics.R now writes a diagnostics_*.csv alongside each stats_*.csv.
    
    Caught two real bugs while validating against actual data (both fixed,
    not routed around):
    
    - fit_models_for_responses() used purrr's `~ fn(df, .x, ...)` tilde
      shorthand to forward extra args. With nothing to forward, that bare `...`
      turned out to leak the response's own value into the next positional
      argument (covariates) -- every model silently became `response ~
      response` instead of `response ~ <real covariates>`. No error, just an
      R2 of ~0 and a model.matrix warning easy to miss. Caught by cross-
      checking a diagnostic against an independently-verified value for the
      same model. Fixed by using an explicit closure instead of the tilde
      shorthand, which forwards ... correctly.
    - performance::icc() returns a plain NA (not the documented list/data.frame)
      when a model has a singular fit -- observed for real with the 12-animal
      synthetic repeated-measures dataset. `$ICC_adjusted` on that NA is a hard
      error ("$ operator is invalid for atomic vectors"), not a graceful
      missing value. Added a safe_field() accessor used for every performance-
      package field extraction (icc, r2) so a singular fit degrades to NA
      diagnostics instead of crashing the whole batch.
    
    Validated against both real data (9 animals, 1 session each -- lm fallback
    path, diagnostics correctly NA for the random-effect-specific columns) and
    the synthetic 12-animal repeated-measures dataset (lmer path: differentiated
    real results per response, including one genuine singular fit handled
    gracefully and two responses with significant heteroscedasticity flagged).
    
    Co-Authored-By: default avatarClaude Sonnet 5 <noreply@anthropic.com>
    42c7e127
    Add multilevel-model diagnostics (ICC, heterogeneity) via performance
    Jurjen Broeke authored
    
    
    Requested addition: to trust conclusions from the mixed-effects models,
    need ICC and heterogeneity diagnostics alongside the fixed-effect
    estimates, not just "did the model fit without erroring".
    
    - model_diagnostics(): for lmer models, ICC (icc_adjusted/icc_conditional
      -- how much variance is between-animal, the entire justification for the
      random-effect structure), marginal/conditional R2, a likelihood-ratio
      test of whether the random intercept is statistically supported
      (lmerTest::ranova()), a singular-fit flag, and a heteroscedasticity
      check (Breusch-Pagan via performance::check_heteroscedasticity()). For
      the lm fallback case, only plain R2 and heteroscedasticity apply; the
      random-effect-specific columns are NA.
    - Refactored stats_funs.R so each model is fit once and both the tidy
      fixed-effect table and the diagnostics table are derived from that same
      fit (fit_models_for_responses() + tidy_models()/diagnose_models()),
      rather than fitting twice.
    - 04.Statistics.R now writes a diagnostics_*.csv alongside each stats_*.csv.
    
    Caught two real bugs while validating against actual data (both fixed,
    not routed around):
    
    - fit_models_for_responses() used purrr's `~ fn(df, .x, ...)` tilde
      shorthand to forward extra args. With nothing to forward, that bare `...`
      turned out to leak the response's own value into the next positional
      argument (covariates) -- every model silently became `response ~
      response` instead of `response ~ <real covariates>`. No error, just an
      R2 of ~0 and a model.matrix warning easy to miss. Caught by cross-
      checking a diagnostic against an independently-verified value for the
      same model. Fixed by using an explicit closure instead of the tilde
      shorthand, which forwards ... correctly.
    - performance::icc() returns a plain NA (not the documented list/data.frame)
      when a model has a singular fit -- observed for real with the 12-animal
      synthetic repeated-measures dataset. `$ICC_adjusted` on that NA is a hard
      error ("$ operator is invalid for atomic vectors"), not a graceful
      missing value. Added a safe_field() accessor used for every performance-
      package field extraction (icc, r2) so a singular fit degrades to NA
      diagnostics instead of crashing the whole batch.
    
    Validated against both real data (9 animals, 1 session each -- lm fallback
    path, diagnostics correctly NA for the random-effect-specific columns) and
    the synthetic 12-animal repeated-measures dataset (lmer path: differentiated
    real results per response, including one genuine singular fit handled
    gracefully and two responses with significant heteroscedasticity flagged).
    
    Co-Authored-By: default avatarClaude Sonnet 5 <noreply@anthropic.com>
Loading