Model diagnostics for Bayesian models for coverage indicators. Diagnostics include smoothing-based analyses of model residuals and innovation terms, and analyses based on out-of-sample validation model fits.

See package website https://alkemalab.github.io/bayescoveragemodelchecks for documentation and vignettes.

Installation

# Install from GitHub
remotes::install_github("AlkemaLab/bayescoveragemodelchecks")

Checks based on smoothing

This package provides tools to detect systematic patterns in residuals (or innovation terms).

Our goal is to examine how residuals and innovations relate to two covariates of interest: time and coverage level. Under correct model assumptions, residuals and innovation terms should be centered around zero for all years and coverage levels; systematic departures from zero indicate model misspecification.

To make these patterns easy to detect, we developed a visual diagnostic with a simple summary measure that highlights systematic departures from zero.The approach is as follows: For each posterior draw and covariate $x$ , we fit a weighted penalized smoothing spline $f(x)$ to the posterior sample of residuals $\{z_1^{(j)}, \hdots, z_n^{(j)}\}$ , with weighting based on the precision of each $z_i^{(j)}$ . At each $x$ , we approximate the posterior probability that the fitted smoother does not deviate from zero, $2 \cdot \min\{P(f(x) > 0), P(f(x) < 0)\}$ . We call this quantity the probability of no deviation at that value of $x$ : values near 0 provide strong evidence of a deviation from zero, while values near 1 indicate no evidence of deviation (residuals centered around zero). As a single summary measure, we report the minimum of the probability of no deviation over a grid of $x$ -values. We refer to this measure as Prob(no deviation).

Input Data Format

The main function expects a tibble with the following columns:

Column	Description
`draw`	Posterior draw number
`residual`	Residual value
`sd_y`	Standard deviation for the observation
`level_prop`	Predictor variable (or other predictor column)

Optional columns for subsetting or visualization:

pop_id / iso: Population/country identifier
year: Calendar year
obs_index: Observation index
y, yhat: Observed and predicted values
region, data_type: For faceting or coloring

Usage

library(bayescoveragemodelchecks)

# Run the residual smoothing check
results <- residual_model_check(
  res_data = residuals,
  predictor = "level_prop",
  n_draws = 500
)

# Visualize the smoothing fits
plot_smoothing_fits(results, x_lab = "Coverage level")

# Plot the summary measures against x
plot_smoothing_summary(results, x_lab = "Coverage level")

# Get Prob(no deviation) summary 
min_evidence <- summarize_smoothing_results(results, verbose = TRUE)

Interpreting Results

Interpretation of Prob(no deviation):

Values close to 0: Strong evidence of systematic trend (potential model misfit)
Values close to 1: No evidence of trend (residuals centered around zero)
Threshold: Values below 0.1 suggest the model should be investigated

Functions

Function	Description
`residual_model_check()`	Fit smoothing curves and calculate evidence metrics
`plot_smoothing_fits()`	Visualize smoothing curves with credible intervals
`plot_smoothing_summary()`	Plot the evidence metric across predictor values
`summarize_smoothing_results()`	Print summary and return minimum evidence