MSCE / ESS for inference

[mattansb]

Following the discussion in #757, if one were interested in obtaining MSCE or ESS for inferential quantities such as p-direction or p-rope, here's how one would do it:

p-direction

library(bayestestR)

posterior <- rnorm(1000, mean = 1, sd = 1)

ess_pd <- function(posterior, null = 0) {
  x_less <- posterior < null
  x_more <- posterior > null
  min(posterior::ess_mean(x_less), posterior::ess_mean(x_less))
}

msce_pd <- function(posterior, null = 0) {
  x_less <- posterior < null
  x_more <- posterior > null
  max(posterior::mcse_mean(x_less), posterior::mcse_mean(x_less))
}

p_direction(posterior, null = 0)
#> Probability of Direction
#> 
#> Parameter |     pd
#> ------------------
#> Posterior | 83.90%

ess_pd(posterior, null = 0)
#> [1] 918.6881
msce_pd(posterior, null = 0)
#> [1] 0.01213185

^{Created on 2026-04-22 with reprex v2.1.1}

We would report:

pd = 0.839 ± 0.012 (ESS = 918.69)

p-ROPE

ess_p_rope <- function(posterior, range = bayestestR::rope_range(posterior)) {
  x_in <- range[1] <= posterior & posterior <= range[2]
  posterior::ess_mean(x_in)
}

msce_p_rope <- function(posterior, range = bayestestR::rope_range(posterior)) {
  x_in <- range[1] <= posterior & posterior <= range[2]
  posterior::mcse_mean(x_in)
}

p_rope(posterior)
#> Proportion of samples inside the ROPE [-0.10, 0.10]: 0.053

ess_p_rope(posterior)
#> Warning: Could not estimate a good default ROPE range. Using 'c(-0.1, 0.1)'.
#> [1] 1125.387
msce_p_rope(posterior)
#> Warning: Could not estimate a good default ROPE range. Using 'c(-0.1, 0.1)'.
#> [1] 0.006681582

We would report:

p-ROPE [-0.1, 0.1] = 0.053 ± 0.007 (ESS = 1125.39)

[DominiqueMakowski]

So you're saying that ESS is that primarily a property characterizing a parameter (e.g., "The" ESS of the intercept is X, of beta1 is Y) but should be reported differentially for each statistic? And act like an uncertainty index... For the statistics themselves?

Interesting meta-uncertainty concept

[mattansb]

What is "chain-relater information"?

Which sample belongs to which chain. get_parameters gives i (iterations) * c (chains) long vector instead of an i-by-c matrix.

[DominiqueMakowski]

divergent conclusions

What I meant is basically "oh oh be careful, you should probably check your model and refit with more iterations" vs. "should be all good 👌"

[strengejacke]

So what's the decision now?
#758 reports tail ESS by default, but we could make the default to report both?
We could furthermore add an argument (like diagnostic) to report ESS for the other functions (like p_rope() etc), too.

[mattansb]

Yes report both.
I don't think the diagnostic option is needed.

[DominiqueMakowski]

Report both in diagnostic_posterior and one in parameters() as this is where the breaking change might be the most consequential

[avehtari]

x_less <- posterior < null
x_more <- posterior > null
min(posterior::ess_mean(x_less), posterior::ess_mean(x_less))

These two ess_mean should be exactly the same for any continuous posterior (up to floating point accuracy). The same for mcse_mean

We would report:

pd = 0.839 ± 0.012 (ESS = 918.69)

There is no sense in reporting ESS with such accuracy, and it would be better to round and report ESS = 919 to avoid unnecessary clutter

MCSE / ESS represent the (frequentists) uncertainty in estimating some characteristic with sampling

They represent also Bayesian uncertainty and information loss respectively (they can derived using a model and prior etc.)

Bulk-ESS is often higher than Tail-ESS for easy posteriors that are close to normal in which case Stan's NUTS/HMC produces antithetic Markov chains that can have super efficiency for mean and other bulk quantites (ESS is bigger than the sample size). Bulk-ESS is often lower than Tail-ESS in case of bimodal distributions with mean/median being between the modes. ESSs are useful unitless values to get rough idea about the MCMC performance. For reporting quantities of interest MCSE is more specific. See more in Case study: https://avehtari.github.io/Bayesian-Workflow/digits/digits.html

Feel free to ask more about posterior and model diagnostics

[avehtari]

Can you elaborate on the Bayesian interpretation of MCSE?

• First, Bayesians can just take any information and say that it presents what they know so you can always use Bayesian interpretation of probability
• Second, in this specific case we can also use a flat Dirichlet process model (Lo, 1987) for the posterior draws and MCSE is equal to posterior standard deviation of the mean functional, which is available analytically and happens to be familiar MCSE equation. For other than mean, we can use delta method which provides and approximate posterior.

[avehtari]

Adding

• The frequentist interpretation of MCSE: what would be distribution of mean estimates if we would repeat the MCMC sampling infinitely many times
• The Bayesian interpretation of MCSE (the same acronym for convenience): given these posterior draws what is our uncertainty about infinitely long MCMC mean estimate

For practical purposes it really doesn't matter in this case, but just wanted to bring it up that there is no need to use frequentist interpretation for MCSE

[mattansb]

Thanks Aki!

[strengejacke]

Thanks a lot! Just adding that there's probably not the "one" frequentist interpretation, though many come up with the "inifinite sampling". See https://link.springer.com/article/10.1007/s11229-022-03560-x and https://www.sciencedirect.com/science/article/abs/pii/S0378375817301738?via%3Dihub.

[strengejacke]

And of course another, maybe slightly related "school": https://doi.org/10.1186/s12874-020-01105-9 and https://doi.org/10.1177/02683962221105904