Come analyze HEASARC, IRSA, and MAST data in the cloud! The Fornax Initiative is now welcoming all interested beta users.

Model comparison by evidence

The samplers above answer “given this model, what are the parameters?” A distinct question is “which of several competing models does the data prefer?” The Bayesian answer is the evidence, the marginal likelihood already introduced as the by-product of nested sampling,

$\displaystyle Z = p(D\vert M) = \int p(D\vert\bm{\theta},M)\,p(\bm{\theta}\vert M)\,d\bm{\theta}.$

(3.4)

For two models the ratio of evidences is the Bayes factor $B_{12} = Z_1/Z_2$ , and by Bayes' theorem the posterior odds are the Bayes factor times the prior odds,

$\displaystyle \frac{p(M_1\vert D)}{p(M_2\vert D)} = B_{12}\,\frac{p(M_1)}{p(M_2)}.$

(3.5)

With

models and equal prior probabilities the posterior model probabilities are simply $P(M_i\vert D) = Z_i / \sum_j Z_j$ .

The evidence differs from a maximum-likelihood comparison in a crucial way: it integrates the likelihood over the whole prior rather than evaluating it at the single best-fit point. A model with extra parameters can reach a higher peak likelihood, but if those parameters spread the prior mass over regions the data rule out, the integral is penalized. This built-in Occam factor is exactly the prior volume the extra freedom occupies — a continuous, prior-aware version of the parameter-count penalties used by the information criteria AIC and BIC (indeed $\mathrm{BIC} \approx -2\ln Z$ in the large-data limit). Because the penalty is the prior volume, the evidence depends on the prior in a way that posterior parameter estimates usually do not: a needlessly wide prior on a parameter inflates the Occam penalty and can wrongly disfavor a model. Setting meaningful priors is therefore part of the comparison, not an afterthought, and is the reason nest (and hence the evidence-based comparison below) refuses to run under unbounded multi-decade flat priors.

The customary scale for interpreting a Bayes factor is that of Kass & Raftery (1995), stated in terms of twice the log-evidence difference $2\,\Delta\ln Z = 2\ln B$ : a value above 2 is “positive” evidence, above 6 “strong”, and above 10 “decisive” for the better model. A difference of order the $\log Z$ uncertainty reported by the sampler is not significant.

In XSPEC this is automated by compare nest. With bayes on and priors set, the command runs nested sampling on the current model and each supplied alternative, then tabulates $\log Z$ , $\Delta\ln Z$ , the Kass–Raftery quantity $2\,\Delta\ln Z$ , and $P(M\vert D)$ , ranking by evidence rather than by the AIC/BIC of the non-Bayesian compare modes. Priors for each alternative are taken from the current model by parameter name, or are read from a saved-model candidate supplied as @<file.xcm>when a model needs priors of its own; see the compare command description in Chapter 6 for the full syntax and a worked example. The cost is one full nested-sampling run per model, so the same nest live and parallel nest considerations apply as for a single nest run.

NASA | GSFC | Sciences and Exploration

HEASARC

High Energy Astrophysics Science Archive Research Center

Model comparison by evidence