org.replikativ.foerster.diagnostics — org.replikativ/foerster 0.1.71

Checking an inference result.

Markov chains: kernel-infer keeps each chain's draws together (the measure's :chain-lengths), so the draws of a quantity split back into chains (chains) and the convergence diagnostics of Vehtari, Gelman, Simpson, Carpenter & Bürkner (2021) apply — rank-normalized split R-hat, bulk and tail effective sample sizes, and the Monte Carlo standard error of the mean, computed as Stan and ArviZ compute them:

(diagnostics/summary measure :mu) ;; => {:mean … :sd … :quantiles … :rhat 1.002 :ess-bulk 1830.0 ;; :ess-tail 1590.0 :mcse 0.004 :chains 4 :draws 4000}

An R-hat above 1.01 or an effective sample size below about 100 per chain says the chains have not mixed: run longer or change the kernel.

Particle methods: weights carry the information, so summary reports the weighted moments with the weight-based ESS; after resampling that counts particles, not distinct histories — distinct-count of an early site says how many histories survive.

Model comparison: pointwise-log-likelihood gives each draw's log density of each observation, the input of PSIS-LOO and WAIC; particle methods also estimate the evidence (measure/log-marginal).

chains clj/s

(chains measure f)

The draws of f in measure, one vector per Markov chain, in draw order. f is a function of the program's value, a keyword (a field of a map value, else a site address) or a vector (a site address). A measure without chain structure (particle methods) is one chain.

compare clj/s

(compare named & [opts])

Models ranked by PSIS-LOO: named is {name measure} over the same observations. Returns rows {:name :elpd-loo :se :elpd-diff :dse} best first; :dse is the standard error of the difference to the best model, from the pointwise differences.

distinct-count clj/s

(distinct-count measure f)

How many distinct values f (see chains) takes over the particles of measure: for an early site after resampling, the number of histories that survive.

ess-bulk clj/s

(ess-bulk cs)

The bulk effective sample size of cs (draws by chain): of the rank-normalized split chains.

ess-tail clj/s

(ess-tail cs)

The tail effective sample size of cs: the smaller of the effective sample sizes of the indicators of the 5% and the 95% quantile.

loo clj/s

(loo measure & [opts])

PSIS-LOO of measure — equally weighted draws, as Markov chains give — computed as ArviZ's az.loo: {:elpd-loo :se :p-loo :pareto-k {address k̂} :good-k :pointwise {address elpd}}. :reff is the relative efficiency ESS/S (default 1; ArviZ estimates it from the posterior's mean ESS). An observation whose k̂ exceeds :good-k = min(1 − 1/log10 S, 0.7) has an unreliable estimate: the model is sensitive to it.

mcse clj/s

(mcse cs)

Monte Carlo standard error of the mean of cs: the sd of all draws over √ESS, the ESS of the split chains themselves.

pointwise-log-likelihood clj/s

(pointwise-log-likelihood measure)

Each draw's log density of each observation: a vector, one map {address log-p} per particle of measure, in particle order — the input of PSIS-LOO and WAIC (with every draw equally weighted, as Markov chains give; resample a weighted measure first).

rhat clj/s

(rhat cs)

Rank-normalized split R-hat of cs (draws by chain): the larger of the bulk and the folded (tail) statistic. Near 1 when the chains agree; above 1.01 they have not mixed.

summary clj/s

(summary measure f)

Summary of the quantity f (see chains) in measure: weighted mean, sd and quantiles, and — for Markov chains — :rhat, :ess-bulk, :ess-tail and :mcse; for weighted particles :ess, the weight-based effective sample size.

waic clj/s

(waic measure)

WAIC of measure (equally weighted draws), as ArviZ's az.waic: {:elpd-waic :se :p-waic :pointwise}. Prefer loo, whose k̂ says when the estimate is unreliable.