Bayesian inference

FYI, today is what Chinese people would call the Singles' Day. This thing even has a Wikipedia page now...

The Bayesian framework and the posterior p-value

Bayesian interpretation of probability is that it represents uncertainty. Bayesian data analysis is about modeling, and we would like to address and quantify the uncertainty that surrounds the appropriate choice for the model parameters. From a Bayesian perspective, we can use the machinery of probability theory to describe the uncertainty in model parameters, or indeed in the choice of model itself. The basic formula for Bayesian inference is Prior + data = posterior. More specifically, Bayes’ theorem is used to convert a prior probability, which formalizes our pressumption on a model parameter θ, into a posterior probability by incorporating the evidence provided by the observed data. The effect of the observed data y = {y1, . . . , yN} is expressed through the conditional probability p(y|θ). Bayes’ theorem, which takes the form p(θ|y) = p(y|θ)p(θ)/p(y) then allows us to evaluate the uncertainty in w after we have observed y in the form of the posterior probability p(θ|y). There is no difficulty from a Bayesian perspective in employing models for which the number of parameters greatly exceeds the number of data points. Indeed, in a Bayesian model the effective number of parameters adapts automatically to the size of the data set. (Dr. Jeff Tilson, another member of our group and a senior scentist at RENCI, is working with Kirk on predictive genomics. They are building predictive models with the number of parameters at the same magnitude of order as human genome. In constrast, the number of samples they have, or the size of the training set is much fewer. To solve this problem they have been applying a Bayesian method.)
The quantity p(y|θ) on the right-hand side of Bayes’ theorem is evaluated for the observed data set y and can be viewed as a function of the parameter vector θ, in which case it is called the likelihood function. It expresses how probable the observed data set is for different settings of the parameter vector θ. Note that the likelihood is not a probability distribution over θ, and its integral with respect to θ does not (necessarily) equal one. The denominator p(y) in the right side of the above formula is the normalization constant, which ensures that the posterior distribution on the left-hand side is a valid probability density and integrates to one. Indeed, integrating both sides of the formula with respect to θ, we can express the denominator in Bayes’ theorem in terms of the prior distribution and the likelihood function p(y) = ∫p(y|θ)p(θ)d(θ). In both the Bayesian and frequentist paradigms, the likelihood function p(y|θ) plays a central role. However, the manner in which it is used is fundamentally different in the two approaches. In a frequentist setting, θ is considered to be a fixed parameter whose value is to be estimated, and the error bar on this estimate is obtained by considering the distribution of possible data sets y. By contrast, from the Bayesian viewpoint there is only a single data set y (namely the one that is actually observed), and the uncertainty in the parameters is expressed through a probability distribution over θ.
A widely used frequentist estimator is maximum likelihood, in which θ is set to the value that maximizes the likelihood function p(y|θ). This corresponds to choosing the value of θ for which the probability of the observed data set is maximized. In the machine learning literature, the negative log of the likelihood function is called an error function. Because the negative logarithm is a monotonically decreasing function, maximizing the likelihood is equivalent to minimizing the error. One approach to determining frequentist error bars is the bootstrap, in which multiple data sets are created as follows. Suppose our original data set consists of N data points X = {x1, . . . , xN}. We can create a new data set XB by drawing N points at random from X, with replacement, so that some points in X may be replicated in XB, whereas other points in X may be absent from XB. This process can be repeated L times to generate L data sets each of size N and each obtained by sampling from the original data set X. The statistical accuracy of parameter estimates can then be evaluated by looking at the variability of predictions between the different bootstrap data sets.

Now let’s talk about the Bayesian goodness of fit test, posterior p-value. Again, let y be the observed data and θ be the vector of parameters. We define yrep as the replicated data that could have been observed, or, to think predictively, as the data we would see tomorrow if the experiment that produced y today were replicated with the same model and the same value of θ that produced the observed data. We define the distribution of yrep given the current state of knowledge with the posterior predictive distribution Now, we can measure the discrepancy between the model and the data by defining test quantities of the data that we would like to check. A test quantity, or discrepancy measure, T(y,θ), is a scalar summary of parameters and data that is used as a standard when comparing data to predictive simulations. The role test quantities play in Bayesian model testing is the same as the role test statistics play in classical model testing. We define the notation T(y) for a test statistic, which is a test quantity that depends only on data; in the Bayesian context, we can generalize test statistics to allow dependence on the model parameters under their posterior distribution. Classically, the p-value for the test statistic T(y) is