Bayesian data analysis in English

There is a paper currently in press at an important social psychology journal. Using a really simple procedure, this fella Bem has provided some evidence for the existence of precognition (having carefully excluded both psychokinesis and clairvoyance as alternate possibilities). He used the same basic statistics that are the basis of all empirical science for the past century-or-so.

So, do scientists accept the existence of precognition or uproot one of the foundations of science? They are doing the second one. What's even better is that fact that that's OK. It turns out that the statistical foundation of science, null hypothesis significance testing, has a lot of problems. But it is so entrenched that it takes a really compelling false positive to motivate people to do anything about it.

So the alternative statistical approaches--contenders for the title, are getting more attention. One of the more notable approaches is called "Bayesian data analysis." It is nice because it is a mathy way of framing what scientists do in the day-to-day:

You take what you came in believing, then you see something new, and you come out with beliefs that have changed given what you just saw. They might be the same, they might be different. It might be that you didn't know what you believed at first---that a lot of perspectives on the world made sense before---but after seeing what you saw only one or a few of those beliefs make sense anymore. Or it could go the either way, that you had strong beliefs but seeing the world made you realize that other perspectives are possible. Either way, your beliefs change in response to what you are experiencing. Of course, this doesn't represent all of science, but pinning down this small part of it is really useful.

And you actually can describe this same process in math. You had some prior plausible beliefs about what kinds of things are likely to happen in the world (A and B). Then you actually saw things happening in the world (X, the data). What is the liklihood of what just happened from the perspectives of the different things you believe? (What are the probabilities of X given A and of X given B? In symbols, what is p(X|A)). Once you calculate that, you can figure out the reverse: How likely are your beliefs given what just happened---what are the probabilities of A and B given X (p(A|X)). This figuring out the reverse is where the name comes from---Bayes rules is an expression relating p(A|X) to p(X|A). With these values, you can do other cool things, like find the number for how much more likely A is than B (p(A|X)/p(B|X)).

So a good experiment will speak to many possible incompatible beliefs, and make most of them really unlikely no matter how it turns out. Bayesian data analysis is a way of calculating which of your priors you believe in the most after seeing what just happened. Designing a good "what just happened" is up to the scientist, and that is where a lot of the creativity comes in.

Bayesian methods have actually been around for a while, but you need computers to calculate all of the different values involved. So the technique has only started to become popular with the personal computer and modern computing. Another post describes how to do it on a mac.