Bayesian data analysis in English

There is a paper currently in press at an important social psychology journal. Using a really simple procedure, this fella Bem has provided some evidence for the existence of precognition (having carefully excluded both psychokinesis and clairvoyance as alternate possibilities). He used the same basic statistics that are the basis of all empirical science for the past century-or-so.

So, do scientists accept the existence of precognition or uproot one of the foundations of science? They do the second one. What's even better is that that's OK. It turns out that the statistical foundation of 20th century science, null hypothesis significance testing, has a lot of problems. But it is so entrenched that it takes a really compelling false positive to motivate people to do anything about it.

So the alternative statistical approaches---contenders for hegemony over 21st century statistics---are getting more attention. One of the more notable approaches is called "Bayesian data analysis." It is nice because it is a mathy way of framing what scientists do in the day-to-day:

You go into a room with prior beliefs about the world In the room you see something new, and you come out with beliefs that have changed given what you just saw. They might be mostly the same or completely different. It might be that you didn't know what you believed at first---that a lot of perspectives on the world made sense before---but after seeing what you saw only one or a few of those beliefs make sense anymore. Or it could go the either way, that you had strong beliefs but seeing the world made you realize that other perspectives are possible. Either way, your beliefs change in response to what you experienced. Of course, this doesn't represent all of science, but pinning down this small part of it is a really useful way of doing statistics.

You actually can describe this same process in math. You had some plausible prior beliefs about what kinds of things are likely to happen in the world (A and B). Then you actually saw things happening in the world (X, the data). What is the likelihood of what just happened from the perspectives of the different things you believe? (What is the probability of X given A, what is p(X|A)? And what about B). Once you calculate that, you can figure out the reverse: How likely are your beliefs given what just happened---what are the probabilities of A and B given X (p(A|X)). Figuring out the reverse is where the name comes from---Bayes rules is an expression relating p(A|X) to p(X|A). With these values, you can do other cool things, like find the number for how much more likely A is than B (p(A|X)/p(B|X)).

So a good experiment will speak to many possible incompatible beliefs, and make most of them really unlikely no matter how it turns out. Bayesian data analysis is a way of calculating which of your priors you believe in the most after seeing what just happened. Designing a good "what just happened" is up to the scientist, and that is where a lot of the creativity comes in.

Bayesian methods have actually been around for a while, but you need computers to calculate all of the different values involved. So the technique has only started to become popular with the personal computer and modern computing. Another post describes how to do it on a Mac.