Statistics and P value

The How Science is done? (previous article) mentions the Scientific Method. One of the most important steps of this algorithm is formulating a hypothesis, a statement that will undergo validation by design of appropriate experiments.

To validate our hypothesis (alternate hypothesis), we make a null hypothesis. Let’s go with an example again.
The desert area outside West Texas is famous for Marfa Lights which are basically glowing orbs of the size of basketballs. For quite a long time, people were scared of the sight when suddenly glowing gas balls appeared in the midst of air. Local university students thought that they were caused by the car headlights passing along the US Highway 67. Now how does one check this hypothesis.
Let’s give it a try.
Here, the alternate hypothesis would be “Marfa lights are caused by car headlights” and the null hypothesis would be ”Marfa lights are not caused by car headlights”.

Now to validate our hypothesis, we can design an experiment to reject the null hypothesis.

Experiment: “Calculate the number of cars on the US Highway 67 and calculate the number of Marfa lights observed.” Now if this experiment is done just once, we cannot conclude anything about our null hypothesis. The experiment results would vary on a sunny day or on a foggy day. Also, it may simply be possible that you turn out to be lucky to get the number of cars equal to the number of Marfa lights. Thus to make a more precise conclusion, it is essential for the experiment to be repeated several times. Exactly many times depends on the nature of experiment, the error bars involved and the analyst in question.

Suppose that the experiment has been repeated several times, maybe 5 times a day over a year. This would be an enormous data.
Now if our null hypothesis were true, the data obtained from these experiments would fit well with the expected data. The expectation would be that the number of car lights and the number of Marfa lights have a significant difference. Here is where statistics comes into play.
What do we mean by significant?

To answer such questions, statisticians have devised excellent methods to analyze data. Based on the null hypothesis and a significance level determined (like 5% significant or 1% significant), the data generated by experiments or observations is cross-verified against this significance level and something called a p-value is generated.
P-value is nothing but a probability value obtained from the distribution function that fits the expected data.
A significance level is a percentage below which the data is supposed to not fit this distribution function. Usually, a significance level of 0.05 (meaning 5%) or 0.01 (meaning 1%) is chosen, but it really depends on the needs of the test and the analyst. So if the data lies below the significance level, we can reject the null hypothesis, however, there exists a probability of (p * 100 %), (where p is the p-value for the given data) that the null hypothesis may be true. Now suppose that the p-value for the data lies above the significance level. One would assume that the null hypothesis is true. However, that isn’t the case. All we could say is the alternate hypothesis is not true.

In short, there exists no method to truly accept or reject the alternate hypothesis. All that is possible is to verify it to a certain significance level.

These statistical tests are very strong if used wisely. However, most of the times, they are not used properly. This leads to something called p-hacking, which I will address in my next blog.

References:
1. The Manga Guide to Statistics, Shin Takashi
2. What are Marfa Lights
3. 3 Times Science Debunked the Paranormal, SciShow

How Science is done?

One of the things that makes humans so special is probably their curiosity, the urge to know, to observe phenomena, to pose questions and above all trying to answer them. Science is one of the ways devised to answer these questions in a systematic and rational manner.

To begin with, I will introduce scientific method. Everything that scientists do can be roughly formulated into an algorithm called the scientific method. I will try to make it more clear using an example.

The first step is to “observe” some phenomena. Observation requires alert mind and constant vigilance. Observation basically includes anything that answers a question “What?”
For instance, we observe that everyday sun is rising in the east (maybe for 20 years of your life and you also know it from historical reports say of at least a few hundred years).

To make sense of it, one then makes a hypothesis, which is basically an answer that may explain the phenomena. This is an important step. It is essential for the hypothesis to be “falsifiable”: meaning that one should be able to disprove the converse of the statement. For instance, the statement that “Sun daily rises in the East” could be a valid hypothesis. Understand that one cannot prove this statement. To prove the statement the observer has to observe the Sun “daily” which is impossible for a normal human being. What can be done instead is to disprove the converse: “Sun doesn't rise in the West”. As far as we don’t see Sun rising in the West provides us with evidence to support “Sun daily rises in the East”. Stronger and increasing shreds of evidence, convert a hypothesis into a fact, a theory. However, a single valid observation of Sun rising in the West is enough to disprove the statement. Generalizing it, one can clearly see that there is nothing called a proof in Science (except maybe in Mathematics where you can provide definitive proofs).

Based on the hypothesis, predictions are made which can then be tested using experiments or analysis or a model can be built that suits well with the data. Experiments are then repeated and reformulated. New predictions are made which are further tested. All of these provide pieces of evidence supporting our hypothesis, never proves it. A single experiment giving a negative result simply crushes the hypothesis or causes it to be restated.

A well-supported hypothesis becomes a theory, a fact. This gives rise to new observations and the cycle continues.

The cycle appears quite simple. However, it does not really take into account all the aspects of “How Science is done?” It does not provide an answer to “What does a Scientist do?” Science like other disciplines requires a lot of hard work and patience. Apart from these steps (mentioned above), a scientist also plays with various parameters. Many experiments fail not because the hypothesis was false but because of errors in instrumentation, in handling and many times, the sources of these errors are difficult to find out. The results appear simple and interesting, but the work done before arriving at it is mostly unknown. The frustration, the excitement, the competition among labs to arrive at the answer first, the politics behind are hardly even recognized.