A p-value tells you how surprising your data would be if there were actually no real effect happening — that is, if the “null hypothesis” (the assumption of no difference or no relationship) were true. It’s one of the most commonly misunderstood numbers in statistics, largely because its actual definition is subtle and easy to mix up with something it doesn’t mean.
The Core Idea
Imagine you flip a coin 100 times and get 60 heads. That’s more than the 50 you’d expect from a fair coin, but is it surprising enough to conclude the coin is actually biased? A p-value answers a very specific version of that question:
“If the coin really were fair, how likely would it be to see a result this extreme (60 or more heads) just by random chance?”
If that probability is very low, the result is considered “statistically significant” — surprising enough that random chance alone seems like an unlikely explanation, making you doubt the assumption that the coin is fair.
The Formal Definition
The p-value is the probability of observing a result as extreme as, or more extreme than, what you actually observed, assuming the null hypothesis is true.
Breaking that down:
- Null hypothesis (H₀) — the default assumption of “no effect” or “no difference” (e.g., “this coin is fair,” “this new drug has no effect compared to placebo”)
- Extreme result — a result as unusual or more unusual than what you observed
- Assuming H₀ is true — this is the critical, often-missed condition: the p-value is calculated under the assumption that there’s nothing going on, not as a measure of whether that assumption is actually correct
What a P-value Is NOT
This is where most confusion happens. A p-value is not:
- The probability that the null hypothesis is true — a p-value of 0.03 does not mean “there’s a 3% chance there’s no real effect.” It means “if there were no real effect, you’d see a result this extreme only 3% of the time.”
- The probability that your results happened by chance — similar misconception, same underlying error
- A measure of effect size — a very small p-value doesn’t necessarily mean a large or important effect; with a large enough sample size, even a tiny, practically meaningless difference can produce a very small p-value
- The probability that you’d get the same result if you repeated the study — that’s a different concept (related to statistical power and replication), not what the p-value measures
The Significance Threshold (Alpha)
Researchers typically set a threshold, called alpha (α), before running a study — commonly 0.05 — to decide how surprising a result needs to be before rejecting the null hypothesis.
- If p ≤ α (commonly p ≤ 0.05) — the result is considered “statistically significant,” meaning the observed data would be quite unlikely if the null hypothesis were true
- If p > α — the result is not statistically significant; this does not prove the null hypothesis is true, only that this particular study didn’t find strong enough evidence against it
The 0.05 threshold is a convention, not a law of nature — some fields use stricter thresholds (like 0.01 or even 0.001), especially when false positives are particularly costly (e.g., certain areas of medical research or physics).
Worked Example
Suppose a company claims a new studying technique improves test scores. You run a study comparing students using the new technique against students using a standard method, and find the new-technique group scored 4 points higher on average.
- Null hypothesis (H₀): There’s no real difference between the two techniques; any observed difference is due to random variation between students
- Run the statistical test (e.g., a t-test) comparing the two groups
- Result: the test produces p = 0.02
Interpretation: If the new technique truly had no effect, you’d see a difference this large (or larger) only 2% of the time just from random variation between students. Since 0.02 < 0.05, this result is considered statistically significant — evidence against the null hypothesis, suggesting the technique likely has some real effect.
What this does not tell you: exactly how much better the technique is in practice, whether it’s worth the cost/effort to implement, or that there’s a 98% chance the technique works. It only tells you the observed difference would be unlikely under the “no real effect” assumption.
Statistical Significance vs Practical Significance
A result can be statistically significant without being practically meaningful. With a large enough sample size, even a trivial difference (say, an average test score improvement of 0.1 points) can produce a very small p-value. This is why researchers increasingly report effect sizes alongside p-values — a measure of how large the difference actually is, not just how unlikely it is to be due to chance.
| Statistical significance | Practical significance | |
|---|---|---|
| Question answered | Is this result unlikely to be pure chance? | Is this result actually large enough to matter? |
| Affected by sample size | Yes, heavily | No (effect size is independent of sample size) |
| Example | p = 0.001 with a tiny, trivial effect | A large, meaningful effect regardless of p-value |
Common Student Mistakes
- Interpreting p-value as “probability the null hypothesis is true” — the most common error, and technically backwards from what the p-value actually measures
- Treating p = 0.05 as a hard scientific cutoff — it’s a conventional threshold, and results just above or below it (p = 0.048 vs p = 0.052) aren’t meaningfully different in practice, despite one being “significant” and the other not
- Assuming statistical significance means practical importance — always check effect size alongside the p-value
- “P-hacking” — running many tests or repeatedly checking data until a p-value below 0.05 appears by chance, which inflates the risk of false positives; this is considered poor research practice
Frequently Asked Questions
What does it mean if a p-value is very small, like 0.001? It means that, assuming the null hypothesis were true, a result this extreme would be very rare — happening only about 0.1% of the time by chance. This is typically strong evidence against the null hypothesis, though it still doesn’t tell you the size or practical importance of the effect.
Is a p-value of 0.05 the same as a 95% chance the result is real? No — this is one of the most common misinterpretations. A p-value of 0.05 means that if the null hypothesis were true, you’d see a result this extreme 5% of the time by chance. It says nothing directly about the probability that the null hypothesis itself is true or false.
Why is 0.05 the standard threshold? It’s a widely adopted convention, originally popularized by statistician Ronald Fisher, rather than a mathematically derived “correct” cutoff. Different fields and different types of research use stricter or more lenient thresholds depending on the cost of false positives.
Can a study have a high p-value but still show a meaningful effect? Yes — a high p-value might simply reflect a small sample size that doesn’t provide enough statistical power to detect a real effect, rather than proving there’s no effect at all. This is why non-significant results shouldn’t automatically be treated as proof of “no effect.”
