Probability & Statistics

What are probability and statistics?

Probability is a measure of how certain we are about the outcomes of an experiment or activity.^[6] Statistics is the science of collecting, analysing, interpreting and presenting data.^[1] Its general goal is to draw inferences, or reasoned conclusions, from data.^[12]

Summarising data is called descriptive statistics, while inferential statistics uses probability to judge how confident we can be that a conclusion is correct.^[1]

How probability works

An experiment is a repeatable procedure with well-defined possible outcomes, and the sample space is the set of all possible outcomes.^{[13, 6]} An event is any combination of outcomes.^[6] Every probability lies between 0 and 1.^{[13, 6]} The probabilities of all possible outcomes add up to 1.^{[13, 16]}

Two events are independent if knowing that one occurred does not change the probability of the other.^{[7, 14]} The conditional probability P(A|B), read “A given B”, is the probability that A occurs given that B has already occurred.^{[6, 14]} Three rules follow (OpenStax notation):

• Complement: A′ (“A prime”) holds every outcome not in A, and P(A) + P(A′) = 1.^[6]
• Addition: P(A OR B) = P(A) + P(B) − P(A AND B), and the last term is 0 when A and B cannot happen together.^[8]
• Multiplication: P(A AND B) = P(A|B) × P(B).^{[8, 14]} For independent events this becomes P(A) × P(B).^[8]

The law of large numbers says that as an experiment is repeated more and more, the relative frequency of an outcome tends to get closer to its theoretical probability.^[6]

Describing data

The mean (average) and the median, the value that splits the ordered data into two equal parts, are the two most widely used measures of the centre.^[4] The median is generally better when there are outliers, because it is not affected by their exact values.^[4] The mode is the most frequent value.^[4]

The standard deviation measures how far data values are from their mean, in the same units as the data.^[5]

The normal distribution has a location parameter μ and a scale parameter σ; the standard normal has μ = 0 and σ = 1.^{[17, 10]} For normal data, about 68% of values lie within one standard deviation of the mean, about 95% within two and about 99.7% within three.^[10] The central limit theorem, a proved result, says that the sum or average of many independent copies of a random variable is approximately normal.^[15]

Sampling, bias and cause

Measuring a whole population is often too costly or impossible, so statisticians study a sample, which should have the same characteristics as the population.^[2] Sampling bias arises when some members of the population are less likely to be chosen than others; biased samples give invalid results.^[2]

The correlation coefficient r always lies between −1 and +1, but strong correlation does not show that either variable causes the other.^[11] Hidden extra factors that can cloud a study are called lurking variables, and assigning subjects to treatments at random spreads them equally among the groups.^[3]

The math (optional): worked examples

Two spades. Draw two cards without putting the first back, so the draws are dependent.^[7] The first is a spade with probability 1/4, and then the second is a spade with probability 12/51, so both are spades with probability 1/4 × 12/51 = 3/51, about 5.9%.^[14]

Expected value. Multiply each value by its probability and add the products.^[9] A soccer team that plays 0, 1 or 2 days a week with probabilities 0.2, 0.5 and 0.3 plays on average (0)(0.2) + (1)(0.5) + (2)(0.3) = 1.1 days a week.^[9]

The screening test. With a 0.5% base rate, 5% false positives and 10% false negatives, 500 of 100,000 people are ill and 450 of them test positive, but so do 4,975 of the 99,500 healthy people: only 450 of 5,425 positives, about 8.3%, are ill.^[14]

History

Probability theory began with the study of games of chance.^[1] In five letters in the summer of 1654, Blaise Pascal and Pierre Fermat solved the problem of points, how to divide the stakes of an unfinished game, for two players, and they are now regarded as joint founders of probability.^{[19, 20, 31]}

In one letter, 64 pistoles are at stake and one player leads two points to one. Win the next throw and he takes all 64; lose and he takes back 32. So he is sure of 32, with an equal chance at the other 32, and Pascal splits the stakes 48 to 16.^[30]

Huygens’s De ratiociniis in ludo aleae (1657) was the first printed work on probability.^{[21, 31]} Jacob Bernoulli’s Ars Conjectandi appeared in 1713, and Bernoulli advanced the theorem that Poisson later called the law of large numbers.^{[22, 31]} Thomas Bayes’s essay on the doctrine of chances was published in 1764, according to both MacTutor and Shafer, and Bayes’s theorem is named in his honour.^{[24, 31]}

According to MacTutor, Karl Pearson’s papers included the chi-square test (1900), and Fisher introduced randomisation into the design of experiments.^{[27, 28]} In 1933, Kolmogorov’s Grundbegriffe der Wahrscheinlichkeitsrechnung set out the axiomatic basis of modern probability.^{[29, 32]}

Common misconceptions

“After five heads, tails is due.” This is the gambler’s fallacy: coin flips are independent, so a tail is not more likely on the next flip.^[18] Probability describes what to expect in the long term, not short-term results.^[9]

“A 95% accurate test means a positive result is 95% likely to be right.” In the screening example, most positives are healthy people.^[14]

Going further

On this map, this field builds on Numbers and meets Algebra, Logic and the Scientific Method.