Statistics 1 - Lecture Notes Chapter 1
Caio Ibsen
Graduate School of Economics - Getulio Vargas Foundation
April 28, 2009
We want to establish a formal mathematic theory to work with results of experiments that can change randomly. For example, imagine the experiment of rolling a dice, there are six possible outputs,{1,2,3,4,5,6}. But we can think in another event, for instance, an event can be the even numbers. Moreover, we want to know what the chances are for some event to occur, that is, the probabilities of each possible result of an experiment(What is exactly a possible result? And a probability?). How we model this kind of situations is the goal of this chapter.
1 Basic Model
Let Ω be a space of outputs of an experiment, called sample space, in the example above Ω≡ {1,2,3,4,5,6}. We call the elements of Ω as the simple or elementary events.
When we are discussing about the possible results of an experiment, in general, we are interested in more results than the elements of Ω, like the result of the dice being even or odd. Let us define what we will considerer as a possible result of an experiment.
Definition 1.1. Let Ω be a nonempty set of elementary results of an experiment, we define as a field (or an algebra) of events of Ω the set A of subsets of Ω such that
i - Ω∈A;
ii - if A∈A then Ac∈A; and
iii - for any finite family {An} of elements of A we have ∪nAn ∈A.
It is interesting to note the reasons of this definition. Intuitively, Arepresents the set of all events that we can identify at the end of the experiment. Thus, the first property tell us that we always know when Ω happens, the second is if we can tell when an eventA occurs then we can tell when this event does not occur. The last property tells us that if we can identify when an event A occurs and when an eventB occurs, defining the event D as the A orB, we also need to be able to identify when the event D occurs.
Exercise 1.1. Show that the property (iii) of the definition of a field can be exchanged for:
• for any finite family {An} of elements of A we have ∩nAn ∈A.
Solution 1.1. We need to prove that both sentences are equivalent. First assume that for any finite family {An} of elements of Awe have ∪nAn∈A. By item ii, we know that for any finite family {An} of elements of A, {Acn} is a finite family of elements of A, thus,
∪nAcn ∈A=ii⇒(∪nAcn)c∈A. But (∪nAcn)c=∩nAn then we have the result. On the other hand, assuming that for any finite family {An} of elements of A we have ∩nAn ∈ A the result follows likewise.
Example 1.1. For the discussed example of a dice we have Ω ≡ {1,2,3,4,5,6} and a field for this Ω can be {∅,Ω}. We call this kind of field a trivial field of events. It is the most simple field of any set Ω. The largest possible algebra is the set of all subsets of Ω, denoted by P(Ω).
Exercise 1.2. Prove that for any nonempty set Ω, we have that P(Ω) is an algebra of events of Ω.
Solution 1.2. Let us see the three properties:
i - Ω⊂Ω⇒Ω∈ P(Ω);
ii - A⊂Ω⇒Ac⊂Ω⇒Ac ∈ P(Ω); and
iii - {An} ⊂P(Ω)⇒An ⊂Ω ∀n⇒ ∪nAn⊂Ω⇒ ∪nAn∈ P(Ω).
Then we can conclude that P(Ω) is an algebra of events.
Without loss of generality1 we can apply a more useful class of events called σ−field.
The advantage of this definition is that it allows you to work with the limits of unions and intersections, which will be very useful for the discussion about convergence of random variables.
Definition 1.2. Let Σ be a field of a nonempty set Ω. We say that Σ is a σ−field of events of Ω if
iii’ - for any countable family {An} of elements of Σ we have ∪nAn∈Σ.
A pair (Ω,Σ) with a nonempty set Ω and a σ−field Σ is called a measurable space, and an arbitrary element of Σ is called a Σ−measurable set(or simply a measurable set if Σ is implicit in the context.).
Exercise 1.3. Prove thatiii0 impliesiii. Moreover, show that, with Ω = [0,1], the family A defined as the set of all finite unions of intervals belonging to[0,1]is a field of Ω but it is not a σ−field.
1See Billingsley, P “Probability and Measure” for a more detailed discussion on fields’ extension.
Solution 1.3. To see that iii implies iii0 is only necessary define, for all finite family {An}Nn=1, the infinite family {An}∞n=1 with An =∅ for all n=N + 1, . . ., then ∪Nn=1An =
∪∞n=1An ∈ Σ. For the second part it is easy to see the three properties of an algebra. To see that A is not a σ−field note that for all n∈N the set
1− 1
2n,1− 1 2n+1
belongs to A. But the infinite union does not since each element is nonempty and the sets are disjoint.
Given a nonempty set Ω and a nonempty family of subsets of Ω, C, we define as the σ−algebra generated by C, σ(C), the smallestσ−field that contains all elements ofC. An equivalent definition is that σ(C) is the intersection of all σ−fields containing C.
Exercise 1.4. Let {Σλ}λ∈Λ be a family ofσ−fields of a nonempty set Ω. Where Λ is any nonempty family of indices. Show that Σ≡ ∩λ∈ΛΣλ is a σ−field of Ω.
Solution 1.4. Let us see the three properties of a σ−field:
i - Ω∈Σλ ∀λ ⇒Ω∈ ∩λ∈ΛΣλ;
ii - A∈ ∩λ∈ΛΣλ ⇒A∈Σλ ∀λ⇒Ac ∈Σλ ∀λ⇒Ac∈ ∩λ∈ΛΣλ; and
iii’ - {An} ⊂ ∩λ∈ΛΣλ ⇒ {An} ⊂Σλ ∀λ⇒ ∪nAn∈Σλ ∀λ⇒ ∪nAn∈ ∩λ∈ΛΣλ.
Exercise 1.5. Prove that for any nonempty set Ω, we have that P(Ω) is an σ−algebra of events of Ω. Conclude that σ(C), as defined above, is well defined.
Solution 1.5. The first part is exactly the same prove of exercise1.2. To see that σ(C) is well defined note that we can write σ(C) as ∩Σ∈SΣ where S is the set of all σ−fields containing C2. As we see in the exercise, this intersection is a sigma field. Therefore, σ(C) is well defined.
Considering Ω the real numbers. We call Borel σ−field, and denote B, the smallest σ−field that contains all intervals in R. The elements of B are called Borel sets.
Exercise 1.6. Let (Ω,Σ)be a measurable space. Prove that for any nonempty set S ∈Σ, we have that ΣS ≡ {A∈Σ;A⊂S} is a σ−field of S.
Solution 1.6. Let us see the three properties of a sigma field:
i - S ∈Σ and S ⊂S then S ∈ΣS;
ii - A∈ΣS ⇒A ∈Σ⇒Ac ∈Σ⇒Ac∩S ∈Σ⇒Ac∩S ∈ΣS; and
2That is nonempty by the first part of exercise.
iii - {An} ⊂ΣS ⇒({An} ⊂Σ)∧(An⊂S ∀n)⇒(∪nAn ⊂Σ)∧(∪nAn ⊂S)⇒ ∪nAn⊂ ΣS.
Thus, we have the proof.
The last object necessary to have a closed mathematical model to an experiment is the probability of an event. This definition is given below.
Definition 1.3. A probability on (Ω,Σ) is a function P : Σ→R+ such that i - P(Ω) = 1; and
ii - for any countable family {An} of disjoints elements of Σ we have P(∪nAn) = P
nP(An).
The triple (Ω,Σ, P) is called a probability space. All the theory developed in these notes will be based on a probability space.
Example 1.2. Imagine an experiment of heads and tails with a fair coin. We can model this experiment such as Ω ={H, T}, Σ ={∅,Ω,{H},{T}} and the probability P is such that P(∅) = 0, P(Ω) = 1 and P(H) = P(T) = 1/2.
In the proposition below we give some useful properties that come from the definition of probability. It is interesting for the reader to know the basic results of set operations to understand the following properties. If you are not certain about this knowledge, it is advisable to take a look at a basic book of theory of sets before continuing.
Proposition 1.1. Let (Ω,Σ, P) be a probability space. Then we have the following prop- erties.
1. P(∅) = 0;
2. P(A)≤1 ∀A∈Σ;
3. P(Ac) = 1−P(A) ∀A∈Σ;
4. P(B ∩A) =P(B)−P(B∩Ac) ∀A, B ∈Σ;
5. P(A∪B) =P(A) +P(B)−P(A∩B) ∀A, B ∈Σ; and 6. ∀A, B ∈Σ A⊂B ⇒P(A)≤P(B).
Exercise 1.7. Prove proposition 1.1.
Solution 1.7. See Barry James.
Proposition 1.2. Let (Ω,Σ, P)be a probability space and {Ci}i a partition of Ωsuch that {Ci}i ⊂Σ. Then ∀A∈Σ we have P(A) =P
iP(A∩Ci). In addition, for any countable family of sets {Ai} in Σ, we have that P(∪iAi)≤P
iP(Ai).
Proof: To see the first result it is only necessary to note that, since∀A∈Σ and{Ci}i ⊂Σ, we have thatA∩Ci ∈Σ∀i. Moreover, {A∩Ci}is a countable family of disjoint sets, then P(A) =P(A∩Ω) = P(A∩(∪iCi)) = P(∪i(A∩Ci)) =P
iP(A∩Ci). For the last result define Bi ≡ Ai ∩(∪j<iAj)c ∈ Σ. Note that Bi ⊂ Ai ∀i and ∪iBi =∪iAi. By definition, the family {Bi} is countable and disjoint, thus P(∪iBi) = P
iP(Bi)≤ P
iP(Ai), where the last inequality comes from Bi ⊂Ai ∀i, and we have the wanted result.
Exercise 1.8. Let (Ω,Σ, P) be a probability space and {An} a family of elements of Σ such that An+1 ⊂ An and ∩nAn =A, or An ⊂ An+1 and ∪nAn = A. Prove that in both cases limP(An) = P(A).
Solution 1.8. See Barry James.
2 Conditional Probability and Independence
Definition 2.1. Let(Ω,Σ, P)be a probability space andB an element ofΣwithP(B)6= 0.
For all A∈Σ the conditional probability of A given B is P(A|B) = P(A∩B)
P(B) . (1)
For the case of P(B) = 0 we will define P(A|B) =P(A) ∀A∈Σ.
Intuitively, the conditional probability of a set A given another nonempty set B is the relative measure of A in B, that is, the conditional probability is the probability of the intersection but we need to ponderate by the probability of the set that we are conditioning on.
Exercise 2.1. In a talking show, the guest needs to chose one door among three, A, B andC, in one of the doors there is an award. After the guest chooses the door A, the host opens the door B and the award is not there. What is the probability of the award being in door A? And in door C?
Solution 2.1. First not that P(A) = P(B) =P(C) = 1/3. Moreover, P(OB|CA) = P(OB ∩A|CA) +P(OB∩C|CA)
=P(A)P(OB|A∩CA) +P(C)P(OB|C∩CA) = 1 3 1 2 +1
31 = 1 2. Where CI, OI represent the guest chose the door I and the host open the door I. Then,
P(A|OB ∩CA) = P(A∩OB ∩CA) P(OB∩CA)
= P(A)P(CA|A)P(OB|A∩CA) P(OB|CA)P(CA) =
1 3 1 3
1 2
1 2
1 3
−1
= 1 3.
Analogously,
P(C|OB ∩CA) = P(C∩OB∩CA) P(OB∩CA)
= P(C)P(CA|C)P(OB|C∩CA) P(OB|CA)P(CA) =
1 3 1
31 1 2 1 3
−1
= 2 3. Proposition 2.1. Let (Ω,Σ, P) be a probability space and {An+1} any finite family of measurable sets. Then
P(A1∩. . .∩An+1) = P(A1)P(A2|A1)P(A3|A1∩A2). . . P(An+1|A1∩. . .∩An).
Proof: Let us prove by induction.
1◦- For n = 1 we have by definition that P(A2|A1) =P(A1 ∩A2)/P(A1) then, multi- plying both sides by P(A1), we have that P(A1∩A2) =P(A1)P(A2|A1).
2◦- Assume that the sentence is true for n = k, using the previous result applied to A1∩. . .∩Ak+1 and Ak+2 we have that P(A1 ∩. . .∩Ak+1 ∩Ak+2) = P(A1∩. . .∩ Ak+1)P(Ak+2|A1 ∩. . .∩Ak+1). But by induction hypothesis P(A1∩. . .∩Ak+1) = P(A1)P(A2|A1)P(A3|A1∩A2). . . P(Ak+1|A1∩. . .∩Ak), thusP(A1∩. . .∩Ak+1∩ Ak+2) =P(A1)P(A2|A1)P(A3|A1∩A2). . . P(Ak+1|A1∩. . .∩Ak)P(Ak+2|A1∩. . .∩ Ak+1) and the sentence is true fork+ 1.
3◦- The sentence is true forn = 1 and if it is true forn =k then it is true forn =k+ 1.
Therefore, by induction, it is true for all natural numbers.
And we have the proof.
The following theorem is known as the Bayes’ Rule. The Bayesian method of estima- tion comes from this theorem and works with the idea of a priori distributions. That is, imagine that you have beliefs about the probabilities of some events {An} that form a partition of Ω. The question is how you change your beliefs after the realization of some event B. This is the answer that Bayes’ Rule gives us.
Theorem 2.1. Let (Ω,Σ, P)be a probability space and {An} ⊂Σbe a countable partition of Ω. Then, for all measurable set B and all i we have
P(Ai|B) = P(B|Ai)P(Ai) P
nP(B|An)P(An). (2)
Proof: By definition P(Ai|B) = P(Ai ∩B)/P(B), using the previous proposition, we know that P(Ai∩B) = P(B|Ai)P(Ai) then it is only necessary to prove that P(B) = P
nP(B|An)P(An). Note that the family {B ∩An} is a countable family of disjoint events and B = ∪n(B ∩An), hence, P(B) = P
nP(B ∩An). Again, using the pre- vious exercise, P(B ∩ An) = P(B|An)P(An). Therefore, we have that P(Ai|B) = P(B|Ai)P(Ai)/P
nP(B|An)P(An).
Exercise 2.2. In a factory of cellphones you know that five percent of the cellphones have a problem in the screen and three percent of them the sound does not work. Moreover, you know that in half of the cellphones with a problem in the screen, the sound does not work.
If you take one which the sound does not work, what is the probability that the screen has a problem?
Solution 2.2. Define V P, SP as the events of screen problem and sound problem respec- tively. Then,
P(V P) = 5/100, P(SP) = 3/100, P(SP|V P) = 1/2 and
P(SP|V Pc) =P(SP ∩V Pc)
P(V Pc) = P(SP)−P(SP ∩V P)
1−P(V P) = P(SP)−P(SP|V P)P(V P) 1−P(V P)
=5/100−3/200
1−5/100 = 7/200 95/100 = 7
190. Using the Bayes’ Rule
P(V P|SP) = P(SP|V P)P(V P)
P(SP|V P)P(V P) +P(SP|V Pc)P(V Pc)
= (1/2)(5/100)
(1/2)(5/100) + (7/190)(95/100) = 5/200 12/200 = 5
12.
Now we will introduce the concept of independent events. We need to formally define the idea that two events are independent when the occurrence of one does not add any information about the chances of the other occurring.
Definition 2.2. Let (Ω,Σ, P) be a probability space and A, B ∈ Σ two events. We say that the events A, B are independent if we have
P(A∩B) = P(A)P(B).
Note that for two sets,A, B ∈Σ, with positive probability, this definition implies that P(A|B) =P(A) and P(B|A) =P(B).
The concept of independence is that information about one event does not tell us anything about the chances of the other. When we are working with many events, this idea changes a little. To assert that these events are independent, it is necessary that subsets of combined events do not give any piece information about the others. The following definition formalize this idea.
Definition 2.3. Let (Ω,Σ, P) be a probability space and {Ai}i∈I ⊂ Σ a finite family of events. We say that these events are independent if for all set of indices J ⊂I we have
P \
j∈J
Aj
!
=Y
j∈J
P(Aj).
Example 2.1. Consider a fair dice. Then the probability space is (Ω,Σ, P) with Ω = {1,2,3,4,5,6}, Σ = P(Ω) and P is the probability constructed using thatP(ω) = 1/6for all ω∈Ω. Defining the events A1 ={1,2,3,4} and A2 =A3 ={4,5,6} we have that
P(A1)P(A2)P(A3) = 1
6 =P(A1∩A2∩A3).
But the events are not independent, since
P(A1∩A2) = 1 6 6= 1
3 =P(A1)P(A2) and P(A2∩A3) = 1 2 6= 1
4 =P(A2)P(A3).
Exercise 2.3. Consider a probability space (Ω,Σ, P) and two events A, B ∈ Σ. Mark true or false for the following items. When the answer is true, make a proof, and when it is false, give a counter example:
1 - ( ) If A, B are independent then Ac, Bc are independent events.
2 - ( ) If A∩B =∅ then A, B are independent.
3 - ( ) P(A∪B) = 1−P(Ac)P(Bc) when A, B are independent.
4 - ( ) Let σ(A) and σ(B) be the sigma fields generated by A and B respectively. Then C, D⊂Σ are independent for all C∈σ(A) and D∈σ(B).
