Fundamental Priciples of Probability and Statistics

1. Random Variables

I toss a coin five times. This is a random experiment and the sample space can be written as

Note that here the sample space has elements. We can define a random variable whose value is the number of observed heads. The value of will be one of or depending on the outcome of the random experiment.

For example, the random variable defined above assigns the value to the outcome , the value to the outcome , and so on. Hence, the random variable is a function from the sample space to the real numbers (for this particular random variable, the values are always integers between and ).

Random Variables:

A random variable is a function from the sample space to the real numbers.
The range of random variable , shown by or , is the set of possible values of .

In the above example, .

Example 1:

I toss a coin times. Let be the number of heads I observe.
$\Rightarrow$ The random variable can take any integer from to , so .
I toss a coin until the first heads appears. Let be the total number of coin tosses.
$\Rightarrow$ The random variable can take any positive integer, so .
The random variable is defined as the time (in hours) from now until the next earthquake occurs in a certain city.
$\Rightarrow$ The random variable can in theory get any non-negative real number, so .

2. Discrete Random Variables

is a discrete random variable, if its range is countable.
Note: Sets such as and their subsets are countable. While sets such as non-empty intervals in are uncountable.

In Example 1 (from previous section), random variables and are discrete, while the random variable is not discrete.

3. Probability Mass Function (PMF)

Definition:

Let be a discrete random variable with range (finite or countably infinite). The function

is called the probability mass function (PMF) of .

Example 2: I toss a fair coin twice, and let be defined as the number of heads I observe. Find the range of , , as well as its probability mass function .

Solution

Sample space: .
The number of heads will be or . Thus
Since this is a finite (and thus a countable) set, the random variable is a discrete random variable. Next, we need to find PMF of . The PMF is defined as

We have

In general, we can write

Properties of PMF

For any set

Example 3: I have an unfair coin for which , where . I toss the coin repeatedly until I observe a heads for the first time. Let be the total number of coin tosses.

Find the PMF of .
Check that
If , find

Solution

1. Find the PMF of .

2. Check that

We have: for With

So, .

3. If , find

Firstly, we have:

We notice that:

So,

4. Independent Random Variables

Event and are independent if we have .
Note that: .
Example: An airplane has two engines, I and II, which operate independently. The probabilities that engine I and engine II function properly are and , respectively. If we call:
- Let be the event “Engine I functions properly.”
- Let be the event “Engine II functions properly.”
- Let be the event “Both engines function properly.
- So , are independent events and ,

The concept of independent random variables is similar to independent events.

Definition

Consider two discrete random variables and . We say that and are independent if
In general, if two random variables are independent, then you can write
Consider discrete random variables . We say that are independent if

Note: Conditional Probability

With are two random variables independent:

Example 4:

I toss a coin twice and define to be the number of heads I observe. Then, I toss the coin two more times and define to be the number of heads that I observe this time. Find .

Solution

Since and are results of different independent coin tosses, the two random variables and are independent.

So, we can write:

From Example 2 (tossing a fair coin twice):

Substitute the values:

Notation Summary

: a random variable
: a probability distribution. In VN, we call this: "Phân phối xác xuất".
: the random variable follows distribution
: The probability assigned to the event where random variable takes values .
: The conditional probability distribution of given
: a probability density function (PDF) associated with distribution
. In the VN, we call this function: "Hàm mật độ".
: expectation of random variable .
: random variables and are independent.
: random variables and are conditionally independent given .
: standard deviation of random variable . In VN, we call this: "Độ lệch chuẩn của biến ngẫu nhiên ".
: variance of random variable , equal to . In VN, we call this: "Phương sai của ".
: covariance of random variables and . In VN, we call this: "Hiệp phương sai".
: the Pearson correlation coefficient between and , equals .
: entropy of random variable .
: the KL-divergence (or relative entropy) from distribution to distribution .

References

Statistical probability