Skip to content

Probability and Random Processes

Descriptive Statistics


Statistics are the summarization of a set of data that has been collected, which demonstrates random variation. Extracting meaning from data.

Inferential Statistics

Inferential Statistics make inferences about a situation based on data, such as forecasting. Descriptive statistics can be the basis for inferences.

Representative Values

  • Mean - The sum of all numbers in a list divided by the number of items in the list
  • Median - The middle value in a list ordered from smallest to largest
  • Mode - The most frequently occuring value in a list
  • Range - [Min, Max]
  • Variance - Average of deviation from the mean squared
  • Standard Deviation - Measure of average absolute deviation
  • Skewness - Measure of the shape of the distribution function
  • Quantiles - Generalization of the median to percentiles

Observational vs. Experimental Data

Experimental Data involves manipulating objects to determine cause and effect in data. Observational Data refers to data extrapolated from naturally occurring events.

Basic Probability

Probability Calculus

Probability events have a total probability between zero and one.

An event that is sure to happen

The definition of probability for how often an event is observed can be related to the number of repetions of the experiment.

Counting the probability of heads in a set of coin tosses

The larger the number of repetiions, the higher accuracy with which we can predict the likelihood of an event happening.

Probability Model


Events are elements in the set of possible outcomes in an experiment.

Sample Space

The set of all possible outcomes for an experiment.

The sample space for a dice roll

Subsets containing events in the sample space

The complement of a subset is the subset of all other events in the sample space which are not contained in .

Event Algebra


The combination of two or more sets.

For and ,


The set of events which occur in two or more sets.

For and ,

  1. Mutual exclusion:
  2. Inclusion:
  3. Double complement:
  4. Commutation:
  5. Associativity:
  6. Distributivity:
  7. DeMorgan’s Law:

Probability of Events

  1. For any event :
  2. For the full set,
  3. If and are Mutually Exclusive then
  4. Axiom 3 can be extended:

Mutual Exclusivity refers to the fact that and will never occur simultaneously, ie .

Non-Mutually Exclusive

In cases where and can occur in the same set, Axiom 3 will not apply.

This is due to the fact that in overlapping events, the same area of probability will be counted twice, though it has no statistical importance.

Non mutually exclusive OR

Complement of an Event

From expanding on these axioms, it can be seen that the complement of an event has a probability related to subtraction of itself from the sample space probability. If the chance of the event happening is known, the chance of an event not happening is found by subtracting this from absolute certainty.

The complement of a set versus the whole

Statistical Independence

Two events and are said to be statiscally independent if:

Statistical independence

This refers to the fact that statistical data will happen independent of the preceding events. If you flip a coin, the probability of the next coin will be the same. If you take items from a bin, the probability of the next item being picked will go up, and is therefore dependent.

Repeated Independant Trials

From the rule of statistical independence, we can process repeated trials.

Coin Flips

Knowing that the probability of either event is 0.5, we can take the list of possible outcomes and calculate the probability.

Sampling with Replacement

In this case, we consider an event where an event occurring does not subtract from a finite amount of events, ie a coin flip. In the case of heads or tails, there aren't one less heads or tails. So it is as if we replace the event in our sample space.

Sampling without Replacement

Finite amounts of events that can be subtracted from the whole. If this event happens, it won't happen again, as if we have taken our card from a deck of cards and not placed it back in the deck.

Order of Outcomes When Sampling Without Replacement

Cases when its important what order events occur in. Did we draw the Ace of Spades within the first 3 draws? The number of each event is not important.


When a trial is repeated times, we form a sample space of outcomes made up of number of events.

The Rule of Product

How many possibilities are there for the formation of -tuples, if there are choices for the th element?

The Rule of Product

By this rule, the number of possibilities when rolling a dice, then flipping a coin, will be . In the case where same number of outcomes are possible with each experiment, the number of -tuples is , or in the case of repeated coin tosses.

Permutations of Unordered Outcomes

We no longer care in what order outcomes occur, we are only concerned with the number of outcomes of a certain sort across all trials.

This involves the number of ways we can choose objects in choices.

Permutations Without Replacement
  • Experiments with two or more possible outcomes
  • These trials can be repeated independently for times
  • For each th trial the outcome from the previous is removed
  • Probabilities change for each consecutive trial

The resulting set is ordered, but as mentioned before, we only care about the number of possible permutations from these elements.

The number of possible sets is or

Example - The 13 cards of a suit in a deck of cards can be laid out in or different ordered sequences.

If you want just draws from possible ways to draw the object, the number of sets will instead be from to

Consider this problem - Lisa has 13 different ornaments and wants to put 4 ornaments on her mantle. In how many ways is this possible?

Using the product rule, Lisa has 13 choices for which ornament to put in the first position, 12 for the second position, 11 for the third position, and 10 for the fourth position. So the total number of choices she has is . Using the factorial notation, the total number of choices is .

From this example, we can see that if we have objects and want to arrange of them in a row, there are ways to do this.

The notation for permutations is

Combinations of Non-Unique Outcomes

A combination is a way of choosing elements from a set in which order does not matter.

Consider the following example: Lisa has 13 different ornaments and she wants to give 3 ornaments to her mom as a birthday gift (the order of the gifts does not matter). How many ways can she do this?

We can think of Lisa giving her mom a first ornament, a second ornament, a third ornament, etc. This can be done in ways. However, Lisa's mom is receiving all three ornaments at once, so the order Lisa decides on the ornaments does not matter.

There are reorderings of the chosen ornaments, implying the total number of ways for Lisa to give her mom an unordered set of 5 ornaments is .

Rule of Combinations or Unordered Permutations

The notation for combinations is =

Conditional Probability

A conditional probability is a probability that a certain event will occur given some knowledge about the outcome or some other event. is a conditional probability, it is read as "Probability of A given B".

Rule of Conditional Probability

A simple example - A fair 12-sided die is rolled. What is the probability that the roll is a 3 given that the roll is odd?

This is or

Because has already happened, the intersection of and can have the probability removed, because it is statistically redundant.

Conditional Probability if statiscally independent

Bayes Theorem

When attempting to compute the conditional probability of two events, when only one event is known, the Bayes Theorem allows for a workaround.

Consider to be Hypothesis, and to be Evidence

Bayes Theorem

We can expand the equation in the numerator to demonstrate fully:

Therefore, or can be found from and vice versa.

Total Probability

If , and form a partition of the sample space, for each

Total Probability

Knowing this, can be found from

Bayes General Rule

Total Probability, when , , form a partition

Random Variables

Random variables deal with a function which maps a number from the sample space . The number can be placed on the real number line, and a probability assigned to it based on its random occurrence.

Discrete Probabilty Distributions

Discrete random variables involve events with a discrete set of values.

Probability Mass Function

The Probability Mass Function, or PMF is a plotting of the probability of all events associated with a random variable . The sum of all amplitudes of the graph, will be 1.

For a given value , the probability of this value is .

Consider the ranking of weights of each possible outcome. If an outcome is more probable, it is heavier, and plotted above the others.


Probability Mass Function

Bernouli Random Variable

A Bernoulli RV is a discrete variable which will only produce values of and . Therefore, the likelhood of one will be and the other will be .

Binomial Random Variable

The Bernoulli concept can be extended with combinatorics, for example in the base of binary transmission error. When detecting the error in the first -bits of a -bit transmission, .

As shown in earlier sections, there are possible variations of bits in an -bit long transmission. If our likelihood of non-error bits is , and error is , the above will be intuitively correct.

Geometric Random Variable

Geometric RVs concern a wait for an event to happen. Should the expected event be given probability, there will be consecutive events before the occurs. Therefore, as seen in the graph below, the event occuring at the second transmission will be .


Geometric Random Variable

Geometric Random Variable PMF (0 k )

Poisson Random Variable

For a situtation where revents occur randomly at a given rate over a certain time interval , the probability of events happening within this time frame has been experimentally verified with representing the average.

Poisson Random Variable PMF

Note that for finding the probability of an event occurring after time , the probability becomes .

Uniform Random Variable

When all events are equally likely, the probability of each can be found easily from the uniform random variable PMF.

Uniform Distribution


Continous RVs and Their Distributions

For values which can take on a continuum of values, such as voltage, velocity, and mass, new tools are used to analyze their probability. The probability of these events is determined using the Cumulative Distribution Function or CDF, which is written as .

By this notation we can see that by following the graph from left to right, the probability of the event occuring to the left of value will be found by the amplitude of the CDF at that value. Therefore, as , .


When finding the probability of a value occurring between points and , their CDF values can be used. Remember this is Distribution, not Density, as we'll see in the PDF below. By keeping this straight, their purposes should be easy to remember.

CDF Probability Within a Range (b a)

The Probability Density Function or PDF is a derivative of the CDF that can also be used to find this probability:


$\text{Pr}[a < X \leq b] = \int_{b}^{a}f_X(x)dx $

CDF Probability Within a Range (b a) From Integration

This can be seen to be similar to the Probability Mass Function, as it will integrate over its full range to . The difference is due to continous distributions being non-discrete - we can no longer say an item has mass, but points will now be denser (equivalent to heavier).

To use the PDF to find the probability of a number , we can multiply the PDF value at this point by the increment value to find the probability.

PDF Probability Within a Range ()

Integrating over a range () will also produce from the PDF.

Common Continuous RVs

Exponential Random Value

An extension of the Geometric Random Variable to the continuous realm, this represents a continuous graph of wait times where again represents the rate of arrival for an event as in Poisson RVs.

Its PDF follows

Gaussian RV

The Gaussian or "normal" random variable arises naturally in numerous cases. It can be defined by its mean, , which will be the center of its bell shape, and its standard deviation , which denotes the value in each direction it will pass before reaching of its peak value.

The Gaussian PDF


The square of the standard deviation, is known as the variance, and is a measure of the total width of the bell between these points.

The standard form of the PDF, centered at with a can be used to express :

The Standard Gaussian Function

And as with any other CDF,

These values, which can be used as a CDF, as with other standard Gaussian values, can be found by table.

Gaussian Q Values

In cases where the probablity of values at either tail of a Gaussian is required, such as , it is common to use functions.

The Gaussian Q Function

When a value is sought which is less than , the argument of the function will be negative, and define the left tail of the CDF. .


The Gaussian Q Function

The Gaussian Q Function with Negative Argument



Expecation can be considered the average of the expected values in a sample space, where the values are weighted by their probability and summed.

Expectation of a Discrete RV

For continuous random variables, when the PDF exists, the expectation can be calculated whenever the variable converges absolutely:

Expectation of a Continuous RV

An important feature of the expectation is its invariance.


The moment is the produced when the value of the expectation is raised to the power. If for = 1, 2, 3

Continous Moment

Discrete Moment

The first moment is the mean, and each further value is called the th moment of the distribution.

Central Moments

The Central Moment is a mean of a random variable not centered at 0. Of particular importance is the variance, the square root of which will grant the width of the distribution.

The 2nd Central Moment

The mean of

The Standard Deviation is represented as , and is a measure of the width.