Law of Large Numbers Explained using python : A practical approach

Law of Large Numbers 

In this article we attempt to understand the concept of Law of Large numbers in Statistics . The idea is basically to see both the sides of statistics , the experimental side using computation and also the theoretical aspect using mathematics.

An Experiment :

So we take the experiment of rolling a dice (Let this Event be X). Theoretically the Expected Value (Mean) of this event is calculated as follows :

Assuming it's a fair dice , so each of the possible outcomes in the Sample Space = {1,2,3,4,5,6} have an equal probability of occurrence and hence probability of getting say a value ' i ' where i belongs to the sample space

P(K = i) = 1/6

So, we can compute Expectation of this event as E[X] = (1+2+3+4+5+6)/6 = 3.5.

This is the theoretical definition of what an expectation for this event would look like.

Now , comes the experimental part of this exercise , So supposedly we roll a single dice and note down the outcome.

Say we got the outcome Xi = xi for ith roll.

Say we get :

                    X1 = 2

Food for thought!!

X1 is not equal to the expected value , is this something wrong?

Obviously no , what error can one make while rolling a dice :P

                                                                              

For understanding this , we would need to perform this experiment of rolling a dice and noting down the No. repeatedly for say N times.

We get a different result in the experimental part than in the theoretical part. So for making a connection between observational and theoretical probability , we need to understand the Law of Large numbers.

Here comes the law of large Numbers :

The Intuitive Approach:

The law States that as we perform the same experiment N times and record the outcome and each time , and then take the mean of all the X1 , X2 , X3 ,.... obtained the expected value converges to the theoretical expectation value when N is large .

Graphically :

Note : The above image and the animation for the same has been generated using Python.

Now Lets Play with the animated version of this :

The python code for generating this animation has been attached on the following link :

https://drive.google.com/file/d/1-oINPdkB7iDtxlrhKFsz4divV022v8ow/view?usp=sharing

So , in this image we see that as Number of trials increases , the expected value tends to be closer and closer to the theoretical value of the Expected value which is 3.5.

Mathematically :

We have two versions of the law :

Lets talk about what both the versions agree upon first :) 

Both versions agree and state that X1 , X2 , ........ is an infinite sequence 

and are independent and identically distributed , then random variables with

E[X1] = E[X2] = ....... = µ

This converges to the Expected value when n tends to infinity , which is :

Assuming that the variance is finite and there is no correlation between the random variables , the variance of the n random variables is given by :

Note : We assume

Hence ,

The reason for the existence of both (that is the difference between the two..)

Weak Law states that the sample average (mean) converges in probability towards the Expected Value.

Mathematically ,

For any epsilon , a very small positive number :

The Strong Law states that the sample average converges almost surely to the expected value.

Mathematically :

Hence , it is just a stronger version of the weak law of large number

I hope this gives you a better understanding of what the Law of Large numbers both theoretically and also getting an intuition of the experiment from the simulation .

If you like this post , do comment , like and share for more posts in future.

Happy Learning!

Thanks

Sahaj Thareja